Out of memory - Gateway on Windows

Ignition
1

Hello,

i am experiencing periodical gateway crashes caused by memory problems on 32bit windows. The problem is not the java heap memory but the native windows memory. At a certain point the gateway process is not able to get any more heap memory from windows, so creation of new java threads or malloc in native libraries (stack trace below) fails.
When i watch the memory allocation with VMMap, i can see that heap memory usage is constantly increasing up to about 1.6G when the gateway starts to fail.
My prime suspect is the SQLite library since it seems to allocate lots of native heap memory. I am not exactly sure when the problem occcured first, but it might have been together with the change of Ignitions internal db.

Is this a known problem or does anybody know a solution?

INFO   | jvm 7    | 2017/07/17 13:40:13 | Caused by: java.sql.SQLException: [SQLITE_NOMEM]  A malloc() failed (out of memory)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	at org.sqlite.core.DB.newSQLException(DB.java:890)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	at org.sqlite.core.DB.newSQLException(DB.java:901)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	at org.sqlite.core.DB.execute(DB.java:810)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	at org.sqlite.jdbc3.JDBC3PreparedStatement.executeQuery(JDBC3PreparedStatement.java:68)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	at com.inductiveautomation.ignition.gateway.localdb.DelegatingDataSource$DelegatingConnection$DelegatingPreparedStatement.executeQuery(DelegatingDataSource.java:574)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	at simpleorm.sessionjdbc.SQueryExecute.executeQuery(SQueryExecute.java:61)
INFO   | jvm 7    | 2017/07/17 13:40:13 | 	... 32 more
2

What version of Ignition is this happening on?

3

Oh, sorry. It began with 7.7.something, the current version is 7.8.5. OS is Windows 7, 32bit in a VM.

4

If you can get the JDK installed on that box, you could also run jmap to get a .hprof heap dump. I know you said the problem was in the OS memory, but there may be some long-lived objects that are also hanging onto native memory. They may show up in the .hprof file.

5

The JDK is already installed, i made some heap dumps when the problem started but could not find anything unsual there.
The Ignition service restarted two days ago (i am using a trigger in the wrapper.conf on OutOfMemory exceptions). I made a heap dump immediately after the restart and two times a day since then. At the same time, i recorded the Windows memory footprint. You can see that the private data (Java heap) stays constant but the windows heap is continously increasing.
The zip file in this link contains two heap dumps, one directly after the restart, the second one from two days later. Maybe you can find something that i missed.

6

Yeah the heap dumps didn’t have any obvious problems. It looks like this gateway is running reports though. Can you try disabling report execution for a period of time to see whether the reports are causing the memory spikes?

7

Disabling the project with the reports had no effect. In addition i uninstalled the reporting module and restarted the Gateway. The increase of memory usage was the same, so reporting is not the problem here.
Any other ideas?