Gateway Startup Fault – SQLite [SQLITE_FULL] Error

[jacque_trahan]

On our development server we’ve occasionally run into a situation where the Gateway memory usage climbs and does not get released. Restarting the Gateway usually clears this up and results in a significant reduction in memory usage. Otherwise, the memory can remain tied up for hours, or sometimes never gets released at all.

What I did to fix it

In the past, when I’ve run into similar issues, I was able to resolve them by restoring a smaller config.idb file from backup. That approach worked because the Gateway could initialize cleanly with a lighter database.

In this particular case, however, the available backups were actually larger than the current file, so restoring them didn’t help.

As a workaround, I renamed the existing db folder to db_old and restarted the Gateway. After doing this, the Gateway was able to start again. All Gateway configuration was lost, but importantly, all Designer project resources were preserved — which was the most critical aspect for us.

Error details

The following error appears in the wrapper.log when the Gateway fails to start:

INFO | jvm 1 | 2025/12/02 11:39:47 | E [IgnitionGateway ] [17:39:47]: Error during context startup. INFO | jvm 1 | 2025/12/02 11:39:47 | org.sqlite.SQLiteException: [SQLITE_FULL] Insertion failed because database is full (database or disk is full) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.core.DB.newSQLException(DB.java:1179) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.core.DB.newSQLException(DB.java:1190) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.core.DB.throwex(DB.java:1150) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.core.NativeDB._exec_utf8(Native Method) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.core.NativeDB._exec(NativeDB.java:93) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.jdbc3.JDBC3Statement.lambda$executeLargeUpdate$2(JDBC3Statement.java:128) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:454) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.jdbc3.JDBC3Statement.executeLargeUpdate(JDBC3Statement.java:115) INFO | jvm 1 | 2025/12/02 11:39:47 | at org.sqlite.jdbc3.JDBC3Statement.executeUpdate(JDBC3Statement.java:102) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.gateway.localdb.AbstractDBInterface.runUpdateQuery(AbstractDBInterface.java:222) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.gateway.localdb.AbstractDBInterface.runUpdateQuery(AbstractDBInterface.java:236) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1238) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:301) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:759) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:693) INFO | jvm 1 | 2025/12/02 11:39:47 | at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:544) INFO | jvm 1 | 2025/12/02 11:39:47 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) INFO | jvm 1 | 2025/12/02 11:39:47 | at java.base/java.util.concurrent.FutureTask.run(Unknown Source) INFO | jvm 1 | 2025/12/02 11:39:47 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) INFO | jvm 1 | 2025/12/02 11:39:47 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) INFO | jvm 1 | 2025/12/02 11:39:47 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) INFO | jvm 1 | 2025/12/02 11:39:47 | at java.base/java.lang.Thread.run(Unknown Source) INFO | jvm 1 | 2025/12/02 11:39:47 | I [IgnitionGateway ] [17:39:47]: Ignition[state=STARTING] ContextState = FAULTED

My questions

1. Is there a better way to force Java / Ignition’s internal processes to release memory without requiring a full Gateway restart?

2. For this specific error, is there a reason why SQLite fails at around 2 GB of file size on restart? Below is a screenshot of the file size that consistently causes the Gateway to fault on startup.

   

edit: Additional information

• Version: 8.1.36 (b2024010211)
• Gateway memory: 40,960
• The gateway memory is at 39/40 GBs, a restart would drop it to 15/40 and it would settle around 20/40 GBs after a few minutes

[paul-griffith]

@jacque_trahan

Insertion failed because database is full (database or disk is full)

This is an error that most likely indicates a lack of physical disk space. Unrelated to anything with system memory (a full disk is going to be catastrophic for almost any software that needs to read and write files).

@jacque_trahan

Is there a better way to force Java / Ignition’s internal processes to release memory without requiring a full Gateway restart?

No. The JVM will automatically run garbage collection whenever it can (increasing in frequency/consuming more CPU resources as the memory approaches configure limits). You can tune the garbage collection process with a variety of system flags, but it's not generally necessary or recommended. And frankly it sounds like your system is struggling for a variety of reasons, so any tweaking you do to system parameters is probably wasted effort.

@jacque_trahan

For this specific error, is there a reason why SQLite fails at around 2 GB of file size on restart? Below is a screenshot of the file size that consistently causes the Gateway to fault on startup.

As previously mentioned, the most likely cause of the SQLITE_FULL error is a lack of physical disk space on the server overall - SQLite needs some extra headroom so that it can atomically perform file updates to minimize the risk of DB file corruption even in exceptional scenarios.
It's definitely not normal to have an internal database anywhere near 2gB. Especially if the things you care the most about are projects - typically large internal database files are full of lots and lots of tag configuration data.

I would strongly recommend you contact our support department officially, because any help we can offer asynchronously via the forums (on whatever information you can publicly provide) is going to be substantially less targeted than a support rep connecting to your system live and doing in depth troubleshooting.

If you can't/won't/don't want to do that, the next step I would undertake is getting the sqlite_tools, a package of programs to run on the command line, one of which, sqlite_analyzer, can tell you what exactly is taking up all that space in the 2gB IDB file.

[jacque_trahan]

Good information here. We checked the disk space on both the entire server and the E: drive, and both appear to be fine. I’ll include the screenshots we captured during the fault below. Once I have a chance, I’ll dig into this further and may reach out to support if needed. Thank you for the quick response!

Side note: We have multiple tag providers, and some of them are very large. As we work, we typically disable the providers we don’t need at the moment. I usually turn off the biggest ones except for a single provider, then perform a restart. In this case, however, the load was simply too much.

E drive size:

Server size:

[bschroeder]

JAVA will always pull the memory from the OS that corresponds to the initmemory in your ignition.conf up to the maxmemory in the same file. Typically for a production machine you set both of those to the same. Once JAVA has that memory, then it never releases it back to the OS. So if you are looking at taskmanager for memory usage, that isn't valid. You need to look at the gateway status page.