Gateway SingleConnectionDatasource errors preventing normal operation

I am getting tons of errors in the Diagnostics logs relating to the following:

Caused by: java.sql.SQLException: Connection is locked. Datasource only allows one connection at a time. More information was logged to the gateway console.

at com.inductiveautomation.ignition.gateway.localdb.sqlite.SingleConnectionDatasource.getConnection(SingleConnectionDatasource.java:75)

I’m not really sure how this started happening, as i’ve had problems with the gateway starting up when i inherited the project. I"m running Ignition off a Windows Server VM, and its currently just a development system.

I’ve seen on other threads this may be related to internal database problems, which i’m not very familiar with. I presume its not talking about the external database connections? Because they aren’t getting much action.

Something wrong with the auto backup of the internal database?
I’ve even tried a re-install of Ignition in a different directory, but had to restore the same gateway backup. Not sure what else to try

Does the gateway have any jython directly accessing the internal DB? Can you find and post the fragment of the wrapper log that surrounds that log entry?

Any Jython written by me? No not at all. I don’t have any knowledge of how to utilize the internal DB.

INFO   | jvm 2    | 2019/03/06 08:14:33 | E [c.i.i.g.l.s.SingleConnectionDatasource] [19:14:31]: The following stack successfully received a connection. A new attempt was blocked for over 30000 ms 
INFO   | jvm 2    | 2019/03/06 08:14:33 | java.lang.Throwable: null
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at com.inductiveautomation.ignition.gateway.localdb.sqlite.SingleConnectionDatasource.getConnection(SingleConnectionDatasource.java:58)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at com.inductiveautomation.ignition.gateway.localdb.sqlite.SQLiteDBManager$AutoBackupDaemon.run(SQLiteDBManager.java:666)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | 	at java.lang.Thread.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:14:33 | E [i.U.Wrapper                   ] [19:14:31]: Error fetching groups.java.lang.Exception: Unexpected exception during internal group listing. 
INFO   | jvm 2    | 2019/03/06 08:16:17 | W [ClockDriftDetector            ] [19:16:10]: Clock drift, degraded performance, or pause-the-world detected.
INFO   | jvm 2    | 2019/03/06 08:16:17 | Max allowed deviation=1000ms, actual deviation=86966ms 
INFO   | jvm 2    | 2019/03/06 08:16:17 | E [c.i.i.g.l.s.SingleConnectionDatasource] [19:16:15]: The following stack successfully received a connection. A new attempt was blocked for over 30000 ms tag-provider=default, scanclass=System Expression
INFO   | jvm 2    | 2019/03/06 08:16:17 | java.lang.Throwable: null
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.localdb.sqlite.SingleConnectionDatasource.getConnection(SingleConnectionDatasource.java:58)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.localdb.sqlite.SQLiteDBManager$AutoBackupDaemon.run(SQLiteDBManager.java:666)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.lang.Thread.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | E [T.P.Config                    ] [19:16:15]: Error storing tag values. tag-provider=default, scanclass=System Expression
INFO   | jvm 2    | 2019/03/06 08:16:17 | simpleorm.utils.SException$Jdbc: Opening com.inductiveautomation.ignition.gateway.localdb.sqlite.SingleConnectionDatasource@780650fd
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at simpleorm.sessionjdbc.SSessionJdbc.innerOpen(SSessionJdbc.java:113)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.localdb.persistence.PersistenceSession.initialize(PersistenceSession.java:31)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.localdb.PersistenceInterfaceImpl.getSession(PersistenceInterfaceImpl.java:62)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.localdb.PersistenceInterfaceImpl.getSession(PersistenceInterfaceImpl.java:44)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.openIfNot(InternalTagStore.java:1303)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.internalStoreTagValues(InternalTagStore.java:1341)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.storeTagValues(InternalTagStore.java:1250)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider.tagValuesChanged(AbstractStoreBasedTagProvider.java:2433)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.sqltags.scanclasses.SimpleExecutableScanClass$ScanClassTagEvaluationContext.processAndReset(SimpleExecutableScanClass.java:1141)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.sqltags.scanclasses.SimpleExecutableScanClass.run(SimpleExecutableScanClass.java:901)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$SelfSchedulingRunner.run(BasicExecutionEngine.java:483)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.FutureTask.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at java.lang.Thread.run(Unknown Source)
INFO   | jvm 2    | 2019/03/06 08:16:17 | Caused by: java.sql.SQLException: Connection is locked. Datasource only allows one connection at a time. More information was logged to the gateway console.
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at com.inductiveautomation.ignition.gateway.localdb.sqlite.SingleConnectionDatasource.getConnection(SingleConnectionDatasource.java:75)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	at simpleorm.sessionjdbc.SSessionJdbc.innerOpen(SSessionJdbc.java:111)
INFO   | jvm 2    | 2019/03/06 08:16:17 | 	... 18 common frames omitted
INFO   | jvm 2    | 2019/03/06 08:16:17 | W [ClockDriftDetector            ] [19:16:17]: Clock drift, degraded performance, or pause-the-world detected.
INFO   | jvm 2    | 2019/03/06 08:16:17 | Max allowed deviation=1000ms, actual deviation=5968ms 

What version of Ignition? You may need to temporarily turn off autobackups - support can help you out with that process, and some other mitigation strategies for this. Also, how big is the config.idb file (or just how large is a .gwbk of your system?)

v7.9.10. The config.idb file is 57MB, .gwbk is similar at 60MB.
I’ve tried using the DB-ShrinkTool mentioned on another thread but it wouldn’t enable the ‘clean database’ button…

Capture

I’ve also tried uninstalling Ignition and installing in a different directory, and still seem to be having the issues which is surprising, as it points to a gateway/project issue not an install one.

Not sure where that tool came from, but it’s probably only built to work on our legacy internal DB format. I would get in contact with support - there’s some optimization strategies that can help things out, but the underlying issue is typically “your internal database is getting used for too much” which is part of the reason we moved projects onto the file system in 8.0.

Ok thanks, still waiting on response from Support.

Hello Richard, did you get any response on how to prevent that ?

That tool is the pre-SQLite cleanup tool. It doesn’t work on SQLite, iirc.

Hi Francois,

I ended up trying a few things:

  1. Reinstalled Ignition on a completely new and clean Virtual Machine. I couldn’t restore from backup, so i had to export/import items individually like tags/windows etc…

  2. Disabled auto internal backups with the following direction from support:

You can do that by editing the “gateway.xml” file in the Ignition install directory’s “data” folder. The localdb.autobackup.count parameter should be set to 0, and then the Gateway will need to be restarted for the change to take effect.

Hope one of those things will help