Gateway Start forever waiting for ProjectManager

Hi all,

After a gateway restart, I seem to be stuck in an eternal twilight (2+ hours) wherein the gateway starting seems to be blocked by the ProjectManager failing to start. The logs show a huge stream of the message:

ProjectManager not yet running, rescheduling for +2 seconds

I’ve delved into the logs but I don’t know enough the diagnose if this is the issue or just a symptom, much less how to potentially fix it. I’ve tried stopping and starting multiple times to no avail.

Since I can’t upload files as a new user, there’s a pastebin of the logs from a fresh re-start here: https://pastebin.com/cUdqQSce

Any help is appreciated. Thanks!

Hmm. Might be related to one of these errors:

INFO   | jvm 1    | 2019/08/12 23:03:02 | E [t.e.a.history                 ] [04:03:02]: Error storing tag history data store-forward-name=DB
INFO   | jvm 1    | 2019/08/12 23:03:02 | java.lang.IllegalArgumentException: Specified historical sink 'DB' does not exist.
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.history.HistoryManagerImpl.storeHistory(HistoryManagerImpl.java:166)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.gateway.tags.history.actor.HistoryActorFactory$DelayedEvaluationContainer.process(HistoryActorFactory.java:887)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.gateway.tags.history.actor.HistoryActorFactory$DelayedEvaluationContainer.run(HistoryActorFactory.java:848)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.lang.Thread.run(Unknown Source)

or

INFO   | jvm 1    | 2019/08/12 23:03:02 | E [C.BasicExecutionEngine        ] [04:03:02]: One-shot task com.inductiveautomation.ignition.gateway.IgnitionGateway$$Lambda$278/0x000000010038f040@50864d7e threw uncaught exception. 
INFO   | jvm 1    | 2019/08/12 23:03:02 | java.lang.AbstractMethodError: Receiver class com.mysql.jdbc.Connection does not define or inherit an implementation of the resolved method abstract getSchema()Ljava/lang/String; of interface java.sql.Connection.
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at org.apache.commons.dbcp2.DelegatingConnection.getSchema(DelegatingConnection.java:941)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at org.apache.commons.dbcp2.DelegatingConnection.getSchema(DelegatingConnection.java:941)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.datasource.SRConnectionWrapper.getSchema(SRConnectionWrapper.java:546)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.db.schema.DBTableSchema.refreshTableState(DBTableSchema.java:187)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.db.schema.DBTableSchema.verifyAndUpdate(DBTableSchema.java:87)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.journal.JournalTables$EventTable.verifyTable(JournalTables.java:66)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournal.verifyTables(AlarmJournal.java:161)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournal.tryInit(AlarmJournal.java:141)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournal.startup(AlarmJournal.java:121)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournalManager.createJournal(AlarmJournalManager.java:129)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournalManager.startup(AlarmJournalManager.java:112)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.alarming.AlarmManagerImpl.startup(AlarmManagerImpl.java:132)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1068)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:273)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:664)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:602)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2019/08/12 23:03:02 | 	at java.base/java.lang.Thread.run(Unknown Source)

I think you might have a version of the MySQL JDBC driver installed that is too old to run on this version of Java.

2 Likes

I know this is very old, but I have the same problem and it is definitely the MSSQL JDBC driver preventing me from starting. How do you go about updating this driver without access to the gateway? Is there a solution for that?

You need filesystem access to the gateway to delete the bad jar file. There is no way to use Ignition itself to fix it if Ignition won’t start.

Thanks for the quick response. So if I delete the old jar files, replace with the correct one and rename it the same as what was there before, then restart the gateway it could solve the issue?

I would just delete it. Then use the web interface to “upgrade” that driver.

Ok I will try that then, thank you!

Sorry but that would just be in the user-lib folder correct? I know there are temp files created elsewhere.

Yes, user-lib/jdbc. Just the one jar.

Thank you very much, we are back up again. :slight_smile: