HsqlDataStore - Error deserializing data from data store

c.bertulli · May 26, 2020, 2:19pm

Hi All,

I found my gateway logs full of this error message ( one error each second ):
HsqlDataStore - Error deserializing data from data store

java.lang.ClassNotFoundException: com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournal$AlarmJournalSFData

at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)

at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)

at java.base/java.lang.ClassLoader.loadClass(Unknown Source)

at java.base/java.lang.Class.forName0(Native Method)

at java.base/java.lang.Class.forName(Unknown Source)

at java.base/java.io.ObjectInputStream.resolveClass(Unknown Source)

at com.inductiveautomation.ignition.gateway.cluster.ModuleObjectInputStream.resolveClass(ModuleObjectInputStream.java:37)

at java.base/java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)

at java.base/java.io.ObjectInputStream.readClassDesc(Unknown Source)

at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)

at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)

at java.base/java.io.ObjectInputStream.readObject(Unknown Source)

at com.inductiveautomation.ignition.gateway.cluster.ClusterUtil.deserializeObject(ClusterUtil.java:96)

at com.inductiveautomation.ignition.gateway.cluster.ClusterUtil.deserializeObject(ClusterUtil.java:56)

at com.inductiveautomation.ignition.gateway.cluster.ClusterUtil.unzipAndDeserializeObject(ClusterUtil.java:90)

at com.inductiveautomation.ignition.gateway.history.stores.AbstractDatasourceStore.deserializeObject(AbstractDatasourceStore.java:497)

at com.inductiveautomation.ignition.gateway.history.stores.AbstractDatasourceStore.loadTransactions(AbstractDatasourceStore.java:570)

at com.inductiveautomation.ignition.gateway.history.stores.AbstractDatasourceStore.syncdTakeNext(AbstractDatasourceStore.java:513)

at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.takeNext(AbstractStore.java:213)

at com.inductiveautomation.ignition.gateway.history.stores.MultiStageStore.syncdTakeNext(MultiStageStore.java:165)

at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.takeNext(AbstractStore.java:213)

at com.inductiveautomation.ignition.gateway.history.forwarders.ForwarderThread.synchedTakeNext(ForwarderThread.java:200)

at com.inductiveautomation.ignition.gateway.history.forwarders.RedundancyAwareForwarder.synchedTakeNext(RedundancyAwareForwarder.java:68)

at com.inductiveautomation.ignition.gateway.history.forwarders.ForwarderThread.run(ForwarderThread.java:139)

From the first line java.lang.ClassNotFoundException: com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournal$AlarmJournalSFData I think it’s related to the alarm journal Store and Forward property ( Config>Alarming>Alarm Journal Profiles>Use Store and Forward), which is set to true

Any help on what is causing those error messages?

PS: The system is working fine
PS: Ignition 8.0.9

Cheers,

PGriffith · May 26, 2020, 4:56pm

Did you recently upgrade from an older version? There’s data in the datastore that can’t be deserialized in the newer version of Ignition due to changes in the code.

c.bertulli · May 26, 2020, 5:01pm

Nope, last time I upgraded was 2020-02-27 and it was from 8.0.4 to 8.0.9

c.bertulli · May 28, 2020, 12:46pm

I noticed that in the Log folders, there is a file called system_logs.idb-journal that keeps to appear and disappear

PGriffith · May 28, 2020, 3:11pm

The journal meta-file is a SQLite maintenance file - and only has to do with the logs DB, not the store and forward caches, which are HSQL file-based databases.

c.bertulli · May 28, 2020, 3:31pm

Ahm… ok
Do you know what can I check ?

PGriffith · May 28, 2020, 3:33pm

Well, 8.0.4 is ‘old’ enough to not have the changes I described. 8.0.6 is when the remote alarm journal and remote audit log were introduced; which entailed a refactor internally that lead to this. The AlarmJournalSFData inner class just doesn’t exist in 8.0.6+ - so whatever data is in your datacache is from that older version. If you haven’t noticed any issues at this point, it should be safe to delete it from quarantine - which should prevent it from ever showing up again.

c.bertulli · May 28, 2020, 3:57pm

How should I do it ?

PGriffith · May 28, 2020, 4:52pm

Well, if you can’t drop the records from the web interface, then you can delete the datacache entirely.

If you can afford a little downtime, then disable your database connection. If not, then disable the disk cache in store and forward settings for the database.
Navigate to ${ignitionInstallDirectory}/data/datacache/ and delete the entire folder with the name of your database connection.
Re-enable the DB connection or disk cache. A new disk cache should be automatically generated.

c.bertulli · May 28, 2020, 5:02pm

The thing is that from the web interface I have nothing stored in quarantine, should I do the procedure you suggested anyway?

PGriffith · May 28, 2020, 6:40pm

If you’re getting this error consistently, then yes. I think one issue is that this class cast exception isn’t being flagged as a failure that puts the records into quarantine.

c.bertulli · May 29, 2020, 11:39am

Thanks a lot @PGriffith!
I’m really happy about this forum! The Ignition community is the best!
Thanks again and have a wonderful day

Dilan · November 3, 2020, 2:55pm

Hi All,

Just wanted to chime in and say I had this issue as well, upgraded from 7.9.14 and then to 7.9.16 with data in quarantine which caused HsqlDataStore to flood the logs. Deleting the folders as suggested resolved my issue.