Yesterday we performed an upgrade from 7.9.12 to 8.0.8. We are regularly getting a batch of errors in the logs for the HsqlDataStore “Error deserializing data from data store”. We get ~50 of these errors per batch. Everything seems to be working fine, it’s just really filling up the logs. Here is one of the full errors:
java.io.InvalidClassException: com.inductiveautomation.ignition.common.config.BasicProperty; local class incompatible: stream classdesc serialVersionUID = 7327305604978257671, local class serialVersionUID = 8982054396802939425
at java.base/java.io.ObjectStreamClass.initNonProxy(Unknown Source)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.base/java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
at java.base/java.util.HashMap.readObject(Unknown Source)
at java.base/jdk.internal.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
at com.inductiveautomation.ignition.gateway.cluster.ClusterUtil.deserializeObject(ClusterUtil.java:96)
at com.inductiveautomation.ignition.gateway.cluster.ClusterUtil.deserializeObject(ClusterUtil.java:56)
at com.inductiveautomation.ignition.gateway.cluster.ClusterUtil.unzipAndDeserializeObject(ClusterUtil.java:90)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractDatasourceStore.deserializeObject(AbstractDatasourceStore.java:497)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractDatasourceStore.loadTransactions(AbstractDatasourceStore.java:570)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractDatasourceStore.syncdTakeNext(AbstractDatasourceStore.java:513)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.takeNext(AbstractStore.java:213)
at com.inductiveautomation.ignition.gateway.history.stores.MultiStageStore.syncdTakeNext(MultiStageStore.java:165)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.takeNext(AbstractStore.java:213)
at com.inductiveautomation.ignition.gateway.history.forwarders.ForwarderThread.synchedTakeNext(ForwarderThread.java:200)
at com.inductiveautomation.ignition.gateway.history.forwarders.RedundancyAwareForwarder.synchedTakeNext(RedundancyAwareForwarder.java:68)
at com.inductiveautomation.ignition.gateway.history.forwarders.ForwarderThread.run(ForwarderThread.java:139)
This is what I would expect if you're putting data into store and forward from a different version of Ignition. Is it always the same count of records? Are you deleting them from quarantine, and they re-appear, or are you just retrying them? It might just be the same chunk of records, continually retrying - if you haven't noticed missing data, then it's probably safe to drop them and see whether anything new shows up.
Should be unrelated to the JDBC driver issue. The JDBC-ODBC bridge likely doesn't support Java 11.
Update: My co-worker informed me that we actually had around 60 quarantined records immediately following the upgrade to 8.0.8. They were retried and haven’t appeared back in the quarantine since. Possible that they are stuck? Should we try a gateway restart?
Thanks Paul. It does appear to be the same number of records each time (50) but I’ll be honest, I only counted a few of the batches to confirm this. We don’t have anything in quarantine at the moment and haven’t been retrying any. We had a few records (3, if I remember correctly) when we first started up the gateway after the upgrade to 8.0.8. We retried those and they went through. Below is a screenshot of the store and forward status screen.
I’ve been monitoring the store and forward since yesterday. Haven’t seen any records in quarantine. But we are still seeing these errors repeatedly. This error has added roughly 96 pages of logs to the log viewer on the gateway webpage in the past 24 hours.
Sounds to me like you need to open a proper support ticket and let IA dig into your system. You aren’t providing any new details that might indicate a solution to pursue.
(This forum is not official support, in case you didn’t know.)
Thanks Phil. Yeah, it hasn’t been a critical issue from what we can see so I was holding off on opening a ticket. Wasn’t sure - based on the details I provided that we don’t have quarantined items in store and forward - if that would lead to any other suggestions on what to check. But it sounds like I have my answer to that now. I’ll be opening a ticket.
We’re still trying to resolve this issue. The initial test was to try “temporarily disabling the Local Disk Cache on the Store & Forward engine(s)”. This hasn’t cleared up the error. I’ll post updates as we work through the problem. Please share if you find a fix!
Support was able to help us fix this issue. Note: This fix deletes any data held up in the cache, so you may need to contact support on how to get this data out of the cache if it is important to keep.
The fix ended up being to disable the store and forward caches, navigate to the datacache folder in the ignition directory, save a copy of the datacache folder somewhere (just in case you need anything inside of it later down the road), delete the contents of the datacache folder, and then re-enable the store and forward caches. This will force the system to reinitialize the datacache folder from scratch. You do not have to stop or restart the gateway for this fix.
It’s still not clear what caused the issue in the first place. Support was unable to replicate the issue when restoring our gateway and datacache. But one thing that was clear is that we had one store and forward connection that was showing around 3,000 items in the local cache. Those items never moved, they always just showed in the local cache. Never moved on to the database or into quarantine. Not sure if there is anything else we can provide for tracking down the root cause to prevent it for others in the future.
