Tags disappearing after power failure

Ignition
1

Hi

Let me start with the obvious. We are begging the customer to install a UPS.

In the meantime, we have a gateway on 8.1.19 that seems to lose all its tags (and UDT definitions) after power failures. For those not familiar with the current situation in South Africa, we are getting planned, scheduled blackouts due to unplanned, unscheduled reasons.

The only clue I’ve found in the logs is below:

provider	06Sep2022 17:05:02	Error starting tag provider
simpleorm.utils.SException$Jdbc: Executing SELECT tagconfig.id, tagconfig.ProviderId, tagconfig.folderId, tagconfig.cfg, tagconfig.rank, tagconfig.name FROM tagconfig WHERE tagconfig.ProviderId = ? ORDER BY tagconfig.id

at simpleorm.sessionjdbc.SQueryExecute.executeQuery(SQueryExecute.java:63)

at simpleorm.sessionjdbc.SSessionJdbc.queryNoFlush(SSessionJdbc.java:525)

at simpleorm.sessionjdbc.SSessionJdbc.query(SSessionJdbc.java:534)

at com.inductiveautomation.ignition.gateway.localdb.PersistenceInterfaceImpl.query(PersistenceInterfaceImpl.java:147)

at com.inductiveautomation.ignition.gateway.tags.config.storage.internaljson.InternalJsonStorageManager.loadTags(InternalJsonStorageManager.java:341)

at com.inductiveautomation.ignition.gateway.tags.TagProviderImpl.loadTagsFromStorage(TagProviderImpl.java:1308)

at com.inductiveautomation.ignition.gateway.tags.TagProviderImpl.loadTags(TagProviderImpl.java:1301)

at com.inductiveautomation.ignition.gateway.tags.TagProviderImpl.startup(TagProviderImpl.java:356)

at com.inductiveautomation.ignition.gateway.tags.model.impl.GatewayTagManagerImpl.instantiateProvider(GatewayTagManagerImpl.java:591)

at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)

at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)

at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)

at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)

at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)

at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)

at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)

at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)

at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)

at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)

at com.inductiveautomation.ignition.gateway.tags.model.impl.GatewayTagManagerImpl.startup(GatewayTagManagerImpl.java:507)

at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1212)

at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:298)

at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:734)

at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:668)

at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:544)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: org.sqlite.SQLiteException: [SQLITE_CORRUPT] The database disk image is malformed (database disk image is malformed)

at org.sqlite.core.DB.newSQLException(DB.java:909)

at org.sqlite.core.DB.newSQLException(DB.java:921)

at org.sqlite.core.DB.execute(DB.java:825)

at org.sqlite.jdbc3.JDBC3PreparedStatement.executeQuery(JDBC3PreparedStatement.java:80)

at com.inductiveautomation.ignition.gateway.localdb.DelegatingDataSource$DelegatingConnection$DelegatingPreparedStatement.executeQuery(DelegatingDataSource.java:575)

at simpleorm.sessionjdbc.SQueryExecute.executeQuery(SQueryExecute.java:61)

... 30 common frames omitted

We are restoring the tags and UDT’s from an export file after every power failure, but have not noticed this behaviour on other installations and would like to figure out whether there is something we can do to remedy it. We did fix a similar message by deleting the system logs.idb file, but I’m not sure if there is a similar fix for the tag DB? Restoring a gateway backup does fix the tags, but after the next power failure, the same thing happens.

2

Seems pretty clear. You will need to get IA support involved.

1 Like
3

My guess, from https://www.sqlite.org/howtocorrupt.html:

3.1. Disk drives that do not honor sync requests

Unfortunately, most consumer-grade mass storage devices lie about syncing. Disk drives will report that content is safely on persistent media as soon as it reaches the track buffer and before actually being written to oxide. This makes the disk drives seem to operate faster (which is vitally important to the manufacturer so that they can show good benchmark numbers in trade magazines). And in fairness, the lie normally causes no harm, as long as there is no power loss or hard reset prior to the track buffer actually being written to oxide. But if a power loss or hard reset does occur, and if that results in content that was written after a sync reaching oxide while content written before the sync is still in a track buffer, then database corruption can occur.

If this is what's going on then I don't think there's anything we can do...

1 Like
4

Yep, just making sure it’s not something someone has experienced before. Deleting the system log idb file seemed to stop that error so I was hoping there’s a quick fix for the tag idb file as well.

5

Gateway runs on an SSD. Not the most industrial machine, but it is at least not a mechanical drive. Not that SSD’s don’t have their own power loss issues…

EDIT: After reading that extract above more closely, I see it still applies to my situation. Thanks.

If the DB is indeed corrupt, does restoring a tag export rebuild that DB?

6

The DB in question holds everything gateway configuration (except for your projects). So not just tags, also database connections, device connections, etc.

If you’ve got scheduled backups, that’d be a good thing to try. There’s also an autobackup folder next to the config.idb file in the $installDirectory/data/db folder; one of those might not be corrupted, and might have most of your recent work.

But, this is really something support is better equipped to guide you through ‘live’.

7

Will contact support right away. Thanks for the help!

1 Like
8

Not sure what voltage your IPC is at, I use these from Puls to keep OnLogic IPC’s online (24V DC in Panel, I use a Traco 24V->12V step down):

  • Mfr. Part No. UB10.241

  • Brand [PULS]

  • Mfr. Part No. UZK12.071

  • Brand [PULS]

1 Like