Automated quarantine export/delete

Ignition
1

We have a slow, ongoing issue with quarantine history data. We have a lot of Sparkplug edge devices which are cautious about ensuring data makes it through to a listening server, in that they are likely to send a measurement for the same path/timestamp multiple times if there was a networking issue after the prior transmission. That rarely happens with a good Internet connection, but we have a lot of cellular devices and some of them have very bad connections. This happens for us hourly.

The first issue, which is annoying but livable, is that it logs at ERROR level every time this happens (MemoryForwardTransaction or DatasourceForwardTransaction with a cause like java.sql.BatchUpdateException: (conn=1426630) Duplicate entry '908299-1755541696924' for key 'PRIMARY').

Less livable is that it slowly and steadily fills up the quarantine. I have to remember to go to the server and retry/export/delete the quarantines every so often or things go bad. I usually retry the quarantine a couple times to shake out the history that weren’t duplicated but just stuck in the same records, and then export-then-delete and keep the quarantine files around for a while.

What’s my easiest path to automating the quarantine retry/export/delete to run daily or weekly?

(EDIT: I get why most users wouldn’t want it, but in our case I wish I could configure the store-and-forward engine to use something like MySQL’s “INSERT ON DUPLICATE KEY UPDATE” for history tags and just fix the whole issue at the source…)

2

I understand the huge pain it would be, however, there is nothing to stop you from scripting the data entry for these tags. Then, assuming you have a compitent, database you could do this.

I have done similar things when I needed to use the history tables in ways that the history system isn’t really built for.

3

Which scope are you envisioning when you suggest scripting up the data entry? Do you mean a tag change script that does the history injection? Or writing up a new Ignition module to be a new history provider?

We have hundreds of thousands of tags that come in from these devices, so the tag change script is kind of dead in the water due to overhead. And many of these are currently coming in via the MQTT Engine “Store Historical Events” feature. (We are slowly converting those tags over to reference tags on normal providers, but we’re not there yet…)

4

Well, obviously, I have no knowledge of the scope of your tag structure. Knowing that, then, probably a custom history module, but I don’t know all the technical debt that goes into writing a custom history provider.

I was initially envisioning this as a gateway event, perhaps, a timer event to emulate a historical scan class, that would do the injection. Though you could certainly do it in large batches, it does kind of make things less apealing.