Ignition Historian Mystery

Hi, so the fundamental issue is we have boolean OPC tags (polled via OPC-UA, this is important see below) that are valid live and correct, that have historians attached set to record on change, and the changes do not seem to be captured in the data bases.

The full description follows:
We are replacing an old hardware screen based HMI that is failing with an Ignition screen. One feature of this screen was a fault table. There are ~300 or so faults (there are six identical modules that can throw the same faults so ~50 separate faults, on 6 modules).

To avoid crudding up our databases as we integrated this, our IT team had us use a test SQL database for the historian. Everything worked swimmingly.

I then mass selected the tags, Edit Tag, for Storage Provider switched to our production database and... nothing. There are only a few errors that we can safely force high - door opening interlocks, Estop button presses, and errorneous settings requests - and the remaining errors happen to rarely for us to have a pattern on them yet, but it seems that when a fault changes state from high to low or back again, it is not historicized.

The only thing I can think is somehow in transitioning from one historian DB to the other something was missed?
The tags all show the new historian as their storage provider.
No other changes about their History properties were changed from when they were directed to TestDB - and worked fine.
The tag changes after the DB transition are not reflected in the old DB, and the tags are all visible in the sqlth_te database on the production DB - so its not that they're misdirected.
The tags live two folders deep (one folder for the PLC they come off of then a separate tag folder for alarms, as there are so many we don't want our normal tags to be lost in a haystack) this didn't matter for the test DB
When we create a new memory tag in that folder, with the production DB as storage provider and toggle it, we DO see the tag pop up on the DB.
It really does seem like something was lost in pulling the tags from using one historian to the other, as the magic seems to be 'they work out of the box but fail if transferred'.

The answer to the obvious question is the sqth_te database will be 40% by volume fault tags for this one device if I simply delete and recreate these tags anew with the production DB as their storage provider. And 20% will be 'retired', dead tags. Not a catastrophe, but it bugs me a little bit to accept this as a solution.

The code that populates the fault table on the HMI also filters on tag path and as we'll have 2 copies for every tag path, one retired and one active I do worry that will break something else.
Filtering on tagid would work for now as all tags will exist in one contiguous block, but if we ever add another tag, the fragmented nature will make the code a mess.

Out of curiousity are you using a UDT? For my system I have an alarm UDT that I put into another UDT. If I was in your position I would have just gone to the Alarm UDT and turned on history and pointed it to the Historian. Was this not possible for you?

No, these are all standard, out-of-the-box Ignition defined OPC tags. Edited my initial question to be more clear.

I would select the highest level folder that contains all of the tags, right click, and restart tags

That should force a full refresh of the tag properties.

I was unaware that was an option, interesting. Does not seem to have fixed the issue though

So, the mystery has been solved with a call to the tech support group.

If you have configured a historian on a tag, set it to On Change, and are not seeing its changes appear at the top of your sqlt_data_#_YYYY_MM database, it is likely that the PLC does not have the most current time on its internal clock. The Ignition server does NOT append a datetime based comparing the server clock to when a tag change is reported by the various PLCs reporting to it, but RATHER appends a datetime based on what time the PLC reports that it is, when the tag changes. Updating the PLC clock was the solution.

NOTE: THIS ISSUE ONLY ARISES when "PLC/protocol actually supports timestamps, such as DNP3 or 61850, or if you have a direct connection to an onboard OPC UA server on that PLC"

In more detail, what happened is, that our PLC (on an air-gapped private network) had its internal clock fall WELL out of synchonization (specifically running slow). Due to this, when Ignition clocks its 'On Change' tags transitioning states (but seemingly NOT when the sample mode is timebased), it uses the timestamp from the PLC.

I think this is not optimal for a few reasons. Our PLC can't be easily synched with a time server due to security concerns. It can be synchronized with a centralized server on the gapped network using SNTP - once we upgrade the firmware. Until such time, the synchronization has already sensibly decoupled over the weekend (seconds flip a noticeable fraction of a second apart).

Essentially, as things stand, in a facility with multiple PLCs reporting to an Ignition server, there are multiple sources of truth around when things happen. This can be an issue trying to use Ignition historians to reconstruct the root cause of a cascading fault that affects systems managed by different PLCs, in the worst case. In less bad cases, not being able to trust that events happened in any particular order leads to needless math-problem headaches for those operators cognizant of this issue, and needless errors for those operators not cognizant.

The only reason I can think of why the central Ignition server would defer to a reporting PLC on when a boolean bit flipped is in the case of a facility with I/O drops distributed so widely they cross time zones. Nevertheless, the need to understand what happened -first-, i.e. causality/correlation is the point of using UTC across all devices and then converting into local time when and as appropriate.

I suppose after a loss of connection, if multiple reportable events have occurred prior to connection restoration, this might help establish a timeline. I haven't bothered to disconnect the PLC from the network, flip a bit, then reconnect and see. I have a strong sense, however, that the timestamp will be when the network is reestablished, not when the bit flipped.

  • Tested the above, this is NOT the case for our hardware. In fact, not only are bit flips that occur while disconnected not reported, but you get a large amount of spam as every point historicized off that device is uploaded as the change from 'bad quality' to 'good quality' counts as a historicizeable event. Not sure I have an issue with that, but the above corner case can be excluded from consideration.

Nevertheless, these are somewhat edge cases, when the main use case will be a single server networked with PLCs within no more than +/- 500 miles east/west (roughly the width of a time zone). In which case a single clock should be used, and IMO that clock should be the device recording the reports of the various reporting PLCs.

Thanks to Oliver on the tech support team for resolving this.

I believe you are able to select the timestamp source more explicitly as of 8.3.1. For OPC UA connections, anyway…

8.1.50 but good to know.

Hopefully the opt-out default is 'Server = sole source of truth'

To clarify for any future readers, since I don't see relevant details anywhere in this thread: this is only the case when the PLC/protocol actually supports timestamps, such as DNP3 or 61850, or if you have a direct connection to an onboard OPC UA server on that PLC, as opposed to a traditional polling-based protocol.

Ahh, ok thought that calling them 'OPC tags' encapsulated that info but I do understand that we have several PLCs whose tags we poll without an OPC-UA connection. I'll update the problem statement.

An OPC tag only tells you that it comes from an OPC server.

That could be our server, which then has one or more drivers configured, polling various PLCs. It could be Kepware or some other third party OPC server. It could be the OPC UA server built into more modern PLCs.