This has been a common issue, and I think we have it sorted out for 7.7.2. We really should probably write a knowledgebase article about it.
The suggestion above, about deleting the scinfo entries and everything, is something, but not what I’d suggest.
Here’s what happened:
- Pre-7.7, tag history was stored identified by the gateway name. This is bad if there are multiple tag providers, as you have no way to distinguish them. This means that you can’t have the same tag path in two providers and successfully store history.
- In 7.7, we added a “providerName” column to sqlth_drv to make this identification.
As a result…
- New entries were created in sqlth_scinfo, which links driver to scan class.
- New entries were created in sqlth_te, which links tags to scan class (and thereby drivers).
That’s all fine. When querying history, for backwards compatibility, the system is supposed to look at tag paths for the specific provider, as well as older paths for non-specific providers. There is not supposed to be an issue in the tag system for multiple ids for a given path (the system has always expected that).
The problem appears to be in the way that the system calculates data validity, or “staleness”. By default, the system looks at a record of when the scan classes were executing (sqlth_sce), and marks the data valid or not. The goal of this system is to distinguish between periods of time when the system was not running for times when the data simply didn’t change. It appears that the system was using the wrong entries for the tags, and so the data previous to the upgraded was being marked as “stale”.
The most direct solution that we’ve been using is to update the sqlth_sce table to set the “start_time” of the rows for the new, more specific, driver ids to 0, so that it covers “all time”. Since this table is really used as a secondary check when querying, I usually don’t feel too bad about just resetting the values to cover a large range.
The important note about this issue is that while terribly annoying, no actual data has been lost. The issue is purely in the interpretation of the data when querying.
Changing the sqlth_drv tables as suggested above is relatively harmless if you only have one tag provider. If you have multiple, updating the sqlth_sce table is better.
As a final note, not exactly related to this but worth remembering, is that the most crucial part of the history tables is the sqlth_te table. If you lose that, there is no way to make sense of the data, so be careful.