I’m trying to insert some values into the MSSQL Historian connection using a simple script:
ds = self.view.custom.FTHistorian_Data
pyData = system.dataset.toPyDataSet(ds)
paths = []
values = []
timestamps = []
for row in pyData:
paths.append(row["PathIgnition"])
values.append(row["Value"])
# Convert DateTime string into Ignition date object
timestamps.append(system.date.parse(row["DateTime"], "yyyy-MM-dd HH:mm:ss"))
# Execution
system.historian.storeDataPoints(paths, values, timestamps)
I have a test table that contains the Ignition path, the value, and the timestamp.
When I start my project, some entries are correctly inserted into the historian (tags at ID 1 and 2).
My historian is configured with monthly partitions and no pruning.
The problem is:
When I run the script, the data is inserted into the historian, but Ignition creates new partitions with older dates (for example, 2 years back).
I can browse and see the values, but Ignition also creates new entries in sqlth_te with new IDs and different created dates for the same tag.
After that, I get a log like this:
mssql_db Executing partition maintenance. Dropping Tag history partitions:
[sqlt_data_1_2025_05, sqlt_data_1_2025_06, sqlt_data_1_2025_07, sqlt_data_1_2025_08]
So all the old values are being erased.
I tried to manually change the created date to 0 in sqlth_te for the entry that doesn’t have a retired date (the last active mapping of that tag). But when I run the script again, the same issue happens: new partitions and new IDs are generated, and the older partitions are dropped.
Context:
I’m currently on Ignition 8.3.4 Beta (not sure if this is related).
My goal is to import previous data from an FT Historian into the new Ignition Historian, while keeping everything in the same structure and without losing data.
I’m also attaching screenshots of my sqlth_te table and my test source table (right now I’m only inserting test values, not the real ones).
My question:
How can I correctly insert backfill data (historical values) into Ignition’s historian without creating duplicate tag IDs or dropping existing partitions?
Is this a limitation of the current beta or am I missing a specific configuration step?
Creating additional tag IDs cannot be avoided. Ignition makes new tag IDs for a variety of tag configuration operations. Ignition native history operations gather from all applicable tag IDs when given a timespan.
The partition dropping behavior sounds like a bug in the beta.
The Linux image I was using on Docker was: image: inductiveautomation/ignition:8.3.0-beta4.
So if I understand correctly, there’s nothing else I need to configure, and I should just try the new 8.3.0 release with -rc1 to see if the bug is still there? I’m going to give it a try and will post an update if the error persists.
It looks like data isn't being written in order. Is the data being queried out the FTHistorian_Test in order by datetime? The behavior with the new sqlth_te entries being created is due to data being written that contains a timestamp that is prior to the existing unretired entry created time; this would result in the existing entry being retired and a new entry being created.
And, the Allow Back-fill Data property on the tag provider setting is for data being processed directly by the tag. The storeDataPoints method doesn't store the data against the tag; it bypasses that and stores directly to the Historian.
Did you change the pruning settings at all on the MSSQL_HISTORIAN?
Another note. Manually updating the created / retired times within the sqlth_te entry will not affect anything within the SQL Historian unless you disable / enable it. These entries are cached and are not re-queried.
UPDATE:
I dug into this some more and found the bug regarding pruning; the issue is that if pruning was once enabled, it'll remain active for the duration of the Gateway, regardless if it was disabled. In the meantime, make sure that pruning is disabled and restart the Gateway.
Yes, my table FTHistorian_Test is queried in ascending order before inserting the entries, using this named query:
SELECT *
FROM FTHistorian_Test
WHERE PathFTHistorian in ({PathFTHistorian})
order by PathFTHistorian asc,datetime asc
With your last update, I forgot to mention that I had enabled pruning to let Ignition delete my previous data and try the script again. The first time it worked fine, but afterward, when I tried enabling and disabling it again, it stopped working and I started getting critical errors every time, such as:
TagHistoryDatasourceSink (Data sink 'mssql_historian' has encountered critical failure)
I wasn’t sure if I was doing something wrong, so I’m glad you confirmed it’s a real bug. (I’m on RC1 now, not Beta as I previously mentioned in the post.)
Is there a way to know when this issue will be fixed, or should I just keep an eye on the next release’s patch notes?
It may or may not get a release note (we've been lax about release notes for things found and fixed in beta), but it looks like he already fixed this and it's targeted to the final release.
My historian is configured with monthly partitions and no pruning.
Context:
