# Historian tag entry duplicates **URL:** \ **Category:** Ignition\ **Created:** [January 28, 2025, 8:00am UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473 "2025-01-28T08:00:28Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\ **Post date:** [January 28, 2025, 8:00am UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/1 "2025-01-28T08:00:28Z")
Hello Folks, I come to you expecting the answer, but I should ask anyway. First, some context: - I'm historizing tags manually with `system.tag.storeTagHistory`. NONE of them have history enabled in their configuration. - Tags can be renamed and moved around through a webdev API. - I make sure their history paths (in `sqlth_te`) are updated to reflect these changes - All tags are memory and their values may also come through that API. While moving and renaming tags works fine, If I send a value to a tag after it has been renamed or moved, a duplicate entry is created in `sqlth_te`. My guess is there's an internal cache that maps tags paths to tags ids, and this cache is not updated when I change tags paths with a script. Then, when a value is received with the new path, that path doesn't exist in the cache and a new entry is created, which results in a duplicate entry. So, here's the question: How can I handle that ? If my guess is right, can I update the cache myself, can I ask the historian to force a refresh of its cache, can I disable it entirely and at what cost ? If I'm wrong, what causes this and how do I fix it ? Or am I doomed to write some code to scan the historian tables and merge duplicates on a schedule ? edit: I tried sending a value after disabling and restarting the historian, and it didn't create a new entry. I believe this would confirm the cache hypothesis. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [January 28, 2025, 1:15pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/2 "2025-01-28T13:15:33Z")
I absolutely would expect the historian to cache metadata, and to **not** expect outside entities to muck around with its permanent storage. Seems to me that your manual historization should use your own tables. ---
**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\ **Post date:** [January 28, 2025, 1:18pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/3 "2025-01-28T13:18:04Z")
Yes, that's what I'm starting to consider. I wanted to take advantage of the historian's partitioning... ---
**Author:** ![djs](https://avatars.discourse-cdn.com/v4/letter/d/e47774/32.png) [@djs](https://forum.inductiveautomation.com/u/djs)\ **Post date:** [January 28, 2025, 6:50pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/4 "2025-01-28T18:50:04Z")
Does the duplicate entry actually break any trends if you update the old "retired" tagpath to match the new one? If the _retired_ and _created_ timestamps match up correctly and the tagpaths are the same, then I believe the historian trending functions should handle 2 different entries. ---
**Author:** ![dillon](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/dillon/32/81537_2.png) [@dillon](https://forum.inductiveautomation.com/u/dillon)\ **Post date:** [January 28, 2025, 6:56pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/5 "2025-01-28T18:56:34Z")
> [@pascal.fragnoud](#): > > Yes, that's what I'm starting to consider. I've heard good things about [GitHub - timescale/timescaledb: A time-series database for high-performance real-time analytics packaged as a Postgres extension](https://github.com/timescale/timescaledb) ---
**Author:** ![djs](https://avatars.discourse-cdn.com/v4/letter/d/e47774/32.png) [@djs](https://forum.inductiveautomation.com/u/djs)\ **Post date:** [January 28, 2025, 7:07pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/6 "2025-01-28T19:07:03Z")
I don't recommend this approach for anything in production that you plan to support or maintain, because as already stated it is unsupported and could break at any time. However... If you really need the historian to update its cache, you can just restart the historian. As in, edit the config on the gateway and save it again. If you watch the logs, you'll see these entries: ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/4/2/42e5178ecb4453d8d93fd4bc0a212460a477f561.png) which should make it use what you updated in the table. Or if you want an even less stable solution, you could automate it with this: > [@system.tag.storeTagHistory() Creating duplicate IDs in sqlth\_te](https://forum.inductiveautomation.com/t/system-tag-storetaghistory-creating-duplicate-ids-in-sqlth-te/93857/2): > > For anyone curious. This function called from the gateway scope will "restart" the history provider: from com.inductiveautomation.ignition.gateway import IgnitionGateway def restartHist(prov): context = IgnitionGateway.get() tagHistMan = context.getTagHistoryManager() tagHistProv = tagHistMan.getTagHistoryProvider(prov) tagHistProv.shutdown() tagHistProv.startup() (Be aware that messing with the historian DB tables or SDK can seriously mess up your historian or gateway.) Use at your own … But again, not something I'd put in production... ---
**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\ **Post date:** [January 28, 2025, 7:53pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/7 "2025-01-28T19:53:12Z")
The "old" and "new" have the exact same tagpath. That's my issue. I have a tag at path "folder1/foo". I store a value for it, so an entry is created in sqlth\_te. Now, I send a request to the API I made to change its path. Let's make it "folder2/foo". The endpoint does a bunch of things, including changing the tag path in sqlth\_te. It is now "folder2/foo". Now I send a new value to store for this tag. The path I use matches the path in the historian. But because it was cached under a different path, we get a cache miss, and a new entry is created, with the same path: "folder2/foo". Now I have 2 entries in sqlth\_te with that same path. There is no retired timestamp. The initial tag was never retired. This whole thing doesn't happen if i restart the historian, I've tried that in an effort to confirm that it's a cache issue. But all of this needs to be done programmatically, and _might_ happen several times per second. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [January 28, 2025, 7:58pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/8 "2025-01-28T19:58:03Z")
I am not shocked that your own manipulation of the metadata is simply not reflected in the historian. 🤷‍♂️ ---
**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\ **Post date:** [January 28, 2025, 8:15pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/9 "2025-01-28T20:15:22Z")
Yes I'm not surprised either, but I hoped that one of you would have a magical solution. ---
**Author:** ![djs](https://avatars.discourse-cdn.com/v4/letter/d/e47774/32.png) [@djs](https://forum.inductiveautomation.com/u/djs)\ **Post date:** [January 28, 2025, 8:37pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/10 "2025-01-28T20:37:56Z")
What if you manually retire an entry whenever you change its path? Expect the cache miss and let the historian create a new entry. From my testing, trending should still work. Are the duplicate entries actually a problem that you have to solve? I understand the downside, that every time you'd "rename" a tag you'd get an additional entry in sqlth\_te, but unless you expect to rename the same tag 100s of times and the trending works, is it a big deal? ---
**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\ **Post date:** [January 28, 2025, 11:49pm UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/11 "2025-01-28T23:49:47Z")
I also retrieve data manually. I guess I could use that in my queries, but it seems to add another layer of complexity. I think the sane option is the one Phil suggested in his first answer, but I have yet to bounce that on the deciding peoples, and I’m not sure it’ll fly. ---
**Author:** ![djs](https://avatars.discourse-cdn.com/v4/letter/d/e47774/32.png) [@djs](https://forum.inductiveautomation.com/u/djs)\ **Post date:** [January 29, 2025, 12:43am UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/12 "2025-01-29T00:43:25Z")
If you're also retrieving data manually then I would completely agree with Phil and just implement the historian yourself. I don't know about timescaledb but in terms of timeseries databases I have used [QuestDB](https://questdb.com/), and the partitioning of a table is all handled automatically in the background by the database. You create a table, say that you want it partitioned by some period of time and to you (as the user) it will just look like one table even though its partitioned in the background. It's very easy. You can also just use SQL to interact with it even though its not a relational database. QuestDB is opensource and is actually the database that IA's Power Historian in 8.3 is built on top of. A downside of QuestDB is that at the moment the data is not compressed (like timescaledb) so it's not as efficient space wise. Though I heard they're working on remedying that later this year with complete Parquet support... ---
**Author:** ![javier](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/javier/32/72418_2.png) [@javier](https://forum.inductiveautomation.com/u/javier)\ **Post date:** [February 7, 2025, 9:19am UTC](https://forum.inductiveautomation.com/t/historian-tag-entry-duplicates/101473/13 "2025-02-07T09:19:57Z")
Hey Daniel! QuestDB Developer Advocate here. For compression, something you can do is mounting a ZFS filesystem with compression enabled and move your questdb root folder there. We've seen some significant compression ratios with performance being equivalent or better, as QuestDB needs to read less data from disk. More info at [Enable compression with ZFS | QuestDB](https://questdb.com/docs/guides/compression-zfs/) Other than that, you are right that we will be releasing native parquet support later this year 🙂