# Dataset Not Including Duplicate Values

**URL:** <https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356>\
**Category:** Ignition\
**Tags:** scripting\
**Created:** [October 16, 2024, 11:00pm UTC](https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356 "2024-10-16T23:00:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joe\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/b5a626/32.png) [@Joe\_Alcide](https://forum.inductiveautomation.com/u/Joe_Alcide)\
**Post date:** [October 16, 2024, 11:00pm UTC](https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356/1 "2024-10-16T23:00:51Z")

</div>

Hi!

I am having the following dilemma: I am trying to record an average of tag values from tag history for the last twelve hours. The average mass balance over the 12 hours. I have a separate gateway timer event script to do the mass balance ever hour. I have this in a tag value changed event script:

```python
endTime = system.date.now()
startTime = system.date.addHours(endTime,-13)
dataSet = system.tag.queryTagHistory(paths=['[default]New Configuration of Application/GENESIS/Final Mass Balance/Current Total Mass Balance'], startDate=startTime, endDate=endTime, noInterpolation=True, ignoreBadQuality=True)
system.tag.write('[default]New Configuration of Application/GENESIS/Final Mass Balance/Mass Balance History Storage',dataSet)

```

This is pushing the values to a dataset tag. This dataset however wont include duplicate values, for example:

| T\_stamp | Value |
| --- | --- |
| 3 PM | 1 |
| 4 PM | 2 |
| 5 PM | 3 |
| 6 PM | 4 |
| 7 PM | 5 |
| 8 PM | 5 |

is only showing in the ignition dataset tag as

| T\_stamp | Value |
| --- | --- |
| 3 PM | 1 |
| 4 PM | 2 |
| 5 PM | 3 |
| 6 PM | 4 |
| 8 PM | 5 |

The tag still shows that its writing a new value so its still executing the tag event change script just fine. However, for whatever reason it wont keep the older duplicate values with different timestamps. I know its been writing the old value to the database before it duplicates before the next run/change. It just disppears once the script is run again. There is no deadband on the tags.

The non-duplicate values it works fine for. If anyone can think of anything and let me know, I would appreciate it! Stuck on this for awhile.

---

<div class="post-metadata">

**Author:** ![ryan.white](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/ryan.white/32/73193_2.png) [@ryan.white](https://forum.inductiveautomation.com/u/ryan.white)\
**Post date:** [October 16, 2024, 11:26pm UTC](https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356/2 "2024-10-16T23:26:39Z")

</div>

> [@Joe\_Alcide](#):
>
> I have this in a tag value changed event script:

Don't. You should only have fast executing scripts(\<5ms) in a Tag Value Change Event. This script is querying tag history from a db, that is extremely slow. This should be executing in a Gateway Tag Change Event.

Honestly it should just be included at the end of your Gateway Scheduled Script. You have the latest hour value, just grab the preceding hours from tag history and combine before writing to the tag.

As for fixing no duplicates, try including the parameter `returnSize=0` in your call of system.tag.queryTagHistory. `returnSize` defaults to -1, which returns values on change instead of returning all the logged points in the time period.

> **[system.tag.queryTagHistory | Ignition User Manual](https://www.docs.inductiveautomation.com/docs/8.1/appendix/scripting-functions/system-tag/system-tag-queryTagHistory#parameters)**
>
> Issues a query to the Tag Historian.

---

<div class="post-metadata">

**Author:** ![Joe\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/b5a626/32.png) [@Joe\_Alcide](https://forum.inductiveautomation.com/u/Joe_Alcide)\
**Post date:** [October 17, 2024, 12:54pm UTC](https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356/3 "2024-10-17T12:54:32Z")

</div>

Thanks!

---

<div class="post-metadata">

**Author:** ![Joe\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/b5a626/32.png) [@Joe\_Alcide](https://forum.inductiveautomation.com/u/Joe_Alcide)\
**Post date:** [October 17, 2024, 1:40pm UTC](https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356/4 "2024-10-17T13:40:29Z")

</div>

Hey y'all, so this didn't work (changing returnSize to 0). Does anyone have any other ideas?

---

<div class="post-metadata">

**Author:** ![Joe\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/b5a626/32.png) [@Joe\_Alcide](https://forum.inductiveautomation.com/u/Joe_Alcide)\
**Post date:** [October 17, 2024, 6:25pm UTC](https://forum.inductiveautomation.com/t/dataset-not-including-duplicate-values/97356/5 "2024-10-17T18:25:23Z")

</div>

Hey y'all, so the tag history has a deadband. The deadband is what was erasing the previous values. Set the deadband mode to off.