Unexpected Values in historian query CSV

I want to explain that as in the chart plotted below - we can see that the value keeps increasing and then on Reset goes to Zero - it will move only in the direction of increment and then drops to zero because it is a totalizer in PLC to keep incrementing the total flow with a reset button

Now lets have a look on the screenshot of the CSV file generated from the query of the python script

I have a value of 244 then the value drops to 231 .14 and then drops to 10 - these two values are not consistent with the trend because the value does not decrease it goes directly to zero and after that it increases - Now I am not sure why this is happening

The tag is set to periodic at 1 Second - the data set supplied is generated from the query which takes a start time and end time and then takes values in between those two points - so how can I avoid this from happening

What is the scan class of the tag? Considering that record is on a 500ms time stamp, it is likely an interpolated value.

1000

Don’t know what your script looks like, did you set interpolation to false?

If you look close at the data in the t_stamp column you will notice that each record is 2.5 sec’s apart. The particular point in question happens to be at one of these splits. I’m guessing that if you look at other times where a reset happens, unless the starting value was fairly close to zero, that you’ll find a similar situation.

I would say those values are not actual values but interpolated values.

The bigger question is why with a tag set up in this way are you not getting any values at a 1 sec. interval.

Because the script takes the start date and end date and number of data points which are calculated from total seconds divided by user selection (5,10,30,60 sec) and then it captures data for the CSV file - so basically if the data collection timestamp falls after it was rest to zero instead of showing either zero or the incremented value the script thinks that the value is now reducing and then calculates some value and produces that in the CSV

import csv
import datetime
# Obtain the java init date from the Input on the screen 
datastart = event.source.parent.getComponent('Start Date').date
dataend =event.source.parent.getComponent('Start Date 1').date
#Calculate the number of hours in between the two dates 



#Return size is the data points to be collected from starttime to endtime
Return_size=system.date.secondsBetween(datastart, dataend)/event.source.parent.getComponent('Dropdown_Datapoints').selectedValue
print(Return_size)
data = system.tag.queryTagHistory(paths = ["[edge]Engine2_rpm.value","[edge]Pump2_RPM.value","[edge]Pump2_strokes.value","[edge]Pressure_sensor_Pump2.value","[edge]FlowRatePump2.value","[edge]Pump2_flow_total.value"],startDate=datastart, endDate=dataend, returnSize=Return_size, aggregationMode="Average", returnFormat='Wide')

csv = system.dataset.toCSV(data)
 
CustomerName = event.source.parent.getComponent('cstnm').text
JobNumber = event.source.parent.getComponent('Jobnumber').text
From=system.date.format(datastart, "yyyy-MM-dd-HH-mm-ss")
To=system.date.format(dataend, "yyyy-MM-dd-HH-mm-ss")
 

fnm = "PUMP2_DATA_"+From+"_to_"+To+".csv"
filename = system.file.saveFile(fnm)
if filename is not None:
   system.file.writeFile(filename,csv)

what happens if you turn off the interpolation?

system.tag.queryTagHistory(noInterpolation = True, paths = ...)

Remember that it takes time for the reset to occur and the system to log that the value has changed to 0. When reporting analog values the system assumes that it is generally impossible for a value to change instantaneously.

As @lrose mentioned, interpolation is the cause of your issue. Interpolated data is the devil in a lot of cases. It certainly has its place, but if you’re looking for accurate data, then you should only be displaying/exporting the raw data points. Generally I only use interpolation for larger time periods when displaying on charts, and raw for everything else where details matter and especially for exported data. It’s not such a big problem for constantly fluctuating values like an analogue transmitter that change values more or less gradually over time, but it is for things like integers and discrete-value tags (like your totaliser) that change values every so often, as then you’ll get linearly interpolated values between the raw data points that don’t reflect reality

Turned off interpolation and still the same - still seeing those ghost values.

You will need to set the returnSize to -1 as well to return the onChange data and not aggregated data.