Custom Tag History Aggregate

liam1 · January 13, 2025, 8:52am

Hi all,

I'm trying to make a custom Tag History Aggregate which will return the min and max value for a period of time but excludes 0's.The reason for this is because the totalizer tag history has erroneous zero values recorded

Here is the code I wrote so far. Can someone point out what I am doing wrong here?

def myMinMax(qval, interpolated, finished, blockContext, queryContext):
	# Initialize min and max values
	current_min = blockContext.getOrDefault('current_min', float(0))
	current_max = blockContext.getOrDefault('current_max', float(0))
	
	# Check if the incoming value is good and not zero
	if qval.quality.isGood() and qval.value != 0:
	    current_min = min(current_min, float(qval.value))
	    current_max = max(current_max, float(qval.value))
	
	# Debugging output
	print("Current Min: {}, Current Max: {}".format(current_min, current_max))
	
	# Store updated min and max in blockContext
	try:
	    blockContext['current_min'] = current_min
	    blockContext['current_max'] = current_max
	except Exception as e:
	    print("Error updating blockContext: {}".format(e))
	
	# If finished is true, return results
	if finished:
	    # If no valid values were found, return None
	    
	    if current_min == 0 or current_max == 0:
	        return None
	    return  [current_min, current_max]

Here is the stack trace error I'm getting when trying to run it:

Java Traceback:
Traceback (most recent call last):
  File "<input>", line 13, in <module>
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:360)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:500)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:292)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:287)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.runTagCalculationQuery(GatewayInterface.java:828)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.queryTagCalculationsImpl(ClientTagUtilities.java:349)

	at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.queryTagCalculations(AbstractTagUtilities.java:808)

	at jdk.internal.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Unable to read response from Gateway.


	at org.python.core.Py.JavaError(Py.java:545)

	at org.python.core.Py.JavaError(Py.java:536)

	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:192)

	at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:553)

	at org.python.core.PyObject.__call__(PyObject.java:400)

	at org.python.pycode._pyx394.f$0(<input>:20)

	at org.python.pycode._pyx394.call_function(<input>)

	at org.python.core.PyTableCode.call(PyTableCode.java:173)

	at org.python.core.PyCode.call(PyCode.java:18)

	at org.python.core.Py.runCode(Py.java:1703)

	at org.python.core.Py.exec(Py.java:1747)

	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:277)

	at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:130)

	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$ConsoleWorker.doInBackground(JythonConsole.java:628)

	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$ConsoleWorker.doInBackground(JythonConsole.java:616)

	at java.desktop/javax.swing.SwingWorker$1.call(Unknown Source)

	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

	at java.desktop/javax.swing.SwingWorker.run(Unknown Source)

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

	at java.base/java.lang.Thread.run(Unknown Source)

Caused by: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Unable to read response from Gateway.

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:360)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:500)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:292)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:287)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.runTagCalculationQuery(GatewayInterface.java:828)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.queryTagCalculationsImpl(ClientTagUtilities.java:349)

	at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.queryTagCalculations(AbstractTagUtilities.java:808)

	at jdk.internal.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)

	... 18 more

Caused by: com.inductiveautomation.ignition.client.gateway_interface.ResponseParser$1: Error reading data element [row=0, column="AD-HOC-PYTHON", type=Long]
java.lang.NumberFormatException: For input string: "299748.8405131454"

	at com.inductiveautomation.ignition.client.gateway_interface.ResponseParser.endElement(ResponseParser.java:290)

	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)

	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:498)

	... 27 more

Caused by: java.lang.NumberFormatException: For input string: "299748.8405131454"

	at java.base/java.lang.NumberFormatException.forInputString(Unknown Source)

	at java.base/java.lang.Long.parseLong(Unknown Source)

	at java.base/java.lang.Long.parseLong(Unknown Source)

	at com.inductiveautomation.ignition.common.TypeUtilities.coerceLocaleSafe(TypeUtilities.java:899)

	at com.inductiveautomation.ignition.client.gateway_interface.ResponseParser.endElement(ResponseParser.java:273)

	... 38 more

Traceback (most recent call last):
  File "<input>", line 13, in <module>
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:360)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:500)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:292)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:287)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.runTagCalculationQuery(GatewayInterface.java:828)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.queryTagCalculationsImpl(ClientTagUtilities.java:349)

	at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.queryTagCalculations(AbstractTagUtilities.java:808)

	at jdk.internal.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Unable to read response from Gateway.

Here is how I am calling the aggregation function:

# Get current date
end = system.date.now()


start =system.date.addDays(end, -20) #end.replace(day=1, hour=0, minute=0, second=0, microsecond=0)




query=system.tag.queryTagCalculations(
	paths=['[Canary/default:default]HYPERV-CANARY/Ignition/process/p_auxiliary/chlorine/pressure_transmitters/post_chlorine/totalizer'],
	calculations=['shared.myMinMax'],
	startDate=start,
	returnSize=100,
	ignoreBadQuality=True,
	endDate=end)
print query

Transistor · January 13, 2025, 9:58am

I don't think I can help, but what is the line
print("Current Min: {}, Current Max: {}".format(current_min, current_max))
supposed to do?

liam1 · January 13, 2025, 10:14am

It's for debugging.

Transistor · January 13, 2025, 10:41am

I know that, but I'd never seen the, "Current Min: {}, Current Max: {}" syntax before with a .format function. I tried it out in Script Console and I can see the result. Thanks.

JordanCClark · January 13, 2025, 1:00pm

You will need to find where this string is coming from. It's trying to cast it as a long, but it can't, because of the fractional part of the value. I can't tell you if it's because of the aggregate function or not.

Caused by: java.lang.NumberFormatException: For input string: "299748.8405131454"

	at java.base/java.lang.NumberFormatException.forInputString(Unknown Source)

	at java.base/java.lang.Long.parseLong(Unknown Source)

What happens if you do the calculation outside of the query?

end = system.date.now()
start =system.date.addDays(end, -20)

query=system.tag.queryTagCalculations(
	paths=['[Canary/default:default]HYPERV-CANARY/Ignition/process/p_auxiliary/chlorine/pressure_transmitters/post_chlorine/totalizer'],
	startDate=start,
	returnSize=100,
	ignoreBadQuality=True,
	endDate=end)

# Set returns unique values of a list. Subtracting set([0]) removes the zero, if it is present.
values = set(query.getColumnAsString(1)) - set([0])

print 'min: {}, max:{}'.format(min(values), max(values))

lrose · January 13, 2025, 2:44pm

So, I'm not sure where or why your getting a failed conversion of a string to a long, that seems odd to say the least. It says it's coming from your "debugging output", which I would be surprised if that was actually doing what you wanted anyway (perhaps it was, print behaves in different ways depending on the context in which it is called).

Anyway, a couple of things:

Generally you want to initialize a minimum with the max possible so that even a max value will result in it being the minimum. In your code, 0 will always be the minimum (making the assumption that a totalizer will never be negative). And so I would expect it to always return None.
There should be no reason to cast the values type. They should always match.
You should either utilize the queryContext's logging functions or create your own logger for logging errors or other information. I prefer using my own logger as it gives me more options.
You're not handling if the system decides to break the query up across multiple windows, which will lead to you getting an incorrect result at some point.

I would write the aggregation like this:
NOTE: If you use this don't forget to add the import at the top of the library.

import sys.float_info as floatInfo

logger = system.util.getLogger("shared.myMinMax")

def myMinMax(qval, interpolated, finished, blockContext, queryContext):
	
	# Initialize min and max values
	lastMin = queryContext.get('lastMin', floatInfo.max)
	lastMax = queryContext.get('lastMax', 0.0)
	current_min = blockContext.get('current_min', lastMin)
	current_max = blockContext.get('current_max', lastMax)
	
	# Check if the incoming value is good and not zero
	if qval.quality.good and qval.value == 0:
	    current_min = min(current_min, qval.value)
	    current_max = max(current_max, qval.value)
	
	logger.infof("Current Min: %.2f, Current Max: %2f", current_min, current_max)
	
	queryContext['lastMin'] = min(current_min,lastMin)
	queryContext['lastMax'] = max(current_max,lastMax)
	
	# If finished is true, return results
	if finished:
	    return  [current_min, current_max]
	
	blockContext['current_min'] = current_min
	blockContext['current_max'] = current_max

liam1 · January 14, 2025, 6:14am

Thank you for the extra information,

I tried your aggregation function which runs without errors but the output is not as expected:

Here is the query I used

paths=['[Canary/default:default]HYPERV-CANARY/Ignition/process/p_01/ferric/mes/south east ferric usage/running totalizer']
end=system.date.now()
start=system.date.addMonths(end, -1)

query=system.tag.queryTagCalculations(
	paths=paths, 
	calculations=['shared.myMinMax'], 
	startDate=start, 
	endDate=end
	)
print query.getColumnNames()
for row in system.dataset.toPyDataSet(query):
	print row[0],row[1]

here is the output with column headers and a single row

[tagpath, AD-HOC-PYTHON]
running totalizer 153553.197439

Not sure what the AD-HOC-PYTHON column is.
The output isn't the values I would expect fromt he return defined

	if finished:
	    return  [current_min, current_max]

I tried returning a dataset with my own column headers and it still produced the same output which is weird.

	if finished:
	    return  system.dataset.toDataSet(['min','max'], [[current_min, current_max]])

JordanCClark · January 14, 2025, 11:07am

???

lrose · January 14, 2025, 1:10pm

When the historian returns the results for queryTagCalculations() the column names are tagpath and the calculation preformed. Since it doesn't have a set name for the given calculation it just uses AD-HOC-PYTHON to indicate that a custom aggregate was preformed.

In my testing, I am unable to get either queryTagHistory() or queryTagCalculations() to return multiple values, even though the documentation clearly states that they should. It appears that it just returns whatever the first value it comes to, so in this case the minimum.

Perhaps one of the IA guys will come by and look at it.

I would suggest at this point that you contact support.

MMaynard · January 14, 2025, 3:03pm

I use it all the time, the code I use is below

calcs = ['Average','Minimum','Maximum','StdDev']|

dataSetCalc = system.tag.queryTagCalculations(paths=data,calculations=calcs,aliases=chdr, startDate=startTime, endDate=endTime,ignoreBadQuality=1,nointerpolation=0)|
cDS = system.dataset.toPyDataSet(dataSetCalc)|

And it returns

"tagpath","Average","Minimum","Maximum","StdDev"
"C5 Belt - Heatsink Temp","55.85749323949571","55.400001525878906","57.19999694824219","0.8996302901778891"

liam1 · January 14, 2025, 3:15pm

We are referring to the return values of custom aggregation functions

lrose · January 14, 2025, 3:36pm

Me too, I use custom aggregates, but they only return a single value per window. @liam1 is wanting to return multiple values per window. That's what I am referencing.

MMaynard · January 14, 2025, 3:48pm

Gotcha, I will go back to the corner

PGriffith · January 14, 2025, 4:26pm

From a quick glance at the code, I don't think this is possible either; it seems like you're restricted to returning a single column of results from each aggregate.

lrose · January 14, 2025, 4:32pm

Interesting, I wonder if there is a way to pass an object reference back?

Otherwise, seems like a bug.

JordanCClark · January 14, 2025, 4:55pm

So, for now at least, it seems that @liam1 will need to separate his custom aggregates?

PGriffith · January 14, 2025, 4:58pm

Hm, okay, I dug deeper and I think I was mistaken. If you return a list, it should be packing those arguments and returning them from the aggregate as an array, which should work the same way as e.g. MinMax returns two values.

You can also return a list of tuples, where the first item in each tuple is the value and the second item is an (old style) quality code integer value.

At least, according to the code

lrose · January 14, 2025, 5:59pm

Except it doesn't, which is weird.

This:

script = """\
python:def wrapper(qval, interpolated, finished, blockContext, queryContext):

	lastMin = queryContext.get('lastMin', 100000.0)
	lastMax = queryContext.get('lastMax',0.0)
	current_min = blockContext.get('current_min', lastMin)
	current_max = blockContext.get('current_max', lastMax)
	
	if qval.quality.good and qval.value:
		current_min = min(current_min, qval.value)
		current_max = max(current_max, qval.value)
	
	queryContext['lastMin'] = min(current_min,lastMin)
	queryContext['lastMax'] = max(current_max,lastMax)
	
	if finished:
		return [400,500]
	
	blockContext['current_min'] = current_min
	blockContext['current_max'] = current_max
"""

ret = system.tag.queryTagHistory(tagPath, rangeHours=8, aggregationModes=[script],returnSize = 2)
print [ret.getValueAt(row,col) for row in range(ret.rowCount) for col in range(ret.columnCount)]

Returns:

[Tue Jan 14 04:47:23 EST 2025, 400L, Tue Jan 14 08:47:23 EST 2025, 400L]