Better way to deal with large numbers of getTagValue calls

Greg.Buehler · November 3, 2011, 8:16pm

I’m trying to refactor part of a system right now, but I thought this might be a good question to put on the forums.

Doing large amounts of tag reads is slow, especially over a VPN connection, is there a better way of reading in a bunch of values?

Example:

def GetData():
	import system
	headers = ["Alarm", "Warning", "Device", "Sensor Number", "Sensor Description", "Actual", "Warning SP", "Alarm SP", "Overridden"]
	data = []
	# build about 25-100 sensors into list
	data.append([system.tag.getTagValue("xxx"), system.tag.getTagValue("yyy"),"zzz","aaa", "bbb", system.tag.getTagValue("ccc"), system.tag.getTagValue("eee"), system.tag.getTagValue("fff"), system.tag.getTagValue("ggg")])

The example code generates 150 to 600 individual getTagValue calls which makes things slow. Is there a better way of batching data?

Travis.Cox · November 3, 2011, 8:48pm

Well, there is a new function system.tag.getTagValues that allows you to read multiple tags at once. It will be faster than doing individual tag reads. It just takes a list of tag paths.

diat150 · November 4, 2011, 12:07am

Travis, could I just replace getTagValue with getTagValues in this code or would I have to do something else to meet the list requirement?

[code]query = “SELECT name, identifier from sbutest”
if event.propertyName in [“devices”, “refresh”]:

table = event.source.getComponent('Table')
newData = []
header = ["Displayname", "Identifier", "Current Throughput", "Today's Throughput", "devicestate", "good polls", "bad polls"]

res = system.db.runQuery(query)
for row in res:
	displayName = row[0]
	tagPath = row[1]
	
	currThru = system.tag.getTagValue("%s/curr_throughput" % tagPath)
	todThru = system.tag.getTagValue("%s/today_throughput" % tagPath)
	devState = system.tag.getTagValue("%s/PrimaryInterval" % tagPath)
	goodPoll = system.tag.getTagValue("%s/curr_goodscans" % tagPath)
	badPoll = system.tag.getTagValue("%s/curr_failedscans" % tagPath)

	

	newData.append([displayName, tagPath, currThru, todThru, devState, goodPoll, badPoll])

table.data = system.dataset.toDataSet(header, newData)[/code]

Kevin.Herron · November 4, 2011, 1:51am

This is untested, but using system.tag.getTagValues() would look something like this:

query = "SELECT name, identifier from sbutest"

if event.propertyName in ["devices", "refresh"]:
	table = event.source.getComponent('Table')
	newData = []
	header = ["Displayname", "Identifier", "Current Throughput", "Today's Throughput", "devicestate", "good polls", "bad polls"]
	res = system.db.runQuery(query)
	
	for row in res:
		displayName = row[0]
		tagPath = row[1]
		
		tagPaths = map(lambda x: "%s/%s" % (tagPath, x), ["curr_throughput", "today_throughput", "PrimaryInterval", "curr_goodscans", "curr_failedscans"])
		
		results = system.tag.getTagValues(tagPaths)
      
		newData.append([displayName, tagPath, results[0], results[1], results[2], results[3], results[4]])

table.data = system.dataset.toDataSet(header, newData)

diat150 · December 20, 2011, 1:39am

Thanks Kevin. That doesnt seem to work for me.

I get this error:

[code]Traceback (innermost last):
File “event:propertyChange”, line 13, in ?
File “event:propertyChange”, line 0, in
NameError: tagpath

at org.python.core.Py.NameError(Py.java)
at org.python.core.PyFrame.getglobal(PyFrame.java)
at org.python.pycode._pyx30.f$1(<event:propertyChange>)
at org.python.pycode._pyx30.call_function(<event:propertyChange>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyFunction.__call__(PyFunction.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.__builtin__.map(__builtin__.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.pycode._pyx30.f$0(<event:propertyChange>:13)
at org.python.pycode._pyx30.call_function(<event:propertyChange>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:394)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.runActions(ActionAdapter.java:139)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.invoke(ActionAdapter.java:247)
at com.inductiveautomation.factorypmi.application.binding.action.RelayInvocationHandler.invoke(RelayInvocationHandler.java:55)
at $Proxy2.propertyChange(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at java.awt.Component.firePropertyChange(Unknown Source)
at com.inductiveautomation.vision.api.client.components.model.AbstractVisionPanel.fireDynChange(AbstractVisionPanel.java:128)
at com.inductiveautomation.factorypmi.application.binding.util.DynamicPropertyUtil.setPropertyValue(DynamicPropertyUtil.java:217)
at com.inductiveautomation.vision.api.client.components.model.AbstractVisionPanel.setPropertyValue(AbstractVisionPanel.java:112)
at com.inductiveautomation.factorypmi.application.script.PyComponentWrapper.__setattr__(PyComponentWrapper.java:68)
at org.python.pycode._pyx7.f$0(<event:mouseClicked>:1)
at org.python.pycode._pyx7.call_function(<event:mouseClicked>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:394)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.runActions(ActionAdapter.java:139)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.invoke(ActionAdapter.java:247)
at com.inductiveautomation.factorypmi.application.binding.action.RelayInvocationHandler.invoke(RelayInvocationHandler.java:55)
at $Proxy1.mouseClicked(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$2.run(Unknown Source)
at java.awt.EventQueue$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

Ignition v7.2.11-beta1 (b209)
Java: Sun Microsystems Inc. 1.6.0_26
[/code]

Ive tried a few things but cant seem to figure out what is wrong here.

Travis.Cox · December 20, 2011, 4:08pm

The variable is called tagPath with a capital P. Got to love case sensitivity

diat150 · December 20, 2011, 5:04pm

sorry, while playing around and trying different things I guess I mistyped the name. same end result, different cause!

[code]Traceback (innermost last):
File “event:propertyChange”, line 13, in ?
File “event:propertyChange”, line 0, in
NameError: tagPath

at org.python.core.Py.NameError(Py.java)
at org.python.core.PyFrame.getglobal(PyFrame.java)
at org.python.pycode._pyx3.f$1(<event:propertyChange>)
at org.python.pycode._pyx3.call_function(<event:propertyChange>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyFunction.__call__(PyFunction.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.__builtin__.map(__builtin__.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.pycode._pyx3.f$0(<event:propertyChange>:13)
at org.python.pycode._pyx3.call_function(<event:propertyChange>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:394)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.runActions(ActionAdapter.java:139)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.invoke(ActionAdapter.java:247)
at com.inductiveautomation.factorypmi.application.binding.action.RelayInvocationHandler.invoke(RelayInvocationHandler.java:55)
at $Proxy2.propertyChange(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at java.beans.PropertyChangeSupport.firePropertyChange(Unknown Source)
at java.awt.Component.firePropertyChange(Unknown Source)
at com.inductiveautomation.vision.api.client.components.model.AbstractVisionPanel.fireDynChange(AbstractVisionPanel.java:128)
at com.inductiveautomation.factorypmi.application.binding.util.DynamicPropertyUtil.setPropertyValue(DynamicPropertyUtil.java:217)
at com.inductiveautomation.vision.api.client.components.model.AbstractVisionPanel.setPropertyValue(AbstractVisionPanel.java:112)
at com.inductiveautomation.factorypmi.application.script.PyComponentWrapper.__setattr__(PyComponentWrapper.java:68)
at org.python.pycode._pyx2.f$0(<event:mouseClicked>:1)
at org.python.pycode._pyx2.call_function(<event:mouseClicked>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:394)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.runActions(ActionAdapter.java:139)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.invoke(ActionAdapter.java:247)
at com.inductiveautomation.factorypmi.application.binding.action.RelayInvocationHandler.invoke(RelayInvocationHandler.java:55)
at $Proxy1.mouseClicked(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

Ignition v7.2.11-beta1 (b209)
Java: Sun Microsystems Inc. 1.6.0_23
[/code]

Travis.Cox · December 20, 2011, 5:17pm

Just make sure everywhere you use tagPath is it consistent. Can you post your code exactly?

diat150 · December 20, 2011, 5:34pm

should be the same as above.

[code]query = “SELECT name, identifier from sbutest”

if event.propertyName in [“devices”, “refresh”]:
table = event.source.getComponent(‘Table’)
newData = []
header = [“Displayname”, “Identifier”, “Current Throughput”, “Today’s Throughput”, “devicestate”, “good polls”, “bad polls”]
res = system.db.runQuery(query)

for row in res:
displayName = row[0]
tagPath = row[1]

   tagPaths = map(lambda x: "%s/%s" % (tagPath, x), ["curr_throughput", "today_throughput", "PrimaryInterval", "curr_goodscans", "curr_failedscans"])
  
   results = system.tag.getTagValues(tagPaths)
 
   newData.append([displayName, tagPath, results[0], results[1], results[2], results[3], results[4]])

table.data = system.dataset.toDataSet(header, newData)[/code]

Travis.Cox · December 20, 2011, 5:41pm

Not sure exactly where the problem lies. Try replacing the line with the lambda with this one:tagPaths = ["%scurr_throughput"%tagPath, "%stoday_throughput"%tagPath, "%sPrimaryInterval"%tagPath, "%scurr_goodscans"%tagPath, "%scurr_failedscans"%tagPath]Does that work or do you get the same error?

diat150 · December 20, 2011, 7:35pm

It does allow the script to run, but I am not sure that it is making a big difference in time. I went ahead and changed it to only grab 1 tag instead of 5 different tags just to track the time difference without having to wait as long and it seems to only make about a 10 second difference.

to run the script for 465 total tags using the getTagValue method it took 70 seconds

to run the script for 465 total tags using the getTagValues method it took 60 seconds

Travis.Cox · December 20, 2011, 8:03pm

That is because you are calling the system.tag.getTagValues function 465 / 5 = 93 times. You want to only call the function a few times. You can modify the code like this:[code]query = “SELECT name, identifier from sbutest”

if event.propertyName in [“devices”, “refresh”]:
table = event.source.getComponent(‘Table’)
newData = []
header = [“Displayname”, “Identifier”, “Current Throughput”, “Today’s Throughput”, “devicestate”, “good polls”, “bad polls”]
res = system.db.runQuery(query)
tagPaths = []

for row in res:
tagPath = row[1]

   tagPaths.append("%scurr_throughput"%tagPath)
   tagPaths.append("%stoday_throughput"%tagPath)
   tagPaths.append("%sPrimaryInterval"%tagPath)
   tagPaths.append("%scurr_goodscans"%tagPath)
   tagPaths.append("%scurr_failedscans"%tagPath)

results = system.tag.getTagValues(tagPaths)

idx = 0
for row in res:
displayName = row[0]
tagPath = row[1]
newData.append([displayName, tagPath, results[0+idx], results[1+idx], results[2+idx], results[3+idx], results[4+idx]])
idx += 5

table.data = system.dataset.toDataSet(header, newData)[/code]

diat150 · December 20, 2011, 8:35pm

all I can say is WOW.

just grabbed 7915 tags in like 1 second.