Inconsistent Errors / 'Blocked threads' when trying to use system.tag.configure to create multiple tags

Ignition
1

Hi,

I’m still fairly new to Ignition, so apologies if I’m making some very basic mistakes.

I’m using system.tag.configure to create a large set of tags from a .csv file.
I’m finding sometimes it works quickly, sometimes it works very slowly, sometimes the interpreter hangs, sometimes it throws back an error, sometimes it will create 3 tags, then hang.

Can anyone help me understand what would cause the errors I’m seeing below?

When the script errors I’ll see messages telling me the connection to the gateway has been lost.
There are no errors in the Logs, but I’ll see that a thread has been blocked with the following info.

Thread [tag-provider-My_Tags] id=87, (BLOCKED)
app//com.inductiveautomation.ignition.gateway.tags.actors.factories.value.AbstractValueGeneratingActor.setListener(AbstractValueGeneratingActor.java:37)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.ExecutableTag.manageActor(ExecutableTag.java:231)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.ExecutableTag.configureActors(ExecutableTag.java:756)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.ExecutableTag.applyPropertyChangesToTag(ExecutableTag.java:605)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.managePropertyChanges(BasicTagDefinition.java:776)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.manageChangesFromPropModel(BasicTagDefinition.java:758)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition$$Lambda$593/0x000000010063d840.accept(Unknown Source)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.TagPropertyModel.dispatchPropertiesChanged(TagPropertyModel.java:64)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.TagPropertyModel.onPropertiesChanged(TagPropertyModel.java:42)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.InheritedExecutableTag.onPathChanged(InheritedExecutableTag.java:75)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.FolderNode$$Lambda$2541/0x0000000100f5e440.accept(Unknown Source)
java.base@11.0.11/java.util.ArrayList.forEach(Unknown Source)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.FolderNode.onPathChanged(FolderNode.java:403)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.applyPropertyChangesToTag(BasicTagDefinition.java:799)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.managePropertyChanges(BasicTagDefinition.java:776)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.startPropModel(BasicTagDefinition.java:472)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.activate(BasicTagDefinition.java:927)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.startupInternal(BasicTagDefinition.java:911)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.MemberTagDefinition.startupInternal(MemberTagDefinition.java:84)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.startup(BasicTagDefinition.java:897)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.FolderNode.startup(FolderNode.java:293)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.AbstractTopLevelUdtTag.configureChild(AbstractTopLevelUdtTag.java:442)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.AbstractTopLevelUdtTag.manageMemberChanges(AbstractTopLevelUdtTag.java:261)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.AbstractTopLevelUdtTag.onSuperMembersChanged(AbstractTopLevelUdtTag.java:175)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.onSuperMembersChanged(TypesFolder.java:525)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.onValidityChanged(TypesFolder.java:472)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.setValidityState(TypesFolder.java:450)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.refreshValidity(TypesFolder.java:373)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.setSuperType(TypesFolder.java:312)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.notifyParentTypeChanged(TypesFolder.java:290)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder$AbstractTypeNode.attach(TypesFolder.java:282)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.TypesFolder.registerComplexNode(TypesFolder.java:112)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.AbstractComplexTagDefinitionNode.startupInternal(AbstractComplexTagDefinitionNode.java:118)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.startup(BasicTagDefinition.java:897)
app//com.inductiveautomation.ignition.gateway.tags.runtime.nodes.FolderNode.startup(FolderNode.java:293)
app//com.inductiveautomation.ignition.gateway.tags.runtime.TagEvaluationManagerImpl.processNewTag(TagEvaluationManagerImpl.java:546)
app//com.inductiveautomation.ignition.gateway.tags.runtime.TagEvaluationManagerImpl.processEditInternal(TagEvaluationManagerImpl.java:385)
app//com.inductiveautomation.ignition.gateway.tags.runtime.TagEvaluationManagerImpl.processEdit(TagEvaluationManagerImpl.java:299)
app//com.inductiveautomation.ignition.gateway.tags.TagProviderImpl.saveTagConfigInternal(TagProviderImpl.java:758)
app//com.inductiveautomation.ignition.gateway.tags.TagProviderImpl.lambda$saveTagConfigsAsync$11(TagProviderImpl.java:715)
app//com.inductiveautomation.ignition.gateway.tags.TagProviderImpl$$Lambda$2543/0x0000000100f5dc40.get(Unknown Source)
app//com.inductiveautomation.ignition.gateway.tags.TagProviderImpl.lambda$exec$3(TagProviderImpl.java:437)
app//com.inductiveautomation.ignition.gateway.tags.TagProviderImpl$$Lambda$2544/0x0000000100f5d040.get(Unknown Source)
java.base@11.0.11/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)

The error thrown back to me from the interpreter is:

Java Traceback:
Traceback (most recent call last):
  File "<input>", line 104, in <module>
  File "<input>", line 13, in openFile
  File "<input>", line 100, in addUDT
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:543)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:945)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.saveTagConfigs(ClientTagUtilities.java:105)

	at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.configure(AbstractTagUtilities.java:199)

	at jdk.internal.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out


	at org.python.core.Py.JavaError(Py.java:547)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.saveTagConfigs(ClientTagUtilities.java:108)

	at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.configure(AbstractTagUtilities.java:199)

	at jdk.internal.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)

	at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:546)

	at org.python.core.PyObject.__call__(PyObject.java:494)

	at org.python.core.PyObject.__call__(PyObject.java:498)

	at org.python.pycode._pyx46.addUDT$2(<input>:102)

	at org.python.pycode._pyx46.call_function(<input>)

	at org.python.core.PyTableCode.call(PyTableCode.java:173)

	at org.python.core.PyBaseCode.call(PyBaseCode.java:134)

	at org.python.core.PyFunction.__call__(PyFunction.java:416)

	at org.python.pycode._pyx46.openFile$1(<input>:12)

	at org.python.pycode._pyx46.call_function(<input>)

	at org.python.core.PyTableCode.call(PyTableCode.java:173)

	at org.python.core.PyBaseCode.call(PyBaseCode.java:119)

	at org.python.core.PyFunction.__call__(PyFunction.java:406)

	at org.python.pycode._pyx46.f$0(<input>:104)

	at org.python.pycode._pyx46.call_function(<input>)

	at org.python.core.PyTableCode.call(PyTableCode.java:173)

	at org.python.core.PyCode.call(PyCode.java:18)

	at org.python.core.Py.runCode(Py.java:1687)

	at org.python.core.Py.exec(Py.java:1731)

	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:277)

	at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:130)

	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$ConsoleWorker.doInBackground(JythonConsole.java:605)

	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$ConsoleWorker.doInBackground(JythonConsole.java:593)

	at java.desktop/javax.swing.SwingWorker$1.call(Unknown Source)

	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

	at java.desktop/javax.swing.SwingWorker.run(Unknown Source)

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

	at java.base/java.lang.Thread.run(Unknown Source)

Caused by: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:543)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:945)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.saveTagConfigs(ClientTagUtilities.java:105)

	... 34 more

Caused by: java.net.SocketTimeoutException: Read timed out

	at java.base/java.net.SocketInputStream.socketRead0(Native Method)

	at java.base/java.net.SocketInputStream.socketRead(Unknown Source)

	at java.base/java.net.SocketInputStream.read(Unknown Source)

	at java.base/java.net.SocketInputStream.read(Unknown Source)

	at java.base/java.io.BufferedInputStream.fill(Unknown Source)

	at java.base/java.io.BufferedInputStream.read1(Unknown Source)

	at java.base/java.io.BufferedInputStream.read(Unknown Source)

	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)

	at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)

	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)

	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)

	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:437)

	... 38 more

Traceback (most recent call last):
  File "<input>", line 104, in <module>
  File "<input>", line 13, in openFile
  File "<input>", line 100, in addUDT
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:543)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:945)

	at com.inductiveautomation.ignition.client.script.ClientTagUtilities.saveTagConfigs(ClientTagUtilities.java:105)

	at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.configure(AbstractTagUtilities.java:199)

	at jdk.internal.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out
2

It might help if you showed us the script.

hint: I’m guessing you’re making repeated calls to system.tag.configure, one for each tag. Am I right ?

3

Hi, I am, for each new tag.

The vast amount of prints are me trying to fault find.
(FYI it was based on this Mass import of UDT tags via simple csv and script (example))

import csv

def openFile():
	filePath = system.file.openFile("csv","%user%")
	csvFile = open(filePath, 'r')
	print '------------------------------------------------------'
	print "Scanning .csv file for table headers..."
	reader = csv.DictReader(csvFile)
	print ".csv table headers found:"
	print reader.fieldnames
	print '------------------------------------------------------'
	for row in reader:
		if row['UDTType'] == 'UDT':	addUDT(row)
	
def addUDT(csvRow):
	print 'Now in addUDT subroutine.'
	tagProv = '[' + csvRow['provider'] + ']'
	print 'tagProv		=	' + tagProv
	baseTagPath = tagProv + csvRow['folder1'] + '/' + csvRow['folder2']
	print 'baseTagPath	=	' + baseTagPath
	tagName = csvRow['tagname']
	print 'tagName		=	' + tagName
	typeId = csvRow['UDTType']
	print 'typeId		=	' + typeId
	tagType = "UdtInstance"
	print 'tagType		=	' + tagType
	print 'Reading in Tag information...'
	tag = {
		"name" : tagName,
		"typeId" : typeId,
		"tagType" : tagType,
		"parameters" : {
			"item1" : csvRow['item1'],
			"item2" : csvRow['item2'],
			"item3" : csvRow['item3'],
			"item4" : csvRow['item4'],
			"item5" : csvRow['item5'],
			"item6" : csvRow['item6'],
			"item7" : csvRow['item7'],
			"item8" : csvRow['item8'],
			"item9" : csvRow['item9'],
			"item10" : csvRow['item10'],
			"item11" : csvRow['item11'],
			"item12" : csvRow['item12'],
			"item13" : csvRow['item13'],
			"item14" : csvRow['item14'],
			"item15" : csvRow['item15'],
			"item16" : csvRow['item16'],
			"item17" : csvRow['item17'],
			"item18" : csvRow['item18'],
			"item19" : csvRow['item19'],
			"item20" : csvRow['item20'],
			"item21" : csvRow['item21'],
			"item22" : csvRow['item22'],
			"item23" : csvRow['item23'],
			"item24" : csvRow['item24'],
			"item25" : csvRow['item25'],
			"item26" : csvRow['item26'],
			"item27" : csvRow['item27'],
			"item28" : csvRow['item28'],
			"item29" : csvRow['item29'],
			"item30" : csvRow['item30'],
			"item31" : csvRow['item31'],
			"item32" : csvRow['item32'],
			"item33" : csvRow['item33'],
			"item34" : csvRow['item34'],
			"item35" : csvRow['item35'],
			"item36" : csvRow['item36'],
			"item37" : csvRow['item37'],
			"item38" : csvRow['item38'],
			"item39" : csvRow['item39'],
			"item40" : csvRow['item40'],
			"item41" : csvRow['item41'],
			"item42" : csvRow['item42'],
			"item43" : csvRow['item43'],
			"item44" : csvRow['item44'],
			"item45" : csvRow['item45'],
			"item46" : csvRow['item46'],
			"item47" : csvRow['item47'],
			"item48" : csvRow['item48'],
			"item49" : csvRow['item49'],
			"item50" : csvRow['item50'],
			"item51" : csvRow['item51'],
			"item52" : csvRow['item52'],
			"item53" : csvRow['item53'],
			"item54" : csvRow['item54'],
			"item55" : csvRow['item55'],
			"item56" : csvRow['item56'],
			"item57" : csvRow['item57'],
			"item58" : csvRow['item58'],
			"item59" : csvRow['item59'],
			"item60" : csvRow['item60'],
			"item61" : csvRow['item61'],
			"item62" : csvRow['item62'],
			}
		}
	collisionPolicy = "o"
	print 'collisionPolicy	=	' + collisionPolicy
	print 'Now creating Tag...'
	system.tag.configure(baseTagPath, [tag], collisionPolicy)
	print 'Success! ' + typeId + ' ' + tagName + ' inserted.'
	print '------------------------------------------------------'
	
openFile()
4

Then you might want to try to ‘batch’ things.
You can actually build your tag structure/hierarchy with dicts and list, json style, then use that to configure the whole thing at once :


folder_with_three_tags = [
	{
		'name': "Foo",
		'tagType': "Folder",
		'tags': [
			{
				'name': "reference_tag",
				'tagType': "AtomicTag",
				'valueSource': "reference",
				'sourceTagPath': "some_tag_path",
				'dataType': "String"
			},
			{
				'name': "query_tag",
				'tagType': "AtomicTag",
				'valueSource': "db",
				'datasource': "database_name",
				'query': "SELECT something FROM sometable",
				"queryType": "Select"
			},
			{
				'name': "expression_tag",
				'tagType': "AtomicTag",
				'valueSource': "expr",
				'expression': "some expression",
				'dataType': "Boolean"
			}
		]
	}
]

system.tag.configure("[]", folder_with_three_tags, 'o')

Had to do something like that to add tags from a configuration file (and xml if I remember correctly), so I built a tree using the basic principle shown above, with quite a few levels of nesting, then used just one call to configure. It was several orders of magnitude faster than the previous version which configured tags one by one.
This might help with the issue. Or it might not because there’s something else wrong. But I’d do it anyway.

5

It would be good to contact support for this anyways. While making these configure requests more efficiently will almost certainly help, you shouldn’t be able to deadlock threads just by calling system functions directly.

6

Hi, thankyou for responding.

I don’t think creating a nested list / tree structure for importing will work for me as I need to create tags of a single UDT in the hundreds.

Following this train of thought, I tried the same script without trying to nest it into any folder structure, but just to dump the tags straight into the tag provider, but I still have the same issues, so I don’t think it’s a nested folder issue?

Is what I’m trying to do part of the intended use of this system function? or was the intention to create just 1 or 2 tags at a time with it?

7

Whatever you do from the designer’s script console will have to be handed off to the gateway to actually do the work. Each such handoff has a hard-coded 60-second timeout. You will probably have to batch your work.

8

Understood. But why sometimes is it successfully executed, creating 200 in under a couple of seconds, and other times it just hangs without completing a single one?

9

It’s probably competing for access to the internal DB, with other invocations and/or who knows what else your gateway and projects are doing.

10

Is there anything in the error logs I posted that suggests something I can do about this?
I’m currently working in a sandbox project that isn’t doing any heavy lifting.

11

No, the stack trace is a little more useful, but it just shows in progress work. A full thread dump when it gets “stuck” might be more useful.

What version of Ignition are you using?

edit: there’s a change that went into 8.1.20 that may improve things if you’re not already on it. I don’t know if it’s relevant, but something about the stack trace you posted reminded me of it.

12

How do I take a full thread dump?

13

https://docs.inductiveautomation.com/display/DOC81/Diagnostics+-+Threads

14

You can make a list. It doesn't HAVE to be nested.

15

Hi,

Version: 8.1.11

The more I look into it, I am seeing the exact same behaviour with the system ‘locking up’ when I try to delete multiple tags.

Thread dump below of when I tried to run my tag creation script, it created the first 19 UDT tags in under a second then just halted.

Ignition-SCADA-Server_thread_dump20220929-095509.json (142.7 KB)

16

Try 8.1.20 and a support ticket if that doesn’t help.