Alarm Acknowledgement Speed

dan.deason · October 14, 2020, 4:55pm

How long is it supposed to take to acknowledge alarms? If i get more than a few dozen and acknowledge them all at once it takes forever (over 30 seconds) and sometimes leads to a timeout.

I can’t imagine this is normal is it? Surely acknowledging even hundreds of alarms should be quick right?

Any insight or help would be appreciated.

The examples below are done in the script console, but the same thing happens if you use the alarm list to select all and then hit the acknowledge button.

This took roughly 30 seconds to complete the acknowledge:

>>> a = system.alarm.queryStatus(path=['*'], state=['ClearUnacked', 'ActiveUnacked'])
>>> b = [str(alarmevent.getId()) for alarmevent in a]
>>> len(b)
32
>>> system.alarm.acknowledge(b, '')

This timed out before completing:

>>> a = system.alarm.queryStatus(path=['*'], state=['ClearUnacked','ActiveUnacked'])
>>> b = [str(alarmEvent.getId()) for alarmEvent in a]
>>> len(b)
239
>>> system.alarm.acknowledge(b, '')
Java Traceback:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)

	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.acknowledge(ClientAlarmUtilities.java:76)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out


	at org.python.core.Py.JavaError(Py.java:552)

	at org.python.core.Py.JavaError(Py.java:543)

	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)

	at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:524)

	at org.python.core.PyObject.__call__(PyObject.java:497)

	at org.python.core.PyObject.__call__(PyObject.java:501)

	at org.python.pycode._pyx66.f$0(<input>:1)

	at org.python.pycode._pyx66.call_function(<input>)

	at org.python.core.PyTableCode.call(PyTableCode.java:171)

	at org.python.core.PyCode.call(PyCode.java:18)

	at org.python.core.Py.runCode(Py.java:1614)

	at org.python.core.Py.exec(Py.java:1658)

	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:276)

	at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:131)

	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$ConsoleWorker.doInBackground(JythonConsole.java:605)

	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$ConsoleWorker.doInBackground(JythonConsole.java:593)

	at java.desktop/javax.swing.SwingWorker$1.call(Unknown Source)

	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

	at java.desktop/javax.swing.SwingWorker.run(Unknown Source)

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

	at java.base/java.lang.Thread.run(Unknown Source)

Caused by: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)

	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.acknowledge(ClientAlarmUtilities.java:76)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:188)

	... 19 more

Caused by: java.net.SocketTimeoutException: Read timed out

	at java.base/java.net.SocketInputStream.socketRead0(Native Method)

	at java.base/java.net.SocketInputStream.socketRead(Unknown Source)

	at java.base/java.net.SocketInputStream.read(Unknown Source)

	at java.base/java.net.SocketInputStream.read(Unknown Source)

	at java.base/java.io.BufferedInputStream.fill(Unknown Source)

	at java.base/java.io.BufferedInputStream.read1(Unknown Source)

	at java.base/java.io.BufferedInputStream.read(Unknown Source)

	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)

	at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)

	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)

	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)

	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:437)

	... 28 more

Traceback (most recent call last):
  File "<input>", line 1, in <module>
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)

	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)

	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.acknowledge(ClientAlarmUtilities.java:76)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out

Kevin.Herron · October 14, 2020, 4:57pm

Any chance you’re acknowledging alarms belonging to a remote provider?

dan.deason · October 14, 2020, 5:49pm

The alarms are on a different gateway, yes. When this system was set up a gatweway was set up for visualization, a gateway for the data tags, and a final gateway to run logic

Kevin.Herron · October 14, 2020, 5:53pm

Well, unfortunately that’s just slow right now.

Side effect of the alarm system’s API having been designed long before remote gateways were really a thing. The scripting call takes a collection, but it funnels down to acknowledging one alarm at a time, even if that’s against a remote provider, which means one network call at a time.

It may happen to get a little faster in 8.1.0 because of some other GAN changes, but I’ll open a ticket to see if something can be done to speed this specifically up in the future.

dan.deason · October 14, 2020, 5:54pm

Thanks. Unfortunate indeed, but thanks for the quick reply.

dan.deason · October 14, 2020, 7:44pm

As a workaround I added an “alarms” project to the gateway with the alarms and an “Ack_Alarms” message handler. Now instead of calling system.alarm.acknowledge directly, I pass the parameters for the query status in the message payload and ack them on the local gateway. Works sufficiently fast - it takes a second for the round trip, but hundreds of alarms can ack in less than a couple seconds.

When the ack all button is pressed:

system.util.sendMessage('Alarms', 'Ack_Alarms', {'path':['*'], 'state':['ClearUnacked','ActiveUnacked']}, remoteServers=['TAG_SERVER1', 'TAG_SERVER2'])

On the message handler:

def handleMessage(payload):
	"""
	This message handler will be called each time a message of this
	type is received.

	Arguments:
		payload: A dictionary that holds the objects passed to this
		         message handler. Retrieve them with a subscript, e.g.
		         myObject = payload['argumentName']
	"""
	'''
	Payload Description:
	path = ['*']
	state = ['ClearUnacked','ActiveUnacked']
	'''
	logger = system.util.getLogger('Ack_Alarms')
	a = system.alarm.queryStatus(path=payload['path'], state=payload['state'])
	b = [str(alarmEvent.getId()) for alarmEvent in a]
	system.alarm.acknowledge(b, '', '')
	logger.info('Acked {} alarms'.format(len(b)))

FrankP · November 21, 2022, 4:17pm

Experiencing this with Ignition 8.1.21. Any news here? Thanks!

Kevin.Herron · November 21, 2022, 4:23pm

Supposedly this was improved in 8.1.2. If you're still having an issue get in touch with support so they can take a look and maybe create a new issue. 8.1.2 should have added a bulk call at the gateway network layer and sped things up.

FrankP · November 21, 2022, 4:42pm

Thank you for the fast response! I have a followup before I make a ticket. For various historical reasons related to ignition 7.9 and some SSL problems, our spokes are using http on port 80 for the GAN connections (same as GW web interface). Would there be any potential performance improvements moving to port 8060? We're using remote tag providers and alarm history sync.

Kevin.Herron · November 21, 2022, 4:48pm

No, but if any of the gateways involved in the alarm acking are on 7.9 you don't get the performance gains.