Connection to Gateway lost, due to exception - Software caused connection abort: recv failed

v8.0.15

We just upgraded a 7.9.9 system to the latest stable v8 as of today, v8.0.15, and the customer is now seeing constant and consistent dropouts of gateway connection and corresponding red overlays all over the screen after a a time delay between loading a new screen. When the new screen is first loaded, all tags are good. Then after a varying time delay, comms is lost and then there is a red overlay on all objects with tag bindings. We are using a direct scanclass if it’s relevant.

Update 1:
It only appears to affect Windows showing tags from particular PLCs. The ones I’ve noticed are:

  • AB 5069-L320ER (load factor 120%)
  • AB MicroLogix (load factor 20%)

However there are other of the same PLCs that don’t have the issue. For the other Windows displaying other PLC tags, the gateway connection isn’t disrupted and the page never displays red overlays on the whole page.

10:51:11.996 [AWT-EventQueue-0] WARN com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Connection to Gateway lost, due to exception.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Software caused connection abort: recv failed
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)
	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.queryStatusImpl(ClientAlarmUtilities.java:40)
	at com.inductiveautomation.ignition.common.script.builtin.AbstractAlarmUtilities.queryStatus(AbstractAlarmUtilities.java:63)
	at jdk.internal.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:188)
	at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:524)
	at org.python.core.PyObject.__call__(PyObject.java:413)
	at org.python.pycode._pyx80.f$0(<expression:runScript>:1)
	at org.python.pycode._pyx80.call_function(<expression:runScript>)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1614)
	at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:782)
	at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:730)
	at com.inductiveautomation.ignition.common.expressions.functions.ScriptFunction.execute(ScriptFunction.java:78)
	at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)
	at com.inductiveautomation.ignition.common.expressions.DefaultFunctionFactory$CastFunction.execute(DefaultFunctionFactory.java:1784)
	at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)
	at com.inductiveautomation.ignition.common.expressions.DefaultFunctionFactory$IfFunction.execute(DefaultFunctionFactory.java:1717)
	at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)
	at com.inductiveautomation.factorypmi.application.binding.ExpressionPropertyAdapter.runExpression(ExpressionPropertyAdapter.java:83)
	at com.inductiveautomation.factorypmi.application.binding.ExpressionPropertyAdapter$1.run(ExpressionPropertyAdapter.java:59)
	at com.inductiveautomation.ignition.client.util.EDTUtil$ProcessQueue.run(EDTUtil.java:127)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.run(Unknown Source)
Caused by: java.net.SocketException: Software caused connection abort: recv failed
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
	at java.base/java.net.SocketInputStream.read(Unknown Source)
	at java.base/java.net.SocketInputStream.read(Unknown Source)
	at java.base/java.io.BufferedInputStream.fill(Unknown Source)
	at java.base/java.io.BufferedInputStream.read1(Unknown Source)
	at java.base/java.io.BufferedInputStream.read(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:437)
	... 40 common frames omitted
10:51:11.996 [AWT-EventQueue-0] INFO tags.subscriptions - Changing connected quality to 'Bad_Stale'
10:51:11.996 [AWT-EventQueue-0] INFO Scripting.ScriptManager.Berri_Estates_Winery - Pausing scripts...
10:51:11.996 [AWT-EventQueue-0] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Starting reconnect thread.
10:51:12.011 [AWT-EventQueue-0] WARN Vision.Binding.ExpressionPropertyAdapter - Error executing expression binding on
Alarm Count.Alarm Count.count
java.lang.Exception: Error executing expression binding on
Alarm Count.Alarm Count.count
	at com.inductiveautomation.factorypmi.application.binding.ExpressionPropertyAdapter.runExpression(ExpressionPropertyAdapter.java:92)
	at com.inductiveautomation.factorypmi.application.binding.ExpressionPropertyAdapter$1.run(ExpressionPropertyAdapter.java:59)
	at com.inductiveautomation.ignition.client.util.EDTUtil$ProcessQueue.run(EDTUtil.java:127)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.run(Unknown Source)
Caused by: com.inductiveautomation.ignition.common.expressions.ExpressionException: Error executing script for runScript() expression:len(system.alarm.queryStatus(state=["ActiveUnacked","ActiveAcked"], path=["**"]))
	at com.inductiveautomation.ignition.common.expressions.functions.ScriptFunction.execute(ScriptFunction.java:95)
	at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)
	at com.inductiveautomation.ignition.common.expressions.DefaultFunctionFactory$CastFunction.execute(DefaultFunctionFactory.java:1784)
	at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)
	at com.inductiveautomation.ignition.common.expressions.DefaultFunctionFactory$IfFunction.execute(DefaultFunctionFactory.java:1717)
	at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)
	at com.inductiveautomation.factorypmi.application.binding.ExpressionPropertyAdapter.runExpression(ExpressionPropertyAdapter.java:83)
	... 15 common frames omitted
Caused by: com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last):
  File "<expression:runScript>", line 1, in <module>
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)
	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.queryStatusImpl(ClientAlarmUtilities.java:40)
	at com.inductiveautomation.ignition.common.script.builtin.AbstractAlarmUtilities.queryStatus(AbstractAlarmUtilities.java:63)
	at jdk.internal.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Software caused connection abort: recv failed

	at org.python.core.Py.JavaError(Py.java:552)
	at com.inductiveautomation.ignition.common.script.builtin.AbstractAlarmUtilities.queryStatus(AbstractAlarmUtilities.java:65)
	at jdk.internal.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:188)
	at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:524)
	at org.python.core.PyObject.__call__(PyObject.java:413)
	at org.python.pycode._pyx80.f$0(<expression:runScript>:1)
	at org.python.pycode._pyx80.call_function(<expression:runScript>)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1614)
	at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:782)
	at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:730)
	at com.inductiveautomation.ignition.common.expressions.functions.ScriptFunction.execute(ScriptFunction.java:78)
	... 21 common frames omitted
Caused by: org.python.core.PyException: Traceback (most recent call last):
  File "<expression:runScript>", line 1, in <module>
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)
	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.queryStatusImpl(ClientAlarmUtilities.java:40)
	at com.inductiveautomation.ignition.common.script.builtin.AbstractAlarmUtilities.queryStatus(AbstractAlarmUtilities.java:63)
	at jdk.internal.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Software caused connection abort: recv failed

	... 37 common frames omitted
Caused by: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Software caused connection abort: recv failed
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:351)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:530)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:283)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:278)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:934)
	at com.inductiveautomation.factorypmi.application.script.builtin.ClientAlarmUtilities.queryStatusImpl(ClientAlarmUtilities.java:40)
	at com.inductiveautomation.ignition.common.script.builtin.AbstractAlarmUtilities.queryStatus(AbstractAlarmUtilities.java:63)
	... 35 common frames omitted
Caused by: java.net.SocketException: Software caused connection abort: recv failed
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
	at java.base/java.net.SocketInputStream.read(Unknown Source)
	at java.base/java.net.SocketInputStream.read(Unknown Source)
	at java.base/java.io.BufferedInputStream.fill(Unknown Source)
	at java.base/java.io.BufferedInputStream.read1(Unknown Source)
	at java.base/java.io.BufferedInputStream.read(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:437)
	... 40 common frames omitted
10:51:12.042 [GatewayConnection-1] INFO tags.subscriptions - Changing connected quality to 'Good'
10:51:12.089 [GatewayConnection-1] INFO Scripting.ScriptManager.Berri_Estates_Winery - Resuming scripts...
10:51:12.152 [GatewayConnection-1] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Stopping reconnect thread.

runScript() making gateway calls. You’re lucky it worked in 7.9.

I only have the one runScript call, which albeit does run every second… I’ve removed these from other projects, however I haven’t touched this one in a year or so. Looks like it’s time to remove it from this one as well!
I did see that error in there, but didn’t think it was related to the surrounding software / gatewayexception errors.