Gateway read timeout happening extremely frequently

Ever since we have started using Ignition 7.9 as well as Sepasoft’s Track and Trace module we have experiences some really slow runtimes for what seems at surface level, really simple tasks.

This can be anything from creating a Process Segment (as long as 4 minutes) or creating an Equipment instance (sometimes 2 minutes), to simply just getting a script to grab data from our production database and do some transformations.

Recently as well we have had many of our scripts fail shortly into their execution because the gateway has timed out after 30 seconds or so. Given that data is really only flowing between the Designers, HMI’s, and our Server, with no complex computation, why would any of this take so long as to time out a thread that the process is running on. A query to a database no matter where it is located in the world should never take more than 100ms, and we are seeing that in the diagnostics pane, yet creating a piece of Equipment in our designer takes 1 full minute…

We have been grasping at straws to try and infer what is going on under the hood here. Currently, we are at a total loss, seriously in need of a speedup. I realize that some of these libraries give you enough rope to hang yourself, but if we were doing more complex things we wouldn’t as surprised by the poor performance.

Java Traceback:

	at org.python.core.Py.JavaError(Py.java:495)
	at org.python.core.Py.JavaError(Py.java:488)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:188)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
	at org.python.core.PyObject.__call__(PyObject.java:387)
	at org.python.core.PyObject.__call__(PyObject.java:391)
	at org.python.pycode._pyx24.create_equipment$38(<module:shared.mes.equipment>:294)
	at org.python.pycode._pyx24.call_function(<module:shared.mes.equipment>)
	at org.python.core.PyTableCode.call(PyTableCode.java:165)
	at org.python.core.PyBaseCode.call(PyBaseCode.java:149)
	at org.python.core.PyFunction.__call__(PyFunction.java:327)
	at org.python.pycode._pyx24.bulk_create_equipment$39(<module:shared.mes.equipment>:320)
	at org.python.pycode._pyx24.call_function(<module:shared.mes.equipment>)
	at org.python.core.PyTableCode.call(PyTableCode.java:165)
	at org.python.core.PyBaseCode.call(PyBaseCode.java:149)
	at org.python.core.PyFunction.__call__(PyFunction.java:327)
	at org.python.pycode._pyx155.f$0(<buffer>:6)
	at org.python.pycode._pyx155.call_function(<buffer>)
	at org.python.core.PyTableCode.call(PyTableCode.java:165)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1275)
	at org.python.core.Py.exec(Py.java:1319)
	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:215)
	at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:89)
	at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:70)
	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$InterpreterWorker.doInBackground(JythonConsole.java:476)
	at com.inductiveautomation.ignition.designer.gui.tools.jythonconsole.JythonConsole$InterpreterWorker.doInBackground(JythonConsole.java:464)
	at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at javax.swing.SwingWorker.run(SwingWorker.java:334)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.moduleInvoke(GatewayInterface.java:874)
	at com.sepasoft.production.rpc.ProductionRPCProxy.invoke(ProductionRPCProxy.java:91)
	at com.sepasoft.production.rpc.ProductionRPCProxy.saveMESObjects(ProductionRPCProxy.java:722)
	at com.sepasoft.production.script.MESClientScript.saveMESObject(MESClientScript.java:166)
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
	... 30 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:421)
	... 40 more
Traceback (most recent call last):
  File "<buffer>", line 6, in <module>
  File "<module:shared.mes.equipment>", line 316, in bulk_create_equipment
  File "<module:shared.mes.equipment>", line 316, in bulk_create_equipment
  File "<module:shared.mes.equipment>", line 285, in create_equipment
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
	at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.moduleInvoke(GatewayInterface.java:874)
	at com.sepasoft.production.rpc.ProductionRPCProxy.invoke(ProductionRPCProxy.java:91)
	at com.sepasoft.production.rpc.ProductionRPCProxy.saveMESObjects(ProductionRPCProxy.java:722)
	at com.sepasoft.production.script.MESClientScript.saveMESObject(MESClientScript.java:166)
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Read timed out

Have you talked with Sepasoft support about this yet?

Made a ticket today, hoping things sort out.