Ignition Gateway stops responding

SteveW · May 4, 2016, 2:00pm

We’re running 7.7.2 on a Windows 64 bit server. We have around 250 clients running at any given time. We occasionally (like today) are seeing the Gateway become unresponsive, with some clients getting a “gateway timeout” message.

Restarting the gateway solves the problem. When looking through the wrapper log files, the only anomaly that we see is a set of messages related to the timeout issue (see below). This message set repeated every 15 minutes or so in the logs prior to us restarting.

Would anyone have any thoughts on things we can check to zero in on the cause? We’d like to get a better handle on the issue aside from doing planned restarts to prevent the problem.

Thanks in advance.

INFO | jvm 1 | 2016/05/04 07:36:57 | java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.util.BlockingCallback.block(BlockingCallback.java:101)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent(HttpInputOverHTTP.java:62)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.HttpInput$1.waitForContent(HttpInput.java:392)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:161)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:125)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.inductiveautomation.ignition.gateway.servlets.Gateway.doPost(Gateway.java:229)
INFO | jvm 1 | 2016/05/04 07:36:57 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
INFO | jvm 1 | 2016/05/04 07:36:57 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
INFO | jvm 1 | 2016/05/04 07:36:57 | at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:85)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:526)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1110)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:453)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1044)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.Server.handle(Server.java:459)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:280)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.lang.Thread.run(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | Caused by:
INFO | jvm 1 | 2016/05/04 07:36:57 | java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:153)
INFO | jvm 1 | 2016/05/04 07:36:57 | at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.util.concurrent.FutureTask.run(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 1 | 2016/05/04 07:36:57 | at java.lang.Thread.run(Unknown Source)

Curlyandshemp · May 5, 2016, 11:06am

Log onto your Gateway and post these two screen shots.
Are you using the Mobile module on any of the clients? That ‘steals’ CPU resources. You may be running out of allocated memory. There is a parameter in the Ignition.conf file for that, but requires a gateway restart to take effect.

I have found on larger systems, getting the Database onto a separate VM greatly improves overall performance.

[attachment=1]2016-05-05 06_59_36-192.168.11.10 - Remote Desktop Connection.png[/attachment]

[attachment=0]2016-05-05 06_59_56-192.168.11.10 - Remote Desktop Connection.png[/attachment]

anna · June 15, 2016, 11:59pm

SteveW, we would need to be looking at your full wrapper log for a better picture. It is possible that the Gateway is running out of memory or CPU time (see Curlyandshemp’s post); it is also possible that there are network issues coming into play. Honestly, with 250 concurrent clients you are largely in an uncharted territory.
What are the specs for the server running the Gateway? What else is running on the same machine?
Are you running the Gateway on a virtual machine? If so, what resources is the VM given? What resources does the host have? What else does it run?