[Bug-Resolved]8.1 RC2 - "Error handling route" every 10s

Ignition Early Access
1

We’re running Ignition 8.1 RC2 and we just started getting an “Error handling route” message in our log every 10 seconds. Currently utilizing Perspective via mobile and Perspective Workstation. Here’s the stack trace:

java.lang.IllegalStateException: Could not find the web session

at com.inductiveautomation.perspective.gateway.comm.Routes.getOrCreateSession(Routes.java:994)

at com.inductiveautomation.perspective.gateway.comm.Routes$PerspectiveRouteHandlerAdapter.handle(Routes.java:601)

at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:252)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:61)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupCollectionServlet.serviceInternal(RouteGroupCollectionServlet.java:54)

at com.inductiveautomation.ignition.gateway.dataroutes.AbstractRouteGroupServlet.service(AbstractRouteGroupServlet.java:38)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1391)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:500)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
1 Like
3

Seems to be registered as a bug here:

4

Yes, this should already be resolved for 8.1Stable.

5

@cmallonee We’ve now updated to 8.1.0 RC3 and are still experiencing the same issue.

6

Because we haven’t released 8.1.0 Stable. This issue was fixed after the release of 8.1.0RC3.

1 Like
7

We’re experiencing this issue with 8.1.1. Has this issue been fixed in a later version? It obviously wasn’t fixed in 8.1.0 Stable.

8

In reading the notes on the internal ticket, it sounds like a shutdown of the offending open Designer is required to fully remove the issue (the issue stemmed from Designers using out-of-date code because they had not been restarted to use the new code). 8.1.0 Stable should indeed include the fix, but until you restart your Designer it will still be using the old code to communicate with the Gateway.

Please close any Designer which could conceivably be running the “old” code and re-open it. Once all Designers have been restarted, please reach back out if you continue experiencing the issue.

9

I’m confused by your response. This was a fresh install of 8.1.1, so there would be no possibility of ‘out-of-date’ code running. The previous system was 7.9, so no clients or Designers were left running.

10

Oh, that's interesting... I have some other tasks I need to get completed, but I'll try to circle back to this today.

11

I worked with another QA member and we were unable to replicate this issue with a clean 8.1.1 installation, even while having Designers (and Workstation clients) persist during Gateway restarts.

As such, there are only two recommendations I’m able to provide:

  1. Verify there are no open Designers by navigating to your Gateway’s Status > Connections > Designers page. If you see any entries here, they could be the cause of the issue. Close all of these Designers, and then go back to normal usage. If the issue persists you will need to contact support because we’ve done all we can with the information available to us here.
  2. If you’re 100% confident you already closed all Designers, then please contact our Support team.
12

Hello All - we are experiencing this issue on a production environment currently. (currently running 8.0.12) - I am trying to understand what version / release this issue is resolved in to perform an upgrade on this production environment.

13

The issue appears to have been fixed as of 8.1.0. There was a report of it still being encountered in 8.1.1, but we were unable to replicate the behavior internally in that version.

14

Thank you @cmallonee – Follow up question would this issue have caused high CPU usage for any reason on a production gateway?

We are troubleshooting our logs and events leading up to the non-responsive gateway (seen in the CPU usage trend below) – The message for this bug is the only standout error / exception we are seeing.

image

15

Not sure if this is useful, but we recently had this error pop up in 8.1.13. A user was logged in to Ignition with the designer open, we were testing Active Directory (unsuccessfully) and we wound up locking this user's AD account. While his account was locked, the designer session wasn't showing as active on the gateway, but was still open on his desktop. We got this Error Handling Route error every 10 seconds until the session was closed on the user's laptop (after calling IT to get the AD account unlocked). Just FYI.

java.lang.IllegalStateException: Could not find the web session

at com.inductiveautomation.perspective.gateway.comm.Routes.getOrCreateSession(Routes.java:1042)

at com.inductiveautomation.perspective.gateway.comm.Routes$PerspectiveRouteHandlerAdapter.handle(Routes.java:628)

at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:254)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:61)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupCollectionServlet.serviceInternal(RouteGroupCollectionServlet.java:59)

at com.inductiveautomation.ignition.gateway.dataroutes.AbstractRouteGroupServlet.service(AbstractRouteGroupServlet.java:38)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)

at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:116)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:516)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.base/java.lang.Thread.run(Unknown Source)