Hello all,
Our company has been struggling with an issue for awhile now and I was wondering if anyone had some insight here.
We have 6 facilities all under our corporation, and 3 of these facilities have been having issues maintaining a solid connection to the ignition gateway web server. What do I mean by this? I'll show you:
The client connection to the gateway is constantly dropping and reconnecting at these 3 facilities. These 3 facilities are connected to our network via a Sophos Remote Ethernet Device (RED) which is the only difference from our other facilities. After further investigation on a PCAP, it is showing what looks like the client failing authentication when trying to get an accurate update on the gateway status:
Normal HTTP request to gateway success -
HTTP response code of 200 meaning everything was successful.
HTTP request with NTLMAUTH redirection failure -
As you can see, we are receiving a 303 response code with a redirection to our firewall authentication method of NTLM (hcifw04.corpnetportal.com is our firewall). We then see RST/ACK TCP flags being thrown due to the connection dropping. There are about 6-7 clients at one of the problem facilities, and they all do the same thing. It is very random and sporadic; it will be fine for 10 minutes, and then constantly disconnect and reconnect for 60 seconds, then go back to being fine for 10 minutes. I am not very knowledgeable on the side of networking, but from researching this issue, it kind of seems like Ignition is sending POST requests for updates to see if the gateway is online, and will sometimes get asked for authentication, not know what to do, and then the connection drops and reconnects. The only problem is, there are no errors in the firewall log.
We use Sophos XG firewall, and have AD SSO / NTLM configured for authentication. When I disable AD SSO, this issue stops occurring, but we do not want to disable that because then our users are not authenticated properly. We made sure that we set the gateway as an allowed connection in the firewall as well. We have a ticket open with Sophos and have been actively working with their developers on this, but we have gotten nowhere in months.
Currently, the only way to fix this issue is to disable AD SSO inside of the firewall, or add a bypass for all IPs having the issue which we want to avoid doing both of these if possible. One workaround we have is to open a web browser and close it immediately to get the gateway back to a solid connection. I think this is because when opening a web browser, SSO authenticates the user automatically, so since the user is authenticated again, the gateway connection stops trying to authenticate against NTLM.
Any advice/suggestions?
EXTRA INFO:
Gateway Logs -
Full Error in Gateway -
org.xml.sax.SAXParseException: Premature end of file.
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at com.inductiveautomation.ignition.gateway.servlets.Gateway.doPost(Gateway.java:286)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:86)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
IdP Authentication Gateway & Client Turned off -
