DefaultExceptionMapper Unexpected Error Occurred

d3h637 · August 21, 2023, 2:38pm

I have two servers that have been operating for at least 10 years, being upgraded multiple times to the newest version of Ignition. I'm not sure what the original version was, but the first one I remember using was 7.5. Anyway, I've been going through the status log and trying to clear up all the errors. One error that I'm having trouble with is "DefaultExceptionMapper Unexpected Error Occured". Below is the text with the details. I'm not sure how long this error has been there, but I first noticed it about a year ago. Any ideas what's causing this? The error I see is "Caused by: java.lang.IllegalArgumentException: Illegal character in query at index 24: /web/designer-launchers?"=1", but I don't really know where to look.

org.apache.wicket.WicketRuntimeException: Can't instantiate page using constructor 'public com.inductiveautomation.ignition.gateway.web.pages.DesignerLauncherDownloadPage()'. An exception has been thrown during construction!

at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:194)

at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:67)

at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:103)

at org.apache.wicket.DefaultMapperContext.newPageInstance(DefaultMapperContext.java:137)

at org.apache.wicket.core.request.handler.PageProvider.resolvePageInstance(PageProvider.java:268)

at org.apache.wicket.core.request.handler.PageProvider.getPageInstance(PageProvider.java:166)

at org.apache.wicket.request.handler.render.PageRenderer.getPage(PageRenderer.java:78)

at org.apache.wicket.request.handler.render.WebPageRenderer.respond(WebPageRenderer.java:279)

at org.apache.wicket.core.request.handler.RenderPageRequestHandler.respond(RenderPageRequestHandler.java:175)

at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:890)

at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)

at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:261)

at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:218)

at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:289)

at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:259)

at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:201)

at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:282)

at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)

at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:516)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: java.lang.reflect.InvocationTargetException: null

at jdk.internal.reflect.GeneratedConstructorAccessor146.newInstance(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)

at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:175)

... 55 common frames omitted

Caused by: java.lang.IllegalArgumentException: Illegal character in query at index 24: /web/designer-launchers?"=1

at java.base/java.net.URI.create(Unknown Source)

at com.inductiveautomation.ignition.gateway.dataroutes.RequestUtils.createRequestUri(RequestUtils.java:23)

at com.inductiveautomation.ignition.gateway.dataroutes.RequestUtils.getRelativeRequestUri(RequestUtils.java:37)

at com.inductiveautomation.ignition.gateway.web.pages.BasePage.getReturnTo(BasePage.java:249)

at com.inductiveautomation.ignition.gateway.web.pages.BasePage.(BasePage.java:121)

at com.inductiveautomation.ignition.gateway.web.pages.BasePage.(BasePage.java:84)

at com.inductiveautomation.ignition.gateway.web.pages.AuthenticatedPage.(AuthenticatedPage.java:36)

at com.inductiveautomation.ignition.gateway.web.pages.NativeLauncherDownloadPage.(NativeLauncherDownloadPage.java:43)

at com.inductiveautomation.ignition.gateway.web.pages.DesignerLauncherDownloadPage.(DesignerLauncherDownloadPage.java:5)

... 59 common frames omitted

Caused by: java.net.URISyntaxException: Illegal character in query at index 24: /web/designer-launchers?"=1

at java.base/java.net.URI$Parser.fail(Unknown Source)

at java.base/java.net.URI$Parser.checkChars(Unknown Source)

at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)

at java.base/java.net.URI$Parser.parse(Unknown Source)

at java.base/java.net.URI.(Unknown Source)

pturmel · August 21, 2023, 2:49pm

I would think you have an old designer launch shortcut with an old launcher trying to hit your upgraded server.

d3h637 · August 21, 2023, 4:16pm

Any idea what would cause that to happen ~40 times every morning starting at ~00:40? Every day, I get:
DefaultExceptionMapper Connection lost, give up responding
"Caused by: org.eclipse.jetty.io.EofException: null" at ~00:35
LoginRedirectPage Unable to create IdP Log In Request
"Caused by: java.net.URISyntaxException: Illegal character in query at index 31: http://'host'/web/login?<<<<<<<<<>>>>=1" --- twice at 00:38 and 00:39
then dozens of DefaultEceptionMapper Unexpecte Error Occurred right after that. This is happening every day.

pturmel · August 21, 2023, 4:39pm

Huh. Does IT do some kind of scan every morning? Or some automation elsewhere probe the gateway?

PGriffith · August 21, 2023, 4:40pm

Did your IT team recently install an internal "vulnerability scanner" that just spams a bunch of known payloads at every internal endpoint it can discover?

d3h637 · August 21, 2023, 4:55pm

I'm sure my organization does do vulnerability scanning. I'll have to ask around to see if we can whitelist that port. This would make a lot of sense. It doesn't happen on my 3 other servers onsite that haven't been around as long, but they're all running port 8088 instead of port 80.

d3h637 · August 22, 2023, 2:36pm

To close this out, we did a test last night where they turned off the vulnerability scanner, and the errors went away. These servers are still running on port 80, so I'll probably move them to port 8088 to avoid these nuisance errors. Thanks for your help.

PGriffith · August 22, 2023, 3:27pm

If you still want the convenience of port 80, there's some options:

Configure logback.xml to silence these errors - potentially not ideal in the event of legitimate errors being blackholed.
Stand up a "reverse proxy" (specific examples are Traefik, nginx) on the same host as the gateway. Have it listen on port 80 and 'forward' traffic to Ignition listening on whatever port. Configure it, if necessary, to 'blackhole' the snooper traffic.

I'll just point out, for the sake of it, you're probably only buying time until someone realizes they should be scanning for webservers on non-standard ports on your internal network. While I personally don't think security scanners like this are really providing a lot of value, it might be good to prepare for an eventuality where someone turns it on for Ignition again.

d3h637 · August 22, 2023, 4:00pm

Thanks for the reply. As usual, Inductive Automation is very responsive and that's one of the reasons our Ignition deployments are successful. Your points are valid, but being a government lab the vulnerability scanners are here to stay as they're mandated by our client. Agreed that we'll eventually have the port scanners scanning port 8088 if we move Ignition there, but we can get the port whitelisted so that the port scanners won't bug us on that port. I'm working with IT now to see whether we can whitelist port 80, or if we should move ports and whitelist port 8088. It may be more palatable to the auditors to whitelist port 8088 than port 80.

pturmel · August 22, 2023, 5:31pm

If you know what they are, and they are just one batch a day, is there really any need to change anything? I would think that knowing they are probing is a good thing--if they stop, you can let them know their infrastructure is broken.

d3h637 · August 22, 2023, 5:44pm

Agreed. The main concern was identifying the error. Now that I know it's just an artifact of the security probes, it's not that important to me. However, the issue is that this puts 60 items in my log every day and makes it harder to find real errors.