IdpAdapterException: Unable to parse the WebAuthResponse from the HTTP request

Frosty · June 12, 2023, 7:38am

Hi,

I'm having trouble configuring Auth0 as an Identity Provider.

Context: We are embedding the Perspective pages in our company web app, which is also secured with Auth0. To enable SSO and prevent any cross-site issues, the web app, Perspective and Auth0 are all served from the same top-level domain. E.g.:

Web app: app.company.com
Perspective: perspective.company.com
Auth0: login.company.com

IdP setup: The settings were imported using the .well-known/openid-configuration endpoint. After the import, I updated the Authorization URL, Token URL and Issuer to use the custom domain (login.company.com) instead of the default auth0 domain.

Problem: When performing a Test login for the first time (or after waiting for a while), it fails with the following exceptions in the logs (see full stacktrace below):

Unable to handle login response
com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapterException: Unable to parse the WebAuthResponse from the HTTP request
Caused by: com.inductiveautomation.ignition.gateway.auth.web.strategy.WebAuthStrategyException: Unable to exchange the auth code for the token
Caused by: com.inductiveautomation.ignition.gateway.auth.oidc.client.service.OIDCClientServiceException: Unable to execute HTTP request
Caused by: java.net.SocketTimeoutException: Read timed out

However, when I start another Test Login right after the first attempt, everything works as expected. The token is returned and I can perfectly map user attributes. The iss claim in the token corresponds to https://login.company.com.

What could be the cause for the timeout on the first login?

Full stacktrace from logs:

com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapterException: Unable to parse the WebAuthResponse from the HTTP request
at com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapter.parseAttributes(IdpAdapter.java:122)
at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.onLoginResponseInternal(WebAuthSessionImpl.java:225)
at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.lambda$onLoginResponse$2(WebAuthSessionImpl.java:250)
at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.mdc(WebAuthSessionImpl.java:108)
at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.onLoginResponse(WebAuthSessionImpl.java:250)
at com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapterConfigRoutes$TestLoginWebAuthResponseHandler.handle(IdpAdapterConfigRoutes.java:301)
at com.inductiveautomation.ignition.gateway.auth.federation.FederationRoutes.callback(FederationRoutes.java:274)
at com.inductiveautomation.ignition.gateway.auth.federation.FederationRoutes$CrossSiteRouteHandler.handle(FederationRoutes.java:122)
at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:254)
at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:61)
at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupCollectionServlet.serviceInternal(RouteGroupCollectionServlet.java:59)
at com.inductiveautomation.ignition.gateway.dataroutes.AbstractRouteGroupServlet.service(AbstractRouteGroupServlet.java:38)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.inductiveautomation.ignition.gateway.auth.web.strategy.WebAuthStrategyException: Unable to exchange the auth code for the token
at com.inductiveautomation.ignition.gateway.auth.web.strategy.oidc.OIDCWebAuthStrategy.parseWebAuthResponse(OIDCWebAuthStrategy.java:204)
at com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapter.parseAttributes(IdpAdapter.java:112)
... 52 common frames omitted
Caused by: com.inductiveautomation.ignition.gateway.auth.oidc.client.service.OIDCClientServiceException: Unable to execute HTTP request
at com.inductiveautomation.ignition.gateway.auth.oidc.client.service.HttpOIDCClientService.doGetToken(HttpOIDCClientService.java:279)
at com.inductiveautomation.ignition.gateway.auth.oidc.client.service.HttpOIDCClientService.getToken(HttpOIDCClientService.java:322)
at com.inductiveautomation.ignition.gateway.auth.web.strategy.oidc.OIDCWebAuthStrategy.parseWebAuthResponse(OIDCWebAuthStrategy.java:202)
... 53 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.inductiveautomation.ignition.gateway.auth.oidc.client.service.HttpOIDCClientService.doGetToken(HttpOIDCClientService.java:277)
... 55 common frames omitted

jspecht · June 12, 2023, 3:11pm

Is there any middleware such as a NAT or firewall which might be evicting idle TCP connections silently after a some timeout period? We've seen this issue before, and we introduced a few new system properties to help alleviate this issue (docs).

For example: in the past, we had a customer deploy Ignition in AWS. The Ignition instance connected to Auth0 over the Internet through the AWS NAT Gateway. After logging in a few times in a short period of time, if no other logins occur for 350 seconds, the AWS NAT Gateway would close its end of the TCP connection without Ignition being aware until the next time a user tried to login (say 450 seconds later). The stack trace matched what you posted.

jspecht · June 12, 2023, 3:16pm

Actually, the stack trace was similar but the root cause was either "Connection Reset" or "Broken Pipe" instead of "Read timed out". Still look into what I mentioned above, but you may have other network issues as well (perhaps a slow or choppy connection?).

Frosty · June 15, 2023, 11:38am

Thanks for the pointers @jspecht. I've set the following values:

# ignition.conf
socket.keepalive=true
pool.idleConnectionCheckInterval=60
pool.maxIdleConnectionTime=300
proxy.enabled=true

# /etc/sysctl.conf
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 5

Do these seem like sensible settings to you? Using these settings, the first-time test login does seem to be fixed, and the SSO seems to work better as well.

jspecht · June 15, 2023, 7:48pm

I can't speak to whether or not the proxy setting is appropriate for you. That depends on whether or not you need Ignition to connect to your IdP through a proxy.

As you have it now, Ignition will check if your connections to the IdP are idle every 60 seconds. The connection is considered idle if it has not sent or received data in the past 300 seconds. So in the worst case Ignition might try to use a connection which has been idle for ~360 seconds. Do you have any middleware which evicts connections idle for less than that amount of time? If so, you may want to decrease one or both of those values...

You also enabled keepalive on the TCP sockets Ignition create which connect to the IdP (socket.keepalive=true). Looks like every 600 seconds your OS will send a keepalive packet on those sockets. If it doesn't receive a keepalive packet in response after 60 seconds, it will try sending another keepalive packet. It will attempt to send a max of 5 consecutive unanswered keepalive packets before giving up.

If Ignition will evict connections after 300-360 seconds of inactivity, why have the keepalive packets enabled? If you'd rather keep the connection alive and avoid Ignition evicting them, alter your keepalive time to be less than your maxIdleConnectionTime + idleConnectionCheckInterval. Usually customers rely on the socket keepalive mechanism or the idle connection eviction mechanism but not both, though that's not to say the two can't play together. You have to find what works best for you.

Sourabh_Ambildhuke · August 5, 2023, 2:21am

Hi @Frosty,

Dis this settings helps you to resolve above issue or have you done extra settings. Facing the same issue

Yogesh_Garud · August 7, 2023, 11:49am

@jspecht @Frosty
Adding more inputs from @Sourabh_Ambildhuke

We are facing the same issue, PFA screenshots and please let us know if this is resolved from your end.

Frosty · November 21, 2023, 12:30pm

Hi @Sourabh_Ambildhuke, a bit late to the party but for us this resolved the issues yes. Another configuration change that we made earlier was enabling or disabling (I'm not sure) same site cookies somewhere in the Gateway Configuration File (Gateway Configuration File Reference - Ignition User Manual 8.1 - Ignition Documentation).

dhiraj.pawar · March 19, 2024, 4:42am

Hello @Frosty,
We are facing same issue. As suggested i have added below properties in ignition.conf file & added property into registry but it's not working for us. Please let me know if anything i missed.

ignition.conf

socket.keepalive=true
pool.idleConnectionCheckInterval=60
pool.maxIdleConnectionTime=300
proxy.enabled=true

# /etc/sysctl.conf
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 5