[IGN-7255] SAML Auth Not working after 8.1.25 update

nicholas.robinson · February 23, 2023, 7:38pm

@jspecht today I had a gateway that was running 8.1.19 that had functional SAML IDP.

After upgrading to 8.1.25 via EAM with no other change to the gateway, the following comes up when trying to authenticate:

This is the full trace:

java.lang.NullPointerException: null

at java.base/java.util.Objects.requireNonNull(Unknown Source)

at java.base/java.util.Optional.(Unknown Source)

at java.base/java.util.Optional.of(Unknown Source)

at com.inductiveautomation.ignition.gateway.auth.web.attr.XmlAttributeSource.get(XmlAttributeSource.java:118)

at com.inductiveautomation.ignition.gateway.auth.web.attr.AttributeSourceWrapper.get(AttributeSourceWrapper.java:20)

at com.inductiveautomation.ignition.gateway.auth.expr.AuthExpressionParseContext$IdpAttributesExpression.getValue(AuthExpressionParseContext.java:60)

at com.inductiveautomation.ignition.gateway.auth.expr.AuthExpressionParseContext$IdpAttributesExpression.execute(AuthExpressionParseContext.java:80)

at com.inductiveautomation.ignition.common.expressions.ArithmeticExpression.execute(ArithmeticExpression.java:64)

at com.inductiveautomation.ignition.common.expressions.ArithmeticExpression.execute(ArithmeticExpression.java:63)

at com.inductiveautomation.ignition.common.expressions.functions.ScriptFunction.execute(ScriptFunction.java:40)

at com.inductiveautomation.ignition.common.expressions.FunctionExpression.execute(FunctionExpression.java:69)

at com.inductiveautomation.ignition.gateway.auth.mapper.attr.expr.AttributeMapperExpression.execute(AttributeMapperExpression.java:74)

at com.inductiveautomation.ignition.gateway.auth.mapper.attr.expr.ExpressionAttributeMapper.map(ExpressionAttributeMapper.java:38)

at com.inductiveautomation.ignition.gateway.auth.mapper.attr.user.UserAttributeMapper$AttributeSourceWrapper.invokeOptionalMapper(UserAttributeMapper.java:90)

at com.inductiveautomation.ignition.gateway.auth.mapper.attr.user.UserAttributeMapper.map(UserAttributeMapper.java:109)

at com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapter.mapUser(IdpAdapter.java:132)

at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.onLoginResponseInternal(WebAuthSessionImpl.java:236)

at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.lambda$onLoginResponse$2(WebAuthSessionImpl.java:250)

at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.mdc(WebAuthSessionImpl.java:108)

at com.inductiveautomation.ignition.gateway.auth.idp.WebAuthSessionImpl.onLoginResponse(WebAuthSessionImpl.java:250)

at com.inductiveautomation.ignition.gateway.auth.idp.IdpAdapterConfigRoutes$TestLoginWebAuthResponseHandler.handle(IdpAdapterConfigRoutes.java:301)

at com.inductiveautomation.ignition.gateway.auth.federation.FederationRoutes.callback(FederationRoutes.java:274)

at com.inductiveautomation.ignition.gateway.auth.federation.FederationRoutes$CrossSiteRouteHandler.handle(FederationRoutes.java:122)

at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:254)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:61)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupCollectionServlet.serviceInternal(RouteGroupCollectionServlet.java:59)

at com.inductiveautomation.ignition.gateway.dataroutes.AbstractRouteGroupServlet.service(AbstractRouteGroupServlet.java:38)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)

at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)

at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.server.Server.handle(Server.java:563)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)

at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)

at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)

at java.base/java.lang.Thread.run(Unknown Source)

So functionally, upgrading to 8.1.25 seemed to have break the SAML IDP. I can send you the SAML traces on DM if you need.

Thanks,

Nick

nicholas.robinson · February 23, 2023, 7:39pm

Adding, when I say SAML IDP not working, after the upgrade via EAM was done, when I tried to login to the gateway, I could not and had to break in through the backdoor using GWCMD.

Nick

nicholas.robinson · February 23, 2023, 7:51pm

Update: by creating a new IDP, I can get a successful login, so this makes it seem as though the upgrade corrupted the existing SAML IDP conenction.

Nick

jspecht · February 23, 2023, 8:53pm

If you can DM me a GWBK of the 8.1.19 Gateway and SAML trace, we can try to reproduce the issue on our side. Do you have logs which have more detailed MDC keys / values? That might help diagnose exactly which attribute mapper is failing. Based on the stack, it seems like there is a problem with the roles expression attribute mapper that you have configured...

nicholas.robinson · February 23, 2023, 9:12pm

Will DM you. Just to note, here is the roles mapping expression we have been using since 2 years back.

The expectation is that if it worked in previous versions, it should not stop working upon upgrade.

runScript("system.idp.processIdpResponse("+{idp-attributes://saml:Attribute[@Name='adgroups']}+", system.net.getHostName())")

This is calling a function in our scripting module that grabs the roles. We do this because there is no functionality in Ignition to get from AD groups contained in the SAML response to roles.

public interface SamlResponseHandler {
    ArrayList<String> processIdpResponse(List<String> samlResponse, String hostName, String country);
    ArrayList<String> processIdpResponse(List<String> samlResponse, String hostName);
}

So just now I added that expression to the new test IDP that I had setup in 8.1.25 that was working and it did indeed make it stop working so it seems related to this expression but we need to know why.

Nick

jspecht · February 23, 2023, 9:26pm

If you can send me the SAML trace I might have everything I need to reproduce it over here so that I can find the answer to your question of "why".

There was a change in 8.1.25 to how we process SAML responses due to a vulnerability which was disclosed to us. It's possible that this change could have caused a regression which we may have missed during testing. But before I can determine that, I need to reproduce the issue on our side...

So just now I added that expression to the new test IDP that I had setup in 8.1.25 that was working and it did indeed make it stop working so it seems related to this expression but we need to know why.

Is this the same new test IdP you mentioned above? What kind if IdP is this? And how does its configuration differ from the pre-existing IdP which is failing after upgrade?

nicholas.robinson · February 23, 2023, 9:35pm

DM sent. To clarify on the IDP's, here is the order of events:

Before upgrade (still at 8.1.19), SAML IDP existing on gateway was able to authenticate OK
After upgrade to 8.1.25, once the gateway was started, unable to login (HTTP 500)
Regained gateway access using GWCMD
Setup new SAML IDP with exact same settings as the existing one, but with no attribute mapping. This IDP was able to get a response just fine.
After adding the roles expression to the new IDP, it also became affected producing HTTP 500

Here SSO is the original one, SAML_Test is the new one setup in step 4 above:

Thanks,

Nick

jspecht · February 23, 2023, 9:37pm

That helps. Thanks for the clarification.

jspecht · February 25, 2023, 12:01am

For posterity, we were able to reproduce the issue thanks to Nick's info. Looks like it is a regression introduced by a SAML vulnerability fix in 8.1.25. We have created internal ticket IGN-7255 to track this issue. A fix is underway targeting 8.1.26. The fix should continue to mitigate the vulnerability while reverting the regression and maintaining proper backwards compatibility for XPath expressions.

jspecht · March 6, 2023, 10:39pm

The fix for the XPath regression is available in the latest Early Access Build.

nicholas.robinson · March 15, 2023, 7:39pm

@jspecht sorry for my slow work, we tested this version today:

Using our original short form addressing in the roles mapping:

runScript("system.idp.processIdpResponse("+{idp-attributes://saml:Attribute[@Name='adgroups']}+", system.net.getHostName())")

Here is the mapped data from a test login:

So the improvements in 8.1.26 seem effective.

Rgds,

Nick

jspecht · March 15, 2023, 8:30pm

Excellent - thank you for testing and confirming