F5 Load Balancer

tbackus · July 22, 2025, 1:10pm

This morning we tried to stand up two Perspective servers behind an F5 load balancer. One server is already configured and running in preparation for this. When we brought up the second server, we started getting errors in existing clients. As they would navigate they would get the initial perspective loading page, but it would stop at "Connecting....."

Log file shows these errors on both servers during the time both servers were active in the load balancer:

java.lang.UnsupportedOperationException: Feature unsupported after Upgraded to WebSocket

at org.eclipse.jetty.websocket.core.server.internal.UpgradeHttpServletRequest.getSession(UpgradeHttpServletRequest.java:293)

at com.inductiveautomation.perspective.gateway.comm.Routes.getOrCreateSession(Routes.java:1765)

at com.inductiveautomation.perspective.gateway.comm.PerspectiveWebSocketServlet$PerspectiveWebSocketCreator.createWebSocket(PerspectiveWebSocketServlet.java:185)

at com.inductiveautomation.perspective.gateway.comm.PerspectiveWebSocketServlet$PerspectiveWebSocketCreator.createWebSocket(PerspectiveWebSocketServlet.java:123)

at org.eclipse.jetty.websocket.server.JettyWebSocketServlet$WrappedJettyCreator.createWebSocket(JettyWebSocketServlet.java:276)

at org.eclipse.jetty.websocket.core.server.internal.CreatorNegotiator.lambda$negotiate$0(CreatorNegotiator.java:64)

at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1451)

at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1488)

at org.eclipse.jetty.websocket.core.server.internal.CreatorNegotiator.negotiate(CreatorNegotiator.java:63)

at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker.upgradeRequest(AbstractHandshaker.java:76)

at org.eclipse.jetty.websocket.core.server.internal.HandshakerSelector.upgradeRequest(HandshakerSelector.java:39)

at org.eclipse.jetty.websocket.core.server.WebSocketMappings.upgrade(WebSocketMappings.java:241)

at org.eclipse.jetty.websocket.server.JettyWebSocketServlet.service(JettyWebSocketServlet.java:181)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)

at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:86)

at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)

at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.server.Server.handle(Server.java:563)

at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)

at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)

at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)

After trying different things it seemed like we never could get good stable connections.
We did confirm that both servers are setup identical and had all the projects copied via EAM. We are able to use both servers independently without issue.

Wondering if anyone has any words of wisdom or gotcha's we should be looking for at this point?

Thanks

Kevin.Herron · July 22, 2025, 1:14pm

A couple support tickets where this error was encountered were solved by clearing browser cache/cookies... unclear if a load balancer was involved in these.

The only piece of advice I have for load balancer setup is that sessions must be sticky. Others have more experience I'm sure.

Denny_Sheng · August 11, 2025, 10:31pm

We are experiencing the same issue. Have you identified a resolution? is there any specific setting need to be done at F5 side ?

Thanks

tbackus · August 12, 2025, 12:03pm

There is. I'm actually at my corporate offices today, I'll try and talk to the F5 guru and get the exact setting name, but I basically gave him the same info as above to make it sticky, and they use the port number 8043 as the test for the F5 to know if a server is online and should participate in the load balance activity.

Denny_Sheng · August 12, 2025, 1:41pm

Great to hear you have a resolution. Kindly update us on the testing progress—we’d like to replicate the setup into our F5 environment

Thanks

Denny

tbackus · August 13, 2025, 8:18pm

Default Persistence Profile is what was needed:
This is the screenshot they gave me, hopefully this helps.

Denny_Sheng · August 14, 2025, 7:00pm

Thanks you so much , the default persistence profile : source_addr works well for us !

David_Nicks · November 10, 2025, 5:02pm

Would you be willing to share more of your F5 setup details? We’ve just stood up two independent gateways behind our F5 and are having some quirky behavior regarding SSO, IdP, etc. I’m working with an F5 administrator who doesn’t know anything about Ignition and has limited availability…so I’m hoping to find plenty of tips & tricks from someone who’s walked this road before.

Currently running 8.1.28, but will soon upgrade to 8.1.48 and next year to 8.3.

tbackus · November 10, 2025, 8:07pm

I don’t really have much more than what’s posted here. We are not using SSO yet, we will be transferring Tenants soon (company was bought out and moving domains) so I didn’t have to deal with that, the Identity provider stuff was pretty straight forward, Setup identical on both servers and not having any issues. We periodically take server in the load balance cluster down to do maintenance and the only hitch is that the clients affected have to go back to the start of the project and re-navigate to the page they need. But otherwise it seems to work great. The one big issue biting us right now is that in Status / Connections / Perspective Sessions all originate from the IP address of the Load Balancer… Not something we have put a lot of time into solving yet.. All of this on 8.1.44.