Module with embedded jetty server for websockets break Ignition GAN connections

mazeyrat · March 14, 2024, 3:16pm

I write a module who need to start an embedded jetty server to reply to web sockets.
The server run on the same ip address as Ignition but on another port (not 8060).

When the first client connect on the websocket Ignition GAN connections are suddenly disconnected (redundancy connection or connection with other gateway).

I don't understand what can interact with Ignition websocket communication ?

// Create a Server instance.
this.server = new Server(this.threadPool);

// connector http ?
if (this.port != -1){
	httpConnector = new ServerConnector(server);
	if (!host.isEmpty()) {
		httpConnector.setHost(host);
	}
	httpConnector.setPort(port);
	server.addConnector(httpConnector);
}

// connector https ?
if (this.httpsPort != -1){
	// The HTTP configuration object.
	HttpConfiguration httpConfig = new HttpConfiguration();
	// Add the SecureRequestCustomizer because we are using TLS.
	//httpConfig.addCustomizer(new SecureRequestCustomizer());

	SecureRequestCustomizer src = new SecureRequestCustomizer();
	src.setSniHostCheck(false);
	httpConfig.addCustomizer(src);

	// The ConnectionFactory for HTTP/1.1.
	HttpConnectionFactory http11 = new HttpConnectionFactory(httpConfig);

	// Configure the SslContextFactory with the keyStore information.
	SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();

	if (sniRequired != null){
		sslContextFactory.setSniRequired(sniRequired);
	}

	if (needClientAuth != null) {
		sslContextFactory.setNeedClientAuth(needClientAuth);
	}

	if (wantClientAuth != null) {
		sslContextFactory.setWantClientAuth(wantClientAuth);
	}

	sslContextFactory.setKeyStorePath(keyStorePath);
	if (!keyStorePassword.isEmpty()){
		sslContextFactory.setKeyStorePassword(keyStorePassword);
	}

	// The ConnectionFactory for TLS.
	SslConnectionFactory tls = new SslConnectionFactory(sslContextFactory, http11.getProtocol());

	// The ServerConnector instance.
	httpsConnector = new ServerConnector(server, tls, http11);
	httpsConnector.setPort(httpsPort);

	if (!host.isEmpty()) {
		httpsConnector.setHost(host);
	}
	server.addConnector(httpsConnector);

}

// Setup the basic application "context" for this application at "/"
// This is also known as the handler tree (in jetty speak)
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
context.setContextPath("/");

ResourceHandler resource_handler = new ResourceHandler();
resource_handler.setDirectoriesListed(true);
resource_handler.setResourceBase(pathForStaticFiles);
HandlerList handlers = new HandlerList();
handlers.setHandlers(new Handler[] { resource_handler, context});
server.setHandler(handlers);

// Configure specific websocket behavior
JettyWebSocketServletContainerInitializer.configure(context, (servletContext, wsContainer) ->
{
	// Configure default max size
	wsContainer.setMaxTextMessageSize(65535);
	// Add websockets
	wsContainer.addMapping("/signaling", new JettyEventEndpointCreator());
});

...


server.start();
server.join();


    public class JettyEventEndpointCreator implements JettyWebSocketCreator
    {
        @Override
        public Object createWebSocket(JettyServerUpgradeRequest jettyServerUpgradeRequest, JettyServerUpgradeResponse jettyServerUpgradeResponse)
        {
            return new JettyEventEndpoint();
        }
    }

    public class JettyEventEndpoint extends WebSocketAdapter
    {
        private final Logger logger = LoggerFactory.getLogger(getClass());
        private final CountDownLatch closureLatch = new CountDownLatch(1);

        @Override
        public void onWebSocketConnect(Session session)
        {
            super.onWebSocketConnect(session);
            try {
                String sessionId = String.valueOf(session.hashCode());
                logger.debug("sessionId = {} - onConnect", sessionId);
                sendMessage(session, createSignalingMessage(MESSAGE_READY, ""));
            } catch (Exception e) {
                logger.error("onConnect : Exception : ", e);
            }
        }

        @Override
        public void onWebSocketText(String message)
        {
            super.onWebSocketText(message);
			...
		}
	}

As soon as my client connect to th websocket I have the following message:

java.lang.Exception: A connection with system name or id 'ignition-tz4-backup' already exists on the GatewayNetwork! The new connection from https://10.0.2.12:8060/system has been rejected

at com.inductiveautomation.metro.impl.protocol.websocket.MetroWebSocket.onConnect(MetroWebSocket.java:170)

at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onOpen(JettyWebSocketFrameHandler.java:177)

at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$onOpen$6(WebSocketCoreSession.java:411)

at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1469)

at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1488)

at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)

at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.onOpen(WebSocketCoreSession.java:411)

at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onOpen(WebSocketConnection.java:542)

at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:451)

at org.eclipse.jetty.server.HttpConnection.upgrade(HttpConnection.java:419)

at org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:450)

at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:968)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:485)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)

at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)

at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)

at java.base/java.lang.Thread.run(Unknown Source)

bmusson · March 15, 2024, 1:04am

You should really be registering a servlet instead of starting a whole new server.

context.getWebResourceManager().addServlet(name, class)

~~Check out org.eclipse.jetty.websocket.servlet.WebSocketServlet for WebSocket support.~~

Edit: Advice is for the wrong version of Jetty.

mazeyrat · March 15, 2024, 6:59am

Can you please share an exemple, for adding a websocket ?
Sure it will be better if I can avoid to start a whole new jetty server.

or perhaps someone at IA ? @PGriffith ? can share a snipped to add some custom WebSocket without launching another jetty server.

PGriffith · March 15, 2024, 6:18pm

Register your websocket servlet:

WebResourceManager web = gatewayContext.getWebResourceManager();
web.addServlet("pws", PerspectiveWebSocketServlet.class);

Extend JettyWebSocketServlet:

public class PerspectiveWebSocketServlet extends JettyWebSocketServlet {

Override your creator and any other websocket initialization settings in configure:

@Override
public void configure(final JettyWebSocketServletFactory factory) {
    factory.setMaxTextMessageSize((long) maxSizeKb * KB);
    factory.setCreator(new PerspectiveWebSocketCreator());

Implement JettyWebSocketCreator:

private static class PerspectiveWebSocketCreator implements JettyWebSocketCreator {

Conditionally (by whatever authorization logic you need), create a new websocket class in createWebSocket:

@Override
public WebSocketChannel createWebSocket(JettyServerUpgradeRequest req, JettyServerUpgradeResponse resp) {

Create your websocket class instance, which should be annotated with @WebSocket and have various annotated methods for the things you care about listening to:
```
@WebSocket
public class WebSocketChannel
```
```
@OnWebSocketMessage
public void onMessage(Session client, String rawMessage) throws IOException {
```

mazeyrat · March 15, 2024, 7:36pm

In your example server will be configured to reply to : ws://localhost:8088/pws ?

PGriffith · March 15, 2024, 8:49pm

/system/pws. Pick a different endpoint so you don't conflict with Perspective

mazeyrat · March 18, 2024, 4:41pm

Conditionally (by whatever authorization logic you need), create a new websocket class in createWebSocket:

@Override
public WebSocketChannel createWebSocket(JettyServerUpgradeRequest req, JettyServe

You mean ?

    private static class PerspectiveWebSocketCreator implements JettyWebSocketCreator {

        @Override
        public Object createWebSocket(JettyServerUpgradeRequest jettyServerUpgradeRequest, JettyServerUpgradeResponse jettyServerUpgradeResponse) {
               return new WebSocketChannel();
           }
    }

why PerspectiveWebSocketCreator must be static ?

PGriffith · March 18, 2024, 5:51pm

It doesn't have to be static, but in our codebase it's a nested class inside the servlet and in general it's better to make things static if they can be, to help avoid memory leaks and enforce separation of concerns.

Yes. But creating websockets is relatively "expensive", so you may want to put some kind of authorization on that endpoint before you actually create a real connection. Depends what your use case is and what your threat model is.

mazeyrat · March 18, 2024, 6:55pm

I probably miss something, when I connect the client websocket I have the following error:

public class WebRTCJettyServer{

	public void init(){

		WebResourceManager web = gatewayContext.getWebResourceManager();
		web.addServlet("byes-signaling", PerspectiveWebSocketServlet.class);

	...
	}

	public class PerspectiveWebSocketServlet extends JettyWebSocketServlet {
        @Override
        protected void configure(JettyWebSocketServletFactory jettyWebSocketServletFactory) {
            jettyWebSocketServletFactory.setMaxTextMessageSize(65536);
            jettyWebSocketServletFactory.setCreator(new PerspectiveWebSocketCreator());
        }
    }

	public class PerspectiveWebSocketCreator implements JettyWebSocketCreator {

        private final Logger logger = LoggerFactory.getLogger(getClass());

        @Override
        public Object createWebSocket(JettyServerUpgradeRequest jettyServerUpgradeRequest, JettyServerUpgradeResponse jettyServerUpgradeResponse) {

            logger.info("createWebSocket");
            return new WebSocketChannel();
        }
    }

	@WebSocket
	public class WebSocketChannel{
        private final Logger logger = LoggerFactory.getLogger(getClass());

        @OnWebSocketConnect
        public void onWebSocketConnect(Session session)
        {

            try {
                sendMessage(session, createSignalingMessage(MESSAGE_READY, ""));
            } catch (Exception e) {
                logger.error("onConnect : Exception : ", e);
            }
        }
		
		...		
	
	}

Error detail:

java.lang.InstantiationException: com.bouyguesenergiesservices.ignition.gateway.videoviewer.WebRTCJettyServer$PerspectiveWebSocketServlet

at java.base/java.lang.Class.newInstance(Unknown Source)

at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:74)

at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)

at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)

at org.eclipse.jetty.server.Server.handle(Server.java:563)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: java.lang.NoSuchMethodException: com.bouyguesenergiesservices.ignition.gateway.videoviewer.WebRTCJettyServer$PerspectiveWebSocketServlet.()

at java.base/java.lang.Class.getConstructor0(Unknown Source)

pturmel · March 18, 2024, 7:02pm

You must have a no-args constructor on your servlet class. That is what will be created by jetty to handle your endpoints. You must use static methods elsewhere for your servlet to get its initialization data.

mazeyrat · March 18, 2024, 7:37pm

I have the same error with adding a no-arg constructor:

    public class PerspectiveWebSocketServlet extends JettyWebSocketServlet {

        public PerspectiveWebSocketServlet() {
        }

        @Override
        protected void configure(JettyWebSocketServletFactory jettyWebSocketServletFactory) {
            jettyWebSocketServletFactory.setMaxTextMessageSize(65536);
            jettyWebSocketServletFactory.setCreator(new PerspectiveWebSocketCreator());
        }
    }

What do you mean by "its initialization data" ?

PGriffith · March 18, 2024, 7:38pm

You're not explicitly marking your inner class static, so it requires an implicit reference to the outer class to be constructed. Make it static and you should be able to construct it.

mazeyrat · March 18, 2024, 9:41pm

Yes it works better with static. Thanks again !
Launching another Jetty inside the same JVM seems was the conflict source.
I need to refactor portion of my initial code and run more test to confirm that I have no remaining conflicts with GAN ad redundancy websockets.

mazeyrat · March 19, 2024, 10:33am

My code doesn't launch anymore a separate Jetty server.

It's fine for websockets, but I need to expose few static HTML and Javascript files

For HTML it's ok with: (in the module Gateway Hook)


    @Override
    public Optional<String> getMountPathAlias(){
        return Optional.of(MOUNT_PATH_ALIAS);
    }

	@Override
	public void mountRouteHandlers(RouteGroup routes){

	routes.newRoute("/index.html")
				  .handler((req, res) -> getMountedData(req, res, "index.html"))
				  .type(RouteGroup.TYPE_TEXT_HTML)
				  .mount();


        routes.newRoute("/test.js")
                .handler((req, res) -> getMountedData(req, res, "test.js"))
                .type(RouteGroup.TYPE_PLAIN_TEXT)
                .mount();

But for JS RouteGroup.TYPE_PLAIN_TEXT generate the error in the browser:

Refused to execute script from 'http://127.0.0.1:8088/data/test.js' because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.

RouteGroup type have no type for JS MIME

@PGriffith, @pturmel Is there a way to expose JS file thru the SDK with ignition embedded Jetty ???otherwise than copying the JS files in:

C:\Program Files\Inductive Automation\Ignition\webserver\webapps\main

As a site note: I don't have webdev module.

PGriffith · March 19, 2024, 1:23pm

You don't have to use one of the constants in RouteGroup. You can just use any string you want as the content type.

mazeyrat · March 19, 2024, 1:50pm

Thanks again

        routes.newRoute("/test.js")
                .handler((req, res) -> getMountedData(req, res, "videoplayer.js"))
                .type("text/javascript")
                .mount();