[IGN-13832] Ignition Docker Image at Kubernetes with Error to show the projects

Hello!,

I'm testing the Kubernetes integration with the Docker Image.
I'm able to deploy and use the gateway, but when try to open the Launch page for Perspective projects have this error:

com.inductiveautomation.perspective.gateway.comm.Routes$MissingGatewayAddressException

I'm able to open the project by the link on perspective with no issue just the Gateway page not show the projects.

Using Ingress with Nginx to access the internal service, gateway is not using TLS, only at Ingress have TLS

com.inductiveautomation.perspective.gateway.comm.Routes$MissingGatewayAddressException: null

at com.inductiveautomation.perspective.gateway.comm.Routes.getRunnableProjects(Routes.java:1854)

at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:361)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:120)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupCollectionServlet.serviceInternal(RouteGroupCollectionServlet.java:133)

at com.inductiveautomation.ignition.gateway.dataroutes.AbstractRouteGroupServlet.service(AbstractRouteGroupServlet.java:40)

at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)

at org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1379)

at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)

at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)

at com.inductiveautomation.catapult.filters.GatewayFilter.doFilter(GatewayFilter.java:116)

at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)

at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)

at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)

at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)

at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:819)

at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)

at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)

at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)

at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)

at org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)

at org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)

at org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)

at org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)

at org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)

at org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)

at org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)

at org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)

at org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)

at org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)

at org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)

at org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)

at org.eclipse.jetty.server.Server.handle(Server.java:182)

at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:662)

at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:418)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)

at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)

at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)

at java.base/java.lang.Thread.run(Unknown Source)

I think the mixed negotiation is what's confusing things.

If we're detecting the incoming request as secure (via HTTPS protocol) but you don't have any HTTPS settings configured, you'll get that error:

if (!req.getRequest().isSecure() && settings.httpPort().isPresent()) {
    gatewayAddress = new HttpURL(
        "http", settings.address(), settings.httpPort().getAsInt(), null).toString();
} else if (settings.httpsPort().isPresent()) {
    gatewayAddress = new HttpURL(
        "https", settings.address(), settings.httpsPort().getAsInt(), null).toString();
} else {
    throw new MissingGatewayAddressException();
}

I don't know if there's some better way to have the traffic show up as HTTP to the Ignition gateway while still negotiating as HTTPS outside of Kubernetes. @kcollins1 might :smiley:

If it is sitting behind an Ingress, you'll probably want to:

  1. Configure your public address settings in Ignition.
  2. Supply gateway.useProxyForwardedHeader=true gateway arg after reading this.

Number 2 there will bridge the gap of communicating to Ignition that you're terminating TLS upstream.

1 Like

Bonus: the Helm Chart we're working on will make it relatively easy to use the same Secret that is linked to your Ingress for TLS termination with the Gateway web server for end-to-end encryption without a lot of setup headaches. Stay tuned!

2 Likes

Interesting, I use the same settings for 8.1.48 gateways, today I manage around 9 Gateways using the same settings, so I created a new deploy with the new 8.3 to check the changes. They have the proxy forwarded header, but the public address is auto. I'll try to setup the public address with the right link first and if not work also create a certificate for the gateway. @kcollins1 I'm using most of the scripts you build for the ICC and the unofficial docker image repo for some scripting to automate a few task, thanks for all the work building that scripts.

Just a missing comment, Everything works except the Perspective Launch page app/home/perspective/session-launcher

Appling a TLS at the Gateway Web Settings worked. Now the Launch Page is loading correctly

2 Likes

I am experiencing the same issue, but I am using Docker with Traefik as a reverse proxy, which handles the certificates and terminates the HTTPS connection. The connection between Traefik and Ignition is not encrypted (port 8088). Therefore, I am unable to set up TLS in Ignition because the certificates are managed by Traefik.
This works perfectly in Ignition 8.1 without a TLS setup, but the problem occurs in 8.3.
(Forwarded headers are enabled and the public address and ports are set.)

Is there an alternative to the TLS setup in Ignition to avoid this problem?

Some years ago I have a Docker deployment, but on this case is using the nginx and lets encrypt images to manage the certificate

version: "3.7"

services:

    reverse-proxy:
        image: "jwilder/nginx-proxy:latest"
        container_name: "reverse-proxy"
        volumes:
            - "html:/usr/share/nginx/html"
            - "dhparam:/etc/nginx/dhparam"
            - "vhost:/etc/nginx/vhost.d"
            - "certs:/etc/nginx/certs"
            - "/run/docker.sock:/tmp/docker.sock:ro"
            - "acme:/etc/acme.sh"
            - "/home/ubuntu/ignition/Nginx/proxy.conf:/etc/nginx/proxy.d/proxy.conf:ro"
            - "/home/ubuntu/ignition/Nginx/my_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro"
        restart: "unless-stopped"
        networks:
            - "net"
        ports:
            - "80:80"
            - "443:443"
    letsencrypt:
        image: "jrcs/letsencrypt-nginx-proxy-companion:latest"
        container_name: "letsencrypt-helper"
        volumes:
            - "html:/usr/share/nginx/html"
            - "dhparam:/etc/nginx/dhparam"
            - "vhost:/etc/nginx/vhost.d"
            - "certs:/etc/nginx/certs"
            - "/run/docker.sock:/var/run/docker.sock:ro"
            - "acme:/etc/acme.sh"
        environment:
            NGINX_PROXY_CONTAINER: "reverse-proxy"
            DEFAULT_EMAIL: "email@domain.com"
        restart: "unless-stopped"
        depends_on:
            - "reverse-proxy"
        networks:
            - "net"
    ignition-home:
        image: "kcollins/ignition:latest"
        container_name: "ignition-home"
        stop_grace_period: "30s"
        secrets:
           - "gateway-password"
        volumes:
           - "gateway-data:/var/lib/ignition/data"
           - "/home/ubuntu/ignition/Modules:/modules"
           - "/home/ubuntu/ignition/JDBC:/jdbc"
        logging:
            driver: "json-file"
            options:
               max-size: "200k"
               max-file: "10"
        environment:
            ACCEPT_IGNITION_EULA: "Y"
            GATEWAY_SYSTEM_NAME: "ignition-home"
            IGNITION_EDITION: "maker"
            TZ: "America/Sao_Paulo"
            IGNITION_LICENSE_KEY: "LICENSE"
            IGNITION_ACTIVATION_TOKEN: "ACTIVATION"
            GATEWAY_ADMIN_USERNAME: "ADMIN"
            GATEWAY_ADMIN_PASSWORD_FILE: "/run/secrets/gateway-password"
            GATEWAY_HTTP_PORT: "8088"
            GATEWAY_HTTPS_PORT: "8043"
            GATEWAY_MODULES_ENABLED: "alarm-notification,allen-bradley-drivers,logix-driver,modbus-driver-v2,omron-driver,opc-ua,perspective,reporting,serial-support-gateway,sfc,siemens-drivers,sql-bridge,tag-historian,udp-tcp-drivers,web-developer"
            GATEWAY_MODULE_RELINK: "true"
            GATEWAY_JDBC_RELINK: "true"
            GATEWAY_PUBLIC_ADDRESS: "localhost"
            GATEWAY_PUBLIC_HTTP_PORT: "9088"
            GATEWAY_PUBLIC_HTTPS_PORT: "9043"
            GATEWAY_MAX_MEMORY: "4096"
            GATEWAY_SKIP_COMMISSIONING: "0"
            VIRTUAL_HOST: "domain.com"
            LETSENCRYPT_HOST: "domain.com"
            LETSENCRYPT_EMAIL: "email@domain.com"
            VIRTUAL_PORT: "8088"
        restart: "unless-stopped"
        depends_on:
             - "reverse-proxy"
        networks:
             - "net"
        command: >
           -- gateway.resolveHostNames=false gateway.useProxyForwardedHeader=true
        ports:
            - "1521:1521"

    db:
         image: mysql:latest
         container_name: "ignition-db"
         volumes:
             - "db_data:/var/lib/mysql"
         logging:
             driver: "json-file"
             options:
                max-size: "200k"
                max-file: "10"
         secrets:
             - "mysql-root-password"
             - "mysql-password"
         environment:
             MYSQL_ROOT_PASSWORD_FILE: "/run/secrets/mysql-root-password"
             MYSQL_DATABASE: "ignition"
             MYSQL_USER: "ignition"
             MYSQL_PASSWORD_FILE: "/run/secrets/mysql-password"
         networks:
             - "net"
secrets:
   mysql-root-password:
      file: ./secrets/MYSQL_ROOT_PASSWORD
   mysql-password:
      file: ./secrets/MYSQL_PASSWORD
   gateway-password:
      file: ./secrets/GATEWAY_PASSWORD

volumes:
  certs:
  html:
  vhost:
  dhparam:
  acme:
  gateway-data:
  db_data:
networks:
  net:
    external: true

May help this image from jrcs/letsencrypt-nginx-proxy-companion like you can create a certificate for the gateway and need some maybe bash script to take the files from the lets script and build a new ssl.pfx file for ignition

I'm going to debug and investigate this further--I'm seeing the very same on my local stack which is very much the same (Docker, Traefik, and step-ca sidecar that Traefik uses TLS-ALPN-01 challenges to request local certs from). Applying TLS at the gateway as well is a workaround but shouldn't be required. I'll report back findings.

EDIT: I've confirmed the issue and wrote up ticket IGN-13832 on our side. Thank you @Marco_Keller and @AGDC04 for taking the time to report this!

2 Likes

Thank you @kcollins1!
In that context an ENV Variable for “Force Secure Redirect*“ would be verry valueble as well.

It’s not possible to set it when HTTPS is handled outside Ignition so no redirection happens when opening Ignition with http.

But there is a workaround, I can set the redirection on traefik as well.

This has been resolved and will be in 8.3.0-beta4 next week. Also, @Marco_Keller , the force secure redirect setting can now be applied even when https is not configured in Ignition, though it requires being driven externally via a gateway argument gateway.forceSecureRedirect=true, applied like shown in the example here:

3 Likes

Wow, that was a quick fix. Thank you verry much!