Traefik + Docker Gateways - Premature EOF

scott · April 29, 2025, 6:13am

Has anyone had success using Traefik reverse proxy to forward Designer connections to docker Ignition containers?

I am getting frequent (like every 10-15 seconds) timeout/reconnects only in the Designer (gateway web UI works fine).
Designer console shows me this is caused by a Premature EOF exception.

Has anyone had similar issues and/or know a way to resolve this?

Designer console output:

com.inductiveautomation.ignition.client.gateway_interface.GatewayException: Premature EOF
  at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:360)
  at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:556)
  at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:366)
  at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.ping(GatewayInterface.java:1024)
  at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.ping(GatewayInterface.java:1009)
  at com.inductiveautomation.ignition.client.tags.impl.GatewayTagInterface.runPoll(GatewayTagInterface.java:128)
  at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
  at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Premature EOF
  at java.base/sun.net.www.http.ChunkedInputStream.readAheadBlocking(Unknown Source)
  at java.base/sun.net.www.http.ChunkedInputStream.readAhead(Unknown Source)
  at java.base/sun.net.www.http.ChunkedInputStream.read(Unknown Source)
  at java.base/java.io.FilterInputStream.read(Unknown Source)
  at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$TrailingMiscDriver.next(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
  at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
  at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:498)
  ... 11 common frames omitted
15:30:35.560 [DesignerExecEngine-2] INFO tags.subscriptions -- Changing connected quality to 'Bad_Stale'
15:30:35.561 [DesignerExecEngine-4] INFO Scripting.ScriptManager.Dev_Project -- Pausing scripts...
15:30:35.561 [DesignerExecEngine-4] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager -- Starting reconnect thread.
15:30:35.642 [GatewayConnection-1] INFO tags.subscriptions -- Changing connected quality to 'Good'
15:30:35.642 [GatewayConnection-1] INFO Scripting.ScriptManager.Dev_Project -- Resuming scripts...
15:30:35.691 [GatewayConnection-1] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager -- Stopping reconnect thread.

For Context, I've set up a docker environment consisting of

Traefik Reverse Proxy
Ignition Gateway "enterprise"
Ignition Gateway "site"

The Traefik reverse proxy listens on ports 80/443 and forwards traffic on subdomains to the correct ignition gateway at the correct port, it will also upgrade the connection from HTTP to HTTPS.

based off of this compose file:

services:
  reverse-proxy:
    image: traefik:v3.3
    container_name: traefik
    command:
      - --log.level=INFO
      - --api.insecure=true
      - --providers.docker
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.email=admin@admin.com
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    networks:
      - dev
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - traefik-letsencrypt:/letsencrypt

  ign-enterprise:
    image: inductiveautomation/ignition:8.1.43
    container_name: enterprise
    networks:
      - dev
    labels:
      - "traefik.http.routers.ign-enterprise.rule=Host(`enterprise.example.com`)"
      - "traefik.http.services.ign-enterprise.loadbalancer.server.port=8088"
      - "traefik.http.routers.ign-enterprise.tls.certresolver=myresolver"
    volumes:
      - ign-ent-data:/usr/local/bin/ignition/data
    environment:
      - ACCEPT_IGNITION_EULA=Y
      - GATEWAY_ADMIN_USERNAME=admin
      - GATEWAY_ADMIN_PASSWORD=password
      - IGNITION_EDITION=standard
    command: >
      -n Enterprise-Server
      -m 2048
      --
      wrapper.java.initmemory=512
      -Dignition.allowunsignedmodules=true
    restart:
      unless-stopped

  ign-site:
    image: inductiveautomation/ignition:8.1.43
    container_name: site
    networks:
      - dev
    labels:
      - "traefik.http.routers.ign-site.rule=Host(`site.example.com`)"
      - "traefik.http.services.ign-site.loadbalancer.server.port=8088"
      - "traefik.http.routers.ign-site.tls.certresolver=myresolver"
    volumes:
      - ign-site-data:/usr/local/bin/ignition/data
    environment:
      - ACCEPT_IGNITION_EULA=Y
      - GATEWAY_ADMIN_USERNAME=admin
      - GATEWAY_ADMIN_PASSWORD=password
      - IGNITION_EDITION=standard
    command: >
      -n Site-Server
      -m 2048
      --
      wrapper.java.initmemory=512
      -Dignition.allowunsignedmodules=true
    restart:
      unless-stopped

volumes:
  ign-ent-data:
  ign-site-data:
  traefik-letsencrypt:

networks:
  dev:
    external: true

Ignition gateway web servers have also been configured with Use Proxy Forwarded Headers ticked and Public Address, Public HTTP Port, Public HTTPS Port all configured to match configuration in the docker-compose i.e. Public Address = enterprise.example.com for the enterprise server.

kcollins1 · April 29, 2025, 4:10pm

I have seen this behavior before, but it has been quite a while. Lately I've been doing more testing with Traefik as the ingress controller on K8s hosted gateways and have not seen this there.

I'll review my local Docker reverse proxy stack (which is setup similarly to be able to do TLS termination, though I'm using ACME (in Traefik) against a locally hosted step-ca server) and do some more testing.

Traefik is usually pretty quiet, even at INFO level, but I'd be interested if there are any notable logs on that side.

kcollins1 · April 29, 2025, 4:13pm

Oh, also, I see you're pinned against v3.3 tag, can you share the specific version of Traefik that you're using? (i.e. v3.3.6 ?)

scott · April 30, 2025, 5:10am

Traefik version 3.3.5

logs were set at INFO level and nothing useful there - just a prompt to update to 3.3.6 and a recurring ACME letsencrypt renwal.

scott · May 5, 2025, 3:04am

After all this it may just have been a resources issue.

I doubled the memory allocation to the docker host and it appears to have greatly reduced the timeout issues.
Hopefully was just a silly little error on my part with system sizing but will continue to monitor just in case.

kcollins1 · May 6, 2025, 2:30pm

That is good to hear, @scott .. That may have very well been the scenario I had seen this under in the past and just not realized. Earlier last week I did bring up my Docker reverse proxy stack and left a designer up and didn't see any issues.