8.3.9: Gateway Network route stays down after a back-end stall while the front end shows Running

If I pause the back end for 60 seconds while a remote history call from the front end is outstanding, the Gateway Network route sometimes stays down until I restart the front end. Across the same stall, system.tag.queryTagHistory returns a partial dataset or no rows, with no exception.

Setup: two inductiveautomation/ignition:8.3.9 containers (b2026082511), standard edition, Docker on macOS. The back-end gateway has a SQL Historian on Postgres and a Core Historian. The front-end gateway holds no history: one outgoing Gateway Network connection to the back end, a remote historian for each back-end historian, a remote tag provider. Gateway Network SSL and two-way auth are off, security policy Unrestricted on both.

Steps:

  1. On the front end, a WebDev doGet resource runs this; undisturbed it returns 9,636 rows in 368 ms:
paths = [...]  # 22 paths: histprov:{remote historian}:/drv:{back end}:default:/tag:{folder}/{tag}
start = system.date.fromMillis(1792825200000)  # 8-day window
end = system.date.fromMillis(1793516400000)
ds = system.tag.queryTagHistory(paths=paths, startDate=start, endDate=end,
    returnSize=-1, returnFormat='Tall', noInterpolation=True, includeBoundingValues=False)
return {'json': {'rows': ds.getRowCount()}}
  1. Call it, then docker pause the back end about a second or less later.
  2. docker unpause it 60 seconds after the pause.
  3. Every 30 seconds for 8 minutes, read GET /data/api/v1/gateway-network/connections on both gateways and run testConnectionToServer from the front end.

A second resource calls system.tag.browseHistoricalTags on one folder, sent less than a second after the pause (it answers in about 4 ms).

Cycles still down 8 minutes after the unpause:

  • queryTagHistory outstanding: 1 of 5
  • browseHistoricalTags sent into the pause: 1 of 5
  • no call outstanding (90 second pauses): 0 of 3; the route was back 6.3 to 6.6 seconds after each unpause

The same failure shows with a module's history calls on this pair (2 of 5).

In every stuck cycle the back end listed its incoming connection from the front end as Initialized at +30 seconds, then logged Initialized to Faulted: onClose has been called on web socket:Connection Idle Timeout and kept it Faulted. The front end listed its outgoing connection as Running throughout, testConnectionToServer failed in 1 to 16 ms, and every remote history call failed at once with:

ServerUnavailableException: The server '_0:0:{back end}_defaultQueue' is not currently available.

The idle-timeout line also appeared in 8 of 11 recovered cycles; only Initialized at +30 seconds marked the stuck ones. A front-end thread dump at +60 seconds showed no thread still in either call. Restarting the front end recovered it within 16 seconds every time.

In all five queryTagHistory cycles, stuck or not, the call returned after about 60 seconds with no exception: once with 2,000 of the 9,636 rows, four times with 0 rows. With 2,000 rows the front end logged Error deserializing streaming dataset with java.net.SocketTimeoutException: Read timed out at the unpause; with 0 rows, Error executing historical tag read. The script saw neither.

I expected the route to reconnect as it does with no call outstanding, and the query to raise an error when its stream breaks.

Is there a supported way for a script or module to detect a remote connection that is stalled while it shows Running, or a remote history result that was cut short?

Interesting bug. I haven't seen anything like that here. You should open a support ticket and help IA look closer.