Timeout error after upgrading to Ignition 8.3.7

Michal_Goralczyk · July 15, 2026, 9:13am

Hello,

I upgraded our Ignition system from version 8.1.52 to 8.3.7 last night. We have one central Gateway connected to five I/O Gateways.

I upgraded all of them, but something went wrong on one of the I/O Gateways. When I try to change any settings, a timeout error appears. However, the changes are still applied and visible in the Gateway.

I do not experience this problem on any of the other Gateways, and the migration completed successfully on all of them.

What could be causing this issue?

Best regards,
Michał

paul-griffith · July 15, 2026, 5:18pm

The error is just the "frontend" (the code running in the web page you're looking at) giving up on waiting for the "backend" (the actual Ignition gateway) to finish, due to some likely hardcoded timeout. That it's taking longer than whatever arbitrary threshold we decided on is potentially concerning - it could be an indicator that the gateway is struggling for some other reason(s).

Michal_Goralczyk · July 20, 2026, 7:27am

After upgrading our Ignition architecture from 8.1.52 to 8.3.7, I noticed a significant difference in the time required to save Gateway configuration changes.

Our architecture consists of:

Five large IO Gateways with different configurations and different numbers of tags
One Frontend Gateway containing all Perspective projects
Remote Tag Provider connections from the Frontend Gateway to the five IO Gateways

The Frontend Gateway does not contain device connections, local OPC connections, databases, Store and Forward pipelines, or local tag configurations. It mainly contains Perspective projects and connections to remote tag providers.

On the Frontend Gateway, configuration changes are saved almost immediately, and the confirmation is returned within a few seconds, similarly to a clean Gateway.

The IO Gateways behave differently:

Two IO Gateways successfully save the configuration and return a confirmation, but the confirmation time is significantly longer than on the Frontend Gateway.
The remaining three IO Gateways exceed the 60-second timeout. The configuration change is usually applied successfully, but the web interface displays a timeout error because it does not receive confirmation within 60 seconds.

I performed several additional tests to identify the cause.

1. Number of tags

I could not find a direct relationship between the number of tags and the problem.

The IO Gateways contain between several hundred thousand and several million tags. However, a Gateway with more tags does not necessarily have a longer save time than a Gateway with fewer tags.

2. Gateway resources

All Gateways appear to have sufficient CPU, memory, and disk resources for normal operation. I therefore assume that the problem is not simply caused by insufficient system resources.

I also do not see any obvious resource saturation during the configuration save operation.

3. Disabling external connections and services

On one of the IO Gateways that always exceeds the timeout, I disabled almost all major configurations using the “select all” and “disable” functions, including:

Device connections
OPC connections and OPC servers
Database connections
Store and Forward / Journal configurations
Tag providers
Connections to other Gateways

Disabling these configurations did not improve the save confirmation time. The operation still exceeded 60 seconds and returned the same timeout error.

Based on these tests, I suspect that the delay may be caused by an internal Gateway operation triggered when a configuration change is saved.

It appears that the overall amount or complexity of the Gateway configuration may influence the save confirmation time, even when the configured resources are disabled. The IO Gateways contain thousands of configuration entries in total, including device connections, tag providers, OPC configurations, journals, databases, and other Gateway resources.

My assumption is that the Gateway may be performing an internal configuration scan, validation, synchronization, serialization, resource reload, or another operation before returning the save confirmation.

However, I cannot determine exactly what is happening internally or which configuration type has the greatest impact.

Questions

What internal operations are performed by the Gateway after saving a configuration change?
Does Ignition scan, validate, serialize, synchronize, or reload all Gateway configuration resources before returning the response?
Can disabled resources still affect the save confirmation time?
Are there specific loggers that can be enabled to identify which internal operation is taking more than 60 seconds?
Is there a thread dump, diagnostic bundle, or performance recording that would be useful for investigating this?
Is the 60-second timeout hardcoded in the Gateway web interface, or can it be configured?
Are there any known performance issues in Ignition 8.3.7 related to saving configuration on large IO Gateways?

Because each IO Gateway contains thousands of configuration entries, testing every device, provider, journal, and connection individually would be very difficult.

Before opening an official support ticket, I would like to understand whether there is a recommended way to diagnose what the Gateway is doing between receiving the save request and returning the confirmation.

Thank you for any suggestions.

paul-griffith · July 20, 2026, 5:09pm

You should just contact support. They're going to have more useful advice, having dealt with a slew of customers with similar problems, than I am just off reading the code.

I think that the short answer is yes, there are some known performance degradations with lots of tags because of the fact that we're putting tags through the resource system; this is a net win for version control, remote configuration, etc, but it is not a free lunch; updating a single SQLite DB is faster than updating many files across deeply nested folders, particular on slower filesystems (cough NTFS, cough Windows).

A series of thread dumps while you're waiting on a save to complete is the useful diagnostic signal. You can also try setting the gateway.ConfigurationManagerImpl logger to debug or trace.

I looked into it further, and the "Timed out waiting for push to complete." message is actually coming from the backend, but it's a hardcoded 1 minute wait on any resource update request.

Noah_Burnette · August 10, 2026, 3:30pm

Running into this same issue.

Michał, were you able to contact support? Did they have any suggestions?

Thanks in advance!

Michal_Goralczyk · August 11, 2026, 6:17am

It has been reported to support, but no solution has been found. Try reporting it again to get more attention from support.

Noah_Burnette · August 18, 2026, 5:50pm

After speaking with support, this bug will be fixed in 8.3.9

kyvski · September 11, 2026, 3:04pm

I have the same issue but support wasn't able to identify the problem, nor did they mention that others were having the popup on 8.3.7. I would think that "Timed out waiting for push to complete" is specific enough to be labeled a bug when multiple people are experiencing it - and they should pull the release (or any related like 8.3.4 when this first started) like they've done before with production-breaking degradations. The only metric I have that seems related is the configuration "notification queue" in the Metrics dashboard suddenly spikes to 1k or even 8k at times, and the error seems to follow that.

My support rep said that it could be fixed with a performance improvement coming in 8.3.11, 8.3.12, or later since it isn't a high priority. Which seems a bit dismissive due to the severity of the issue. There was no mention of 8.3.9 providing a fix for this either...

Michal_Goralczyk · September 11, 2026, 3:39pm

Based on an AI-assisted analysis of the decompiled Ignition jar files, we suspect that saving a configuration change triggers an expensive scan operation across all resource collections on the gateway. This scan appears to run over every collection - including those unaffected by the change - because the check for whether a collection is actually affected seems to happen only after the full scan, not before. On gateways with large configuration sets and a lot of tags, this unnecessary work adds up and causes the save to exceed the 60-second limit, resulting in a timeout in the UI even though the save itself completes. This is our current suspicion based on code analysis and thread dumps, but we are not able to confirm with certainty that this is the underlying cause - that is why we asked the questions above, to verify this hypothesis

kyvski · September 11, 2026, 4:06pm

But that reason would imply that the popup would always happen on a configuration change.
At least for us, the issue starts only after that "notification queue" isn't 0 anymore
That seems to happen due to a specific event after the gateway has been running for an extended period. Maybe that event is related for the reasons you listed, but something has to change to allow it to work before but suddenly it doesn't later.
I'm not sure exactly what the "notification queue" is (it isn't documented on the Metrics Dashboard docs page), but maybe whatever processes the queue suddenly stops working due to some kind of hang?

Noah_Burnette · September 11, 2026, 6:47pm

As per support:

Hi Noah,

I came across an internal bug describing this issue.

In your thread dumps, I saw a blocked thread that was trying to compute the OPC UA server's datatype tree:

org.eclipse.milo.opcua.stack.core.util.Lazy.reset(Lazy.java:84) 
org.eclipse.milo.opcua.sdk.client.OpcUaClient.resetDataTypeTree(OpcUaClient.java:926) 
org.eclipse.milo.opcua.sdk.client.OpcUaClient.lambda$new$10(OpcUaClient.java:488) 
org.eclipse.milo.opcua.sdk.client.OpcUaClient$$Lambda$4728/0x00000008017808b0.initialize(Unknown Source) 
org.eclipse.milo.opcua.sdk.client.session.SessionFsmFactory.runSequentially(SessionFsmFactory.java:1584) 
org.eclipse.milo.opcua.sdk.client.session.SessionFsmFactory.lambda$runSequentially$86(SessionFsmFactory.java:1598) 
org.eclipse.milo.opcua.sdk.client.session.SessionFsmFactory$$Lambda$5407/0x00000008019293c8.apply(Unknown Source) 
java.base@17.0.18/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) 
java.base@17.0.18/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
java.base@17.0.18/java.util.concurrent.CompletableFuture.complete(Unknown Source) 
org.eclipse.milo.opcua.stack.transport.client.AbstractUascClientTransport.lambda$handleResponse$4(AbstractUascClientTransport.java:149) 
org.eclipse.milo.opcua.stack.transport.client.AbstractUascClientTransport$$Lambda$4569/0x000000080175d1d0.run(Unknown Source) 
java.base@17.0.18/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
java.base@17.0.18/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
java.base@17.0.18/java.lang.Thread.run(Unknown Source)

This corresponds to our bug ticket, where a deadlock was identified with this operation. To confirm this, I wanted to ask if this issue was reproducible on your dev server?

I've attached a module that you can upload to your dev gateway to test if this will fix it. This bug is fixed in 8.3.9, however instead/before upgrading, uploading a new OPC UA module will be simpler for testing.

Let me know if uploading a new module is possible. I have tested this on my own gateway and have not seen any issues.

Based off of my thread dump they did have my specific bug already tracked. The error Timed out waiting for push to complete seems like a generic catch to me so it might not relate to the issues you are running into if you already shared your thread dump and diagnostic bundle