Timeout error after upgrading to Ignition 8.3.7

Ignition
1

Hello,

I upgraded our Ignition system from version 8.1.52 to 8.3.7 last night. We have one central Gateway connected to five I/O Gateways.

I upgraded all of them, but something went wrong on one of the I/O Gateways. When I try to change any settings, a timeout error appears. However, the changes are still applied and visible in the Gateway.

I do not experience this problem on any of the other Gateways, and the migration completed successfully on all of them.

What could be causing this issue?

image
image1919×835 62.1 KB

Best regards,
Michał

2

The error is just the "frontend" (the code running in the web page you're looking at) giving up on waiting for the "backend" (the actual Ignition gateway) to finish, due to some likely hardcoded timeout. That it's taking longer than whatever arbitrary threshold we decided on is potentially concerning - it could be an indicator that the gateway is struggling for some other reason(s).

3

After upgrading our Ignition architecture from 8.1.52 to 8.3.7, I noticed a significant difference in the time required to save Gateway configuration changes.

Our architecture consists of:

  • Five large IO Gateways with different configurations and different numbers of tags

  • One Frontend Gateway containing all Perspective projects

  • Remote Tag Provider connections from the Frontend Gateway to the five IO Gateways

The Frontend Gateway does not contain device connections, local OPC connections, databases, Store and Forward pipelines, or local tag configurations. It mainly contains Perspective projects and connections to remote tag providers.

On the Frontend Gateway, configuration changes are saved almost immediately, and the confirmation is returned within a few seconds, similarly to a clean Gateway.

The IO Gateways behave differently:

  • Two IO Gateways successfully save the configuration and return a confirmation, but the confirmation time is significantly longer than on the Frontend Gateway.

  • The remaining three IO Gateways exceed the 60-second timeout. The configuration change is usually applied successfully, but the web interface displays a timeout error because it does not receive confirmation within 60 seconds.

I performed several additional tests to identify the cause.

1. Number of tags

I could not find a direct relationship between the number of tags and the problem.

The IO Gateways contain between several hundred thousand and several million tags. However, a Gateway with more tags does not necessarily have a longer save time than a Gateway with fewer tags.

2. Gateway resources

All Gateways appear to have sufficient CPU, memory, and disk resources for normal operation. I therefore assume that the problem is not simply caused by insufficient system resources.

I also do not see any obvious resource saturation during the configuration save operation.

3. Disabling external connections and services

On one of the IO Gateways that always exceeds the timeout, I disabled almost all major configurations using the “select all” and “disable” functions, including:

  • Device connections

  • OPC connections and OPC servers

  • Database connections

  • Store and Forward / Journal configurations

  • Tag providers

  • Connections to other Gateways

Disabling these configurations did not improve the save confirmation time. The operation still exceeded 60 seconds and returned the same timeout error.

Based on these tests, I suspect that the delay may be caused by an internal Gateway operation triggered when a configuration change is saved.

It appears that the overall amount or complexity of the Gateway configuration may influence the save confirmation time, even when the configured resources are disabled. The IO Gateways contain thousands of configuration entries in total, including device connections, tag providers, OPC configurations, journals, databases, and other Gateway resources.

My assumption is that the Gateway may be performing an internal configuration scan, validation, synchronization, serialization, resource reload, or another operation before returning the save confirmation.

However, I cannot determine exactly what is happening internally or which configuration type has the greatest impact.

Questions

  1. What internal operations are performed by the Gateway after saving a configuration change?

  2. Does Ignition scan, validate, serialize, synchronize, or reload all Gateway configuration resources before returning the response?

  3. Can disabled resources still affect the save confirmation time?

  4. Are there specific loggers that can be enabled to identify which internal operation is taking more than 60 seconds?

  5. Is there a thread dump, diagnostic bundle, or performance recording that would be useful for investigating this?

  6. Is the 60-second timeout hardcoded in the Gateway web interface, or can it be configured?

  7. Are there any known performance issues in Ignition 8.3.7 related to saving configuration on large IO Gateways?

Because each IO Gateway contains thousands of configuration entries, testing every device, provider, journal, and connection individually would be very difficult.

Before opening an official support ticket, I would like to understand whether there is a recommended way to diagnose what the Gateway is doing between receiving the save request and returning the confirmation.

Thank you for any suggestions.

4

You should just contact support. They're going to have more useful advice, having dealt with a slew of customers with similar problems, than I am just off reading the code.

I think that the short answer is yes, there are some known performance degradations with lots of tags because of the fact that we're putting tags through the resource system; this is a net win for version control, remote configuration, etc, but it is not a free lunch; updating a single SQLite DB is faster than updating many files across deeply nested folders, particular on slower filesystems (cough NTFS, cough Windows).

A series of thread dumps while you're waiting on a save to complete is the useful diagnostic signal. You can also try setting the gateway.ConfigurationManagerImpl logger to debug or trace.

I looked into it further, and the "Timed out waiting for push to complete." message is actually coming from the backend, but it's a hardcoded 1 minute wait on any resource update request.