Hello,
After upgrading our Ignition architecture from version 8.1 to 8.3.8, we started experiencing a gradual loss of stability in OPC UA client connections.
The same devices and network infrastructure worked correctly with Ignition 8.1.
Environment
- Ignition version: 8.3.8, build b2026071409
- OPC UA module version: 8.3.8
- Several IO Gateways working as OPC UA clients
- Approximately 200 outgoing OPC UA connections per IO Gateway
The Gateway analysed in detail has:
- 204 configured OPC UA connections
- 190 connections in the Connected state
- 14 connections in the Error state
The main OPC UA servers are Phoenix Contact PLCnext AXC F 2152 controllers.
Most remote devices are connected over WAN links. Short communication interruptions may occasionally occur on these links.
The connections use the following security settings:
- Basic256Sha256
- Sign
- SignAndEncrypt
Problem description
Immediately after the upgrade, all OPC UA devices connected correctly and worked without problems for several hours.
After some time, individual connections started entering the following error state:
java.lang.Exception: session inactive:
id=NodeId{ns=1, id=28278825-f823-4274-a9e8-1350e8d4dfdd}
name=ignition_1784617805703
at com.inductiveautomation.ignition.gateway.opcua.client.connection.OpcUaConnection$MiloSessionActivityListener.onSessionInactive(OpcUaConnection.kt:446)
at org.eclipse.milo.opcua.sdk.client.session.SessionFsmFactory.lambda$configureActiveState$42(SessionFsmFactory.java:737)
at org.eclipse.milo.opcua.sdk.client.session.SessionFsmFactory.lambda$configureActiveState$43(SessionFsmFactory.java:736)
Once a connection enters this state, it does not recover automatically. It remains in the Error state until the entire Ignition Gateway is restarted.
The same behaviour occurs on several IO Gateways, so this does not appear to be a local problem limited to one installation.
Impact on the system
The problem causes a gradual degradation of the OPC UA client subsystem.
On the affected Gateways, we observe:
- Approximately 160–290 active
opc-ua-executorthreads - Thread numbering increasing into the thousands
- Approximately 8–10 threads permanently remaining in the
BLOCKEDstate - Low Gateway CPU usage
- OPC UA connections that do not recover automatically
This does not appear to be related to CPU usage, memory usage, network bandwidth, or general Gateway performance.
Diagnostic results
1. Thread dump analysis
All blocked opc-ua-executor threads are waiting for the same lock related to lazily initialized data type structures in the Milo library:
org.eclipse.milo.opcua.stack.core.util.Lazy.getOrThrow(Lazy.java:62)
org.eclipse.milo.opcua.sdk.client.OpcUaClient.getDataTypeTree(OpcUaClient.java:918)
Other affected threads show the following stack:
org.eclipse.milo.opcua.stack.core.util.Lazy.getOrThrow(Lazy.java:62)
org.eclipse.milo.opcua.sdk.client.OpcUaClient.getDynamicEncodingContext(OpcUaClient.java:707)
org.eclipse.milo.opcua.sdk.client.subscriptions.OpcUaSubscription.notifyDataReceived(...)
We also found the following reset path:
org.eclipse.milo.opcua.stack.core.util.Lazy.reset(Lazy.java:84)
org.eclipse.milo.opcua.sdk.client.OpcUaClient.resetDataTypeTree(OpcUaClient.java:926)
org.eclipse.milo.opcua.sdk.client.OpcUaClient.lambda$new$10(OpcUaClient.java:488)
The last stack frame indicates that resetDataTypeTree() is called during session initialization.
This may mean that the data type tree is removed and rebuilt each time the OPC UA session is re-established.
2. Lock dependency analysis
We analysed the lock ownership chain in the thread dumps.
There is no cycle, so this is not a classic logical deadlock. However, each blocked thread chain ends with a thread in the WAITING state inside:
CompletableFuture.get()
The thread appears to be waiting for a network read of data type definitions from the OPC UA server.
This means that one hanging network read may hold the lock while many other OPC UA threads wait behind it.
3. Request Timeout is not enforced
All connections have the following setting:
Request Timeout: 60000 ms
We would expect the hanging request to be cancelled after 60 seconds. However, the blocked threads remain in the same state for several hours.
It appears that the network read used to build the data type tree does not respect the configured Request Timeout.
4. The same device works from another Gateway
When a device is in the Error state on one Gateway, we can connect to the same device from another Ignition Gateway without any problems.
The second Gateway successfully establishes and maintains the OPC UA session.
This indicates that the following elements are working correctly at that time:
- OPC UA server
- Network path
- Certificates
- Security configuration
The issue appears to remain within the OPC UA client state of the affected Gateway.
5. Endpoint discovery still works
On the affected Gateway, the Edit Endpoint function can still discover the device and its available endpoints correctly.
Endpoint discovery is a short operation and does not require a complete OPC UA session or the creation of the data type tree.
This is consistent with the assumption that the problem occurs during session initialization.
6. Packet capture analysis
We captured the network traffic for one connection in the Error state.
The observed sequence is:
- TCP SYN and SYN-ACK are successful.
- OPC UA HEL and ACK messages are successful.
- OpenSecureChannel and certificate exchange are successful.
- No OPC UA MSG frames are sent.
- The CreateSession request is not sent.
- After approximately 18 seconds, the server closes the TCP connection with a FIN packet.
- The Ignition client immediately tries to reconnect.
- The same sequence repeats approximately every 20 seconds.
The Secure Channel is opened correctly by the Netty event loop. However, the executor thread that should continue the process and send CreateSession does not move to the next step.
7. Gateway restart temporarily resolves the issue
A full Gateway restart clears the condition:
- The executor pool is recreated
- Thread numbering returns to low values
- Blocked threads disappear
- OPC UA connections are restored
However, the problem gradually returns as the Gateway continues to operate.
We observed the following trend in consecutive thread dumps:
The number of blocked threads increases over time.
This indicates gradual and permanent degradation rather than a temporary increase in activity after Gateway startup.
Restarting the Gateway provides only short-term relief.
8. Disabling connections does not release the threads
We tested disabling individual OPC UA connections and then disabling all OPC UA connections on the Gateway.
The frozen executor threads remained inside CompletableFuture.get().
Disabling a connection does not interrupt or cancel the hanging data type read. Only a full Gateway restart immediately clears the thread pool.
Suspected cause
We suspect that the problem is related to dynamic data type loading in the Milo OPC UA client used by Ignition 8.3.
Our current hypothesis is:
- The data type tree is initialized lazily while holding a lock.
- Building the tree requires network reads from the OPC UA server.
- The tree is reset whenever the OPC UA session is re-established.
- A short WAN interruption causes a reconnect and another data type tree rebuild.
- If the OPC UA server responds slowly or the connection is interrupted during the read, the request remains blocked without respecting the configured timeout.
- The thread continues to hold the lazy initialization lock.
- Other threads calling
getDataTypeTree(),getDynamicEncodingContext(), ornotifyDataReceived()become blocked. - Because approximately 200 OPC UA connections share the Gateway’s executor resources, the entire OPC UA client subsystem gradually loses available threads.
- This causes additional connection timeouts and reconnect attempts.
- Each reconnect may trigger another reset and rebuild of the data type tree, creating a self-sustaining loop.
This could also explain why an affected connection does not recover after the device becomes available again.
The original network read may remain blocked on an old or inactive TCP connection. The device reconnects using a new socket, but the old request is not cancelled.
Phoenix Contact devices
We believe the Phoenix Contact devices may trigger the problem, but they are not necessarily the root cause.
These devices:
- Expose custom OPC UA data structures
- May have relatively large data type trees
- Are connected over WAN links
- Re-establish their sessions more frequently
Ignition 8.1 handled the same Phoenix Contact controllers under the same network conditions without this problem.
Even if one OPC UA server responds slowly, one network read should not block the processing of approximately 200 independent OPC UA connections.
Summary
After the Gateway has been running for some time, some OPC UA server connections become unstable and enter the Error state.
A large number of opc-ua-executor threads is visible. Approximately 8–10 of these threads enter the BLOCKED state and remain there permanently, while the remaining threads have different states.
Restarting the Gateway restores the connections, but the issue returns after some time.
I am attaching logs captured while the issue is present. We can also provide:
- Full Gateway diagnostic bundles
- Thread dumps from the degraded state
- Thread dumps captured after a Gateway restart
- A packet capture of a failing OPC UA connection
- OPC UA connection configuration details
Please let me know which diagnostic materials would be most useful for further investigation.
Best regards, Michał
