I have a client attempting to use Ignition in a lab environment for R&D purposes.
They have multiple test control system racks, each having upwards of 20 standalone distributed devices.
They’ve been experiencing a severe issue where OPC-UA tags stop updating completely in both Ignition designers and clients.
In some cases, the quality will go bad and read “config error” under diagnostics. Other times the quality stays good but the tags simply don’t update. The scan class claims to have executed but the values are not updated.
If I use the OPC quick client or another OPC client to check the values, they do report correctly, so I know it’s not a connection issue.
Additional Details:
No OPC-UA tags are updating, all values are held at ‘null’
This is true even for new tags whose OPC-UA connection in the Gateway is reading ‘good’
The scanclasses are executing, as expression tags update fine
The Gateway is not under heavy load. Peaks at about 20% CPU and 25% memory usage before GC (21 GB total)
They have configured 262 OPC-UA connections, of which only 16 are active, the rest are faulted (test devices currently disconnected)
They default realtime tag provider has 4913 Tags registered
The obvious solution would be to manage these connections in a third party OPC-UA server, and have Ignition only connect to that one server. Any advice?
Am I right in understanding that these are all OPC UA connections directly to the devices and that Ignition’s built-in OPC UA server is not really involved here?
Ok, well definitely start with upgrading to 7.9.10.
From there, you can go to the logging area in the gateway and configure the logging level for the logger you find searching “UaSubscriptionChangeListener” to TRACE. And “OpcUaConnection” to DEBUG. Leave these on and get a log export next time there’s an issue.
You might also get Wireshark installed in case we need to use that for troubleshooting as well.
Yeah, those are important. The “Failed to create item” messages are likely why you’re seeing Bad_ConfigError quality on your tags. What’s the StatusCode returned by the server?
No, unless a scan class is in Read Mode it’s almost entirely irrelevant when dealing with OPC tags. Only the scan class rate is used to determine the sampling interval of monitored items belonging to the corresponding subscription.
Whatever the StatusCode is for those items items that were failing to be created is probably important and your issue will probably come back again…
The servers connections that I removed were physically connected in the past.
Though I think you're right, the issue could be caused by some of the still-connected devices.
There are additional warnings of the form:
"Failed to set publishingInterval=5000 on subscription ... scanclass details ...
UaServiceFaultException: status=BAD_ServiceUnsupported, message=The server does not support the requested service."
That only happened after attempting to change the scanclass again.
Though it doesn't report which UA servers are causing the issue.
Oh boy. Bad_ServiceUnsupported in response to setting the publishing interval means the server isn’t implementing the Modify Subscription service, which is not optional if you support subscriptions.
Unfortunately I was never able to pinpoint the specific fault in the custom server software. The team responsible for the the server implementations were already in the process of switching from open62541 to a commercial MatrikonOPC stack when I inquired about implemented services. With the new server software in place, we haven’t experienced any further connectivity issues with those devices. Safe to say your evaluation was correct.
However, I’ve now identified a separate OPC issue with tags exposed in NXT Control (now Schneider Electric) soft controllers via their proprietary OPC UA server… The logs are relaying similar errors as before, with UaSubscriptionChangeListener reporting “The server does not support the requested service” I’m not seeing an error code this time, so unsure what that unsupported service is. In this instance the scanclass is still functioning as expected. Only tags from the offending server are faulting out. Tag Diagnostics reports quality “Unknown”. Using a third party OPC-UA client (Unified Automation’s UaExpert) the tags are still reading correcting. Any advice?
On the log level config page in the gateway find the logger for “UaSubscriptionChangeListener” and turn it to TRACE, then edit/save the OPC UA connection to the NXT server and collect the logs.
Disabling the OPC UA server connection does stop the warnings
Strangely when re-enabling the connection all the tag values report correctly, but the Bad_ServiceUnsupported messages resume and the values/timestamps no longer update after the initial read.
The sequence of UaSubscriptionChangeListener logs:
[D] Synchronizing subscriptions…
[D] Creating/deleting items on subscription ‘Scanclass ‘Default[default]’’
[D] nodesToAdd.size()=0
[D] nodesToRemove.size()=0
[D] Setting MonitoringMode to Reporting for 16 of 16 items on subscription ‘Scanclass ‘Default[default]’’
[W] Failed to set MonitoringMode.Reporting on 16 items on subscription ‘Scanclass ‘Default[default]’’: UaServiceFaultException: status=Bad_ServiceUnsupported, message=The server does not support the requested service.
