OPC-UA Server Connections and Tag Issues

I have a client attempting to use Ignition in a lab environment for R&D purposes.
They have multiple test control system racks, each having upwards of 20 standalone distributed devices.

They’ve been experiencing a severe issue where OPC-UA tags stop updating completely in both Ignition designers and clients.

In some cases, the quality will go bad and read “config error” under diagnostics. Other times the quality stays good but the tags simply don’t update. The scan class claims to have executed but the values are not updated.

If I use the OPC quick client or another OPC client to check the values, they do report correctly, so I know it’s not a connection issue.

Additional Details:

  • No OPC-UA tags are updating, all values are held at ‘null’
  • This is true even for new tags whose OPC-UA connection in the Gateway is reading ‘good’
  • The scanclasses are executing, as expression tags update fine
  • The Gateway is not under heavy load. Peaks at about 20% CPU and 25% memory usage before GC (21 GB total)
  • They have configured 262 OPC-UA connections, of which only 16 are active, the rest are faulted (test devices currently disconnected)
  • They default realtime tag provider has 4913 Tags registered

The obvious solution would be to manage these connections in a third party OPC-UA server, and have Ignition only connect to that one server. Any advice?

What version of Ignition is this?

Am I right in understanding that these are all OPC UA connections directly to the devices and that Ignition’s built-in OPC UA server is not really involved here?

7.9.7. Will be pushing them to update soon.

That’s correct, they’re all configured as their own OPC UA servers, no usage of the Ignition server.

Ok, well definitely start with upgrading to 7.9.10.

From there, you can go to the logging area in the gateway and configure the logging level for the logger you find searching “UaSubscriptionChangeListener” to TRACE. And “OpcUaConnection” to DEBUG. Leave these on and get a log export next time there’s an issue.

You might also get Wireshark installed in case we need to use that for troubleshooting as well.

1 Like

Changing UaSubscriptionChangeListener to TRACE released a deluge of “Unable to get client to synchronize, rescheduling…” messages.

Also interesting sequences of:

  • synchronizing subscriptions
  • creating/deleting items on subscription …
  • nodesToAdd.size() = N
  • nodesToRemove.size() = 0
  • Creating N items on scanclass
  • Failed to create item '… details … ’ x N

Yeah, those are important. The “Failed to create item” messages are likely why you’re seeing Bad_ConfigError quality on your tags. What’s the StatusCode returned by the server?

The tag quality is no longer bad since I’ve deleted over 60 faulted (unused) UA servers and increased the scanclass rate from 1 to 5 seconds.

Seems like the scanclasses just get overloaded with too many inactive UA connections?

No, unless a scan class is in Read Mode it’s almost entirely irrelevant when dealing with OPC tags. Only the scan class rate is used to determine the sampling interval of monitored items belonging to the corresponding subscription.

Whatever the StatusCode is for those items items that were failing to be created is probably important and your issue will probably come back again…

Ah I see…

StatusCode for the ‘failToCreate’ messages is:
{name=Bad_NodeIdUnknown, value=0x80340000, quality=bad}

Sidenote: altering the scanclass triggers the tag issue again, causing all qualities to go bad indefinitely.

Assuming those NodeIds are/were valid, this looks more like an issue with the Servers rather than the Client.

1 Like

The servers connections that I removed were physically connected in the past.

Though I think you’re right, the issue could be caused by some of the still-connected devices.

There are additional warnings of the form:

“Failed to set publishingInterval=5000 on subscription … scanclass details …
UaServiceFaultException: status=BAD_ServiceUnsupported, message=The server does not support the requested service.”

That only happened after attempting to change the scanclass again.

Though it doesn’t report which UA servers are causing the issue.

Oh boy. Bad_ServiceUnsupported in response to setting the publishing interval means the server isn’t implementing the Modify Subscription service, which is not optional if you support subscriptions.

What are these and who makes them?

1 Like

There is a variety, so it’s hard to nail down.

DYNSIM 3 jumps out in the list as the server type is legacy: OPC_COM_ServerType

Some devices are also running open62541, though I wouldn’t suspect that.

None of this logging or troubleshooting applies to COM connections.

open62541 is generally solid but it’s just an SDK so you can still screw things up with it.

1 Like

Great points.

Thanks for your help @Kevin.Herron, I’ll relay your wisdom to their development team and update this post with what we find. Cheers!

Good luck! Let me know what you find.

Unfortunately I was never able to pinpoint the specific fault in the custom server software. The team responsible for the the server implementations were already in the process of switching from open62541 to a commercial MatrikonOPC stack when I inquired about implemented services. With the new server software in place, we haven’t experienced any further connectivity issues with those devices. Safe to say your evaluation was correct.

However, I’ve now identified a separate OPC issue with tags exposed in NXT Control (now Schneider Electric) soft controllers via their proprietary OPC UA server… The logs are relaying similar errors as before, with UaSubscriptionChangeListener reporting “The server does not support the requested service” I’m not seeing an error code this time, so unsure what that unsupported service is. In this instance the scanclass is still functioning as expected. Only tags from the offending server are faulting out. Tag Diagnostics reports quality “Unknown”. Using a third party OPC-UA client (Unified Automation’s UaExpert) the tags are still reading correcting. Any advice?

On the log level config page in the gateway find the logger for “UaSubscriptionChangeListener” and turn it to TRACE, then edit/save the OPC UA connection to the NXT server and collect the logs.

Disabling the OPC UA server connection does stop the warnings
Strangely when re-enabling the connection all the tag values report correctly, but the Bad_ServiceUnsupported messages resume and the values/timestamps no longer update after the initial read.

The sequence of UaSubscriptionChangeListener logs:

  • [D] Synchronizing subscriptions…
  • [D] Creating/deleting items on subscription ‘Scanclass ‘Default[default]’’
  • [D] nodesToAdd.size()=0
  • [D] nodesToRemove.size()=0
  • [D] Setting MonitoringMode to Reporting for 16 of 16 items on subscription ‘Scanclass ‘Default[default]’’
  • [W] Failed to set MonitoringMode.Reporting on 16 items on subscription ‘Scanclass ‘Default[default]’’: UaServiceFaultException: status=Bad_ServiceUnsupported, message=The server does not support the requested service.

No code reported in the warning stack trace.

That’s everything I needed.

Again, the server you’re connected to is not implementing a mandatory service (SetMonitoringMode this time).