OPC subscription failing

Ignition
,
1

Hi,

We use Ignition together with a couple of KepServers. We connect all machines to these different KepServers (around 7), and then connect to the KepServers with OPC UA. The reason for this is that we already had the KepServers setup before we installed Ignition. This has worked well for us without any real hiccups. Until this Friday.

Someone noticed that one KepServer was low on RAM and asked the IT-department if they could increase it as they run on VMs. So they did and after that the subscription started to fail. The servers says "Connected" in the status page, we can browse the OPC-server (though a bit slow), but we don't get any tag-values.

I tried to reboot both the KepServer VM and Ignition. Tried to delete the connection and add it again and anything else I could come up with. The server we have an issue with doesn't really have any critical connections so I figured I would wait it out over the weekend as the IT-department was about to have a maintenance and everything would be offline for a couple of hours. This didn't help either. Today I tried to rename the connection and instantly it started working. Same connection, same settings, but I changed the name. I only removed a hyphen in the name. "KEPServerEX/UA@ABB-C12" -> "KEPServerEX/UA@ABBC12". But of course this isn't something that we want to do on all our tags.

Any clue to what happens here?

Here are a screen from when the error first occurred:

image
image1210×434 30.8 KB

The "Failed to create"-errors reoccurs every minute if I keep the connection enabled.

2

I ended up removing the server, disabling all tags bound to this server with "Tag Report Tool" (~21000), added the server again and enabled 2-3000 at a time. I was expecting it to fail on me again at a certain point, but it never did. All tags are enabled now, and everything seems to work. Only thing left is to try to restart both Ignition and KepServer to make sure it doesn't happen again.

Even though this ended up being the solution, it would be nice to know why this happened if anyone has any clues.

3

It seems like Kepware was overwhelmed and unable to respond to the CreateMonitoredItemsRequests.

If/when it happens again you can turn that OpcUaSubscriptionSynchronizer logger to DEBUG or TRACE for more information. There's also an advanced setting on the OPC UA connection called Max Per Operation that may need to be lowered by half and then half again and again if it still doesn't solve the problem.