OPC Tags freezing

Ignition
1

Hi all.
I am working on a large system and I noticed some OPC tags freezing. The tags in question are integers that do not change a lot, I am using them to inform summary of status and mode of operation, like:
Normal mode = 0
Locked out = 1
Maintenance mode = 3

So, it is not exactly a tag that changes that much.

We already tried multiple troubleshooting procedures here, the observed behaviour when the tags freezes are:
a. The tag keeps its quality as “Good” and the time stamp from the last change that was captured by the system.
b. Using the OPC quick client we see the variable with the right value.
c. If I change anything in the tag, or even copy and paste it, it starts to respond and works well for a few days.

Frozen tag

image
image1110×508 49.4 KB

OPC quick client

image
image1472×143 7.27 KB

Tag configuration:

image
image1021×967 102 KB

Thank you.

2

Hi,
We have a similar issue in our system.
Have you found any solution or workaround?
/Erik

3

I have seen this issue. Typically restarting the communication’s driver or gateway fixes it.

I have seen this occur more often on a system that is absolutely hammered (90+ CPU utilization) because they were subbing and data logging a very large number of tags. If your system is running that hard you should consider doing less data logging or branching out some of that activity onto additional gateways and sharing the historical data source to your visualization gateway. Basically, share the load across multiple gateways if you need to collect a lot of data points.

I have seen this occur on gateways that aren’t absolutely hammered but it seems to happen much more frequently when Ignition is not being given enough resources to handle what it is being asked to do.

4

Hi.
I haven’t found any really good workaround. However, I did a few things that helped.

  1. I reduced the amount of active subscriptions. The solution is quite dirty, any variable that is not data logged and not part of the main visualization stream is disabled. Then I enable/disable this variables as the user navigates down to specific screens. It reduced drastically the workload on the OPC side.

  2. I changed the templates to pass Tag Paths instead of UDTs, it reduced the amount of “leased” variables drastically.

  3. I updated the firmware on the OPC Server side (Ignition is reading from multiple Pilz OPC Servers). This procedure solved the problem of freezing variables, but now the OPC crashes all together. At least this behaviour is better than the previous one because we can visualize the problem and take action when the OPC Server crashes.

  4. I added an extra OPC server to share the workload.

All in all. I am not happy with the performance of the communication so far. But it has shown some improvements. I am still in doubt about whether the problem lies on the OPC server or on the Ignition side. I have a few lines of investigation open at the moment.

Let me know if you find a solution to your problem too.

Thanks!

5

Hi.

I can tell you that the project here is large, if not huge. But the CPU utilization isn’t bad, same for Network and disk:

image

I am exploring some possibilities of sharing the workload and changing the network topology. But it is quite blurry to be honest.

Splitting the application in two seems to be a good test.

6

Yeah, that’s not bad. The system I’ve seen have frequent problems with this is literally over 90% all of the time.

7

Hello rcarrilhof thanks for sharing this problem. Have you managed to solve it?

8

Hi Pablo, after all the workarounds I listed in this post the system stopped freezing variables. I also reworked the network topology. Later I found a managed switch reporting that one branch of the network had a higher workload compared to the others, nothing major, but we made some changes that reduced it.

We also upgraded the server with more memory and a better CPU. So, the answer I guess was to reduce the communication workload and improve the infrastructure.

Good luck there.

1 Like