Performance problems with Tag Provider

[philipp.melzer]

We are currently developing our own MES, using the Java modules.

Do to this we need to read, write and subscribe to PLC tags. The communication between the MES and PLC is based on handshake, i.e. setting a Boolean tag to true to triggers functions to be executed in the MES and vice-versa.

Our current setup looks like this

1. Connect to the device using the gateway UI. In this case it’s a Allen-Bradley Logix Driver
2. Setup an OPC UA Client using the gateway UI. The UA Server is the interal one from Ignition itself
3. Setup a Tag Provider using the gateway UI
   
4. Import the tags using the designer
   
   
5. Note that currently all tags use the same Tag Group

Now on the Java site, we use the gateway context to get the Tag Provider to do our read, write and subscribe stuff.

We are doing some testing and it technically it works, but ran into some heavy performance issues, that I feel like are either because of wrongly configured settings, or because of how the Tag Provider works internally.

For that I have some questions:

• How many tags does one Ignition support per device and in which rates?
  • We are looking at at least 10k+ tags that we need to read and write and roughly 700 tags that have a subscription
• The Tag Provider seems to update ALL its known tag on EVERY single interval, even if I set the Data Mode to “Subscribed”
  • I know that there are different modes (direct, driven, least) to change the update interval
  • In the logs I can clearly see, that all tags are pooled on each interval, even if there was no change on the value on the UA server site
  • I set the update interval to 1000ms. This causes the Tag Provider to query all 10k tag every 1000ms, even if there was no change on the tag value on the UA server site
  • This understandably kills the performance on the device site, since all tags are constantly polled, since the client doesn’t use the publishing system that OPC UA generally provides
  • Same thing when I try to read a tag. Why does it wait for the next poll interval for the value to update in the Tag Provider, instead of just making a read request to the UA server?
  • 

Either I misunderstand the concept of the tag provider, but I feel like the idea is, that the tag provider tries to update all its value on each interval (depending on the tag group), and does not listen to publishing changes from the UA server. My module than uses the tag provider for these operations, but the tag provider kind of acts as it’s own server, instead of just relaying the commands to the actual UA server.

I even tested this behaviour like this:

• Connect an UA client (UaExpert) to the UA Server
• Connect an UA client (UaExpert) to the tag provider
• If I browse the tags, I can see that the UA Server knows all the device tags and updates their values on value change accordingly without any performance hits
• If I do the same on the tag provider, it’s values are updated only on it’s configured interval and not on value change, causing the performance to drop the more tags that are configured on the provider

Splitting in up to more providers doesn’t help, since everything still goes through the same device.
Using different tag groups and even dynamically adjusting the polling interval using driven would not really help, since we would have ~700 handshakes with it's own data block, which would required the same amount if tag groups to keep in sync.

Am I misunderstanding the concept, or is the Tag Provider really polling everything every time to keep it's values interally up-to-date? This seems like a mayor performance issue on large amount of tags, instead of just updating it on value changes on the UA server - or the Data Mode setting in the tag group does not work as expected.

My only solution is to not use the tag provider from Ignition and rather connect my own OPC client and handle the request as would be expected in the UA standard.

[pturmel]

A 100ms leased rate for Logix tags is usually unreasonable, and leased/unleased changeover is pathological in anything prior to Ignition v8.3.

Subscribed tags only have changes reported through OPC, but the driver itself has to poll to notice changes in the PLC.

[Plantformance]

I think you narrowed it there

This understandably kills the performance on the device site, since all tags are constantly polled, since the client doesn’t use the publishing system that OPC UA generally provides

If a native device just doesn't support pushing changes when needed, you're pretty much stuck with polling. Slipping an OPC UA server in the middle speeds up the link between OPC UA and Ignition, but unfortunately it doesn't remove the need to poll the device itself—it just shifts where that poll originates.

You'll probably need to do some tuning across the board here. Splitting tags into different scan classes or execution groups (slow vs. fast), switching non-critical data to on-demand reads, or using event-driven polls should help lighten the load quite a bit.

[philipp.melzer]

Thanks for the replies so far!

@pturmel

Subscribed tags only have changes reported through OPC, but the driver itself has to poll to notice changes in the PLC.

From what I am seeing the Device tags are updated correctly based on the supported sampling (=polling changed from the device into the UA Server) and published to the listing clients (TagProviders, other clients, etc.) just fine. If I connect my own client to the device tag directly using the Ignition OPC UA Server, and therefor skip the TagProvider, the values on the UA tags are updated as expected.

Meaning that the configured polling interval is just for the TagProvider, on how and when values are updates there.

After some further digging, could you please confirm if my understanding of the TagProvider is correct:

1. All device tags are known to the OPC UA server under the device folder
2. All device tags are updated using the sampling interval that the UA server and device (in this case Allen Bradley) allows
3. TagProvider tags are a mirror of the device UA tag, and are updated using the configured polling interval
4. The Java module talks with the TagProvider tags, and therefor has the polling interval as a delay for read, write and subscribe

I don't really see the reason as to why the TagProvider needs to exist in the first place, since it feels like an additional layer of complexity and performance in between with it's own polling (=mirroring device tags) mechanics, instead of relying on the mechanisms of UA.

If my assumptions are correct (and please correct me if I am wrong), than I see 3 possible solution, depending on if they are possible:

1. Configure the TagProvider / TagGroups somehow that they only behave as pass-through on the tags, and are updated not via internal polling, but rather in value changes of the device tag, which in turn is updated via the sampling mechnism
2. Use the internal Tag Manager from the Gateway Context to access the device tags directly, and skip the whole TagProvider mechanism -> is this possible?
3. Write and connect my own OPC Client to the device tags directly

[philipp.melzer]

To add some more context to how I am testing these update intervals between device and TagProvider tags:

I subscribted on the same tags once on the device and once in the TagProvider on the same UA Server using the Ignition quick-client:

With the following testing environment:

• Set the polling rate in the quick-client to 1000ms
• Set the polling interval of the TagProvider TagGroup to 5000ms
• -> The device tag correctly updates every 1000ms
• -> The TagProvider tag somehow updates every 2000ms -> Why?
• If I remove the device tag from the subscrubted tags, the TagProvider tag correcly updates every 5000ms

Meaning that the device tag correctly updates it's value either on the polled read or on the configured sampling interval of the UA Server, outside of the TagProvider system.
The TagProvider copies(?) the values to its mirrored tag, but somehow still is effected by the device tags publishing interval as well.

[pturmel]

The "Tag Providers" tree within the Ignition OPC server is meant to expose all kinds of Ignition tags to OPC clients. While it may re-expose items from the "Devices" tree, that is simply an artifact of the "expose tags" feature of the OPC server. Ignition OPC tags can point at items in other OPC servers, not just the internal loopback server.

FWIW, I recommend NOT using the expose tags feature. For anything that is repeating an OPC tag, its timing will be the slowest of any constraint in the entire chain. I recommend exposing Ignition tags to external systems by deliberately publishing via MQTT or another Cloud connector, or if necessary, using my Modbus Server driver.

[philipp.melzer]

@pturmel

WIW, I recommend NOT using the expose tags feature. For anything that is repeating an OPC tag, its timing will be the slowest of any constraint in the entire chain.

I agree that going via the exposed tags of the TagProvider is not a goods idea. We would have both the mentioned performance problems that the TagProvider has, as well as anything that comes after that.

But what we are now trying to do is use the device tags directly, and skip the whole TagProvider. The device tags seem to update via the normal behaviour of OPC UA and are therefor via its subscription mechanism very performant (i.e. they do not poll every tag each cycle like the TagProvider does).

When having a look at the the performance metrics of the device driver (Connections -> Devices -> Connections -> View Details on the Device) you can see the monitored item count. I assume this is the actual number of monitored items of the OPC UA server in the namespace that the device is assigned to. And as soon as I add a tag to the TagManager and set the TagGroup to "subscribed", it adds this tag to the monitored items, even though no one is currently using said tag. If would make mor sense for the TagProvider to only add the tag to the monitored items if:

1. Someone is subscribed to the tag via code
2. The tag is in use via an HMI screen

...but because it is always monitored, it uses unneeded system resources to keep values up to date in the TagProvider, or even worse being polled by the TagProvider even if it is not set to be polled (as confirmed by the device connection logs).

Because of this we are trying to now skip the TagProvider on the Java site. To do this, we are now using the the OpcManager from the GatewayContext:

context.getOpcManager().getConnectionState(OPC_CONNECTION_NAME);

And adding our own monitored items via subscriptions:

subscriptionConnection = context
        .getOpcManager()
        .createSubscription(OPC_CONNECTION_NAME, subscriptionNameConnection, properties());

Which increases the monitored item count by 1 and works with the publishing mechanism of UA.

Reads and write on the UA Service work like normal as well, without having to be subscribed to the tag (and it being constantly polled like the TagProvider) by having the UA Server handle the request

private QualityCode write(String nodeId, Object value) {
    return context.getOpcManager()
            .write(List.of(new BasicWriteRequest(new ServerNodeId(OPC_CONNECTION_NAME, nodeId), value)))
            .get(0);
}

private QualifiedValue read(String nodeId) {
      return context.getOpcManager()
              .read(List.of(new ServerNodeId(OPC_CONNECTION_NAME, nodeId)))
              .get(0);
}

We will try and reimplement our logic using that approach to see if performance improves. Obviously we will never get better than the underlying Logics Driver that connects to the PLC.

Any thoughts on this or the potential drawbacks of doing it like this? i.e. is it wise to skip the TagProvider to improve performance?
Also, do you know if the subscribe, unsubscribe, read and write operation of the OpcConnection are threadsave? We expect to have multiple mechanism run in parallel on different threads, and all of them would go through the same OpcManager connection for their read and write operations.

[pturmel]

@philipp.melzer

If would make mor sense for the TagProvider to only add the tag to the monitored items if:

1. Someone is subscribed to the tag via code

2. The tag is in use via an HMI screen

This is what leased tags do. Set the leased rate to the desired rate and the unleased rate to zero (disabled).

Only do this with non-historized tags. The subscription must be providing current data for history storage to work.

Be aware that prior to v8.3, there is a constraint on subscription changes that makes leased tags behave poorly system-wide if ANY tag in the same OPC connection is missing or misconfigured.

@philipp.melzer

Also, do you know if the subscribe, unsubscribe, read and write operation of the OpcConnection are threadsave?

I think so. If you encounter relevant bad behavior, open a ticket.