Can someone tell me a story here? Network problems, data latency, hope the image uploads!

Okay, if you can see the above image, can you tell me what this means?
The information in the manual is insufficient to understand what is happening.
I am trying to read a couple of UDTs. Within the UDT there is an array with 500 entries - they are logs that I am trying to capture and process to store in a database. The total UDT I am trying to read is only about 57Kb of data! I have a couple of instances though. So total not more than about 120Kb of data at this scan rate. I am having all sorts of problems with latency - The handshake/new data signal arrives well before the data fills in and I end up reading old data in with the new. I’ve tried adding in delays, and they work up to a point, but I hate to count on timers for data update. Is there any way to confirm that an entire topic has been fully updated? A timestamp perhaps that can be looked at when the topic completes?
And back to the screenshot - why is the mean response time down to the few ms, while the queue is in the 3-4S range? What does this mean, why?

The quantity of tags you are trying to read at a 1500ms pace is boiled down by the driver’s optimizer into 144 actual requests that need to go over the wire. The average response time for those requests is 63.22 ms. When handled one at a time, that would be ~ 9.1 seconds. Your actual sampling interval is half that, so your driver must have concurrency = 2, and it actually is helping. (In many cases, the bottleneck is message processing on the PLC end, and concurrency can’t fix that.)

So you want 1.5 seconds, but you are getting 4.5 seconds, so you are overloaded by ~200%.

You have more tags subscribed than your device/connection can support. Modern PLCs tend to have response times in the single digit milliseconds when not bottlenecked, so I’m going to go out on a limb and guess that your Ignition server is in the cloud. Query/response PLC protocols operating across a WAN simply suck. As you see.

Try bumping your concurrency setting up. But you are likely to max out the available buffers in most PLCs if you go very far with that.

2 Likes

Thank you so much, that is a great explanation.
A little more info on hardware: we are using a 1756-L84 via a dedicated EN3TR so the PLC side is blazing and shouldn’t be the bottleneck. We already figured it was out network side - we are on a test setup and from Ignition on a VM on a server to the PLC there are several hops, skips and jumps that is almost certainly the issue. We are hoping once on real hardware a lot of this goes away.
The bigger issue is the realization that there is no guarantee that just because some data is updated, all the data is updated. It seem like time delays for data to update is just not a bulletproof way to know that all the data has been received, so we are hoping there is some way to verify. So if we get our hardware installed at the customer, and its all humming along nicely, and we introduce say a 2 second delay to read the data and today its working fine - a year from now something changes on the network, or someone goes and adds bunch more tags - and now are test records are messing up because the 2 second delay isn’t enough any more.
Thanks again for your help on this!

To be sure all data is recieved you can create a handshake… a bunch of extra work tho as all the plcs will have to be updated too. idk if there is another way, i dont rly work with plcs xd

Since this is a controllogix… then be aware that reading the data directly from an AOI has very poor overall performance. See here: Cannot read some tags inside Rockwell AOI - #19 by nminchin

Thanks for that input, that is really good to know, as a general thing, but does not apply in this case. This is just a UDT housing an array of other UDTs, making up a “RecordsToIgnition” tag.

If you only need the data when some other data changes (triggered in some way), consider not subscribing to all of these tags. Instead place all of your triggers in another PLC tag and subcribe to those. Use a tag change event with system.opc.readValues() to pull the rest of the data on-demand. This has the side-benefit of ensuring that the data isn’t racing with the trigger.

1 Like

Well Phil just beat me to it, but I was going to add a little off topic: if you care about the atomicity of the data you’re acquiring then subscriptions are the wrong approach. You need to be issuing explicit reads based on some trigger or using a transaction group to do that for you.

First off, I increased the concurrency to 4 and things have gotten way better on the loading. Thanks for that suggestion Phil!
I am going to dig in to this system.opc.readValues() business as it sounds like the better way to do it. The reality is that we are actually going to expect some new records to read on just about every scan, but if it makes the data retrieval more deterministic then it will be the way to go. Thanks guys for all your input.