Hi everyone,
We have a project where we need to read 20,880 tag values at once with a tag change script and we're trying the different methods and wanted to get some thoughts from the experts. We need to read these as fast as possible (preferably in less than 1 second). We've used examples and read theories from @PGriffith, @pturmel, and @lrose
This is a sample we've built and shortened for brevity. So we will have a parent folder called 'array'. Inside of array is 360 sub folders (point1-point360). Inside of each folder is 58 float tags (temp1-temp58). This gives a total of 20,0880 tags that need to be read at once.
Method1 - using jsonValues metadata
from java.lang import System
path = "[default]dev/af/array.jsonValues"
start_ts = System.nanoTime()
v = system.tag.readBlocking(path)[0].value
readBlocking_ts = System.nanoTime() - start_ts
print readBlocking_ts
Why less than 1 second? Honestly reading 20K tags in 5 seconds is pretty good, IMHO.
Depending on what exactly you're needing the speed for, there may be different ways to handle this.
If I were forced to do this with just the Ignition Core package, I would probably look at multi-threading the read, but that's going to get complex quick, and isn't necessarily guaranteed to give you the result you're looking for.
The current requirement is to go no slower than 4 seconds because those tags will be overwritten with new values.
We actually got it down to 1.3 seconds! I'll post the code shortly. This is turning more into a challenge at this point.
We considered a multithread approach by breaking down the array into chunks and using system.tag.readAsync() and then re-assembling but it seems that most people seem to recommend minimizing the number of calls to system.tag.read____() functions.
Don't use jsonValues.
Put this in a project library script:
from java.lang import System
arrayPaths = ['[default]dev/af/array/point%d/temp%d' % (p+1, t+1) for p in range(360) for t in range(58)]
def getArray():
ts0 = System.nanoTime()
qvList = system.tag.readBlocking(arrayPaths)
values = [qv.value for qv in qvList]
ts1 = System.nanoTime()
return "%.1fms" % (0.000001 * (ts1-ts0)), values
Note that this will be much slower when called from the designer script console than from an actual event.
Part of the key to speed is to pre-compute the list of tag paths. It becomes a pseudo-constant in that script library.
Keep in mind that reading tags like this does not cause any PLC traffic--it just gathers the last values delivered from OPC. If you really need a proper snapshot of the PLC values after some trigger, you need to use system.opc.readValues(), and that speed will be very dependent on your OPC driver.
That code is just getting the paths, it is not reading the values.
1.3 seconds to read the paths, is slow. Use's Phil's approach for pre-computing the paths.
Yes, notice the minimizing key word there. Generally speaking you can read a single group of tags with a single call, however, there are some outlier cases where it makes since to break that idiom.
Personally, I would use a buffer in the device and read at a reasonable pace from the buffer (with system.opc.readValues() ), because at this type of speed, you don't want to be depending on network latencies to guarantee that you don't miss values.
Sorry, here is the code to actually get the values with system.tag.browse() and it is running at 1400ms
from java.lang import System
pathSD = '[default]dev/af/array'
start_ts = System.nanoTime()
tagresults = system.tag.browse(path = pathSD, filter = {'tagType':'AtomicTag', 'recursive':1})
for result in tagresults.getResults():
pName = result["name"]
value = result["value"].getValue()
tagbrowse_ts = System.nanoTime() - start_ts
print tagbrowse_ts
I also ran @pturmel suggestion and it was 2200ms. And yes, all tests are in the Designer script console so they will be faster on actual events.
But I'm quite surprised that system.tag.browse() appears to be performing slightly better than precomputing the paths with system.tag.readBlocking(). In fact, system.tag.browse() WITHOUT the filter {'tagType':'AtomicTag', 'recursive':1} is lightning fast! But then we lose time going into the loop to get the values...and the total time ends up being almost exactly the same.
There is a benefit of just being able to specify a top-level parent folder and using the recursive filter in system.tag.browse() because of less code.
Regarding system.opc.readValues() and buffering. It is a ControlLogix PLC and is currently buffering that array for us and holds until the next buffer refresh at approximately 3.7 seconds. But we are curious to see how system.opc.readValues() performs since we never used it before. We can test Ignition AB drivers and Kepware. And I've been keeping my eye on @pturmel's OPC module
If you use system.tag.readBlocking() as soon as your trigger tag says the fresh buffer is ready, you will likely gather stale values from the previous cycle. Because OPC subscriptions are unordered.
I would use only that, and not have tags for the items at all. Eliminate all opportunity for stale values in your downstream operation.
Just tested a Gateway Tag Change with system.tag.readBlocking() and pre-building a big list. You were right, Phil. It's taking about 350ms to read all 21K tags. That is super fast for us.
Getting ready to play with system.opc.readValues(). Do you know if we can read the top level parent folder in the PLC or will we need to pre-build a list like we did with system.tag.readBlocking()? We'll begin with Kepware and AB drivers if that matters.
Prebuild your list. Note that it is OPC Item Paths, not Ignition tag paths. Delete all the tags that subscribe to the same items for best results. (Will also speed up your other tags, as you won't be polling 20,000 items all the time.)
It will not be as fast as readBlocking for tags. (Well, unlikely to be as fast.) But it is way less work for both Ignition and for your PLC. And ensures your values aren't stale.
20,000 tags at 4 bytes each (REAL) in a dense array will require ~ 21 round trips with 4k driver buffers. With concurrency 2, and a typical array read response time of ~5ms, you might be fastest with this method. Depends on how busy that PLC is, and how new. (L8x should be scorching fast for such array reads.)
And, with a simple dense array workload, I'll even bet IA's driver outperforms Kepware.
Just wanted to update everyone. We switched to using one system.opc.readValues() call and deleted all of Ignition OPC tags. The execution time for only the opc call for 20,880 tags is about 1.4 seconds compared to 350ms milliseconds for system.tag.readBlocking(). This is using IA's driver on Ignition Edge 8.1.21 and an L8 processor. So it is quite a bit slower, but according to what Phil said, it should be a better method because we won't grab stale values right?
I tried changing Max Concurrent Requests to a number higher than 2 but doesn't seem to affect the speed of IA's driver. Are there any other ways to tune this to make it faster?
Do you have a continuous task or only periodic?
Are your tags all in an array? Or are they scattered about in the PLC? (Sorry if I missed that in the discussion above)
You can also change CIP Connection Size in the IA driver. If everything supports it, try going to 4000.
If you haven't already, change the connection size from the default of 500 bytes to 4000 bytes. I also noticed that max concurrent requests seems to be in the sweet spot at 2 as decreasing or increasing yielded lower performance in all of my testing.
I bumped up the connection size to 4000 and the task now completes in 950ms. Thanks for the suggestion. I have found out it is an L83 processor. Still trying to track down someone who can tell me SOTS and type of task. All tags are in one array. 56 floats and 2 LINTS (timestamps)..
Doesn't seem relavent to you, but for reference, note that you must call the system.opc.* functions from the gateway that has the OPC-UA server installed and obviously the gateway that has the connection to the device you want to read from. So this means using system.util.sendRequest* functions to execute the opc functions on the remote server in cases where the OPC-UA module is on another gateway
