I have a gateway that is monitoring ~300 devices by reading UDP streams of JSON objects.
Currently I have a startup script that opens up a port and reads the stream of data flowing in.
The script will invoke another script to parse the object and store the tags into our tag provider.
This has been working fine for a while, but we are now experience issues with performance of other areas of ignition; the most glaring is that it takes the designer a solid 2-5 minutes to load a tag provider.
When we turn off the long running script, the performance issues go away entirely. CPU and RAM usage stay under 30%, so I'm inclined to believe its a performance issue related to the way Ignition handles threading (pools, queues, etc.)
I plan to change device communication to be through Ignitions OPCUA using UDP devices. I can wild card a tag change script in gateway events to handle the parsing. From the research I've done and the experience we stumbled through; this seems like the best method.
One problem immediately remains; what to do with the tag change scripts for specific tags in the devices?
Example:
I have a tag ('device_type_a/device_a_number_123/tag_i_want_a_scipt_to_execute_against')
I can't place a wildcard in the middle of the path for tag change event declarations.
i.e.: 'device_type_a/*/tag_i_want_a_scipt_to_execute_against'
Organizationally is my only path forward to have another folder?:
'tag_change_scripts/device_type_a/tag_i_want_a_scipt_to_execute_against/device_a_number123'
where device_a_number123 is a reference tag looking at the original tag structure's tag_i_want_a_scipt_to_execute_against
Ignition Devs: Can you make wildcards fit in the middle of our paths? I'll send you a case of beer...
Is there a way to import tag change scripts via from external files? You'd be able to generate them all in a batch script that reads the device tags and generates their tag change scripts, if this is the case. I'm doing this for transaction groups -- it has a legacy system,groups.readFromFile method that lets me ingest XML files to dictate the configuration of the transaction group.
I don't believe so.
the resource lives at: C:\Program Files\Inductive Automation\Ignition\data\projects\project_name\ignition\event-scripts,
but its a bin file.
I could do it manually with a script that generates the paths for me to copy paste, but I'd really rather have it automated.
I'd be more interested in fully understanding the how/why there is a performance issue caused by your current setup. You haven't described it in enough to detail for me to really guess yet. Some thread dumps and details about what the "long running" scripts are doing might help.
All I've really gleaned is that at a high level you are parsing incoming UDP packets containing JSON data and then setting values on a bunch of memory tags you've created. You have also defined a bunch of tag change scripts on those memory tags, though it's not clear if those scripts are related to the performance issues.
Can I email you the thread dumps I just took?
The specific issue I'm seeing:
when trying to use the tag browser in the designer there is a significant delay when I open the tag editor by clicking a tag. A popup appears "listing security levels".
More on the "long running script"
I've defined a script that invokes Java to open a UDP port. I had to use java here because there's a bug with Jython (Netty specifically) that doesn't allow you to increase the buffer size. The objects are large so I have to unfortunately.
It then starts a while true loop that reads the buffer, and invokes the parser (asynchronosly), then sleeps.
At startup I invoke the script (asynchronosly). I also save the socket object to globals and close it at shutdown.
I've attached the script def and startup call code.
udp_reader_snippet.txt (1.9 KB)
My coworker forwarded me this relevant ticket number:
#110102
I also noticed he's got everything in expressions now, no scripts. Which makes one of my questions more of a thought experiment now.
You can email or DM the thread dumps (zip them first).
I've seen at least one other similar kind of setup come through support and their main issue was that in addition to using the JSON to set tag values, they were also using it to dynamically create the tags if they didn't exist yet. Even once the tags had been created the logic to check all the tags turned out to be quite slow.
Ahhhh guilty, that might have been me. I'm just checking if the return of the writeBlocking to see if any of them were bad. Do you think its that much of an additional delay?
Looking at the running scripts status page, I never see one last more than ~100 ms. And that's when many come in at the same time. Each device sends a message every 10s or so.
1 Like
Don't know, measure it. Might not be a problem in this case - I think in the other ticket the UDP messages were arriving at high speed.
I skimmed your support ticket and looked at the thread dumps real quick... I'm not convinced that switching to the UDP driver will solve any of your issues.
It looks like you're abusing runScript quite a bit; there's a small thread pool dedicated to expression evaluation, and it's all blocked up executing slow running scripts you call via runScript.
I'm not sure yet how/why this is related to the tag browsing.
Coworker says he tried turning off all the runscripts as part of the ticket I mentioned and didn't see a noticeable difference. runscripts bother me as a concept anyway, so I'll see about pruning them.
There's only so much I can tell from a thread dump, and in the 3 you sent me these are the obvious issues.
1 Like