Ooooo! Do you have many tag change event scripts? Any with long executions? Or any form of sleep() function calls? You might be backlogging the tag event thread pool. Instead of using a tag event (on the tag), try a gateway tag change event (in the project's scripting/events part of the designer tree). Those don't have the same kind of queuing issues.
(Set it to have a dedicated thread just to be sure.)
If that makes a difference (and I suspect it will), you will need to visit all of your tags to get rid of the pathological algorithms.
(If you are using sleep() anywhere in Ignition, you are usually screwing up.)
Oooh this is interesting. I didn't code the system initially so I can't attest to how many tags are using the valueChange event, but I do believe there are many. One thing I noticed is the tag values seem to stutter in the tag editor window. Like the tags are just straight up not being updated. Would the backlog of change events cause the opc connection to stop updating tags as well?
Maybe this is my problem. At what point does the script become active? I assumed it was when I click ok on the gateway event menu. Is it only active once the project is saved?
Project script edits you've OK'd become active when project is saved. The gateway tag change event scripts you setup under project tree are part of the project (unlike tag event scripts defined directly on tags).
This was the problem. I wasn't saving the project. I would attempt it without saving it and it didn't work so I disabled the script so as to not cause any unforeseen issues.
It is working now, I did the echo test 10 times...
These response times seem by far more consistent. However, the plant isn't running at capacity right now because its the weekend and therefore ignition isnt loaded down with its other tasks. I will run the test again tomorrow while I am there and we are at fully operation.
Are there any white papers or documentation on best practices on when you should use tag level valueChange scripts vs gateway level valueChange scripts?
So I was able to move the scripts related to omron over to the gateway events and it helped tremendously. I am going to move the problematic scripts for the Siemens side tomorrow. I'm hoping to see the same drastic improvements.
I'm still having a hard time understanding why it was problematic using the tag events. By default, it looks like tags all each have their own queue (3 I believe). Was it a problem with the tag scripts queueing up in that, or a problem some where else? I'd like to better understand why this change helped so much.
The each have their own queue of length 5. But there are only three threads servicing all of the queues. If any event script bogs down, only two threads will be servicing everything else. If three event scripts bog down, nothing gets serviced.
Do note that if you move long-duration event scripts to the unbounded queues, you can still get in trouble. (Out of memory, in particular.) Plan on rewriting those algorithms.
Not really, no. Export all of your tags and search that for the literal sleep(.
After you've fixed those, add uses of java.lang.System.nanoTime() to capture start and end timestamps. Log the delta at the end of every event. Skip logging for anything shorter than five milliseconds.
So I've identified a script that is taking in excess of 30s to run. Sometimes takes over a minute. There are 4 tags that call the script. I frequently see the script occupying all 3 of the available threads. I'm thinking this may be the problem.
While I investigate reworking these scripts to make them more efficient, I was thinking about increasing the number of threads available for tag event processing. If I were to increase it from 3 to say 5, it would allow one thread to always be available even if the other 4 get backed up processing the lengthy script. This would be a temporary solution to help our lack of throughput through our process. Run with 5 threads to help get parts out while I rework the problematic script. Once the script is figured out I would change the number of threads back to 3.
My question with this is if I increase the thread count to 5 does it pull resources from else where that might create a problem in itself?