Slow Expression Tags

Version 8.1.39

I was working on a customer's server when I noticed that expression tags were taking 30-40 seconds to initialize or to update their values.

I tried creating memory and reference tags and those updated immediately.

All of these tags were on my own tag group which had the default settings of 1000ms and a new name to differentiate from "Default".

Remoting into the server showed the cpu was consistently around 40% and the memory allowed for Ignition is 12 of 32GB. Ignition is only using a max of 9GB.

I am curious to know how expression tags execute differently from other types. And I want to know if anyone has seen and/or remedied this behavior before.

Some other information:

  • there are 2 other tag providers that have about 50k tags each
  • otherwise there around 10k tags among 10 other providers
  • this gateway has the MQTT transmission module

If it's still happening get a thread dump and post it here.

Ok here it is. Thanks

Ignition-JOPV9903_thread_dump20241206-143716.json (555.7 KB)

Sorry, use this one. This is while it was executing.

Ignition-JOPV9903_thread_dump20241206-144026.json (580.4 KB)

Same behaviour for me. Im am using Ignition 8.1.36
Is there a solution for that?

The most common cause is use of runScript() in expression tags where the scripts aren't executing quickly (~10ms or less).

Like tag event scripts (on tags, not in projects), expression tags execute in a dedicated thread pool of size three. As soon as you have three slow expressions running, all other expressions (in the whole gateway) have to wait.

All of the constraints discussed on this forum for tag event scripts also apply to expression tags' runScript() calls.

3 Likes

Thanks for the answer. I added a few more threads (described here).

We now have ~150k tags on that server, thus i think it makes sense to increase the number of threads. runScript() is not that often used but there are a lot of expression tags.

Best Regards,
Gerald

1 Like

Yes, that's a lot of tags. If the increased thread pool doesn't help, you should open a support ticket and give IA a thread dump to examine.

I recently ran into the same issue. I would have an opc tag that would update immediately, but I used an expression tag to first validate the quality of the OPC tag before using it and go to a failover if the quality is bad. I also have a lot of runScripts in expression tags. My tags would update very slowly.

I have been working on reducing my use of this. One of the things I have been doing is putting the opc tag quality validation in my view logic, instead of in tags. But in some cases, an expression tag really is the best way to go as far as I can tell.

So in the case that I still end up using a high amount of expression tags and runScripts, the best way would be to increase the thread count based on this discussion.

How do I know the amount of threads I can increase by? I have never done this so I’ll have to look into that link that Gerald added. Thank you by the way!

And are there ways to improve this other than adding more threads?

Fix this. Really.

If you have lots of them, then adding more threads will simply tie up more threads.

Yup, I have been reducing them. So general consensus is to use runscript functions as little as possible?

In general yes. I use them very sparingly and only for things like string lookups or formatting where I have a number that is like a fault code on a VFD and I lookup the associated name/description of the fault.

This is what I have been using it for as well. To be able to refer to tags using wildcards.

But going back to the original topic. Expression tags in general should then also be avoided?

Yes. Where you must use them, you should time them with java.lang.System.nanoTime() and log when >5ms. So you can focus on removing the worst offenders.

Ideally, do not run database or network operations, or make system calls that do database or network operations.

Make sure the tags are in event driven mode, so they only run when a reference changes.

This should not be in an expression tag at all. It will run when no one is looking at it (useless), and cannot be internationalized for multiple simultaneous users. This belongs in UI bindings.

This is a heavy operation. Avoid tag searches except where a user triggers it.

No, expression tags in general are fine, and useful in a variety of cases.

1 Like

I’m not sold on this. Consider you have multiple valve devices that have slightly different status enums resulting in value 1 for example being a different state of the valve for each. (this can happen over time as code is improved, or if you’re incorporating another integrator's plc code).

If you use an expression tag, it's extremely simple to create UDTs for each of the valves and have a unique status descriptive tag for each valve, where 1 results in a different value for each. Then your UI becomes detached from the raw values of the valve tags, because you can simply use the standard descriptive tag values which are all standardised. Eg with states “Opened”, “Closed”, “Opening”, etc. Rather than looking at a status enum value which might represent different things as in my example.

If I instead do the enum lookup in the UI, I now have a lot more work to do with significantly higher risk of mistakes and maintenance. I either need to have a huge case statement first looking at the UDT Type, then another which maps the enum value to the state, which all needs to be done everywhere you want to use or show the status descriptive text :skull:; or I need to have a template view for every single UDT which does the enum mapping in that :skull::skull:

Maybe I'm missing something?

I'd call that design error #1.

Until you need a second or third language, and you are screwed.

And you are still processing string lookups for many tags that aren't being watched.

For such a poorly designed project, I would construct a shared document tag with two layers of keys--outer enum "type" and inner state value to lookup. I would ensure the result strings were unique translation keys. Then any tag value that needs lookup and translation can just have custom tag property indicating the enum "type" for lookup.

UDTs stay lean. Lookup values are centralized. No excess computation is performed.

2 Likes

Sometimes you cant help it when you’re incorporating programmes from multiple OEM's who have fixed code

Certainly. Or rescuing a disaster. That's why I made a concrete suggestion.

1 Like

(sorry got a phone call as I was replying)

Sounds like a great way to do this though while still maintaining the centralised approach that is also maintainable, while also minimising impact on the gateway. Would you do the same if you're expecting large numbers of clients connected at once? Eg 150 or so

Yup. It's all in the gateway. The expression language bracket syntax to extract values from document tags appears to be super efficient.

2 Likes