Embedded views subscribing the same tag (Alarm Aggregation)

In our Perspective application, we have an embedded view that serves as the primary indicator for data. We pass a UDT path to it, and it shows the value and alarm border for that UDT instance.

A gateway script runs every 5 seconds using system.alarm.queryStatus(). I take those results, expand each path, and build an aggregate for every folder. This aggregate is saved as a document in a memory tag. Each embedded view uses an indirect tag binding to read this tag and get the associated border state.

After many iterations, this has been the most reliable solution so far. However, when a few clients open busier views, the tag-provider-subs thread CPU spikes for that provider.

Is there a better way to accomplish what I’m trying to do?

In 8.3, there's an automatic alarm aggregation 'virtual' node below each complex tag (folder or UDT instance):

That will be more efficient than the script + document tag approach, but until 8.3 is generally available I'd probably just keep going with what you have.

Yeah, I saw that… can’t wait for full release :grinning_face: .

Any other ideas to improve performance? Would using a message send from the script to the embedded views work better than subscribing to a tag?

This is the thread I’ve noticed that seems to spike the CPU:

Thread [tag-provider-subs-Project-1] id=120, (RUNNABLE)
  java.base@17.0.14/java.io.StringWriter.write(Unknown Source)
  app//com.inductiveautomation.ignition.common.gson.stream.JsonWriter.newline(JsonWriter.java:654)
  app//com.inductiveautomation.ignition.common.gson.stream.JsonWriter.beforeValue(JsonWriter.java:693)
  app//com.inductiveautomation.ignition.common.gson.stream.JsonWriter.value(JsonWriter.java:470)
  app//com.inductiveautomation.ignition.common.document.DocumentElementTypeAdapter.write(DocumentElementTypeAdapter.java:78)
  app//com.inductiveautomation.ignition.common.document.DocumentElementTypeAdapter.write(DocumentElementTypeAdapter.java:64)
  app//com.inductiveautomation.ignition.common.document.DocumentElementTypeAdapter.write(DocumentElementTypeAdapter.java:71)
  app//com.inductiveautomation.ignition.common.document.DocumentElementTypeAdapter.write(DocumentElementTypeAdapter.java:11)
  app//com.inductiveautomation.ignition.common.document.DocumentElement.toString(DocumentElement.java:347)
  java.base@17.0.14/java.util.Formatter$FormatSpecifier.printString(Unknown Source)
  java.base@17.0.14/java.util.Formatter$FormatSpecifier.print(Unknown Source)
  java.base@17.0.14/java.util.Formatter.format(Unknown Source)
  java.base@17.0.14/java.util.Formatter.format(Unknown Source)
  java.base@17.0.14/java.lang.String.format(Unknown Source)
  app//com.inductiveautomation.ignition.common.model.values.BasicQualifiedValue.toString(BasicQualifiedValue.java:147)
  java.base@17.0.14/java.lang.String.valueOf(Unknown Source)
  java.base@17.0.14/java.lang.StringBuffer.append(Unknown Source)
  app//org.apache.commons.lang3.builder.ToStringStyle.appendDetail(ToStringStyle.java:617)
  app//org.apache.commons.lang3.builder.ToStringStyle.appendInternal(ToStringStyle.java:581)
  app//org.apache.commons.lang3.builder.ToStringStyle.append(ToStringStyle.java:467)
  app//org.apache.commons.lang3.builder.ToStringBuilder.append(ToStringBuilder.java:860)
  app//org.apache.commons.lang3.builder.ReflectionToStringBuilder.appendFieldsIn(ReflectionToStringBuilder.java:654)
  app//org.apache.commons.lang3.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java:841)
  app//org.apache.commons.lang3.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java:312)
  app//org.apache.commons.lang3.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java:165)
  app//org.apache.commons.lang3.builder.ToStringBuilder.reflectionToString(ToStringBuilder.java:162)
  app//com.inductiveautomation.ignition.common.tags.model.event.TagChangeEvent.toString(TagChangeEvent.java:70)
  java.base@17.0.14/java.lang.String.valueOf(Unknown Source)
  java.base@17.0.14/java.util.Objects.toString(Unknown Source)
  com.inductiveautomation.perspective.gateway.binding.tag.DirectTagBinding.tagChanged(DirectTagBinding.java:327)
  app//com.inductiveautomation.ignition.gateway.tags.subscriptions.ProviderSubscriptionManagerImpl$TagChangePublish.run(ProviderSubscriptionManagerImpl.java:1530)
  java.base@17.0.14/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
  java.base@17.0.14/java.util.concurrent.FutureTask.run(Unknown Source)
  java.base@17.0.14/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  java.base@17.0.14/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.base@17.0.14/java.lang.Thread.run(Unknown Source)

Sigh. That particular thread dump is from a logging statement eagerly formatting an object to a string... that will only end up being logged if you have the logger set to TRACE anyways :person_facepalming:

There's not really going to be a way to prevent that particular stack trace from showing up. If you make your document smaller/leaner, it'll be faster.

EDIT: I've created an internal issue to fix this performance issue.

I suppose it’s more of a symptom than the cause? Or is that thread showing me exactly what’s causing the high CPU usage?

I do have some ideas on how I could make the aggregate document tag smaller. I basically generate something like this:

{
  "[area1]": [true, 4],
  "[area1]dh1": [true, 4],
  "[area1]dh1/chiller1": [true, 3],
  "[area1]dh1/chiller1/pump1": [true, 3]
}

The idea is that my UDT path matches one of those keys, and if it doesn’t (meaning queryStatus() didn’t return anything), then I know it’s not in alarm. The views have an indirect tag binding to that tag, and then reference with a transform:

{value}[{view.params.path}]

Would it be better to break the aggregation document into multiple tags, even if it’s still the same size? There’s certainly a balance on what I’m doing on the view itself vs. the script.

Thanks for the help.

I'd be surprised if that particular thread was the true cause of the high CPU unless you have literally thousands of entries in this document tag. JSON isn't a very efficient format, and GSON isn't the fastest encoder, but it's still not going to take that long to mash it into a string. I'd recommend collecting a series of thread dumps while the CPU is spiking to see if you can spot some commonalities. Consider taking them into our support department.

I have faced the exact same issue when dealing with alarm aggregation.

What we ended up doing is putting the whole dictionnary of paths and alarms inside system.util.globals and then calling a runScript expression on each view to get the correct priority for a given tag path in the value of system.util.globals (or return nothing if the tag path is not in system.util.globals).

I’m 100% sure what I wrote above will make lots of people here scream in agony as this is a terrible idea (all embedded views periodically running a script to the gateway and retrieve the associated priority is awful ahah).

But surprisingly, even at large scale with hundreds of active alarms and hundred of embedded views at the same time with at least 20+ clients, the whole thing was quite powerful actually.

We did have the exact same issue as you, our CPU used to spike a lot every 5 seconds as our gateway was doing the queryStatus() computation and then write this huge dictionary in a document memory tag.

Now feel free to try this or not, but I know for a fact, it did solve our use case.

Another thing I do now which I believe might be a bit better (hopefully) is the following:

  • Still perform the computation every X seconds and store everything in system.util.globals
  • Create a UDT that, wherever it is placed, retrieves the associated priority based on where it is placed in the tag tree by using runScript periodically
  • In my embedded views, create an indirect tag binding based on the tag path of my equipment and add ‘AlarmStates’ (I named all the instances of this UDT ‘AlarmStates’ to retrieve the associated priority, etc.
  • I essentially have an instance of AlarmStates in almost all my UDTs, and even in some of my folders to get alarm aggregation data.
  • This way, all embedded views subscribe only to a small portion of the whole alarm aggregation dictionary, putting less pressure on the gateway.

Essentially this is very close to the feature 8.3 implements, but this is a homemade version if I may say so myself.

Experts, feel free to absolutely shame me if what I wrote above are the worst possible ways of doing this!