Perspective, Trying to reduce thread count per session

Ignition
, ,
#1

I’ve restarted work on a project that shelved a while ago due to low priority. Part of my update work has been reducing the load of the session on our gateway. I’m managed to reduce the amount of DB polls from ~400/s per session to ~6/s per session.

The problem I am running into is each session can spawn between 100 and 350 TIMED_WAITING threads. I have tested this and it appears to be tied to the number of instances displayed in a flex repeater on the main page of the project. A large number of the timed_waiting threads are named Perspective_Worker

Part of my work in reducing the number of DB polls by the session was to pull as much data calculation /fetching out of the templates used in the repeater and having it calculated/determined in a session global script and then passed via parameters to each template instance.

Each template contains an embedded view that uses a type parameter to build a path to the required embedded view. This was to replace the original template which had one of each data type (Checkbox, text, number, date, etc) and showed only the one used. These data entry fields have an ‘onActionPerformed’ script that will write the new data value to the database if it changed. Each template also has some labels that either hide or show based on another value that is calculated outside the repeater then passed in. There are still several expressions executing on each template.

I am aware that some of the answers to the questions below will vary with the application. Some more info about the project I am working on:
The project is meant to be a digital buildbook. There are different section numbers the user can select, and a flex repeater displays the steps of the selected section. Current state of the step is displayed, any data entered for that step, and a timestamp with the UID of the user who entered the data/signed off on the step.

There is also a section on a step that will appear if the step requires third party validation. This hidden section is included in the step template and show as needed. This section has a checkbox, label, and a timestamp with the UID and time of signoff. Additionally, if there is something that deviates from norm, there is a label that will show at the top of the step indicating there is additional info available.

I am trying to reduce the amount of timed waiting threads as much as possible. As such I had the following questions:

  • Does having an expression or a transform script spawn a timed thread per expression?
  • What things/operations spawn timed threads?
  • Is there a lower limit that exists for amount of threads per perspective session that increases with project complexity?
  • What is an ideal number of threads per perspective session?
  • Is this a case where it would be better to start from the ground up instead of reworking an old project?

I’ve also attached a thread dump in case it may be of use.Ignition Gateway_thread_dump20210222-212235.txt (427.8 KB)

#2

Not quite - threads are re-used as eagerly as possible (see Executors.newCachedThreadPool()), but in general - yes, expressions, ‘polling’ bindings (tag history, query, HTTP, etc) certain transforms, and various other places in the codebase will in some/all circumstances execute an asynchronous task, which has the possibility of creating a new thread.

No, as mentioned above it’s a dynamically allocated pool with no upper or lower bound (although that’s an interesting idea); it might be better in certain scenarios to pre-allocate a pool of threads on GW startup if you’re going to have a consistent load…

Pretty much impossible to say, but lower is certainly better; Java threads are (currently) essentially tied to heavyweight OS threads. Creating them is not ‘cheap’ - hence our use of a cached thread pool.

It’s very hard to say, but…maybe? It’s my experience that Perspective has much more of a learning curve than Vision, or maybe just more potential ‘bad patterns’ to fall into. Actively spending more time in Perspective you learn ‘better’ ways to do things, that are often less work, more performant, and ‘better’ (mobile responsive, more flexible, etc) - probably the first ‘real’ Perspective lesson is the use of flex containers in abundance. But, it’s also a significant engineering effort to redo something that ‘works’, especially if the goal (less CPU load/threads/etc) will be very hard to measure until you’ve already done all the work. I would start by trying to isolate ‘problematic’ pages, and rework just those pages, rather than a wholesale refactor. (Ground up refactors are often a bad choice; https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/)

1 Like
#3

I don’t think so, it doesn’t sound like there is anything really wrong with your project. May I ask, what has made you focus on this thread pool size in the first place?

#4

Just to tack on to that last question about a full project refactor. I had similar hopes. I think every programmer who gets brought into a legacy code base full of undocumented spaghetti has the same idea at some point. But try to recreate from scratch is almost never better/worth it as the article mentions.

I think you’re better off taking one view, refactoring it nicely with consistent naming conventions and use of scripting modules instead of logic in the buttons etc, and make it as clean as possible. Then, try to the bring the same methodology to the next view. You will probably eventually run into a view that needs a caveat here or there - this is the sort of thing you’d probably miss if you just tried from scratch.

Also, trying from scratch then means everyone has to continue to use your current program until you have your rework done - at which point you will deploy it and realize you may have got rid of the old bugs but also probably introduced new ones.

Unless its a very tiny project and you don’t have a lot to recreate, I would have to say no.

#5

Thank you everyone for the responses.

@Carl.Gould,
As I mentioned before my goal is to reduce the load on our gateway as much as possible. Before I reworked this project to not spam our database with requests it had a similar number of spawned threads as our other projects, ~40.

I also know that large™ numbers of waiting threads will affect the performance of systems. (I accidentally spawned 4000 waiting threads once, which appears to be our current gateway’s limit) I also would like to try to make this project as efficient as possible before throwing more hardware at it.

Optimizing his project is a bit more important than the other projects we currently deploy as we will likely have a large number of sessions active at the same time, as those above me want this build book to replace our current paper books on the floor.

@bkarabinchak.psi,

Going view by view is what I am doing right now. When you say use of scripting modules, could you elaborate? Do you mean not utilizing the ‘OnActionPerformed’ event and using something else? Perhaps I am just misunderstanding.

Right now I do have a lot of expressions on the template components that I use to change displayed text, and hide or show certain items based on data that was passed to the template. Should I just make the template ‘dumb’ and do these determinations at a global level and pass through the full string or show/hide value via parameters?

@PGriffith
One more question about threads spawning, do the event listeners for the ‘onActionPerformed’ events group as much as possible or is it again like you said where there is a possibility of it spawning a new thread?

Thanks again everyone.

#6

do the event listeners for the ‘onActionPerformed’ events group as much as possible or is it again like you said where there is a possibility of it spawning a new thread?

It’s not really either of those. All of these things that need to do “blocking work” (any work that may block: running a script, running a query, running an expression, etc) will put their task into a queue serviced by the perspective-worker threadpool. This threadpool will create threads as needed to empty the queue as quickly as possible. A thread created will then hang out for 60 seconds. If it finds more work to do within 60 seconds, it will be re-used. If no work is available in 60 seconds, it will terminate. In this way, the thread pool can grow to handle big bursts of work, and then will slowly shrink back down if the burst ends.

I think you are going to find diminishing returns of performance gains as you continue down this path. I wouldn’t be scared off of using expressions, etc simply because of a thread count. What matters more to actual performance metrics (cpu, memory) is the sort of work the expression is doing, and whether it is doing blocking work.

I will say, that soon we will have better metrics visibility into more of the metrics that matter. For example: what is the size of the queue of tasks waiting to execute? How long are they waiting in the queue? These sorts of things will give us better insight into the kinds of performance numbers that matter more than a raw thread count.

2 Likes
#7

Yes you would still but doing things in the onActionPerformed aspect of buttons, but the logic inside the button should be minimal - getting info off the screen/user, and feeding it to a function asap.

So imo the onActionPerformed should look like

import myLibrary

some_text = someComponent.text
some_number = someComponent.value.
...
myLibrary.doSomething(some_txt, some_number, ....)

I recommend this for a few reasons. What if you need to integrate the business logic else where? Or from a different data source, say a csv file/database table instead of a GUI? Etc. Plus eventually this means you’ll have to touch the GUI designer less and be able to look at your script modules instead of the windows designer as you work - my personal preference.

As for your second question - I cede to what Ignition employees are telling you as I don’t know the specifics. All I can say for certain is for performance you should most definitely try to avoid using runScript() in any expressions as that is a performance hit. However, I usually like expressions - I think they’re generally very fast.

Based on the design of these templates that you are telling me, they sound pretty lightweight. You just pass a few parameters, have a few expressions that visibility is bound to and that is it? I don’t think your templates are causing a performance issue, I could only see that happening if you had many templates on the same page, but even then I don’t think going from expressions to strings like you’re suggesting would really help. I would leave the expressions as is.

Now if you have scripting on the templates, that’s something else. In my experience, if you are able to do the same with expressions and scripting, the expressions evaluate faster. I can’t empirically prove it but it seems like it lol.

#8

Okay, I guess I did misunderstand. The good news is I am already doing what you described, sort of. All the event script does is gather the needed variables and then pass them to a system.db.runNamedQuery call.

I did a quick browse through my templates for all the step types, and everything uses either direct bindings or expressions. There are only two items that utilize actual scripts, and those are the data entry components with onActionPerformed scripts.

@Carl.Gould,
Thanks for the insight. Do you have a rough timing of when these metrics will become available? Or is it Soon™? In the meantime I’ll set aside the thread reduction effort and continue getting this project back to working status.

Thanks everyone for the feedback!

#9

Can you qualify “soon”? :slight_smile: we have a server running extremely poorly so this would be very handy!

#10

You know I can’t. Or won’t. Or both! All I can say is that this effort is underway and is a “before 8.2” effort, so it’ll get slotted into a 8.1.* maintenance release in the relatively, somewhat, sort-of near future.

That being said, @nminchin, we could probably help troubleshoot / diagnose your performance issues before these metrics improvements land.

2 Likes