Front end performance is dismal

Ignition
,
1

Our Ignition 8.1 system has operated extremely slow since day one. Small improvements have been made, but it's to the point where operators are extremely frustrated with the performance.

Since December 2022 we have been running 8.1.22, but we had not rolled out our clients to all areas of the plant. That has slowly taken place over this year (2023) to the point where almost all the plant is now running Ignition clients.

Today I upgraded to the latest release 8.1.32 hoping that it would resolve these performance issues, but it has not.

If I start the system up from scratch, it's extremely responsive for the first 30-40 minutes, but then it goes downhill. Navigating between our primary screens shows little issues, just a slight delay as the screen tag bindings refresh. Only when I click on an object, which then brings up that object's sidebar do we see major delays. Right now it's about 15 seconds for the sidebar to refresh. During that delay the entire client freezes.

Once the sidebar is loaded, if I click on objects that are similar (such as different analog values - temperature, pressure, flow, etc.) the sidebar contents refresh quickly. But, if the sidebar is still open and I click on a different object type (such as a motor), then I'll experience the same 15 second delay and client freeze up until it loads.

There is an "Alarm" button on each sidebar that, when pressed, opens all alarms for that specific object. The delay in opening it is upward of 30 seconds or more.

Comms on the backend are looking very good as that's where I've been concentrating (since I inherited this system!) because our PLC's were getting overloaded. That's no longer the case. This really appears to be a front end issue. If I open a designer session on the front end, the Tag Browser is almost unusable because of delays to refresh.

I have a video showing navigation between main windows and the speed they update, but then I click on an analog instrument (a tank level in this case) and it demonstrates the issue with loading the sidebar. I tried to attach, but it's too big for the 8M maximum. I'll see if I can get a dropbox link available.

In the meantime, I am attaching a view photos of the client diagnostics pulled up. You can see the gap in data logging caused by the freeze up delay when opening the sidebar (IMG_7802). Image IMG_7809 is the same logging after opening up the alarm page which is an even longer delay. It's quite bad.

IMG_7809
IMG_78091920×1440 240 KB

IMG_7802
IMG_78021920×1440 235 KB

1 Like
2

Are you using any of the red alarm label components on the header?

3

Hello and thanks.

I don't quite follow what your question is referring to. Can you clarify?

4

You have described the pathological situation that is common when naïve programmers do all kinds of busy work in their project in Vision's event scripts and in runScript() bindings. Crushes performance.

I would start by auditing your event scripts by adding System.nanoTime() calls and logging any event that takes more than single digit milliseconds to execute.

(While there, refactor any event script that is more than two lines of code into a project library script function, and delegate to that function.)

10 Likes
5

This sound suspiciously like someone scripted something blocking on the GUI thread.

If I remember correctly there was an included alarm button in the project templates that had horrible performance due to the method it used to get alarms. Your button may be based off this template

Do this.

At this current point, since we cannot look at your project ourselves, we won't be able to provide anything other than general recommendations. You'll need to put in the legwork to find out what your predecessor did and what needs to be corrected.

Once you have more detailed info of what is bogging down your system, we can help to provide more specific fixes/recommendations. You may able to find threads that already talk about some of the issues you find.

7 Likes
6

Thank you for all the quick replies. I'll see what I can find.

As for the comment regarding my project, is that something that people do - post their projects here?

7

Yes, but keep in mind your employer's or client's proprietary information. Check with your boss(es) if in doubt.

IA support can look at stuff under an NDA, and you can engage an integrator under an NDA if you cannot figure it all out with any constraints on what you can share.

4 Likes
8

I'm assuming this is probably what's causing the numerous amounts of "NonResponsiveEdt-.json" files I'm seeing in client cache directories, yes?

5 Likes
9

Yes.

10

It is probably a long executing script on the EDT like Phil and Ryan have stated. I am also going to post this link as it has a bunch of other gotchas that you might have.

7 Likes
11

Hey Guys. I'm working with Travis and wanted to share this. This is the Gateway Network details for the back-end server as seen from the front-end server. It seems there are a high number of Alarm Query invocations per second. Is this a clue we should follow? I know there are some less than ideal coding in the app, and we will work on those. Thanks for all the input.

Screenshot from 2023-10-13 18-50-38
Screenshot from 2023-10-13 18-50-381920×1080 391 KB

12

Yes. That is a common pattern where some alarm information can only be obtained with scripting function calls, which is too conveniently available through runScript, which then blocks your Vision UI for all of those round-trip network requests.

If you treat all database, tag read/write, and alarm information scripting functions as forbidden in anything called from runScript, or called from any propertyChange script, you'll likely fix the bulk of your problems.

The sad part is that IA itself put out a bunch of project templates with embedded bad practices (particularly in docked windows).

7 Likes
13

Likely related to this. For the alarm button, see what view it opens, and then inspect that view's properties and components, most likely the binding on whatever is used to display the list of alarms. Also check the built-in script methods on the table/alarm table/whatever is displaying the alarms.

You can also perform a search for system.alarm.. Tell it to look in all windows, bindings, and templates. Then scroll through the resulting list and see if any are in a binding with runscript. The search tool is not the end all be all but it should get you close.

The other thing I would look at is the sidebar you use for when a user selects an item.

This sounds like it might be a script that fires on a page parameter change that either queries a DB for some information or has some other long running process. That would explain why the freeze appears to only happen on view open or item type change. The system might be loading some heavy configuration or something that is specific to each type of item.

Take a look at the sidebar window's visionWindowOpened, internalFrameOpened and propertyChanged event handlers.

Be very careful of this, this has got me before (even with perspective stuff)

14

To be explicit here, never call system.alarm.queryStatus ever from a vision (or perspective) client. This can cause extreme performance issues.

@Travis_Graham
If you do end up posting the project or even DMing it to some of us, that would certainly help, but otherwise screenshots of some of the slow performing pages might be useful

4 Likes
15

Also, make sure whatever is grabbing your alarm history is not grabbing an obscene amount of records. We had an engineer almost crash our gateway trying to pull/display almost 2 years worth of alarm data from all our connected machines. (We run a small gateway, not a huge operation)

1 Like
16

Just curious, but would an issue like this get amplified as we bring more clients online? And, we're also seeing that the delay time seems to grow over time, which seems odd. I'd think it would max out at some point, maybe 20 seconds of delay, instead of seeming to get worse and worse?

This is more of a question for me to better understand the underlying system and how best to go about things.

Thanks!

17

It could, because each client is issuing its own queries, and query performance on the backend will decline.

Those non-responsive EDT files you mentioned are worth sharing also. They'll probably just point to queries, but maybe there's other things.

1 Like
18

It's always that alarm button that people put on a header.

19

Thanks, Kevin. I’ll gather some of those files and send them. I’ll get some from multiple clients.

Going into this, I thought our back end was the major issue. PLCs were getting extremely overloaded and so a lot of time was spent lining that out. Now that the backend looks good (seriously, the difference is amazing), it seems like it brought to light the client front-end issues. The system is almost unusable at the moment and I haven’t figured out how to give it even a little relief.

I’ll dig in to the suggestions here to see if I stumble on something.

20

This kind of make me feel like we should rebuild those templates with better practices, submit them for peer review, then send them to IA.

12 Likes