Project heavily based on templates gets often stuck

[szcz]

Hello all,

Ignition 7.9.4 user here running on 2012R2 and MSSQL 2016 R2 CU 11. I came upon interesting issue I cannot fix myself. I wanted to create a project for users to push buttons when something happens. There are currently 12 production lines and and the occurence will be tracked for all 12 lines.

The database structure is trivial - there are 8 operations per line and each operation tracks 3 actions (A,D,N). So in the DB, there is basically catalog of machines, catalog of operations, catalog of actions and “fact” table that stores what action happened on which operation in which line and when.

Now to the main problem:

I am terribly struggling with performance and the project behavior. It often happens the CPU load peaks to 100% due to Ignition designer or client running (checked through taskmgr). CPU on PC where that happens is i5-6400 4 cores.

Could the reason be I overshot with templates? On the image, each red rectangle represents one template. Count with me:

1 template instance per line - there are 12 lines
1 template instance per operation - there are 8 operations per line
1 template instance per action - there are 3 actions per operation

All the templates use template repeater - all the template repeaters query database for lines (i.e. how many lines are there and how are they called - e.g. name is put to template as parameter value), operations and actions. A,D,N represents actions and the text behind white rectangles represents operations. Covered for secrecy.

that gives us: 12x8x3 template instances in one main window = 288 template instances!

Often it happens that in the window, instead of template, there is loading progress bar running. Designer gets stuck to the extent where I have to kill it via task manager (always while hammering client’s CPU at 100% - server is basically idling around 3-7% CPU). I saw designer got stuck at “serializing window” message in the left bottom corner.

Of course the templates load some data from database (e.g. what operations are there, what actions are there and what machines are there), but these are tables with < 20 records. That cannot be the reason of my problems.

Every 180 seconds and on the window opening, I call self-made method refresh() within each template.

The “call - stack” is as follows:

line template calls refresh() for line → that calls refresh() for operation → that calls refresh() for each action (there are 3 actions per operation). The last refresh queries fact table to see what was the last action for given line and operation - based on that, some attributes are set. This refresh() chain is called upon window opening and every 180 seconds.

In case you would be missing some crucial informations about my project, I will be happy to answer further.

Take care,
SZ

[bpreston]

There are all kinds of things that can play into the screen freezing.

How is your RAM usage on the gateway when this happens? What does your RAM usage look like on your client or designer when you load the page? What are your gateway and client RAM settings?
When your talking your CPU load going to 100% thats just on your local system, not on your gateway?
How many individual queries are you running at one time when the page loads or refreshes? You mentioned 288 total template instances, if each is running a query at the same time that would be a big load on your system.
Are you seeing any errors logged?
How are you connecting? I’m assuming your connecting to the gateway locally and not over VPN.

With performance issues, they can be hard to track down, especially without access to the system. You may want to call into support and get them to help you look at the screen to suggest ways to improve your performance.

You can also look at splitting off parts of the screen into a test window so you can see how it operates with less running. If an instance for one line causes high CPU or RAM usage then you would want to make changes to improve the performance, then you can slowly add more line instances to it while monitoring system performance.

[nminchin]

Try to remove your individual queries from your templates, create the fields as template parameters, and then write to these properties within your template repeater query. That way, you’re only querying once and not multiple times per template (if I read your post correctly)

[szcz]

Hello,

RAM on gateway usage around normal baseline values. On the client, the RAM saturation is barely touching 50%.

Ignition server: 12GB RAM, Ignition as an application has 10GB assigned. 99.9% of the time using less than 4GB. CPU on gateway basically idling.

Ignition client: This particular client had 24GB RAM, we were faaaaar from saturating it. CPU on client @ 100% when the problem stroke.

Errors were not logged during the problem with 100%CPU, the network connectivity is through IPSec tunnel - the parameters of the tunnel are: latency around 10-20 miliseconds and throughput at least 30Mbps, often more.

Problem seems to be currently solved - I have re-designed the application - I use significantly less templates (under 100) on one page, I query the database less and the problem seems to be gone (which was the objective). I really can’t spend hours or days troubleshooting what the bottleneck really was.

Thank you anyway for the effort.

Take care,
SZ

[szcz]

Hello,

I have re-designed the application, the suggestion of yours was also taken into consideration, the amount of queries was significantly lowered and it seems to be OK now. Take a look at my answer to Mr. BPreston.

Thank you for the effort, much appreciated.

Take care.

[bpreston]

For client memory I was more thinking what is set in your project properties.

Even if the client has 24GB of RAM, the project is running using java which is limited in the memory it can use. How much memory can be assigned to the client also varies based on if your client computer is a 32bit or 64bit machine. If the settings I showed are set to low for the memory needed to load the screen you have then you can also run into issues with the client locking up.

It sounds like your on the right tack though. I have a feeling it was related to your database queries. I have run into an issue in the past with using queries in a template repeater that caused the client and designer to lock up when running them over VPN. In my case everything ran great when I was connected on the local network but it would lock up when connected over VPN. I ended up on the phone with support about it and when we moved the queries out of the templates the issue resolved. I was doing the queries through bindings on the templates. Talking to support they told me the vision screen bindings refresh at 250ms. With the number of queries I was doing and the extra latency from connecting over VPN, it caused the templates screen to lock up waiting on the query results.

[szcz]

Hello, I had the default - 32M and 256M -> maximum lifted to 1024MB. Anyway, it is not relevant right now, the project is re-designed, works good even with lower settings and there is no performance related hiccup. Either the amount of used templates or the amount of SQL queries was of the essence. I was able to diminish both, so we are good.

Anyway, thanks for the effort help and sharing your experience - I could’ve bumped to similar troubles as you did. Much appreciated.

Take care.