Performance issues since 8.1.21 upgrade

Ever since I upgraded to 8.1.21 a week ago I have been having performance issues system wide. I upgraded from 8.1.2.

System Info:
My ignition VM has 16GB of RAM (gateway is only allocated 8GB) and has 8 virtual processors.
mySQL VM has 16GB of RAM and 4 virtual processors.

The physical server is only a few years old and is pretty high end.
64GB RAM
Intel Xeon Gold 6244 (2x)
RAID 5 configuration with SAS SSD drives. 7x drives in the RAID with a few hot spares.

Things I have noticed:

Disk utilization is extremely high all the time (80% or more)

System response time is flickers between 1ms and 1000-2000ms. Clicking through my screens and performing normal tasks are slow.

My CPU usage looks okay. Before the upgrade I averaged 12% CPU usage. After upgrade it sits around 22% give or take 4%. My logfile maintenance thread has been using about 10% CPU on average so this may be the culprit.

Memory profile is also slightly different. I allocate 8192MB of memory to the gateway. Before the upgrade I had a very consistent sawtooth pattern. The heap would usually go up to 5-5.5 GB and then after garbage collection I would drop to 1.8-2.0 GB. After the upgrade I usually go up to 6-6.2 GB and still drop down to the same 1.8-2.0 GB. There is still a sawtooth pattern.

Pretty much right after I upgraded Ignition I also put in a new PLC which has its own OPC-UA server. That is the only other thing that I changed externally.

Can answer any other questions to help figure this out. The disk utilization being very high and logfile maintenance using 10% CPU seem to be the main issues (at least I think).

You should probably contact Support. This forum is not an official support venue.

I would also recommend not upgrading a production system without first testing in a "test" environment.

I just talked to support and I believe the issues are resolved. Just wanted to put it on here as well beforehand.

1 Like

Normally, yes, but I have a unique situation where no clients will be using the system for a while.

Might help others if you can share what resolved your case.

2 Likes

A few months ago I was having problems with gateway crashes. I narrowed it down to problems with my easy charts. In my logback.xml file I increased the amount of logs I was keeping by A LOT in order to figure out and resolve the problem. Because it never noticeably affected my performance, I never changed it back. I lowered the amount of logs I was keeping when talking with support and it helped tremendously.

Unfortunately, my disk utilization is still hovering around 80% and that problem is yet to be solved.

1 Like