We have been working nicely with our SCADA system based on Perspective for the last months. But all of a sudden, today the performance was horrible. I attach some pictures. Does anyone know why this happened?
I did not change anything in the perspective project, just the tablets where the clients are running get completely stuck and nothing can be done. If I check the performance the CPU Trend raises and I have to restart the VM where Ignition is running (five times today).
Find what is hogging your CPU. You can look at threads in the status section, maybe get thread dumps. Note the "clock drift" warnings: the deviations are in milliseconds. You show multiple CPU stalls of multiple seconds. Such stalls can be due to garbage collection, but can also be busy threads hogging the CPU.
Do you have new users for Perspective? Or a new feature you released for Perspective users? New historical chart or report that can bog you down with DB operations, perhaps? Something changed.
Do share your VM setup, and whether your DB is in the same VM. Note any overcommitted VM resources.
FWIW, 3GB of RAM for Ignition seems pretty low for a Perspective setup (which does all scripts and expressions and UI bindings on the gateway), unless you only have a couple users.
Clock drift issues have long been associated with environments which do not have enough allocated resources. This won't explain the change in behavior (performance). Situations like this typically lean toward an Occam's Razor sort of explanation: even if you think nothing's changed, something has.
These could all result in the change in performance:
More users.
Simple/minor change to a View (new charts/graphs).
Something newly installed on the same environment which is now sharing the resources Ignition needs to itself.
Dear @cmallonee and @pturmel, thanks for your info. Please find below more details:
Indeed there are more than a couple of users (but this is not new), this could be up to 15, all of them maybe executing a lot of scripts, bindings and expressions (depending on the view they are). What should be the memory assigned to Ignition then?
The Ignition VM and SQL VM is under the next environment (this has not changed neither). As you can see, both VMs are separated in different servers. Both Ignition and SQL Server have a dedicated VM with Windows Server, no other services and software running.
In a view, the operator must upload some photos and when pressing a button, a report is saved into disk. I think this can be the heaviest operation, but has been working ok for the past few months.
If you need more details, do not hesitate to ask. Thank you so much. Hopefully we will solve this issue soon.
Your VMs are slightly overcommited on CPUs (more CPUs allocated to VMs than present in the host). Work done on the Win10 facilities machine or the Design server can impact the co-resident Ignition without it showing in Ignition's performance metrics. There should always be at lease one core left over for the hypervisor to use, too.
However, the first thing to change is your Ignition memory allocation. You should be able to set Ignition to use 8GB (half of VM memory) without problem. Do be sure to set both initial and max heap memory to 8GB, so it all gets claimed by java on startup.
No more simple ideas. Set up Java Flight Recorder to run all of the time, and save on JVM exit. Set up a recurring OS task to run the Windows equivalent of jcmd $(pgrep java) GC.heap_dump someFileName-$(date +%Y%m%d-%H%M).hprof fairly often for later perusal with the flight recording.
