Ignition performance problems

I do not understand that I am giving Ignition VM 4GB. I have the java max heap size to 3GB right now. Is that what you mean?

Ah sorry, I remembered your screenshot wrong - I thought it was 4096 not 3072, but really it’s neither here nor there; 3-4/32GB is an extremely small % and if you’re not going to give Ignition a bit more, then there’s not much point in giving the VM so much above this. Then again, it doesn’t really matter since you’ve got 96 to play with. It was just an observation really :person_shrugging:

Perfect, all clear. Thanks!

I think I will try this setup a try:

  1. SQL VM: 4 cores and 16GB RAM
  2. Ignition VM: 4 cores and 8 GB RAM

(both VMs on the same server)

Please any more suggestions on how to improve performance would be nice.

well, if it were me, i would increase the memory ignition can use to like 6gb an set ram to 10GB

As far as you can tell is the slow performance happening on the tablet/client, server/gateway, or both?
Does it happen all the time or periodically?
Does it get worse when you add more tablets/clients into the mix?
When nothing is running and the Ignition gateway service is turned off how does the Windows 10 OS behave when using it? Have you tried using the OS while the issue is happening? Do you notice any performance issues on the OS, e.g. Windows Explorer, Start Menu, programs open with a large delay? Does the OS ever seem slow, unresponsive, sluggish overall?

Is your VM part of a domain? If yes then the VM by default will try to synchronize to the active directory domain controller(s) which acts as an NTP server. As already mentioned by others, in that case you will want to make sure that the VMware Tools option to “Synchronize guest time with host” is unchecked. Also, you will need to check if you domain controller’s time is off relative to your VMware server’s time. In any case running “w32tm /query /source” can tell you what the OS is set to sync its time to right now.

The topic of virtualization is very expansive and seems to require technical knowledge as well as knowledge of the dark arts to troubleshoot in some cases. Nowadays virtualization has performance that is comparable to native in non-CPU-bound workloads. Your system seems fine when it comes to CPU/memory allocation. In your case your CPU Ready time seems to be below 1% which is good. That makes sense since VMware is able to give you practically 100% of the CPU. You will start noticing performance issues if your CPU Ready climbs above 5%, sometimes less than that for latency-sensitive or CPU-bound workloads. CPU overcommitting/overprovisioning does not become a problem until you are assigning more vCPUs than you have available physical cores. Usually that happens once you have dozens to hundreds or even thousands of VMs running on the server and even then if done correctly it should not be a problem (long separate topic). I have seen Ignition systems run just fine on two vCPUs but that strongly depends on what you are doing, how many clients you are running, etc. I usually default my Ignition VM to 4 vCPUs and so far have not had to designate more. If fine tuning is required, I generally like to set up my vCPUs to keep my guest OS CPU usage under 30% with allowed spikes up to 80%. Giving a VM more vCPUs presents a problem if you do not have the physical cores to match it because VMware has to schedule all cores at the same time to perform work.

As others already mentioned making sure Ignition is allowed to use more than 2GB is important, although based on your screenshot your system is not using that much. Plan ahead though.

It is recommended to have the database on a different VM from the Ignition install for best performance but stricly speaking it is not absolutely necessary.

If you are using SQL Server Express Edition you are limited to using 1 CPU and/or 4 cores (4 vCPUs if virtualized). You can provide more vCPUs to the VM but SQL server will not use them unless you have one of the paid editions. 2-4 vCPUs are sufficient in the vast majority of cases.

Regarding RAM, I try to make sure my OS has at least 4GB dedicated to it (I don’t notice much of a performance boost after that) after subtracting what Ignition or SQL Server needs. So if your Ignition is given 4GB I would give my Ignition VM 8GB. I also usually limit the amount of RAM I give to my SQL Server to “VM RAM total minus 4GB”. SQL Server will by default try to offload the entire database(s) into RAM for faster performance. SQL Server is supposed to be good at giving back to your OS what it needs but I have seen that fail spectacularly. It definitely does not care about sharing with other instances or programs on the same VM.

What does your storage subsystem look like? Physical drives JBOD/RAID, NAS, or SAN? Are the drives SATA, SAS, or SSD? If you have an I/O heavy worload e.g. a database server that might affect your performance a lot.

Which Ignition version are you on? There have been Perspective performance improvements throughout most of the minor releases.

Is this from MozillaFirefox?

@jespinmartin1 , no, this is from Microsoft Edge

Hi @yburdak , thanks for your reply and support. I am going to try to reply to all your questions:

  • The slow performance is happening on the tablet/client, as far as I can tell.
  • It seems to happen periodically but only during the “night”. During the day maybe I get two ClockDrift events but during the night (let´s say that night is from 12pm to 7am) I get a lot.

  • I could not say if it gets worse when adding more tablets or clients. It has been working well with two tablets and with eight.

  • As we have the Ignition installed on a VM, the idea is to not intensely use the OS itself, the OS is for hosting Ignition and SQL Server (right now). I have used SQL Server Management Studio to do some queries and I have not noticed any issues (even when problems appeared).

  • As far as I know, the VM is not on a domain. The VMWare Tools options “Sync guest time with host” is unchecked and the VM is syncing with an NTP. I do not know how to run “w32tm /query /source”. Is this for cmd?

  • Probably I will move the SQL Server (we run the Standard Edition) to a new VM that will be running in the same server, but separated from the Ignition VM.

  • I will check what you say about storage subsystem.

  • The Ignition version is 8.1.17.

Thank you so much again!

Any OS/IT Scheduled events occurring during this time frame? SQL Backups, Snapshots, etc...

As has been stated highly recommended.

1 Like

@lrose I checked that, but my guess is that it has to be related to OS/IT events during the night. I will check again with our IT service.