Ignition performance problems

Hi all!

We are currently facing a problem that I am not able to solve.

We developed a SCADA system for registering some quality inspections and images in a database during a factory process.

We have the perspective app in approx. 10-12 tablets. The thing is, that during our trial period, using only one tablet we have encountered the system very slow and I am afraid when using all the tablets at the same time.

Ignition is solely installed in a VM with Windows 10 Pro OS and dedicated 32GB RAM. This mentioned VM is solely running on a Dell Server with VMWare Esxi OS, an 8-core Intel(R) Xeon(R) E-2388G CPU @ 3.20GHz and up to 96GB RAM.

We upload the images directly as blob to the database.

I do not know if our mistake is using so many queries, or programming mistakes, please a little help would be useful. See attached a performance screenshot.

In this picture I see the clockdrift warning, but we are not neither using the designer nor the tablets since 3-4 days ago.

It is true that we have an old vision project we do not use anymore (so it is disabled) and an OPC connection to a device we do not use anymore also, but the connection is ok.

Thanks in advance,

First obvious thing I see is that the Memory Ignition is allowed to use it set at 2GB by default, you need to change this in the ignition.conf file if you want to use more, then restart the gateway. I can see on your memory chart that the max is 2048

We’ve had projects max out at the 2GB limit and I had to reconfigure to run on 8GB

Thanks @hayden for your quick response,

where is located the “ignition.conf” file?

Not sure on windows as we mostly develop on Linux. Normally in the Ignition install location, C://…/ignition/data

But I used the below Ignition document, it says for 7.9 but works on version 8.1 as well. Just remember to write the memory in bytes. I’ve crashed gateways before by just writing “8” :smiling_face_with_tear:

https://docs.inductiveautomation.com/display/DOC79/Changing+Gateway+Memory+Allocation

1 Like

@hayden thanks for that, I have already done it!

Now my problem of clock drift events is still not solved. It seems that it is related to this picture. Could anybody give me a tip on this?

How many cores are dedicated to the Ignition gateway? (An 8-core host seems super wimpy.) Ignition runs best with many cores.

dear @pturmel see below image from the server administration panel for settings to the VM

And this is the hw of the actual server:

I’m not a VMware expert. But 4-cores (8 hyperthreads) is a light-duty setup, IMNSHO. If other VMs exist in this box where the total allocated cores is more than 7, you’ve got a problem. (That’s called an over-committed hypervisor.) IT folk think they can do this to hosts containing Ignition guests because Ignition tends to not be a big CPU hog. VMware doesn’t return CPU cores on demand very well, from what I’ve seen.

I don’t know enough about VMware to be more specific. The one customer of mine that relied on it is now running Ignition on bare metal.

Hi @pturmel, there is only one VM in the server, and that VM is where the Ignition gateway is running.

I will make sure that no other VMs are installed (for now) in the server, but then, what could I do to improve performance?

Is this really related to the ClockDrift events that I showed upper in the conversation?

PS: I did not expect that running Ignition on a VM was “that bad”. Our company is on a virtualization process and we think is a good approach to have the main services running on VMs.

PS2: sorry for disturbing, but I am not an IT guy neither.

It can be as drift can be related to CPU load.

If the VM is on a domain, you can have a conflict as the time can bounce between the host and the domain controller. I’ve had best luck turning off the time sync to the host and using other methods (AD or NTP) to sync the time. You can turn it off through the VM settings → Options tab → VMWare Tools

Thanks @JordanCClark, that was unchecked by default. But it is strange to me that the event is every 1 and half hour more or less.

What is the multicast scheduler?

I also disabled a PLC connection that we had (but we do not use anymore) and disabled the OPC server and client connections.

@pturmel @JordanCClark how can I change the gateway time?? Please see the differences of the time between the next to images. (Down on the right it is showed the windows os time, the correct for me).


In this picture, the logs are ahead my computer and gateway time.

Have you looked into VMWare latency sensitivity settings? It can be helpful to set “latency sensitivity” to “high” in ESX/ESXi and dedicate 100% vCPU and memory for resources allocated to Ignition.

VMWare offers resources if you Google “vmware latency sensitivity” such as this:

1 Like

8 cores, but 23000 passmarks for the cpu isn’t that bad. It depends really how large the project is and what it’s doing

Also, where is you sql sever hosted? It shouldn’t be in the same VM as ignition but it should be on the same local network / run in another VM on the same server

Hi all again. Thanks @nminchin and @nathan for your quick responses.

I also read the documentation ClockDriftDetector warnings in the Gateway Console – Inductive Automation Help Center

I have received a lot of inputs on what to modify, but I would like to know what would be the correct order to apply the steps in order to avoid ClockDriftDetector.

Right now, as showed in the pictures posts above, I have Ignition installed in a VM inside a server. The server (96GB RAM and 8 cores) has only this VM running (4 cores and 32GB RAM). It must be said that inside the VM is also installed SQL Server (but the maximum capacity of this service is limited to 4GB).

image

We would like to keep both Ignition and SQL Server in the same VM if possible. If not, we can really do the changes needed to obtain the best performance.

Then, what would be the steps to take/try and in what order:

  • Move SQL Server to another VM in the same server?
  • Move SQL Server to another VM in another server (same network)?
  • Move SQL Server to another server (same network)?
  • Correct Java heap sizes?
  • Name of CPUs allocated to the Ignition VM (out of 8)?
  • Date and time settings or other sync setttings?

Please find attached some pictures from the task manager.


I think I solved some ClockDrift events but I still see big values during the night. Please any advice could be helpful. Thanks.

PS: it must be said that right now SCADA is working well with two tablets (perspective sessions) connected, but we expect in short term to be up to ten.

It’s always recommended to move sql to another machine, doesn’t really matter if that’s another VM or another server altogether as long as they’re on the same local network (remote network is bad unless you don’t care for performance :roll_eyes:) I’d be interested to see what splitting this out does for your setup as I’ve never actually benchmarked it before.

Hi @nminchin, no problem on that, but then how much RAM and how much CPU should I give to this “new VM (dedicated to SQL Server)”? And the Ignition VM should remain untouched regardind to resources? Remind my server installed hardware (shown above).

Well, I would give the sql VM 4-6 cores and 16 GB to SQL, but you’re limited in your number of cores due to the cpu

My setup would be then:

  1. SQL VM: 4 cores and 16GB RAM
  2. Ignition VM: 4 cores and 32 GB RAM

I would explore (but I think this is not going to be immediately) the option to increase CPU on the server.

1 Like

No point having 32GB on the ignition server though if youre only giving ignition 4 though