We are currently facing a problem that I am not able to solve.
We developed a SCADA system for registering some quality inspections and images in a database during a factory process.
We have the perspective app in approx. 10-12 tablets. The thing is, that during our trial period, using only one tablet we have encountered the system very slow and I am afraid when using all the tablets at the same time.
Ignition is solely installed in a VM with Windows 10 Pro OS and dedicated 32GB RAM. This mentioned VM is solely running on a Dell Server with VMWare Esxi OS, an 8-core Intel(R) Xeon(R) E-2388G CPU @ 3.20GHz and up to 96GB RAM.
We upload the images directly as blob to the database.
I do not know if our mistake is using so many queries, or programming mistakes, please a little help would be useful. See attached a performance screenshot.
In this picture I see the clockdrift warning, but we are not neither using the designer nor the tablets since 3-4 days ago.
It is true that we have an old vision project we do not use anymore (so it is disabled) and an OPC connection to a device we do not use anymore also, but the connection is ok.
First obvious thing I see is that the Memory Ignition is allowed to use it set at 2GB by default, you need to change this in the ignition.conf file if you want to use more, then restart the gateway. I can see on your memory chart that the max is 2048
We’ve had projects max out at the 2GB limit and I had to reconfigure to run on 8GB
Not sure on windows as we mostly develop on Linux. Normally in the Ignition install location, C://…/ignition/data
But I used the below Ignition document, it says for 7.9 but works on version 8.1 as well. Just remember to write the memory in bytes. I’ve crashed gateways before by just writing “8”
I’m not a VMware expert. But 4-cores (8 hyperthreads) is a light-duty setup, IMNSHO. If other VMs exist in this box where the total allocated cores is more than 7, you’ve got a problem. (That’s called an over-committed hypervisor.) IT folk think they can do this to hosts containing Ignition guests because Ignition tends to not be a big CPU hog. VMware doesn’t return CPU cores on demand very well, from what I’ve seen.
I don’t know enough about VMware to be more specific. The one customer of mine that relied on it is now running Ignition on bare metal.
Hi @pturmel, there is only one VM in the server, and that VM is where the Ignition gateway is running.
I will make sure that no other VMs are installed (for now) in the server, but then, what could I do to improve performance?
Is this really related to the ClockDrift events that I showed upper in the conversation?
PS: I did not expect that running Ignition on a VM was “that bad”. Our company is on a virtualization process and we think is a good approach to have the main services running on VMs.
PS2: sorry for disturbing, but I am not an IT guy neither.
If the VM is on a domain, you can have a conflict as the time can bounce between the host and the domain controller. I've had best luck turning off the time sync to the host and using other methods (AD or NTP) to sync the time. You can turn it off through the VM settings -> Options tab -> VMWare Tools
@pturmel@JordanCClark how can I change the gateway time?? Please see the differences of the time between the next to images. (Down on the right it is showed the windows os time, the correct for me).
Have you looked into VMWare latency sensitivity settings? It can be helpful to set “latency sensitivity” to “high” in ESX/ESXi and dedicate 100% vCPU and memory for resources allocated to Ignition.
VMWare offers resources if you Google “vmware latency sensitivity” such as this:
8 cores, but 23000 passmarks for the cpu isn’t that bad. It depends really how large the project is and what it’s doing
Also, where is you sql sever hosted? It shouldn’t be in the same VM as ignition but it should be on the same local network / run in another VM on the same server
I have received a lot of inputs on what to modify, but I would like to know what would be the correct order to apply the steps in order to avoid ClockDriftDetector.
Right now, as showed in the pictures posts above, I have Ignition installed in a VM inside a server. The server (96GB RAM and 8 cores) has only this VM running (4 cores and 32GB RAM). It must be said that inside the VM is also installed SQL Server (but the maximum capacity of this service is limited to 4GB).
We would like to keep both Ignition and SQL Server in the same VM if possible. If not, we can really do the changes needed to obtain the best performance.
Then, what would be the steps to take/try and in what order:
Move SQL Server to another VM in the same server?
Move SQL Server to another VM in another server (same network)?
Move SQL Server to another server (same network)?
Correct Java heap sizes?
Name of CPUs allocated to the Ignition VM (out of 8)?
I think I solved some ClockDrift events but I still see big values during the night. Please any advice could be helpful. Thanks.
PS: it must be said that right now SCADA is working well with two tablets (perspective sessions) connected, but we expect in short term to be up to ten.
It’s always recommended to move sql to another machine, doesn’t really matter if that’s another VM or another server altogether as long as they’re on the same local network (remote network is bad unless you don’t care for performance ) I’d be interested to see what splitting this out does for your setup as I’ve never actually benchmarked it before.
Hi @nminchin, no problem on that, but then how much RAM and how much CPU should I give to this “new VM (dedicated to SQL Server)”? And the Ignition VM should remain untouched regardind to resources? Remind my server installed hardware (shown above).
) I’d be interested to see what splitting this out does for your setup as I’ve never actually benchmarked it before.