Hi there,
Since several week we experienced swap memory high usage on our VM which runs the programs.
The IT team report me the machine must be restarted to clean this memory.
Unfortunately it occurs on our main server and our lines runs 24/7 just with two stops (summer holidays, Christmas holidays)
The architecture is: one the top of each line of production we have one IOT edge and then on the top of all the edges we have 2 VMs. One for the programs and another one for the DB.
So I have 2 questions:
- Can I restart the VMs hosting the programs and the DB without inducing a "breakup" on the dataflow ? Normally the edge must store the data and push it once the main server is up again but will the VISION client not be impacted ?
- Why this swap space is used while there is a lot of RAM available ?
If the Vision client is connected to the main gateway, it will be impacted. If the Vision clients are part of Edge Panel, they'll be fine.
Data from the Edge to the main gateway will recover and catch up.
You should investigate whether your VMs are set up properly for Ignition. In particular, you should ensure that the VM for Ignition itself has enough RAM allocated for the max you configured in ignition.conf plus a few gigabytes for the OS to use. The RAM must not be shared with any other VMs (dedicated RAM allocation).
When VM RAM is properly configured, I usually do not enable swap in the gateway's OS at all.
Thank you for the details. By asking to the IT team what has been set in ignition.conf it seems the value is too low (16Go). The server has a capacity of 32Go. So my question: if we reach the 16Go which has been set, if more RAM is needed by Ignition, is it going to take some RAM on the 16Go remaining, or is it going to use the "swap memory" ?
PS: I confirm you, the server which host the Ignition VM has 512Go of RAM. The Ignition VM is not allowed to use more than the 32Go of RAM which has allowed for this VM.
Neither. It will try really hard to do garbage collection to make room (on the java side) for what is immediately needed. If that fails, the gateway will crash.
The settings in ignition.conf for initial and max memory, on a production system, should be the same value, and must be large enough for the worst case workload. The JVM, once it claims memory from the OS, never gives it back. Memory recycling in java is entirely internal to the JVM (aka garbage collection).
A server that runs an Ignition gateway service, and no other service, should be set up with dedicated RAM for the max value in ignition.conf plus ~30%. On VMs, a hypervisor that overcommits RAM will crush Ignition.