I set the JVM max memory in the docker command to be 2048, but when I go to /usr/local/bin/ignition/data/ignition.conf inside the container it shows that the max memory is 1024. Am I looking at the wrong file?
You’re not actually modifying the ignition.conf file with that command. What ends up happening is a temp file is created with any additional arguments (all the stuff after --) and wrapper.java.additional_file gets pointed at that file.
Any wrapper.* arguments are processed directly by the Java Service Wrapper (and will override any settings initially loaded from ignition.conf). Any other arguments are compiled into a temporary file that is referenced by the wrapper.java.additional_file setting and will be passed directly to the JVM–these are properties that you’d otherwise have to add into those numeric-indexed wrapper.java.additional.<n> lines within ignition.conf.
If you're driving the JVM memory settings either by the wrapper argument (as shown above) or our -m runtime argument, it will apply to that launch of the gateway. As discussed, it will not mutate the underlying ignition.conf.
If you're running a container, you should be using a persistent data volume against /usr/local/bin/ignition/data (for our official image). This will enable you to recreate the container with the new configuration (container config is mostly immutable once created, though there are a few aspects you can adjust on Docker Engine with docker update).
The max memory was set with -m 10000 which is ~2gb below the pod resource request of 12000. The JVM restarted 42 times, about every 10 minutes, so we took some corrective actions:
Increased pod limits to be more than the requests, from 12000 memory to 16000 memory, and 2 CPU to 8 CPU.
Updated ignition.conf to have init-memory and max-memory limits to be at 10000 to match -m 10000 configuration.
As a background, I have restored a .gwbk for initial configuration from another gateway, which had a setting of 56000 max memory in the file. Since the change where ignition.conf setting matched -m, it only went above the redline once, (I think thanks to max pod limit increase to 16000 the wrapper didn't send a SIGKILL (9)?) and has been stable since then.