[quote=“pturmel”]It is normal for Ignition (and java in general) to cycle between low and high memory usage. In a nutshell, java will defer difficult garbage collection tasks until memory pressure makes it necessary. If there is a great deal of garbage objects when that happens, java will stall briefly while it frees memory. If you have tweaked anything else on your system, you are probably still running what is called the Concurrent-Mark-Sweep garbage collection algorithm (standard for java7). A new garbage collection algorithm is now available for production use in java 8 called the Garbage First Garbage Collector, aka G1GC. This new algorithm breaks memory management into much smaller pieces, minimizing stalls when each piece is cleaned up. Stalls are still possible, but in my experience, very rare.
You can also instruct the garbage collector keep a separate log file – this can show the actual stall times and a variety of memory usage statistics.
This ignition.conf excerpt might help you get started:[code]
Java Additional Parameters
#wrapper.java.additional.1=-XX:PermSize=3076m
#wrapper.java.additional.2=-XX:MaxPermSize=4096m
wrapper.java.additional.1=-XX:+UseG1GC
wrapper.java.additional.2=-XX:MaxGCPauseMillis=100
#wrapper.java.additional.4=-Xincgc
wrapper.java.additional.3=-Xloggc:/var/log/ignition/javagc-%WRAPPER_TIME_YYYYMMDDHHIISS%.log
#wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled
#wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.4=-Ddata.dir=/var/lib/ignition/data
wrapper.java.additional.5=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.8=-Xdebug
#wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
wrapper.java.additional.6=-XX:+PrintGCDetails
wrapper.java.additional.7=-XX:+PrintGCTimeStamps
wrapper.java.additional.8=-XX:+PrintGCDateStamps
Initial Java Heap Size (in MB)
wrapper.java.initmemory=1536
Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=3072
[/code]Note that the PermSize and MaxPermSize directives are not used with G1GC (and cause problems if set with G1GC).
You might want to consider turning on the GC logging options first, then running for a few days to establish a baseline. Then switching algorithms to see the improvements.[/quote]
Phil, this is what we have currently, I believe the only thing that has been messed with is the Max heap size:
# Java Additional Parameters
wrapper.java.additional.1=-XX:PermSize=64m
wrapper.java.additional.2=-XX:MaxPermSize=128m
wrapper.java.additional.3=-XX:+UseConcMarkSweepGC
wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.6=-Ddata.dir=data
wrapper.java.additional.7=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.8=-Xdebug
#wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=3072
# Application parameters. Add parameters as needed starting from 1
wrapper.app.parameter.1=com.inductiveautomation.catapult.Catapult
wrapper.shutdown.timeout=120
wrapper.jvm_exit.timeout=120How can I instruct GC to keep a separate log file? you’ve been using G1 for a while right? how do you feel about it compared to CMS GC?
Would any tweak to what I have right now could help with smoother GC process? I see your Perm and MaxPerm values are considerably higher than the default.
I’d really like to explore every option so hopefully I can find the best way to deal with the clockDrift issue.
Thanks