Our memory usage has increased to the point of causing clock drift events/gateway restarts after upgrading from 7.8.5 to 7.9.13. I have done a some trimming by uninstalling modules that were in a trial version, and other modules that I knew were unused. I’m really unsure where to start on troubleshooting the issue, but I’d add that all of our vision/designer clients combined are using about 500MB consistently. Also I tried changing the garbage collection method to G1GC but it didn’t make any impact. Our overall usage is up to 5.7/6.0GB, from 3.8/4.0GB. Add to that the clock drifts, and I believe we have a memory leak. Attached is a screenshot of all of our modules, as well as a thread dump during the high memory usage event. Let me know if there is anything else I can add that might assist in troubleshooting.
ThreadDump.txt (50.6 KB)
I tried posting the thread dump as a block text, but it hit the max character limit for the body of the post.
Figured I’d post a reply in case anyone stumbles across this thread. Our issue ended up not being related to Ignition. Our third party Data Archival system had about 30k tags set to expire on the 1/1/2020. This software resides on the same server as Ignition, and began eating up resources that Ignition needed. Moral of the story is that VM’s are cheap, and you should always run Ignition on its own standalone VM.
Or better yet, its own hardware. VMs are too often under-provisioned or running on hardware hosting too many other VMs.
I completely agree Kevin, and if I had my way then it would. However, it takes an act of congress to get our infrastructure group to place a physical device.