# Gateway Performance Issues and Tuning **URL:** \ **Category:** Ignition\ **Tags:** ignition81, perspective\ **Created:** [November 30, 2022, 8:27pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109 "2022-11-30T20:27:05Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![bschroeder](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bschroeder/32/13260_2.png) [@bschroeder](https://forum.inductiveautomation.com/u/bschroeder)\ **Post date:** [November 30, 2022, 8:27pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/1 "2022-11-30T20:27:05Z")
I've got a gateway that our company is maintaining for a customer, and we are trying to track down a performance issue and I need some advice on where to go. This is a gateway that is a perspective and webdev system. This gateway is a central gateway in the customer's data center that is communicating to multiple remote smaller sites via the gateway network. Some data is being synced across the gateway network using gateway timer scripts. The issue is that every so often the CPU usage in Ignition will spike, memory will spike and ClockDriftDetector warnings will occur. Usually, the GC will be able to drop that memory down, however, in extreme situations, the java heap seems to fill up and then things start to crash. I'm trying to isolate the cause, or more specifically prove where it isn't. For instance, this is a cut from the performance page: ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/1/a/1aa839e6fea3135e547ef5ed54e6cf4eaf9e2ac8.png) This gateway is installed on Windows Server 2016, in a Hyper-V environment. I don't have access to the Hyper-V Host side of things so I can't verify that end. I've been told that all the resources are properly allocated and it isn't overcommitted. The only software on the guest is Ignition, Windows and Carbon Black Cloud Sensor Anti-virus. Currently, the java heap is set for 8GB. Where do I start looking into things to try and start to get this working better? Is this a simple memory issue? Or possibly something else? Thanks in advance... ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 30, 2022, 8:36pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/2 "2022-11-30T20:36:29Z")
Bit of an odd memory usage pattern. Typical is a sawtooth. A healthy sawtooth tends to ramp from 40%-ish (or less) to 80%-ish, then GC knocks it back down. Except. Long time spans for requested graphs, especially in the reporting module, randomly spike memory. A couple or more such requests together could push a server over the edge. Perspective also loads a server much more than Vision, so a bunch of busy clients can do similar. Consider this similar topic: > [@Ignition performance issues](https://forum.inductiveautomation.com/t/ignition-performance-issues/66887/9): > > No more simple ideas. Set up Java Flight Recorder to run all of the time, and save on JVM exit. Set up a recurring OS task to run the Windows equivalent of jcmd $(pgrep java) GC.heap\_dump someFileName-$(date +%Y%m%d-%H%M).hprof fairly often for later perusal with the flight recording. You might want support to help you set this up. Maybe also set up GC detail logging in ignition.conf. See how long your pauses usually are. ---
**Author:** ![bschroeder](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bschroeder/32/13260_2.png) [@bschroeder](https://forum.inductiveautomation.com/u/bschroeder)\ **Post date:** [November 30, 2022, 8:41pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/3 "2022-11-30T20:41:36Z")
Yeah... that is something I recognized as well on the memory. Specifically, on this gateway there are not connected devices etc... This is used as a clearing house for remote data monitoring and then handling specific truck ticketing located at remote sites. Do you have any recommended sites so I can lear how to setup Java Flight Recorder? And any specific flags to add to the ignition.conf to get the GC detail logging? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 30, 2022, 8:54pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/4 "2022-11-30T20:54:13Z")
I happen to be using it right now on one of my VMs for driver testing, with this in `ignition.conf`. ```auto wrapper.java.additional.9=-XX:+FlightRecorder wrapper.java.additional.10=-XX:StartFlightRecording=maxsize=2g,filename=porthos81z.jfr ``` 2GB is probably too big. (: And you probably want the `dumponexit` option. Some [docs here](https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm). Note that the `UnlockCommercialFeatures` clause went away at some point. Follow the links for the `jcmd` docs for things you can do while running, including getting early exports from JFR. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 30, 2022, 8:56pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/5 "2022-11-30T20:56:03Z")
For the GC reporting, I use this: ```auto wrapper.java.additional.3=-Xloggc:/var/log/ignition/javagc-%WRAPPER_TIME_YYYYMMDDHHIISS%.log wrapper.java.additional.6=-XX:+PrintGCDetails ``` Adjust that file name pattern to suit your needs. As shown, you'll need to clean them out manually. ---
**Author:** ![bschroeder](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bschroeder/32/13260_2.png) [@bschroeder](https://forum.inductiveautomation.com/u/bschroeder)\ **Post date:** [November 30, 2022, 8:59pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/6 "2022-11-30T20:59:21Z")
Thanks! I've got a local gateway in our office I can play with to try and learn how to decipher the data. Do you think that maybe there just isn't enough memory assigned? I can add quite a bit more to the gateway., I just don't want to go that route and just cover something else up. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 30, 2022, 8:59pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/7 "2022-11-30T20:59:42Z")
To look at the files you get, you will want "JDK Mission Control" and/or "VisualVM". I like the latter for looking at HPROF dumps. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 30, 2022, 9:21pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/8 "2022-11-30T21:21:43Z")
> [@bschroeder](#): > > maybe there just isn't enough memory assigned? It's possible. I'd look at the GC details first. And if you get another crash after JFR is set up, look at that, too. ---
**Author:** ![bschroeder](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bschroeder/32/13260_2.png) [@bschroeder](https://forum.inductiveautomation.com/u/bschroeder)\ **Post date:** [November 30, 2022, 9:23pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/9 "2022-11-30T21:23:54Z")
Thanks. I also reached out to support to see if they have any ideas as well. ---
**Author:** ![bschroeder](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bschroeder/32/13260_2.png) [@bschroeder](https://forum.inductiveautomation.com/u/bschroeder)\ **Post date:** [November 30, 2022, 9:24pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/10 "2022-11-30T21:24:51Z")
Any good tools for examining the GC logs? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 30, 2022, 9:25pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/11 "2022-11-30T21:25:49Z")
The GC dumps are the HPROF "heap profile" files I mention above. The GC logs are just text files. I use `grep` to pick out the pauses. ---
**Author:** ![bschroeder](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bschroeder/32/13260_2.png) [@bschroeder](https://forum.inductiveautomation.com/u/bschroeder)\ **Post date:** [November 30, 2022, 10:19pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/12 "2022-11-30T22:19:40Z")
Thanks. Anything in particular I'm looking for the heap dumps? I finally got my local VM working correctly and VisualVM is working. I can dump the heaps and take a look locally. This isn't the same as the problem child, but at least I can get a bit of familiarity with what I'm seeing. And I can get familiar with the process as well. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [December 1, 2022, 12:35am UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/13 "2022-12-01T00:35:51Z")
When there's a memory leak, something usually sticks out like a sore thumb. I'm usually focused on my own stuff, looking at allocations by class. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [February 7, 2026, 4:10pm UTC](https://forum.inductiveautomation.com/t/gateway-performance-issues-and-tuning/67109/14 "2026-02-07T16:10:45Z")
2 posts were split to a new topic: [Document Manager and Quick Buffer Errors](https://forum.inductiveautomation.com/t/document-manager-and-quick-buffer-errors/113831)