[TUTORIAL] How to Debug High RAM Usage

[flexware]

Hey, everyone!

Has anyone ever had the issue where you look at your RAM trend on your gateway and it is either constantly increasing over time or it is just in general higher than you would expect? Well, I had a gateway that in general was using more RAM than I thought it should, and I wanted to understand why and how to lower it. While high RAM isn’t necessarily an issue if it is within the server specs, it certainly is something you will want to optimize early on if you are looking to scale.

While debugging, I have seen several different forum posts about this topic, each with different pieces of information that helped in debugging this issue. These were instrumental in me being able to get this solved, but I did want to create a tutorial that aggregates this info into one spot. Hopefully, someone besides future me will find this helpful.

Disclaimer: In this tutorial I am using Windows. I am sure that a similar flow works in Linux too, but that is not what I am most familiar with. I will also be using VisualVM and JMap. These are just the tools I selected, and there are others as well. If you have used other cool tools for this let me know!

Step 1: Download a JDK

In order to see what is causing the high RAM usage, we will need to get a heap dump. The heap is a memory area that is allocated to each program and is used for short term data storage.

To get a heap dump from Java (which is what Ignition runs on) we will need a JDK, or a Java Development Kit. To get one:

1. Sign into the server that is hosting Ignition.
2. Navigate to Java Downloads | Oracle (Note: I am using JDK24 since it is the newest at this time. Just use whatever is the most recent at the time)
3. Select the installer for your OS. Since I am on windows, I will select the below

4. This will download an executable. Run the executable.
5. Click through all the prompts. I just left everything at default. On the last screen, do not worry about the “Next Steps” and hit Close.

Step 2: Running JMap

WARNING: This step will lock your gateway for part of the time that it is running, and your system will be unusable. In my experience this lasts anywhere from a couple of seconds to several minutes. DO NOT RUN THIS UNLESS YOU HAVE SCHEDULED DOWNTIME.

The reason we had to download the JDK was because we needed access to JMap. JMap is a Java tool that allows for taking heap dumps, which is exactly what we want to do. To take the heap dump:

1. Open a Command Line, making sure to Run as Administrator.
2. In CMD, navigate to the folder where your JDK was installed, and then into the bin folder. Assuming JDK24 and you left everything default, the command should be

cd C:\Program Files\Java\jdk-24\bin

3. We need to find the process id of Java. There are some command line ways to do this, but my favorite way to do this is to:
   a. Open Task Manager
   b. Go to the Details tab
   c. Look for java.exe (not IgnitionGateway.exe… Only got stuck on that for an hour or so )
   d. Your process id is the number in the PID column of that row. So, my process id is 4196 in this case.

4. Back in command line, run the following command:

jmap -dump:file=<C:\path\to\heap.bin>

As an example, I used the below command

jmap -dump:file=C:\dump\heap.bin 4196

One note, this command will not auto create the folders in the path. So, I had to create the dump folder before running this command.

5. If everything is successful, then you should get a message like the below and have a .bin file in the folder you specified.

Step 3: Download VisualVM

VisualVM is a tool that will allow us to analyze the heap dump generated in Step 2. There are a lot of tools that can do this, but this is just the one I happened to choose and will use for this tutorial. To use VisualVM:

1. Go to VisualVM: Download
2. Click the VisualVM 2.2 link to download the zip file.
3. Extract the downloaded zip file.
4. In the extracted folder, find the visualvm.exe file in the bin folder.
5. Run the visualvm.exe file and accept the license agreement (make sure you read the whole thing of course).

Step 4: Inspecting the Heap Dump

Now that VisualVM is running, it is time to put our detective hat on . We need to inspect the heap file to figure out what is causing high RAM, and this can be kind of challenging. What you will see is we get clues of what the problem may be, but not the whole picture. I will walk through an example though of a common thing you will find. But first, let’s look at the heap.

1. In Visual VM, click on File > Load…
2. Select the .bin file we exported in Step 2. This will take a little bit of time to load. When it is done, you should see something like the below:

3. From here there are a lot of things you can do, but my recommendation is to switch from Summary to Objects:

4. In the table you see here, make sure it is sorted by size (should be by default). Your top item will probably be byte[].
5. Expand the top line. Once this is done computing, you should get a bunch more sub lines. These lines are the clues to what is causing the high ram. The top items will be the heaviest hitters. To see more info about them, click on the first item and select Preview. This will open a window at the bottom.

6. In this bottom view, look at the text on the right side. This is our clue. As you can see, this doesn’t tell us what the issue is completely, but it gives us a hint. From this I can tell that the issue must be caused by some custom property called buildLotDS. So, we then go into debugging mode.

Step 5: Fixing the Problem

Now that we have a clue, we need to both find the issue and then fix it. This part is the most ad hoc, and you kind of just have to experiment. To find where this hint is in the system, I recommend opening every project in the designer and utilizing the Edit > Find/Replace tool to search for the keywords in your hint. Make sure that when doing this search you change the dropdowns to search everything and not just open items:

I will say, in this issue, and in most cases that I have looked at, the issue is that in the designer a large default value was saved into a custom property. In this case, that buildLotDS default value in the designer was a dictionary with 200+ items. To fix the issue I just changed the binding default value to be something substantially lower than it was before. In my case, this alone took my RAM usage from hovering around 6GB to 4GB.

Conclusion

We have now successfully gotten the clues to help us debug our high RAM issues and can start the investigation process. Let me know if this is helpful. And feel free to let me know if there are any other ways anyone has debugged this issue.