We are using version 8.1.28, with a server configuration of 16 cores and 48GB of memory, running on VMware with no other software running except Ignition.
Yesterday, we encountered a issue where the gateway memory was exhausted. We tried restarting, but within a few minutes, all the memory was used up again. We even restarted the server once, but after recovery, it continued to consume all the memory. We reverted all the changes, but the problem persisted. After leaving the program closed for an hour, restarting the gateway restored normal operation.
The first occurrence of the problem was at 21:39:45 (wrapper.log.3 Line 131464), which triggered the first FullGC and a clock drift of 7833ms. After that, the gateway was basically unresponsive. At 21:50:44, we stopped the Ignition service.
At 21:56:41, we attempted to restart it (wrapper.log.1 Line 3406), but four minutes later, all the memory was filled again (wrapper.log.1 Line 5191). The gateway automatically restarted at 22:12:34 (wrapper.log.1 Line 42136), but at 22:16:43, the same problem occurred. After a few more similar incidents, we shut down the service. After more than an hour, when we restarted it, everything returned to normal.
How should we troubleshoot this issue?Thanks.
WrapperLog.zip (1.2 MB)
You've got all those flight recording and remote debugging arguments enabled... did you collect any heap dumps while it was in this state?
It's close to impossible to debug a memory leak without one.
Sorry to forget attach these JFR files. When the memory was exhausted, I tried to dump the JFR manually, but there was no response. These are JFR files exported before the memory was exhausted. I hope this helps.
FlightRecording.zip (4.2 MB)
flight_recording_localhost1089_3_2024-04-02_22-27-36_b95c2009.zip (1.9 MB)
This is the JFR file exported when the issue occurs. I tried to open it with JMC and JCMD, but both attempts failed. Did I do something wrong?
org.openjdk.jmc.flightrecorder.internal.InvalidJfrFileException: No readable chunks in recording
at org.openjdk.jmc.flightrecorder.internal.FlightRecordingLoader.readChunks(FlightRecordingLoader.java:248)
at org.openjdk.jmc.flightrecorder.internal.FlightRecordingLoader.loadStream(FlightRecordingLoader.java:96)
at org.openjdk.jmc.flightrecorder.internal.FlightRecordingLoader.loadStream(FlightRecordingLoader.java:77)
at org.openjdk.jmc.flightrecorder.ui.RecordingLoader.doCreateRecording(RecordingLoader.java:173)
at org.openjdk.jmc.flightrecorder.ui.RecordingLoader.run(RecordingLoader.java:97)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Have you opened a support ticket? Relying on Paul checking the forum is not a good plan, nor fair to him. I doubt any volunteer here with the knowledge to review your files is going to spend the hours examining them in detail. IA support can do that.
Also, you need a heap dump to debug memory leaks. A flight recording is not a heap dump. You should use jcmd to obtain one when your memory usage hits ~90%.
1 Like
I have submitted a support ticket, but I have not received reply yet, so I am trying my luck in the forum.
I'm wondering if I can see what is in the Old region in the flight recording.
That is a heap region. For which you need a heap dump.
1 Like
