Java has garbage collection, it will free memory back up by removing garbage (=unreferenced objects) and reuse it. Some memory is easier to remove than other memory (f.e. try cleaning up memory where everything still has a circular reference to itself, but isn’t referenced from the main thread anymore).
This is why, by default, Java will do the easy garbage collections (f.e. the big amount of variables with a very short lifespan) quite often, certainly every time the memory needed reaches the current memory allocated to Java. But it will wait with the hard garbage collections until it absolutely has to do those.
If you allow the process to reach 4 GB, while you actually only need 1 GB or less at any time, there will be 3 GB or more garbage laying around to be cleaned up. As it’s the kind of garbage that takes a lot of work to get cleaned up, this will stop the main thread for quite a while during the cleaning. It gets even worse when the total RAM of the computer is too low to give 4 GB to Java. As then most of that 3 GB of garbage will be written to the HDD (as Windows prefers to keep the RAM filled with stuff that often gets accessed). So the process will need to load 3 GB of garbage from the HDD into RAM, just to remove it. All of that will happen while all Java threads are halted.
So while it’s possible there’s a memory leak, it would be best to first evaluate your memory needs, and find a sweet spot between enough memory to have most pages cached, and not too much memory to have more often garbage collections.
If that doesn’t help, you can also look into alternative garbage collection algorithms. The G1-GC algorithm f.e. does garbage collection in small batches. So it normally doesn’t even reach complete memory usage, and if it does, the pauses won’t be that long.
Only when those two things failed, it might be worth to look into memory leaks. Memory leaks are very hard to find, partially due to the time it takes for them to build up.