Ok, then 5GB would be too low. The idea is that garbage collecting 8.6GB (11GB => 2.4GB) takes time. If the max limit is lower, GC is going to happen earlier and more often, with a smaller amount collected. Hopefully faster. Java mostly garbage collects just the easiest stuff until memory pressure makes it collect more difficult stuff. (That's the peak of the sawtooth.)
If you can't get the a max below 11GB to yield a faster, smaller sawtooth (with shorter pause the world), then you will need better hardware. Especially more cores.