How long does it take for the ram to grow? Is it a matter of minutes or a matter of days?
I have to say I think 8 GB is quite a lot to assign when you only need 200-300 MB typically. Java has a tendency to build up RAM usage until it really needs to collect the garbage. If there’s a lot of garbage to collect, it might put extra stress on the system. Certainly if parts of the old data were already moved to the slow pagefile.
If the memory grows over a long time, and then crashes, it might be worth to lower the maximum memory and check what happens. 1GB should be plenty if it typically only needs 200-300 MB. You may also want to look into better garbage collectors, like the G1GC.
If it builds up in a matter of minutes, then there’s probably a real bug somewhere in a script or a configuration. The first thing I would do would be symptom hunting: check which parts you can disable on a test server until it doesn’t happen again.
If that doesn’t help, or if the RAM issues are too infrequent to know when you solved it, attaching a debugger is probably the only way to go, though I don’t know about good ones for Java.