Java does not return memory to the OS. Ever. That is how it is designed. You configure the RAM Ignition is allowed to use for its heap in ignition.conf. In production, the initial value and maximum value should be the same so that Java gets its allotment immediately.
That allotment should be significantly less than the total in the machine in order to leave RAM for the OS to use, especially for I/O buffers. If installing on a laptop for local development, you must leave a lot of RAM for the OS, your development DB, your graphical desktop, and any designer(s) you might run (which also are java, and use gobs of memory).
If your situation is Ignition using all of its allowed memory, while the OS still has plenty available, then adjust ignition.conf to give it more. You might simply be doing more with Ignition that will fit in its allowance.
For more specific advice, share more specific information. Screen shots of Ignition's performance page and of your OS's memory usage would help. How many tags are defined? How many clients are running? What else is running in the same machine as the gateway?
Include a snippet from ignition.conf to show your setup.