I have seen an increase in RAM usage all of a sudden with no apparent reason why. We used to be only using 1gig at the peak and at most 2gig. Now I see 3.5 at the peaks and it was like a switch was flipped and the usage increased around 11:45 yet no changes to projects had been made in this time frame.
Are there good ways to see what projects are eating up the most RAM or what kind of of tools are there in Ignition to track down what is using the most recourses.
I wish there were better tools built into Ignition to see what's causing the sudden CPU and memory spikes. I was troubleshooting performance issues over the course of many months to see why I would get random 100% CPU/memory usage that would cause the gateway to crash and ran into this same problem, where there is limited visibility into what causes those jumps.
What might be a likely culprit or something to check are the open Perspective sessions and running scripts - the spike might be due to a large number of users logging in for the day, for example. This wouldn't fully explain why this would occur, based on what you describe. I would also check gateway and timer scripts, especially if they're run at an interval.
What you can do in the meantime (what I did) was to historize a lot of the System tags (CPU usage, memory, sessions, etc) and also enable audit logging (or, in general, as much logging as you can) so that when the spikes happen, you can go back and look at tag history and audit logs. This is far from ideal, though.
The gateway is just a regular Java app, so there's lots of off the shelf tools that do a better job providing low level insights than we ever could. We're also restricted to running 'inside' the process, so there's lots of stuff we simply can't do.
One thing you can opt in to is generating a heap dump with a diagnostics bundle:
https://docs.inductiveautomation.com/display/DOC81/Gateway+Configuration+File+Reference#GatewayConfigurationFileReference-GatewayDiagnosticBundleExportMemoryDump
A heap dump is the ultimate tool in terms of analyzing what's using up memory.
Care with heap dumps though, as they will pause the jvm and hence ignition while it dumps the memory. For a large project, this can be in the 10s of minutes. For large projects also, you'll need to increase the ping timeout in the ignition.conf otherwise the jvm will reboot due to it being paused for too long