Few minutes ago, I realized that my gateway restarted automatically.
All started at around 09:02 AM.
I tried reading logs and below is what I found :
Any ideas on possible cause ?
cpu/memory seems to be going up before 9 already so these logs arent rly covering enough
if you think its that, then you should try to replicate that query (in a test environment) and see if it happens again.
This is gateway event script running every 5 second. I doubt that database got overloaded or something.
Thanks,
Kaushik
Did someone try to plot 8GB worth of history?
Shoot, I wouldn’t think so. But, probably it got tangled up with something else ?
I am also getting this one once a while.
But, my number of queries are not that high enough and I am not limiting connections either
This happened again last night.
There was a memory spike and the garbage collector is gone nuts since then.
And as always, can’t find anything in the Logs.
Consider disabling one project at a time to see any impact. Note that if this is script-related (and I would judge that likely), you need to have logging in your scripts to have anything useful in your gateway log. At least at the debug level so you can turn them on/off.
@pturmel, This is Production gateway and I can't really disable the project. Other thing is that the frequency of the error. it's not predictable.
Do you think it might could be something with the database ?
Not likely. More likely some script that runs regularly and quickly is encountering a condition that makes it run long with extra memory usage. Without instrumenting your scripts, this sort of thing is very hard to find.
So, in that case, if I make the gateway restart, the memory trend should be the same right ?
Maybe. Depends on the circumstances that trigger the excessive memory usage.
Add loggers to your scripts. Capture timestamps and measure execution time from within the scripts. Collect for a while to determine “normal”. Then you can log/flag/alarm for some multiple of normal.
I know I am late to the party, but what are your settings in the ignition.conf file for initial Java Heap size and Max Heap size? Looks like below
Initial Java Heap Size (in MB)
wrapper.java.initmemory=2048
Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=4096
Initial Java Heap is - 1024 and maximum is - 8192 (8 gbs)
Make them the same. If your application will ever need 8GB, you should allocate it up front, or your OS might drag its feet when you need it.
1 Like
I have run into issues with having them the same. It maxes out the RAM and you don’t know when you are getting close to the max amount unless you are watching a different way. When it is actually using the max amount of RAM and it tries to add the overhead space, then Ignition starts having issues. I went with the recommendation of an IA technical rep and lowered the initial heap size and I have not had any issues since. I was having a problem several times a week with the initial and the max set to the same. But that could be isolated to my environment.
Plus, you need to make sure you limit the MAX so you leave room for the OS to run. (I know that point is pretty obvious but I figured I should point it out anyway) 
Hmm,
to @pturmel, I never knew that when ignition doesn’t need, the memory could possibly used by OS somewhere else. I thought when we allocate Max heap to let say 8GB, ignition always has that 8GB.
No you're telling Ignition that the max it is allowed to use is 8GB.