Memoryleak, how to find and fix?

Ignition
1

Hi, in one of our customers ignition installation, they seem to run out of memory, causing production stop.
Last time, they had a stop for 188 minutes because the server of ignition (VM:ed) runs out of ram.

How to find what causes (most likely) ignition to run out of available ram?

Thinking on another thing, i am not sure if the memory problem only exists if the clienmt is running, or if its still exists if the gateway only runs (on the same machine).

Would an idle-exit function be viable for the client? Lets say, if it has not been used(how to determine?) for X hours, then exit.

I was thinking, a dirty fix, would be schedule the gateway restart every night.

EDIT: Other inputs. We use a couple of gateway scripts, the plc requests information for item XYZ, then the script loads the information from SQL database and puts into the tags the plc wants information in, and then a bit is set that tells the plc, data available & ready.
In my understanding, i dont need to .dispose() (C# reference) or set anything “= null/nothing” since java should fix that by it self.

EDIT2: Atm i cannot connect to the server vpn/remote desktop and cannot verify, but iirc, the server allows either 2 or 4 GB of ram for the Win 2012 Server installation, is there some Java settings i should change so the server wont run out of ram and become slow?

2

There’s no simple answer here. But there are several things you should check.

  1. Look in your Ignition server’s ignition.conf file at the memory settings. The maximum memory setting there must be less than your server VM’s allocated memory, minus the memory used by that server’s OS and other applications. If Ignition’s max memory plus all of the other memory used in your server is greater than the physical memory allocated to that VM, your system will almost certainly “thrash” with swap.
  2. Any designer or client run directly on the Ignition server counts as an “application”. I would absolutely forbid running any user applications on an Ignition server that is critical for production.
  3. Monitor your Ignition server’s Heap Memory Usage. Normal behavior is for memory consumption to slowly climb over a few to several minutes, approaching 90+ % of max, then drop due to garbage collection. And then begin ramping again. In my experience, the bottom of that drop should be less than 50%. If your server stays above 50% all of the time, give Ignition more memory, even if you must upgrade your hardware. If Ignition runs with a normal ramp and drop action, but the “bottom” after GC creeps up over time, you have a memory leak.
3

[quote=“pturmel”]There’s no simple answer here. But there are several things you should check.

  1. Look in your Ignition server’s ignition.conf file at the memory settings. The maximum memory setting there must be less than your server VM’s allocated memory, minus the memory used by that server’s OS and other applications. If Ignition’s max memory plus all of the other memory used in your server is greater than the physical memory allocated to that VM, your system will almost certainly “thrash” with swap.
  2. Any designer or client run directly on the Ignition server counts as an “application”. I would absolutely forbid running any user applications on an Ignition server that is critical for production.
  3. Monitor your Ignition server’s Heap Memory Usage. Normal behavior is for memory consumption to slowly climb over a few to several minutes, approaching 90+ % of max, then drop due to garbage collection. And then begin ramping again. In my experience, the bottom of that drop should be less than 50%. If your server stays above 50% all of the time, give Ignition more memory, even if you must upgrade your hardware. If Ignition runs with a normal ramp and drop action, but the “bottom” after GC creeps up over time, you have a memory leak.[/quote]

1)
The server is assigned 8GB of ram and around 3.5-4GB is used currently (incl. java ~858MB)

[code]# Java Additional Parameters
wrapper.java.additional.1=-XX:+UseConcMarkSweepGC
wrapper.java.additional.2=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.3=-Ddata.dir=data
wrapper.java.additional.4=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.5=-Xdebug
#wrapper.java.additional.6=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

Initial Java Heap Size (in MB)

wrapper.java.initmemory=1024

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=2048[/code]

2)
There is a possible chance that i have let the client running when i logged of. But if the client it self do a memory leak, should not the client do that the same out in the factory where it should run 24/7?
But after the restart of the server, the client is not started in the server, and should not cause memory usage.

3)
I tossed together an application to log the heap memory of the java.exe and store it into a CSV file, i am going to let it run for a couple days or so, and then analyze how the memory behaves for that process.

Another note; They operators claims that when the server is restarted, the client never re-connects, but as far as i understand it, the client should always try reconnect infinity, right?
According to description, they say they have to restart the gateway server, then the plcs, then the clients, so get it up and running, which sounds wierd to me.
The gateway is connected by opc to the siemens s7-1200 plc, and the connections are initiated from the gateway.

EDIT: Also, how do the gateway handle the scripts? when it has finished, will it be nulled/garbage collectable? Or do the script them self need to handle disposing created variables in the scripts?

4

[quote=“StefanGronberg”]1)
The server is assigned 8GB of ram and around 3.5-4GB is used currently (incl. java ~858MB)[/quote]OK. Sounds reasonable to start. But, you have:wrapper.java.maxmemory=2048in your configuration. Java won’t use any more than 2GB, even if the complexity of your gateway needs it. That might be contributing to your problem.[quote=“StefanGronberg”]2)
There is a possible chance that i have let the client running when i logged of.[/quote]When I wrote [quote=“pturmel”]absolutely forbid running any user applications on an Ignition server that is critical for production[/quote]that means YOU, too.
Run your designer and any test clients across your network. NO ONE should ever run a client or designer directly on a production server.[quote=“StefanGronberg”]but if the client it self do a memory leak, …[/quote]Correct. A memory leak in your client will not use memory on your server.[quote=“StefanGronberg”]3)
I tossed together an application to log the heap memory of the java.exe and store it into a CSV file, i am going to let it run for a couple days or so, and then analyze how the memory behaves for that process.[/quote]Heh. You could have just turned on tag history for the memory stats in [System]Performance. Then you could watch a realtime chart from your office. (-:

[quote=“StefanGronberg”]They operators claims that when the server is restarted, the client never re-connects, but as far as i understand it, the client should always try reconnect infinity, right?[/quote]Yes. Sounds like a separate problem. You’ll have to look at your Ignition logs after it starts up and while the clients are failing to connect.[quote=“StefanGronberg”]Or do the script them self need to handle disposing created variables in the scripts?[/quote]No, jython will garbage collect your script variables. Java and Jython don’t even have an analog to .dispose()