Gateway / Java question 7.7.2

General Discussion
1

I have a general question about the performance of the gateway.

As a background, we’ve had Ignition for a few years and have upgraded our server a couple of times as the number of clients we have has grown (using in-house MES application). We were on a 32 bit version with a couple of CPUs and 4gb of ram.

As we continued expanding use of Ignition, the CPU usage gradually bumped up to where it was hovering in the 70% range with occasional spikes.

We switched to a 64bit/Windows Server 2012 with 2 CPUs and 8 gb of RAM. Within a few days, I noticed that the CPU was moving back up to 60% or so. We had one occurrence of the gateway shutting down. This was during 1st shift, and I and another developer noticed the designer getting really slow just before the system shut down. A quick restart and everything was fine.

Our IT decided to up the server to 4 CPUs and 16gb of memory. Since then, the server is hovering at 20% or so. When this upgrade was done, I found that the max memory setting for the gateway was still set at 1024mb, which is the default from the original install.

This was bumped up to 4gb.

We’re currently running our Ignition MES project across 200 or so clients in 2 facilities (one gateway).

Sorry for the long winded background…

When checking the gateway, I’ve seen the memory usage hovering around 600mb - 1.5gb for days, then today it starts climbing up to near the max of 4gb. No performance issues yet, but it’s likely that the previous setting of 1024mb wasn’t helping.

The memory usage stays near the max for several hours then drops back to <1gb. The number of clients really does not fluctuate.

Is 4gb as a max heap space too low for this setup, could there be other issues causing this much memory usage, or could this just be normal?

I realize that this is a high level description…does anyone have any general suggestions on what to check / where to look in the gateway for more detailed information? Part of the thought process was trying to determine if we have a specific app that is causing a big hit on the gateway due to poor design. Can memory/cpu usage be traced to a specific project/client?

Any help is appreciated.

2

It is normal for Ignition (and java in general) to cycle between low and high memory usage. In a nutshell, java will defer difficult garbage collection tasks until memory pressure makes it necessary. If there is a great deal of garbage objects when that happens, java will stall briefly while it frees memory. If you have tweaked anything else on your system, you are probably still running what is called the Concurrent-Mark-Sweep garbage collection algorithm (standard for java7). A new garbage collection algorithm is now available for production use in java 8 called the Garbage First Garbage Collector, aka G1GC. This new algorithm breaks memory management into much smaller pieces, minimizing stalls when each piece is cleaned up. Stalls are still possible, but in my experience, very rare.

You can also instruct the garbage collector keep a separate log file – this can show the actual stall times and a variety of memory usage statistics.

This ignition.conf excerpt might help you get started:[code]

Java Additional Parameters

#wrapper.java.additional.1=-XX:PermSize=3076m
#wrapper.java.additional.2=-XX:MaxPermSize=4096m
wrapper.java.additional.1=-XX:+UseG1GC
wrapper.java.additional.2=-XX:MaxGCPauseMillis=100
#wrapper.java.additional.4=-Xincgc
wrapper.java.additional.3=-Xloggc:/var/log/ignition/javagc-%WRAPPER_TIME_YYYYMMDDHHIISS%.log
#wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled
#wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.4=-Ddata.dir=/var/lib/ignition/data
wrapper.java.additional.5=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.8=-Xdebug
#wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

wrapper.java.additional.6=-XX:+PrintGCDetails
wrapper.java.additional.7=-XX:+PrintGCTimeStamps
wrapper.java.additional.8=-XX:+PrintGCDateStamps

Initial Java Heap Size (in MB)

wrapper.java.initmemory=1536

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=3072
[/code]Note that the PermSize and MaxPermSize directives are not used with G1GC (and cause problems if set with G1GC).

You might want to consider turning on the GC logging options first, then running for a few days to establish a baseline. Then switching algorithms to see the improvements.

4 Likes
3

[quote=“pturmel”]It is normal for Ignition (and java in general) to cycle between low and high memory usage. In a nutshell, java will defer difficult garbage collection tasks until memory pressure makes it necessary. If there is a great deal of garbage objects when that happens, java will stall briefly while it frees memory. If you have tweaked anything else on your system, you are probably still running what is called the Concurrent-Mark-Sweep garbage collection algorithm (standard for java7). A new garbage collection algorithm is now available for production use in java 8 called the Garbage First Garbage Collector, aka G1GC. This new algorithm breaks memory management into much smaller pieces, minimizing stalls when each piece is cleaned up. Stalls are still possible, but in my experience, very rare.

You can also instruct the garbage collector keep a separate log file – this can show the actual stall times and a variety of memory usage statistics.

This ignition.conf excerpt might help you get started:[code]

Java Additional Parameters

#wrapper.java.additional.1=-XX:PermSize=3076m
#wrapper.java.additional.2=-XX:MaxPermSize=4096m
wrapper.java.additional.1=-XX:+UseG1GC
wrapper.java.additional.2=-XX:MaxGCPauseMillis=100
#wrapper.java.additional.4=-Xincgc
wrapper.java.additional.3=-Xloggc:/var/log/ignition/javagc-%WRAPPER_TIME_YYYYMMDDHHIISS%.log
#wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled
#wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.4=-Ddata.dir=/var/lib/ignition/data
wrapper.java.additional.5=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.8=-Xdebug
#wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

wrapper.java.additional.6=-XX:+PrintGCDetails
wrapper.java.additional.7=-XX:+PrintGCTimeStamps
wrapper.java.additional.8=-XX:+PrintGCDateStamps

Initial Java Heap Size (in MB)

wrapper.java.initmemory=1536

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=3072
[/code]Note that the PermSize and MaxPermSize directives are not used with G1GC (and cause problems if set with G1GC).

You might want to consider turning on the GC logging options first, then running for a few days to establish a baseline. Then switching algorithms to see the improvements.[/quote]

Phil, this is what we have currently, I believe the only thing that has been messed with is the Max heap size:

# Java Additional Parameters
wrapper.java.additional.1=-XX:PermSize=64m
wrapper.java.additional.2=-XX:MaxPermSize=128m
wrapper.java.additional.3=-XX:+UseConcMarkSweepGC
wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.6=-Ddata.dir=data
wrapper.java.additional.7=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.8=-Xdebug
#wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=3072

# Application parameters.  Add parameters as needed starting from 1
wrapper.app.parameter.1=com.inductiveautomation.catapult.Catapult

wrapper.shutdown.timeout=120
wrapper.jvm_exit.timeout=120

How can I instruct GC to keep a separate log file? you’ve been using G1 for a while right? how do you feel about it compared to CMS GC?
Would any tweak to what I have right now could help with smoother GC process? I see your Perm and MaxPerm values are considerably higher than the default.
I’d really like to explore every option so hopefully I can find the best way to deal with the clockDrift issue.
Thanks

4

[quote=“Mr.K001”]How can I instruct GC to keep a separate log file?[/quote]See wrapper.java.additional.3, .6, .7, and .8 in my config.[quote=“Mr.K001”]You’ve been using G1 for a while right? how do you feel about it compared to CMS GC?[/quote]Yes, I been using G1GC since it was introduced in Java7. The performance gain is so great, I wouldn’t use anything else.[quote=“Mr.K001”]Would any tweak to what I have right now could help with smoother GC process? I see your Perm and MaxPerm values are considerably higher than the default.[/quote]No, my Perm and MaxPerm are commented out. You should not have them set at all, as I stated immediately after my suggested config.

1 Like
5

[quote=“pturmel”]See wrapper.java.additional.3, .6, .7, and .8 in my config.[/quote]Thank you, I will try that

[quote=“pturmel”]Yes, I been using G1GC since it was introduced in Java7. The performance gain is so great, I wouldn’t use anything else.[/quote]That’s what I keep seeing online, I wouldn’t mind switching at some point. I assume the way your code is set up will switch to G1GC correct? I mean if I decide to switch to G1 is that all I need to change?

[quote=“pturmel”]No, my Perm and MaxPerm are commented out. You should not have them set at all, as I stated immediately after my suggested config.[/quote]My question on this was about the CMS GC not the G1. I was asking whether or not tweaking those two values will have any positive impact.

6

[quote=“Mr.K001”][quote=“pturmel”]Yes, I been using G1GC since it was introduced in Java7. The performance gain is so great, I wouldn’t use anything else.[/quote]That’s what I keep seeing online, I wouldn’t mind switching at some point. I assume the way your code is set up will switch to G1GC correct? I mean if I decide to switch to G1 is that all I need to change?[/quote]Yes. Please take a look at the manual page for the java command. It explains how all this works. The UseG1GC option is required in Java7. It is already the default in Java8.[quote=“Mr.K001”][quote=“pturmel”]No, my Perm and MaxPerm are commented out. You should not have them set at all, as I stated immediately after my suggested config.[/quote]My question on this was about the CMS GC not the G1. I was asking whether or not tweaking those two values will have any positive impact.[/quote]Yes, but they are somewhat sensitive to your actual workload. Typically the MaxPerm would be 1/2 to 2/3 of the max heap size, but must be at least large enough for all of Ignition’s long-lived objects. The “Memory” tab of Ignition’s system console page shows the breakdown of permanent vs. non-permanent memory consumption. I haven’t used CMS in a long while, so maybe someone else can chip in with advice. { My advice is to switch to the G1GC. }

7

Hi, had a similar problem and just update to JAVA 8 update 144; the problem has gone thanks for the tips :slight_smile:

8

I have java 11
Ignition 8.1.0

I observed a stall I think.
Yesterday we had some high performance, and a clockdriftdetection before some issues.

Do you still get the issues?
How often are you restarting your gateway?

9

You don’t need Java Installed for Ignition 8.

Read this article.
ClockDriftDetector warnings in the Gateway Console – Inductive Automation Help Center

10

Thanks

I still wonder how often they are restarting.
After the restart, our memory trend showed a third of the load it had either side of the stall we had.

I like the comment in the article "In the image above, the Task Manager is reporting memory usage of over 90%. This system does not have enough RAM to run all of the processes it is running. "

11

Hi Phil,

I created a post similar to this one, as I think I have the same problem. Please could you take a look at it and confirm if my problem is related to this, is that configuration the recommended one to avoid high CPU performance?