I’m using Ignition 7.9 but this problem is a little old.
I have a serious problem with my gateway. Suddenly, the CPU usage increments and everything in my server falls down, network, file and print, things like that. I need to restart the Gateway and the CPU usage decrements again.
This is the detail of the thread that uses a lot of cpu:
Try increasing your initial heap size to 2048. then restart. setting the initial heap to 2gb will allocate more memory to ignition at the start. Give this a shot.
In addition, the particular thread you highlighted is trying to generate the stacktrace of an error coming from SQL Server. Is your database running on the same physical machine/VM instance as Ignition? If so, I would recommend moving it to a different system to avoid a conflict of resources. How many queries / second are you executing against the DB server, and how many connections are you using at a time? (by default, Ignition allocates 8 connection threads)
The Ignition gateway and the database server are in diferent machines. We are currentle running at 200 - 300 qrys/ sec. I attachen a screenshot of the values of our Ignition Server. The graphic has data logger from tags of gateway performance.
As you can see, nowadays is very stable, but in every minute the CPU load goes to 80% - 90% and the exception that I posted before comes out.
This afternoon, my gateway used a lot of CPU and I restarted again.
This is the current configuration of the ignition.conf file:
#encoding=UTF-8
# Configuration files must begin with a line specifying the encoding
# of the the file.
#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
# Professional and Standard Editions of the Wrapper require a valid
# License Key to start. Licenses can be purchased or a trial license
# requested on the following pages:
# http://wrapper.tanukisoftware.com/purchase
# http://wrapper.tanukisoftware.com/trial
# Include file problems can be debugged by removing the first '#'
# from the following line:
##include.debug
# The Wrapper will look for either of the following optional files for a
# valid License Key. License Key properties can optionally be included
# directly in this configuration file.
#include ./wrapper-license.conf
# The following property will output information about which License Key(s)
# are being found, and can aid in resolving any licensing problems.
#wrapper.license.debug=TRUE
#********************************************************************
# Wrapper Localization
#********************************************************************
# Specify the locale which the Wrapper should use. By default the system
# locale is used.
#wrapper.lang=en_US # en_US or ja_JP
# Specify the location of the Wrapper's language resources. If these are
# missing, the Wrapper will default to the en_US locale.
wrapper.lang.folder=../lang
#********************************************************************
# Wrapper Java Properties
#********************************************************************
# Java Application
# Locate the java binary on the system PATH:
wrapper.java.command=C:\ProgramData\Oracle\Java\javapath\java.exe
# Specify a specific java binary:
#set.JAVA_HOME=/java/path
#wrapper.java.command=C:\ProgramData\Oracle\Java\javapath\java.exe
# Tell the Wrapper to log the full generated Java command line.
#wrapper.java.command.loglevel=INFO
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
# Java Classpath (include wrapper.jar) Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=lib/wrapper.jar
wrapper.java.classpath.2=lib/catapult.jar
# Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=lib
# Java Bits. On applicable platforms, tells the JVM to run in 32 or 64-bit mode.
wrapper.java.additional.auto_bits=TRUE
# Java Additional Parameters
#wrapper.java.additional.1=-XX:PermSize=64m
#wrapper.java.additional.2=-XX:MaxPermSize=128m
wrapper.java.additional.1=-XX:+UseG1GC
wrapper.java.additional.2=-XX:MaxGCPauseMillis=100
#wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled
#wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.3=-Ddata.dir="C:\Program Files\Inductive Automation\Ignition/data"
wrapper.java.additional.4.stripquotes=TRUE
wrapper.java.additional.5=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.8=-Xdebug
#wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
#wrapper.java.additional.6=-XX:+PrintGCDetails
#wrapper.java.additional.7=-XX:+PrintGCTimeStamps
#wrapper.java.additional.8=-XX:+PrintGCDateStamps
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=2048
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=4096
# Application parameters. Add parameters as needed starting from 1
wrapper.app.parameter.1=com.inductiveautomation.catapult.Catapult
wrapper.shutdown.timeout=120
wrapper.jvm_exit.timeout=120
# Set this value to TRUE to allow ignition.conf to be reloaded when the JVM is restarted
wrapper.restart.reload_configuration=TRUE
#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Enables Debug output from the Wrapper.
# wrapper.debug=TRUE
# Format of output for the console. (See docs for formats)
wrapper.console.format=PM
# Log Level for console output. (See docs for log levels)
wrapper.console.loglevel=INFO
# Log file to use for wrapper output logging.
wrapper.logfile=C:\Program Files\Inductive Automation\Ignition/logs\wrapper.log
# Format of output for the log file. (See docs for formats)
wrapper.logfile.format=LPTM
# Log Level for log file output. (See docs for log levels)
wrapper.logfile.loglevel=INFO
# Maximum size that the log file will be allowed to grow to before
# the log is rolled. Size is specified in bytes. The default value
# of 0, disables log rolling. May abbreviate with the 'k' (kb) or
# 'm' (mb) suffix. For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=10m
# Maximum number of rolled log files which will be allowed before old
# files are deleted. The default value of 0 implies no limit.
wrapper.logfile.maxfiles=5
# Log Level for sys/event log output. (See docs for log levels)
wrapper.syslog.loglevel=NONE
#********************************************************************
# Wrapper General Properties
#********************************************************************
# Allow for the use of non-contiguous numbered properties
wrapper.ignore_sequence_gaps=TRUE
# Title to use when running as a console
wrapper.console.title=Ignition Gateway
#********************************************************************
# Wrapper JVM Checks
#********************************************************************
# Detect DeadLocked Threads in the JVM. (Requires Standard Edition)
wrapper.check.deadlock=TRUE
wrapper.check.deadlock.interval=60
wrapper.check.deadlock.action=RESTART
wrapper.check.deadlock.output=FULL
wrapper.on_exit.default=SHUTDOWN
wrapper.on_exit.2=RESTART
# Out Of Memory detection.
#wrapper.filter.trigger.1000=java.lang.OutOfMemoryError
#wrapper.filter.action.1000=RESTART
#wrapper.filter.message.1000=The JVM has run out of memory.
wrapper.on_exit.default=SHUTDOWN
wrapper.on_exit.2=RESTART
#********************************************************************
# Wrapper Email Notifications. (Requires Professional Edition)
#********************************************************************
# Common Event Email settings.
#wrapper.event.default.email.debug=TRUE
#wrapper.event.default.email.smtp.host=<SMTP_Host>
#wrapper.event.default.email.smtp.port=25
#wrapper.event.default.email.subject=[%WRAPPER_HOSTNAME%:%WRAPPER_NAME%:%WRAPPER_EVENT_NAME%] Event Notification
#wrapper.event.default.email.sender=<Sender email>
#wrapper.event.default.email.recipient=<Recipient email>
# Configure the log attached to event emails.
#wrapper.event.default.email.attach_log=TRUE
#wrapper.event.default.email.maillog.lines=50
#wrapper.event.default.email.maillog.format=LPTM
#wrapper.event.default.email.maillog.loglevel=INFO
# Enable specific event emails.
#wrapper.event.wrapper_start.email=TRUE
#wrapper.event.jvm_prelaunch.email=TRUE
#wrapper.event.jvm_start.email=TRUE
#wrapper.event.jvm_started.email=TRUE
#wrapper.event.jvm_deadlock.email=TRUE
#wrapper.event.jvm_stop.email=TRUE
#wrapper.event.jvm_stopped.email=TRUE
#wrapper.event.jvm_restart.email=TRUE
#wrapper.event.jvm_failed_invocation.email=TRUE
#wrapper.event.jvm_max_failed_invocations.email=TRUE
#wrapper.event.jvm_kill.email=TRUE
#wrapper.event.jvm_killed.email=TRUE
#wrapper.event.jvm_unexpected_exit.email=TRUE
#wrapper.event.wrapper_stop.email=TRUE
# Specify custom mail content
wrapper.event.jvm_restart.email.body=The JVM was restarted.\n\nPlease check on its status.\n
#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled.
# Name of the service
wrapper.name=Ignition
# Display name of the service
wrapper.displayname=Ignition Gateway
# Description of the service
wrapper.description=Ignition Gateway
# Service dependencies. Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=
# Mode in which the service is installed. AUTO_START, DELAY_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START
# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false
#VERSION - DO NOT MODIFY: 7
You are having the wrapper do deadlock detection every minute. That is the only thing that looks even remotely suspicious. Consider changing the interval and seeing if your CPU spike interval changes with it.
I’m thinking to program a some kind of “automatic restart” inside of Ignition.
I have some scripts, but fails 8/10 times. It consist in watch the CPU load using Ignition tags, and if the CPU is greather than a reference, try to restart the service. But it’s sensible to any spikes…
This is not the “good” solution, but decreases the CPU load, the problem is when fails.
Anybody knows a way to restart automatically the ignition service when the CPU Load is high?
I’m using os calls… but I don’t know if there exists another better way…
[quote=“jruben84”]I change the line of dead lock detection to 10 minutes.
I have a question,
what are this lines for:
# Java Additional Parameters[/quote]See this older post and this more recent followup.[quote=“jruben84”]I’m thinking to program a some kind of “automatic restart” inside of Ignition.[/quote]Ewww! Please don’t do this. It like asking a criminal to police himself. When Ignition is choking on some unexpected workload, you can’t expect a “policeman” inside Ignition to get CPU time except by chance. You really need to figure out what is causing this, not just paper over the problem.
Consider turning on gargage collection statistics logging, as GC events can be disruptive, too.
Hi. I read the post that you suggested me. So… If I understand well, I have to modify my ignition.conf in this way (just show the part of the JVM parameters):
Java Additional Parameters
#wrapper.java.additional.1=-XX:PermSize=64m #wrapper.java.additional.2=-XX:MaxPermSize=128m
wrapper.java.additional.1=-XX:+UseG1GC [color=#FF0000](thats OK)[/color]
wrapper.java.additional.2=-XX:MaxGCPauseMillis=100 [color=#FF0000](thats OK too)[/color] #wrapper.java.additional.4=-XX:+CMSClassUnloadingEnabled #wrapper.java.additional.5=-XX:+CMSPermGenSweepingEnabled
wrapper.java.additional.3=-Ddata.dir=“C:\Program Files\Inductive Automation\Ignition\data\jvm_logs.log” [color=#FF0000](I think that in this line I need to add the name of log file…)[/color]
wrapper.java.additional.4.stripquotes=TRUE
wrapper.java.additional.5=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false #wrapper.java.additional.8=-Xdebug #wrapper.java.additional.9=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
[quote=“jruben84”]wrapper.java.additional.3=-Ddata.dir=“C:\Program Files\Inductive Automation\Ignition\data\jvm_logs.log” [color=#FF0000](I think that in this line I need to add the name of log file…)[/color]
Is that correct?[/quote]No, you can’t change the data directory. Look at PrintGCDetails and related options. This link in the old post explains the details you need for the command line, including how to do the logging.
noone Knows what youre are doing in youre Project, you dont give any details.
I can give you some Details on mine:
VritualVM 8 virtual Cores Cisco UCS Cluster with Xeon(R) CPU E5-2630 v3 @ 2.40GHz
~20 Machine Connections Via Opcua
12 Databases Connected sum queries 60/s
17k Tags 2 providers
ca. 20 machines which are communicating via WebDEV Api Calls
ca. 25 Active Vision Clients
ca. 20 Active Perspective Sessions
4 connected remotegateways
