I’m teaching an introduction class to Ignition to some new hires next week and there will be a lab portion. I’m hoping to spin up an Ignition instance for each student (about 20), but it looks like Docker may have a limit of 10 instances? When starting the 11th instance, it shows that the “Gateway is FAULTED”. Honestly, it makes sense to me that this would be the case, more just curious if this is a hard set number or if there is some way to have more than 10 instances.
Neither Docker (the core product) nor Ignition have any limit on the number of instances deployed.
Without knowing the nature of the fault itself, it’s hard to say, but if I had to guess:
- If you’re using Docker Desktop (on Windows or MacOS) it might have some limitation on the number of running containers at once, or
- Your host system itself may not have sufficient resources to dedicate to these containers.
The entire point of Docker/containers is that the instances are isolated and effectively “on their own”. You could also be hitting port conflicts, potentially; I would expect the Docker daemon to handle that for you unless you specified certain ports to use.
There definitely are some situations where you can run into resource constraints, even ones other than CPU/memory. Maximum number of filesystem watches (per user) is one that comes to mind [that I’ve tripped over recently on a densely populated host]. Setting fs.inotify.max_user_instances to 1024 is not atypical.
You might share a little more about your environment. Is this Docker Engine running on Linux? Or are you attempting this under Docker Desktop? If the latter, there are separate resource allocations for the under-the-hood Linux VM that you may need to tune. The former? Possibly some other sysctl values that need adjustment. Either way, it would also be helpful to know the specs of the underlying hardware as well (to try and speculate on what you might be running into). Oh, and output from docker logs <one of the containers that failed> wouldn’t hurt either. 
1 Like
Thanks for the response. docker logs <container> yields the following. The thing that stuck out to me at the bottom was
jvm 1 | 2022/06/30 21:06:20 | W [g.LicenseManager ] [21:06:20]: Could not read license from HASP key. Driver not found. But I don’t know what to make of that.
In response to your other questions, this is on Docker Desktop. I haven’t dug much into the Linux VM resource allocations, but I should have plenty of juice on my machine.
>docker logs 957d5ff53022
<abbreviated for length>
init | 2022/06/30 21:06:08 | Creating init.properties file
init | 2022/06/30 21:06:08 | Creating gateway.xml
init | 2022/06/30 21:06:08 | Adding gateway.port=8088 to gateway.xml
init | 2022/06/30 21:06:08 | Adding gateway.sslport=8043 to gateway.xml
init | 2022/06/30 21:06:08 | Adding gateway.metroSSLPort=8060 to gateway.xml
init | 2022/06/30 21:06:08 | Starting Ignition gateway
wrapper | 2022/06/30 21:06:08 | --> Wrapper Started as Console
wrapper | 2022/06/30 21:06:08 | Java Service Wrapper Standard Edition 64-bit 3.5.42
wrapper | 2022/06/30 21:06:08 | Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
wrapper | 2022/06/30 21:06:08 | http://wrapper.tanukisoftware.com
wrapper | 2022/06/30 21:06:08 | Licensed to Inductive Automation for Inductive Automation
wrapper | 2022/06/30 21:06:08 |
wrapper | 2022/06/30 21:06:08 | Launching a JVM...
jvm 1 | 2022/06/30 21:06:08 | WrapperManager: Initializing...
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,026 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [data//logback.xml] at [file:/usr/local/bin/ignition/data/logback.xml]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,077 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,079 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAppender]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,081 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,092 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [com.inductiveautomation.logging.SQLiteAppender]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,097 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DB]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,175 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@6e4074f1 - Driver name=SQLite JDBC
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,175 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@6e4074f1 - Driver version=3.23.1
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,175 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@6e4074f1 - supportsGetGeneratedKeys=true
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,178 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,180 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAsync]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,180 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAppender] to ch.qos.logback.classic.AsyncAppender[SysoutAsync]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,180 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Attaching appender named [SysoutAppender] to AsyncAppender.
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,181 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Setting discardingThreshold to 51
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,182 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,184 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DBAsync]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,184 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DB] to ch.qos.logback.classic.AsyncAppender[DBAsync]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,184 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Attaching appender named [DB] to AsyncAppender.
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,184 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Setting discardingThreshold to 51
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,184 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,185 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAsync] to Logger[ROOT]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,185 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DBAsync] to Logger[ROOT]
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,185 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,185 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@6cb302e - Registering current configuration as safe fallback point
jvm 1 | 2022/06/30 21:06:09 | 21:06:09,208 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@1f9b6a8a - Propagating INFO level on Logger[ROOT] onto the JUL framework
jvm 1 | 2022/06/30 21:06:09 | W [WebResourceManagerImpl ] [21:06:09]: Unable to read "gateway.metroKeystoreAlias" gateway.xml property. Assuming metro keystore alias is: metro-key
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.u.log ] [21:06:09]: Logging initialized @806ms to org.eclipse.jetty.util.log.Slf4jLog
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: Starting setup
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.forceSecureRedirect property is not set. Defaulting to false
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.includedCipherSuites property is not set. Defaulting to []
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.excludedCipherSuites property is not set. Defaulting to []
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.publicAddress.autoDetect property is not set. Defaulting to true
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.publicAddress.address property is not set. Defaulting to
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.publicAddress.httpPort property is not set. Defaulting to
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.publicAddress.httpsPort property is not set. Defaulting to
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.useProxyForwardedHeader property is not set. Defaulting to false
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: gateway.resolveHostNames property is not set. Defaulting to false
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: Setup complete
jvm 1 | 2022/06/30 21:06:09 | I [g.WebServerManager ] [21:06:09]: Starting up
jvm 1 | 2022/06/30 21:06:09 | I [C.SecureRandomProvider ] [21:06:09]: Secure random seed generated in 0ms
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.s.Server ] [21:06:09]: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.14.1+1-LTS
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.s.session ] [21:06:09]: DefaultSessionIdManager workerName=node0
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.s.session ] [21:06:09]: No SessionScavenger set, using defaults
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.s.session ] [21:06:09]: node0 Scavenging every 600000ms
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.Application ] [21:06:09]: [WicketFilter] init: Wicket core library initializer
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.RequestListenerInterface] [21:06:09]: registered listener interface [RequestListenerInterface name=IBehaviorListener, method=public abstract void org.apache.wicket.behavior.IBehaviorListener.onRequest()]
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.RequestListenerInterface] [21:06:09]: registered listener interface [RequestListenerInterface name=IFormSubmitListener, method=public abstract void org.apache.wicket.markup.html.form.IFormSubmitListener.onFormSubmitted()]
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.RequestListenerInterface] [21:06:09]: registered listener interface [RequestListenerInterface name=ILinkListener, method=public abstract void org.apache.wicket.markup.html.link.ILinkListener.onLinkClicked()]
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.RequestListenerInterface] [21:06:09]: registered listener interface [RequestListenerInterface name=IOnChangeListener, method=public abstract void org.apache.wicket.markup.html.form.IOnChangeListener.onSelectionChanged()]
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.RequestListenerInterface] [21:06:09]: registered listener interface [RequestListenerInterface name=IRedirectListener, method=public abstract void org.apache.wicket.IRedirectListener.onRedirect()]
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.RequestListenerInterface] [21:06:09]: registered listener interface [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.Application ] [21:06:09]: [WicketFilter] init: Wicket extensions initializer
jvm 1 | 2022/06/30 21:06:09 | I [IgnitionGateway ] [21:06:09]: Ignition[state=STOPPED] ContextState = STARTING
jvm 1 | 2022/06/30 21:06:09 | I [o.a.w.p.h.WebApplication ] [21:06:09]: [WicketFilter] Started Wicket version 6.30.0 in DEPLOYMENT mode
jvm 1 | 2022/06/30 21:06:09 | I [IgnitionGateway ] [21:06:09]: Starting Ignition 8.1.15 (b2022030114)
jvm 1 | 2022/06/30 21:06:09 | I [IgnitionGateway ] [21:06:09]: Reloading JDBC classloader.
jvm 1 | 2022/06/30 21:06:09 | I [R.S.S.InternalDb ] [21:06:09]: Internal db synchronizer disk cache initialized. State id=153e1f45-fbb1-430f-a341-fe43042c20db
jvm 1 | 2022/06/30 21:06:09 | I [R.StateMonitoring ] [21:06:09]: Redundancy state changed: Role=Independent, Activity level=Active, Project state=Good, History level=Full
jvm 1 | 2022/06/30 21:06:09 | I [g.InternalDatabaseManager ] [21:06:09]: Starting up...
jvm 1 | 2022/06/30 21:06:09 | I [P.InternalDatabase ] [21:06:09]: Looking for existing internal database "config.idb"...
jvm 1 | 2022/06/30 21:06:09 | I [P.InternalDatabase ] [21:06:09]: ... found existing.
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.s.h.ContextHandler ] [21:06:09]: Started c.i.c.MainWebAppContext@5ec6e8eb{Ignition,/,file:///usr/local/bin/ignition/webserver/webapps/main/,AVAILABLE}
jvm 1 | 2022/06/30 21:06:09 | I [o.e.j.s.AbstractConnector ] [21:06:09]: Started ServerConnector@2fd040d8{HTTP/1.1, (http/1.1)}{0.0.0.0:8088}
jvm 1 | 2022/06/30 21:06:10 | I [P.InternalDatabase ] [21:06:09]: internal database "config.idb" started up successfully.
jvm 1 | 2022/06/30 21:06:10 | I [P.InternalDatabase ] [21:06:09]: Registering autobackup task [owner=Internal DB Autobackup, name=internal database "config.idb"]
jvm 1 | 2022/06/30 21:06:10 | WARNING: An illegal reflective access operation has occurred
jvm 1 | 2022/06/30 21:06:10 | WARNING: Illegal reflective access by com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl (file:/usr/local/bin/ignition/lib/core/gateway/gateway-8.1.15.jar) to field java.lang.ClassLoader.usr_paths
jvm 1 | 2022/06/30 21:06:10 | WARNING: Please consider reporting this to the maintainers of com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl
jvm 1 | 2022/06/30 21:06:10 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
jvm 1 | 2022/06/30 21:06:10 | WARNING: All illegal access operations will be denied in a future release
jvm 1 | 2022/06/30 21:06:10 | I [g.ModuleManager ] [21:06:10]: Loading modules....
jvm 1 | 2022/06/30 21:06:10 | I [o.e.j.s.AbstractConnector ] [21:06:10]: Started ServerConnector@2a34d93f{SSL, (ssl, http/1.1)}{0.0.0.0:8060}
jvm 1 | 2022/06/30 21:06:10 | I [o.e.j.s.Server ] [21:06:10]: Started @1648ms
jvm 1 | 2022/06/30 21:06:11 | I [A.Notification ] [21:06:11]: Alert notification system created.
jvm 1 | 2022/06/30 21:06:11 | I [g.PersistentRecordSecurityLevelConfigService] [21:06:11]: Set up in 2 ms
jvm 1 | 2022/06/30 21:06:11 | I [g.PersistentRecordIdpAdapterConfigService] [21:06:11]: Set up in 5 ms
jvm 1 | 2022/06/30 21:06:11 | I [g.PersistentRecordIdpAdapterMetricsService] [21:06:11]: Set up in 4 ms
jvm 1 | 2022/06/30 21:06:11 | I [IgnitionGateway ] [21:06:11]: System properties exist in IDB. Checking if system IdP is set...
jvm 1 | 2022/06/30 21:06:11 | I [g.InternalDatabaseRememberedSubjects] [21:06:11]: Set up in 4 ms
jvm 1 | 2022/06/30 21:06:11 | I [g.OIDCProviderManager ] [21:06:11]: Set up in 4 ms
jvm 1 | 2022/06/30 21:06:11 | I [G.Manager ] [21:06:11]: Setting up trust list managers...
jvm 1 | 2022/06/30 21:06:11 | E [IgnitionGateway ] [21:06:11]: Error during context startup.
jvm 1 | 2022/06/30 21:06:11 | java.io.IOException: User limit of inotify instances reached or too many open files
jvm 1 | 2022/06/30 21:06:11 | at java.base/sun.nio.fs.LinuxWatchService.<init>(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at java.base/sun.nio.fs.LinuxFileSystem.newWatchService(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.metro.pki.GatewayNetworkTrustListManager.<init>(GatewayNetworkTrustListManager.java:134)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.gan.GatewayAreaNetworkManagerImpl.createTrustListManager(GatewayAreaNetworkManagerImpl.java:212)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.gan.GatewayAreaNetworkManagerImpl.setupTrustListManagers(GatewayAreaNetworkManagerImpl.java:436)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.gan.GatewayAreaNetworkManagerImpl.setup(GatewayAreaNetworkManagerImpl.java:462)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1157)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:290)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:735)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:669)
jvm 1 | 2022/06/30 21:06:11 | at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:539)
jvm 1 | 2022/06/30 21:06:11 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | at java.base/java.lang.Thread.run(Unknown Source)
jvm 1 | 2022/06/30 21:06:11 | I [IgnitionGateway ] [21:06:11]: Ignition[state=STARTING] ContextState = FAULTED
jvm 1 | 2022/06/30 21:06:20 | W [g.LicenseManager ] [21:06:20]: Could not read license from HASP key. Driver not found
A license wouldn’t equate to gateway is faulted, it would just show in status page unlicensed.
All barr 1 of these is unlicensed (2 hour trial) anyway;
Thanks for this! We’re getting far outside the scope of Ignition, but I’m trying to find out how to change fs.inotify.max_user_instances to no avail. Any insight there? Is it on a container basis? Or a docker setting?
Some quick googling suggests that it’s a setting to apply to the Docker Desktop host VM, but I didn’t see a conclusive “do this to fix it” setting anywhere.
If you’ve got any experience running a virtualization tool, you could run your own Linux VM and install Docker into it. It won’t be quite as seamless as Docker Desktop (tries to) make things but isn’t too much more effort. WSL (the Windows Subsystem For Linux) may also make things easier.
You should be able to gain access to the underlying Linux VM using this solution (from Justin Cormack, CTO Docker):
docker run -it --rm --privileged --pid=host justincormack/nsenter1
Once in, you can view the current sysctl settings with sysctl -a (it looks like the default for this one is 128) and update with:
sysctl user.max_inotify_instances=1024
(then just exit that shell). Try spinning up your containers and see if it works… What I don’t know yet is how to persist this setting (with /etc/sysctl.d being read-only). It will need to be applied each time Docker Desktop VM starts.
1 Like
This is great, thank you for this solution! Fortunately, this is a short term thing that I need 20 ignition servers for.