Cannot start gateway after upgrade to 8.1.16

Gateway failed to start. Anyone can help to advise?

Running Ignition-Gateway...
wrapper  | --> Wrapper Started as Console
wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.42
wrapper  |   Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
wrapper  |     http://wrapper.tanukisoftware.com
wrapper  |   Licensed to Inductive Automation for Inductive Automation
wrapper  |
wrapper  | Launching a JVM...
jvm 1    | WrapperManager: Initializing...
jvm 1    | 16:59:50,964 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [data//logback.xml] at [file:/usr/local/bin/ignition/data/logback.xml]
jvm 1    | 16:59:51,057 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
jvm 1    | 16:59:51,061 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAppender]
jvm 1    | 16:59:51,074 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
jvm 1    | 16:59:51,111 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [com.inductiveautomation.logging.SQLiteAppender]
jvm 1    | 16:59:51,124 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DB]
jvm 1    | 16:59:51,247 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@3f34d645 - Driver name=SQLite JDBC
jvm 1    | 16:59:51,248 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@3f34d645 - Driver version=3.23.1
jvm 1    | 16:59:51,248 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@3f34d645 - supportsGetGeneratedKeys=true
jvm 1    | 16:59:51,253 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
jvm 1    | 16:59:51,255 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAsync]
jvm 1    | 16:59:51,255 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAppender] to ch.qos.logback.classic.AsyncAppender[SysoutAsync]
jvm 1    | 16:59:51,255 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Attaching appender named [SysoutAppender] to AsyncAppender.
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Setting discardingThreshold to 51
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DBAsync]
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DB] to ch.qos.logback.classic.AsyncAppender[DBAsync]
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Attaching appender named [DB] to AsyncAppender.
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Setting discardingThreshold to 51
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAsync] to Logger[ROOT]
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DBAsync] to Logger[ROOT]
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
jvm 1    | 16:59:51,259 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@679c5dc0 - Registering current configuration as safe fallback point
jvm 1    | 16:59:51,297 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@1ce3f954 - Propagating INFO level on Logger[ROOT] onto the JUL framework
jvm 1    | W [WebResourceManagerImpl        ] [08:59:51]: Unable to read "gateway.metroKeystoreAlias" gateway.xml property. Assuming metro keystore alias is: metro-key
jvm 1    | I [o.e.j.u.log                   ] [08:59:51]: Logging initialized @1494ms to org.eclipse.jetty.util.log.Slf4jLog
jvm 1    | I [g.WebServerManager            ] [08:59:51]: Starting setup
jvm 1    | I [g.WebServerManager            ] [08:59:51]: Setup complete
jvm 1    | I [g.WebServerManager            ] [08:59:52]: Starting up
jvm 1    | W [g.SslManager                  ] [08:59:52]: Unable to verify chain of trust state=NO_CERTIFICATE
jvm 1    | java.lang.Exception: PKIX certificate path validation failed
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:292)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:109)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager.validateKeyStore(SslManager.java:261)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager$AbstractStateReader.readState(SslManager.java:301)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager.refreshInternal(SslManager.java:423)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager.startup(SslManager.java:460)
jvm 1    |      at com.inductiveautomation.catapult.IgnitionServer.setupConnectors(IgnitionServer.java:1152)
jvm 1    |      at com.inductiveautomation.catapult.IgnitionServer.<init>(IgnitionServer.java:272)
jvm 1    |      at com.inductiveautomation.ignition.gateway.web.WebResourceManagerImpl.<init>(WebResourceManagerImpl.java:163)
jvm 1    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.<init>(IgnitionGateway.java:508)
jvm 1    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.main(IgnitionGateway.java:294)
jvm 1    |      at com.inductiveautomation.catapult.Catapult.main(Catapult.java:8)
jvm 1    |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1    |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1    |      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
jvm 1    |      at java.base/java.lang.Thread.run(Unknown Source)
jvm 1    | Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
jvm 1    |      at java.base/java.security.cert.PKIXParameters.setTrustAnchors(Unknown Source)
jvm 1    |      at java.base/java.security.cert.PKIXParameters.<init>(Unknown Source)
jvm 1    |      at java.base/java.security.cert.PKIXBuilderParameters.<init>(Unknown Source)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:230)
jvm 1    |      ... 17 common frames omitted
jvm 1    | I [g.SslManager                  ] [08:59:52]: State refreshed state=CA_SIGNED_CERTIFICATE
jvm 1    | I [o.e.j.s.Server                ] [08:59:52]: jetty-9.4.45.v20220203; built: 2022-02-03T09:14:34.105Z; git: 4a0c91c0be53805e3fcffdcdcc9587d5301863db; jvm 11.0.14.1+1-LTS
jvm 1    | I [C.SecureRandomProvider        ] [08:59:52]: Secure random seed generated in 0ms
jvm 1    | I [o.e.j.s.session               ] [08:59:52]: DefaultSessionIdManager workerName=node0
jvm 1    | I [o.e.j.s.session               ] [08:59:52]: No SessionScavenger set, using defaults
jvm 1    | I [o.e.j.s.session               ] [08:59:52]: node0 Scavenging every 600000ms
jvm 1    | I [o.a.w.Application             ] [08:59:52]: [WicketFilter] init: Wicket core library initializer
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IBehaviorListener, method=public abstract void org.apache.wicket.behavior.IBehaviorListener.onRequest()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IFormSubmitListener, method=public abstract void org.apache.wicket.markup.html.form.IFormSubmitListener.onFormSubmitted()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=ILinkListener, method=public abstract void org.apache.wicket.markup.html.link.ILinkListener.onLinkClicked()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IOnChangeListener, method=public abstract void org.apache.wicket.markup.html.form.IOnChangeListener.onSelectionChanged()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IRedirectListener, method=public abstract void org.apache.wicket.IRedirectListener.onRedirect()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
jvm 1    | I [o.a.w.Application             ] [08:59:52]: [WicketFilter] init: Wicket extensions initializer
jvm 1    | I [IgnitionGateway               ] [08:59:52]: Ignition[state=STOPPED] ContextState = STARTING
jvm 1    | I [o.a.w.p.h.WebApplication      ] [08:59:52]: [WicketFilter] Started Wicket version 6.30.0 in DEPLOYMENT mode
jvm 1    | I [IgnitionGateway               ] [08:59:52]: Starting Ignition 8.1.16 (b2022040511)
jvm 1    | I [IgnitionGateway               ] [08:59:52]: Reloading JDBC classloader.
jvm 1    | I [R.S.S.InternalDb              ] [08:59:53]: Internal db synchronizer disk cache initialized. State id=e92ae623-85c9-418a-8ed4-7bb0527403fb
jvm 1    | I [R.StateMonitoring             ] [08:59:53]: Redundancy state changed: Role=Independent, Activity level=Active, Project state=Good, History level=Full
jvm 1    | I [o.e.j.s.h.ContextHandler      ] [08:59:53]: Started c.i.c.MainWebAppContext@38ae9b27{Ignition,/,file:///usr/local/bin/ignition/webserver/webapps/main/,AVAILABLE}
jvm 1    | I [g.InternalDatabaseManager     ] [08:59:53]: Starting up...
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: Looking for existing internal database "config.idb"...
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: ... found existing.
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: internal database "config.idb" started up successfully.
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: Registering autobackup task [owner=Internal DB Autobackup, name=internal database "config.idb"]
jvm 1    | WARNING: An illegal reflective access operation has occurred
jvm 1    | WARNING: Illegal reflective access by com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl (file:/usr/local/bin/ignition/lib/core/gateway/gateway-8.1.16.jar) to field java.lang.ClassLoader.usr_paths
jvm 1    | WARNING: Please consider reporting this to the maintainers of com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl
jvm 1    | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
jvm 1    | WARNING: All illegal access operations will be denied in a future release
jvm 1    | I [o.e.j.s.AbstractConnector     ] [08:59:53]: Started ServerConnector@cd201fb{SSL, (ssl, http/1.1)}{0.0.0.0:8060}
jvm 1    | WrapperSimpleApp:
jvm 1    | WrapperSimpleApp Error: Encountered an error running main:
jvm 1    | WrapperSimpleApp Error: MultiException[java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:80, java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:443]
jvm 1    | WrapperSimpleApp Error:      at org.eclipse.jetty.util.MultiException.ifExceptionThrow(MultiException.java:122)
jvm 1    | WrapperSimpleApp Error:      at org.eclipse.jetty.server.Server.doStart(Server.java:413)
jvm 1    | WrapperSimpleApp Error:      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.catapult.IgnitionServer.start(IgnitionServer.java:1169)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.ignition.gateway.web.WebResourceManagerImpl.start(WebResourceManagerImpl.java:335)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.ignition.gateway.IgnitionGateway.start(IgnitionGateway.java:563)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.ignition.gateway.IgnitionGateway.main(IgnitionGateway.java:296)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.catapult.Catapult.main(Catapult.java:8)
jvm 1    | WrapperSimpleApp Error:      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    | WrapperSimpleApp Error:      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
jvm 1    | WrapperSimpleApp Error:      at java.base/java.lang.Thread.run(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:80
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.Server.doStart(Server.java:401)
jvm 1    | WrapperSimpleApp Error:              ... 12 more
jvm 1    | WrapperSimpleApp Error:      Caused by: java.net.SocketException: Permission denied
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind0(Native Method)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
jvm 1    | WrapperSimpleApp Error:              ... 17 more
jvm 1    | WrapperSimpleApp Error:      Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:443
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.Server.doStart(Server.java:401)
jvm 1    | WrapperSimpleApp Error:              ... 12 more
jvm 1    | WrapperSimpleApp Error:      Caused by: java.net.SocketException: Permission denied
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind0(Native Method)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
jvm 1    | WrapperSimpleApp Error:              ... 17 more
jvm 1    | WrapperSimpleApp Error: Caused by: [CIRCULAR REFERENCE: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:80]
jvm 1    | I [SystemInfo                    ] [08:59:53]: os.name="linux", os.arch="amd64".
jvm 1    | I [SystemInfo                    ] [08:59:53]: Validating native library "ignition-x64"...
jvm 1    | I [SystemInfo                    ] [08:59:53]: Validated ignition-x64 (libignition-x64.so)
jvm 1    | I [SystemInfo                    ] [08:59:53]: Loading native library "ignition-x64"...
jvm 1    | I [SystemInfo                    ] [08:59:53]: Native library load complete
jvm 1    | I [g.ModuleManager               ] [08:59:53]: Loading modules....
wrapper  | <-- Wrapper Stopped

Looks like you are trying to use the standard HTTP & HTTPS ports with a non-root user. It can be done, but only by adding a clause to your service file. See this topic:

Thanks for the reply. AmbientCapabilities=CAP_NET_BIND_SERVICE was added in the earlier version. Seem like a the service file is overwritten during the upgrade so have to add it again.

Make sure to use systemctl edit <service name> to supplement your systemd service configurations. This creates an override file that won’t be overwritten with baseline updates to the service.

2 Likes

In case it helps the next guy on this thread, you should be adding

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

to the window that opens when you run this command, then hit ctrl+x, enter to save and exit

3 Likes