Cannot start gateway after upgrade to 8.1.16

Gateway failed to start. Anyone can help to advise?

Running Ignition-Gateway...
wrapper  | --> Wrapper Started as Console
wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.42
wrapper  |   Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
wrapper  |     http://wrapper.tanukisoftware.com
wrapper  |   Licensed to Inductive Automation for Inductive Automation
wrapper  |
wrapper  | Launching a JVM...
jvm 1    | WrapperManager: Initializing...
jvm 1    | 16:59:50,964 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [data//logback.xml] at [file:/usr/local/bin/ignition/data/logback.xml]
jvm 1    | 16:59:51,057 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
jvm 1    | 16:59:51,061 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAppender]
jvm 1    | 16:59:51,074 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
jvm 1    | 16:59:51,111 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [com.inductiveautomation.logging.SQLiteAppender]
jvm 1    | 16:59:51,124 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DB]
jvm 1    | 16:59:51,247 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@3f34d645 - Driver name=SQLite JDBC
jvm 1    | 16:59:51,248 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@3f34d645 - Driver version=3.23.1
jvm 1    | 16:59:51,248 |-INFO in ch.qos.logback.core.db.DataSourceConnectionSource@3f34d645 - supportsGetGeneratedKeys=true
jvm 1    | 16:59:51,253 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
jvm 1    | 16:59:51,255 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SysoutAsync]
jvm 1    | 16:59:51,255 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAppender] to ch.qos.logback.classic.AsyncAppender[SysoutAsync]
jvm 1    | 16:59:51,255 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Attaching appender named [SysoutAppender] to AsyncAppender.
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.classic.AsyncAppender[SysoutAsync] - Setting discardingThreshold to 51
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [DBAsync]
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DB] to ch.qos.logback.classic.AsyncAppender[DBAsync]
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Attaching appender named [DB] to AsyncAppender.
jvm 1    | 16:59:51,256 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Setting discardingThreshold to 51
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SysoutAsync] to Logger[ROOT]
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [DBAsync] to Logger[ROOT]
jvm 1    | 16:59:51,257 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
jvm 1    | 16:59:51,259 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@679c5dc0 - Registering current configuration as safe fallback point
jvm 1    | 16:59:51,297 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@1ce3f954 - Propagating INFO level on Logger[ROOT] onto the JUL framework
jvm 1    | W [WebResourceManagerImpl        ] [08:59:51]: Unable to read "gateway.metroKeystoreAlias" gateway.xml property. Assuming metro keystore alias is: metro-key
jvm 1    | I [o.e.j.u.log                   ] [08:59:51]: Logging initialized @1494ms to org.eclipse.jetty.util.log.Slf4jLog
jvm 1    | I [g.WebServerManager            ] [08:59:51]: Starting setup
jvm 1    | I [g.WebServerManager            ] [08:59:51]: Setup complete
jvm 1    | I [g.WebServerManager            ] [08:59:52]: Starting up
jvm 1    | W [g.SslManager                  ] [08:59:52]: Unable to verify chain of trust state=NO_CERTIFICATE
jvm 1    | java.lang.Exception: PKIX certificate path validation failed
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:292)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:109)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager.validateKeyStore(SslManager.java:261)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager$AbstractStateReader.readState(SslManager.java:301)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager.refreshInternal(SslManager.java:423)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.SslManager.startup(SslManager.java:460)
jvm 1    |      at com.inductiveautomation.catapult.IgnitionServer.setupConnectors(IgnitionServer.java:1152)
jvm 1    |      at com.inductiveautomation.catapult.IgnitionServer.<init>(IgnitionServer.java:272)
jvm 1    |      at com.inductiveautomation.ignition.gateway.web.WebResourceManagerImpl.<init>(WebResourceManagerImpl.java:163)
jvm 1    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.<init>(IgnitionGateway.java:508)
jvm 1    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.main(IgnitionGateway.java:294)
jvm 1    |      at com.inductiveautomation.catapult.Catapult.main(Catapult.java:8)
jvm 1    |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1    |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1    |      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
jvm 1    |      at java.base/java.lang.Thread.run(Unknown Source)
jvm 1    | Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
jvm 1    |      at java.base/java.security.cert.PKIXParameters.setTrustAnchors(Unknown Source)
jvm 1    |      at java.base/java.security.cert.PKIXParameters.<init>(Unknown Source)
jvm 1    |      at java.base/java.security.cert.PKIXBuilderParameters.<init>(Unknown Source)
jvm 1    |      at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:230)
jvm 1    |      ... 17 common frames omitted
jvm 1    | I [g.SslManager                  ] [08:59:52]: State refreshed state=CA_SIGNED_CERTIFICATE
jvm 1    | I [o.e.j.s.Server                ] [08:59:52]: jetty-9.4.45.v20220203; built: 2022-02-03T09:14:34.105Z; git: 4a0c91c0be53805e3fcffdcdcc9587d5301863db; jvm 11.0.14.1+1-LTS
jvm 1    | I [C.SecureRandomProvider        ] [08:59:52]: Secure random seed generated in 0ms
jvm 1    | I [o.e.j.s.session               ] [08:59:52]: DefaultSessionIdManager workerName=node0
jvm 1    | I [o.e.j.s.session               ] [08:59:52]: No SessionScavenger set, using defaults
jvm 1    | I [o.e.j.s.session               ] [08:59:52]: node0 Scavenging every 600000ms
jvm 1    | I [o.a.w.Application             ] [08:59:52]: [WicketFilter] init: Wicket core library initializer
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IBehaviorListener, method=public abstract void org.apache.wicket.behavior.IBehaviorListener.onRequest()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IFormSubmitListener, method=public abstract void org.apache.wicket.markup.html.form.IFormSubmitListener.onFormSubmitted()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=ILinkListener, method=public abstract void org.apache.wicket.markup.html.link.ILinkListener.onLinkClicked()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IOnChangeListener, method=public abstract void org.apache.wicket.markup.html.form.IOnChangeListener.onSelectionChanged()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IRedirectListener, method=public abstract void org.apache.wicket.IRedirectListener.onRedirect()]
jvm 1    | I [o.a.w.RequestListenerInterface] [08:59:52]: registered listener interface [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
jvm 1    | I [o.a.w.Application             ] [08:59:52]: [WicketFilter] init: Wicket extensions initializer
jvm 1    | I [IgnitionGateway               ] [08:59:52]: Ignition[state=STOPPED] ContextState = STARTING
jvm 1    | I [o.a.w.p.h.WebApplication      ] [08:59:52]: [WicketFilter] Started Wicket version 6.30.0 in DEPLOYMENT mode
jvm 1    | I [IgnitionGateway               ] [08:59:52]: Starting Ignition 8.1.16 (b2022040511)
jvm 1    | I [IgnitionGateway               ] [08:59:52]: Reloading JDBC classloader.
jvm 1    | I [R.S.S.InternalDb              ] [08:59:53]: Internal db synchronizer disk cache initialized. State id=e92ae623-85c9-418a-8ed4-7bb0527403fb
jvm 1    | I [R.StateMonitoring             ] [08:59:53]: Redundancy state changed: Role=Independent, Activity level=Active, Project state=Good, History level=Full
jvm 1    | I [o.e.j.s.h.ContextHandler      ] [08:59:53]: Started c.i.c.MainWebAppContext@38ae9b27{Ignition,/,file:///usr/local/bin/ignition/webserver/webapps/main/,AVAILABLE}
jvm 1    | I [g.InternalDatabaseManager     ] [08:59:53]: Starting up...
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: Looking for existing internal database "config.idb"...
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: ... found existing.
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: internal database "config.idb" started up successfully.
jvm 1    | I [P.InternalDatabase            ] [08:59:53]: Registering autobackup task [owner=Internal DB Autobackup, name=internal database "config.idb"]
jvm 1    | WARNING: An illegal reflective access operation has occurred
jvm 1    | WARNING: Illegal reflective access by com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl (file:/usr/local/bin/ignition/lib/core/gateway/gateway-8.1.16.jar) to field java.lang.ClassLoader.usr_paths
jvm 1    | WARNING: Please consider reporting this to the maintainers of com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl
jvm 1    | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
jvm 1    | WARNING: All illegal access operations will be denied in a future release
jvm 1    | I [o.e.j.s.AbstractConnector     ] [08:59:53]: Started ServerConnector@cd201fb{SSL, (ssl, http/1.1)}{0.0.0.0:8060}
jvm 1    | WrapperSimpleApp:
jvm 1    | WrapperSimpleApp Error: Encountered an error running main:
jvm 1    | WrapperSimpleApp Error: MultiException[java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:80, java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:443]
jvm 1    | WrapperSimpleApp Error:      at org.eclipse.jetty.util.MultiException.ifExceptionThrow(MultiException.java:122)
jvm 1    | WrapperSimpleApp Error:      at org.eclipse.jetty.server.Server.doStart(Server.java:413)
jvm 1    | WrapperSimpleApp Error:      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.catapult.IgnitionServer.start(IgnitionServer.java:1169)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.ignition.gateway.web.WebResourceManagerImpl.start(WebResourceManagerImpl.java:335)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.ignition.gateway.IgnitionGateway.start(IgnitionGateway.java:563)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.ignition.gateway.IgnitionGateway.main(IgnitionGateway.java:296)
jvm 1    | WrapperSimpleApp Error:      at com.inductiveautomation.catapult.Catapult.main(Catapult.java:8)
jvm 1    | WrapperSimpleApp Error:      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    | WrapperSimpleApp Error:      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
jvm 1    | WrapperSimpleApp Error:      at java.base/java.lang.Thread.run(Unknown Source)
jvm 1    | WrapperSimpleApp Error:      Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:80
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.Server.doStart(Server.java:401)
jvm 1    | WrapperSimpleApp Error:              ... 12 more
jvm 1    | WrapperSimpleApp Error:      Caused by: java.net.SocketException: Permission denied
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind0(Native Method)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
jvm 1    | WrapperSimpleApp Error:              ... 17 more
jvm 1    | WrapperSimpleApp Error:      Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:443
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.Server.doStart(Server.java:401)
jvm 1    | WrapperSimpleApp Error:              ... 12 more
jvm 1    | WrapperSimpleApp Error:      Caused by: java.net.SocketException: Permission denied
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind0(Native Method)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.Net.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
jvm 1    | WrapperSimpleApp Error:              at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
jvm 1    | WrapperSimpleApp Error:              ... 17 more
jvm 1    | WrapperSimpleApp Error: Caused by: [CIRCULAR REFERENCE: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:80]
jvm 1    | I [SystemInfo                    ] [08:59:53]: os.name="linux", os.arch="amd64".
jvm 1    | I [SystemInfo                    ] [08:59:53]: Validating native library "ignition-x64"...
jvm 1    | I [SystemInfo                    ] [08:59:53]: Validated ignition-x64 (libignition-x64.so)
jvm 1    | I [SystemInfo                    ] [08:59:53]: Loading native library "ignition-x64"...
jvm 1    | I [SystemInfo                    ] [08:59:53]: Native library load complete
jvm 1    | I [g.ModuleManager               ] [08:59:53]: Loading modules....
wrapper  | <-- Wrapper Stopped

Looks like you are trying to use the standard HTTP & HTTPS ports with a non-root user. It can be done, but only by adding a clause to your service file. See this topic:

Thanks for the reply. AmbientCapabilities=CAP_NET_BIND_SERVICE was added in the earlier version. Seem like a the service file is overwritten during the upgrade so have to add it again.

Make sure to use systemctl edit <service name> to supplement your systemd service configurations. This creates an override file that won’t be overwritten with baseline updates to the service.

In case it helps the next guy on this thread, you should be adding

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

to the window that opens when you run this command, then hit ctrl+x, enter to save and exit