Ignition Gateway stuck in starting - probably tags not loading correctly

Hello

we have Ignition Gateway run on customer server that have been down for around 6-7 hours.
I need to get it back on by morning (in around 8 hours) when the systems will record new data.
I’ve searched through forums and have not yet found the solution to this issue.
it seems to me that we have some issue loading the tags during the ignition Startup.
I’ve tried so far deleting the jre-win folder and increasing the java heap size.
I upload the wrapper logfiles from today.
wrapper.zip (1007.2 KB)

Also short part of wrapper in quotes where I see that “pv_tags” does not finish with “complete tag loadin in xx ms” similar to previous tag providers.

INFO   | jvm 1    | 2022/09/14 23:08:49 | I [G.Manager                     ] [21:08:49]: Setting up trust list managers... 
INFO   | jvm 1    | 2022/09/14 23:08:49 | I [G.Manager                     ] [21:08:49]: Trust list managers setup completed in 13 ms 
INFO   | jvm 1    | 2022/09/14 23:08:50 | I [c.i.i.g.p.r.ProjectRecordConverter] [21:08:50]: Conversion finished. Elapsed time: 15 ms 
INFO   | jvm 1    | 2022/09/14 23:08:52 | I [g.ModuleManager               ] [21:08:52]: Setting up modules 
INFO   | jvm 1    | 2022/09/14 23:08:52 | I [G.L.A.AlarmNotificationService] [21:08:52]: Remote Alarm Notification Manager initialized successfully. 
INFO   | jvm 1    | 2022/09/14 23:08:52 | I [c.i.i.g.o.KeyStoreManager     ] [21:08:52]: Loading KeyStore at C:\Program Files\Inductive Automation\Ignition\data\opcua\client\security\certificates.pfx 
INFO   | jvm 1    | 2022/09/14 23:08:52 | I [c.i.i.g.o.KeyStoreManager     ] [21:08:52]: Loading KeyStore at C:\Program Files\Inductive Automation\Ignition\data\opcua\server\security\certificates.pfx 
INFO   | jvm 1    | 2022/09/14 23:08:53 | I [o.e.m.o.s.s.OpcUaServer       ] [21:08:53]: Java version: 11.0.14.1 
INFO   | jvm 1    | 2022/09/14 23:08:53 | I [o.e.m.o.s.s.OpcUaServer       ] [21:08:53]: Eclipse Milo OPC UA Stack version: 0.6.4 
INFO   | jvm 1    | 2022/09/14 23:08:53 | I [o.e.m.o.s.s.OpcUaServer       ] [21:08:53]: Eclipse Milo OPC UA Server SDK version: 0.6.4 
INFO   | jvm 1    | 2022/09/14 23:08:54 | I [o.e.m.o.s.s.n.OpcUaNamespace  ] [21:08:54]: Loaded 2004 nodes in 775ms. 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.PersistentRecordSecurityLevelConfigService] [21:08:59]: Started up in 0 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.SecurityLevelManager        ] [21:08:59]: Started up in 1 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.PersistentRecordIdpAdapterConfigService] [21:08:59]: Started up in 19 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.PersistentRecordIdpAdapterMetricsService] [21:08:59]: Started up in 0 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.IdpAdapterManager           ] [21:08:59]: Started up in 0 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.InternalDatabaseRememberedSubjects] [21:08:59]: Started up in 5 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [g.OIDCProviderManager         ] [21:08:59]: Started up in 6 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.m.provider                  ] [21:08:59]: Tag provider initialized. Starting to load tags... provider=System
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.m.provider                  ] [21:08:59]: Complete tag loading in 92 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.manager                     ] [21:08:59]: Instantiating provider 'default' of type 'STANDARD' 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.m.provider                  ] [21:08:59]: Tag provider initialized. Starting to load tags... provider=default
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.m.provider                  ] [21:08:59]: Complete tag loading in 17 ms 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.manager                     ] [21:08:59]: Instantiating provider 'pv_tags' of type 'STANDARD' 
INFO   | jvm 1    | 2022/09/14 23:08:59 | I [t.m.provider                  ] [21:08:59]: Tag provider initialized. Starting to load tags... provider=pv_tags
INFO   | jvm 1    | 2022/09/14 23:09:00 | I [t.e.dispatcher                ] [21:09:00]: ProjectManager not yet running, rescheduling for +2 seconds tag=valAppRefresh
INFO   | jvm 1    | 2022/09/14 23:09:01 | W [g.LicenseManager              ] [21:09:01]: Could not read license from HASP key. HASP login() failed. Last error = 14. 
INFO   | jvm 1    | 2022/09/14 23:09:02 | W [p.HistoryManager              ] [21:09:02]: Tried to store history while history manager was not running - data will be lost. store-forward-name=pv_db
INFO   | jvm 1    | 2022/09/14 23:09:02 | W [a.Execution                   ] [21:09:02]: (prov:pv_tags:/tag:farm23/gInverters/gInverter3/alValue2:/alm:Alarm_10) Some bound alarm properties failed to compile. They will be replaced with static default values. For more information, turn logger 'alarm.Execution' to 'Debug', and reinitialize the alarm. tag=alValue2
INFO   | jvm 1    | 2022/09/14 23:09:02 | W [a.Execution                   ] [21:09:02]: (prov:pv_tags:/tag:farm23/gInverters/gInverter3/alValue2:/alm:Alarm_11) Some bound alarm properties failed to compile. They will be replaced with static default values. For more information, turn logger 'alarm.Execution' to 'Debug', and reinitialize the alarm. tag=alValue2
INFO   | jvm 1    | 2022/09/14 23:09:03 | W [a.Execution                   ] [21:09:03]: (prov:pv_tags:/tag:farm23/gInverters/gInverter3/alValue2:/alm:Alarm_12) Some bound alarm properties failed to compile. They will be replaced with static default values. For more information, turn logger 'alarm.Execution' to 'Debug', and reinitialize the alarm. tag=alValue2
INFO   | jvm 1    | 2022/09/14 23:09:03 | W [a.Execution                   ] [21:09:03]: (prov:pv_tags:/tag:farm23/gInverters/gInverter3/alValue2:/alm:Alarm_6) Some bound alarm properties failed to compile. They will be replaced with static default values. For more information, turn logger 'alarm.Execution' to 'Debug', and reinitialize the alarm. tag=alValue2
INFO   | jvm 1    | 2022/09/14 23:09:03 | W [a.Execution                   ] [21:09:03]: (prov:pv_tags:/tag:farm23/gInverters/gInverter3/alValue2:/alm:Alarm_7) Some bound alarm properties failed to compile. They will be replaced with static default values. For more information, turn logger 'alarm.Execution' to 'Debug', and reinitialize the alarm. tag=alValue2
INFO   | jvm 1    | 2022/09/14 23:09:04 | W [a.Execution                   ] [21:09:04]: (prov:pv_tags:/tag:farm23/gInverters/gInverter3/alValue2:/alm:Alarm_8) Some bound alarm properties failed to compile. They will be replaced with static default values. For more information, turn logger 'alarm.Execution' to 'Debug', and reinitialize the alarm. tag=alValue2

Would be worth trying an upgrade to Ignition 8.1.20 and seeing if that fixes or speeds up the startup.

And start a support ticket! This forum is not official support.

@pturmel
I’ll surely open support ticket - thank You for reminding me that possibility.
I thought to write forum post as possibly someone could’ve had similar issue before.

Update: I’ve left the Gateway trying to startup for 6 hours and with no success so far.
I can safely assume in current stat the Gateway will not startup without additional actions.

The gateway started after increasing the Memory after the actions mentioned in first post in around 1h of starting.
Currently while server is running it have high memory usage around 7.7mb out of 8.2mb of ram, and low CPU usage. Previous allocation was 4.1mb.
@Kevin.Herron
Should I update this directly from installation file? I’m working on “windows server 2019”, os.arch=“amd64”.