Hello all. My gateway has started having a problem, shutting down after less than an hour, sometimes restarting, and sometimes not.
The environment is Ignition 8.1.10 on a Windows Server 2016 machine with 32Gb RAM, the SQL server is on the same machine, we have the Vision and Sepasoft OEE modules.
The problem(from what little I can figure out) seems to be the JVM stopping. I’ve read a number of threads of similar problems, but nothing I’ve found or done has resolved the problem, so I’m reaching out to the community(thanks in advance!). I have increased the JVM heap size all the way to 13Gb, and don’t seem to be running out of resources from looking at memory use trends. I’m attaching some log files, and trend of memory, if I can figure out how to. I started the gateway at 6:00cst this morning(2022-02-24), the logs and trend should encompass a shutdown of the gateway. If anyone can point me towards where the problem might be, I would appreciate it. This(or a similar) problem happened a few months ago, and then stopped when I added more RAM. It was running well since, until earlier this week I created and enabled a new production line and equipment in the OEE setup. I have disabled the new line, thinking I did something wrong with it, but the gateway shutdowns are continuing.
You should really contact support.
At a glance, it sounds like your underlying system is simply overwhelmed. The big telltale are messages like this:
2022-02-23 00:16:33:000
Pinging the JVM took 28 seconds to respond.
2022-02-23 00:16:33:000
Pinging the JVM took 23 seconds to respond.
2022-02-23 00:16:33:000
Pinging the JVM took 18 seconds to respond.
2022-02-23 00:16:33:000
Pinging the JVM took 13 seconds to respond.
2022-02-23 00:16:33:000
Pinging the JVM took 8 seconds to respond.
2022-02-23 00:17:09:000
JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
2022-02-23 00:17:10:000
JVM exited after being requested to terminate.
These are being logged semi-frequently by the ‘service wrapper’ (the long-lived process that actually starts Ignition) and indicate that the IPC between the wrapper and Ignition itself is failing. That’s pretty unusual outside of severe resource starvation issues.
How’s the CPU usage of this server? Is it a dedicated physical server, or a virtual machine? If it’s a VM, how are its resources allocated compared to the host/what else is the VM host running?
I would generally recommend not having SQL server (or any other RDBMS) on the same server as Ignition. DBs and Ignition both tend to ‘compete’ pretty strongly for resources like CPU time and memory.
In addition to Paul’s sound guidance and inquiries, I’ll also mention that I’ve had to explicitly set memory limits on SQL Server in the past (in situations where services were already co-located) to keep it from scooping up all available memory over longer uptime durations. It tends to squeeze out competing applications on a given machine if you’re not careful… Check out some of the SQL Server configurability options here for more info: Server memory configuration options - SQL Server | Microsoft Docs
Thanks for the responses. I intend to contact support, but was waiting for the workday to start out west. I’ll look into SQL resource use. Migrating that to a different box will have to be a longer term project. This is a dedicated server, not a VM.
In the short term, set your min and max memory to the same value in ignition.conf. That way Ignition claims it “up front” and SQL Server cannot steal it.
Thank you, everyone, for your suggestions. It’s been back online for over two hours, so I am cautiously optimistic.
I implemented a combination of things: set the min and max heap size to the same value in ignition.conf, and set the max allowed memory in SQL. Then I rebooted the server, just to flush any caches and allow everything to reset and restart fresh.
I’ll continue to watch it, while putting together a request for anew server to host the gateway.