Ignition Service Half Crash

Ignition
1

Hey guys,

Sometime last night our Primary ignition server had some type of error that caused the system to stop working. CPU usage was pegged, OPC communications seemed to have stopped and trending was not working at all. The gateway was still up however, so all of the clients were connected to the Master and the backup never took over because technically the master was still up but hours of data was lost. I restarted the Ignition service this morning when I got in and everything seemed to be ok. I looked through the logs and there is 50 MB of the following error:

INFO | jvm 1 | 2015/05/13 07:33:38 | ERROR [PublishRequestPump ] [07:33:38,545]: Received failure notification for request '[PublishRequest requestHandle=43875664]'. INFO | jvm 1 | 2015/05/13 07:33:38 | StatusCode[Severity=Bad, Subcode=Bad_SessionIdInvalid]: ServiceFault: StatusCode[Severity=Bad, Subcode=Bad_SessionIdInvalid] INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.client.stack.TCPClientChannel.validateResponseType(TCPClientChannel.java:824) INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.client.stack.TCPClientChannel.receiveMessage(TCPClientChannel.java:785) INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.common.stack.UAChannel$1DeliverMessage.deliver(UAChannel.java:967) INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.common.stack.UAChannel$DeliverToDelegate.run(UAChannel.java:1599) INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.client.stack.SerialExecutionQueue$RunnableExecutor.execute(SerialExecutionQueue.java:100) INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.client.stack.SerialExecutionQueue$RunnableExecutor.execute(SerialExecutionQueue.java:97) INFO | jvm 1 | 2015/05/13 07:33:38 | at com.inductiveautomation.xopc.client.stack.SerialExecutionQueue$PollAndExecute.run(SerialExecutionQueue.java:75) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.util.concurrent.FutureTask.run(Unknown Source) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) INFO | jvm 1 | 2015/05/13 07:33:38 | at java.lang.Thread.run(Unknown Source)

I don’t know if this is the cause or if this is an effect. I can’t go back to when it happened because that 50 MB of logs spans about 3 minutes, all this error. Can someone tell me what this error is pointing to? Could it be a memory issue (memory was pretty close to being maxed out)?

2

Can you email your logs into support?

It’s hard to tell if the Bad_SessionIdInvalid is a side effect or the cause, but either way that particular bug been fixed for the latest release.

3

Hey Kevin,

The log files were too big to e-mail so I e-mailed a link to a shared google drive. FYI.

4

Got em.

For future reference, when you download the logs.bin.gz from the gateway console they will be compressed for you.

When you send wrapper.log files in you can zip them first and they will compress to a trivial size.

5

Will do. I will try and log that into my brain, but I seem to forget so many things these days…

6

Was this connection to the local Ignition OPC-UA server or a 3rd party (Kepware) server?

Either way, the logs got filled by the SessionIdInvalid spam and there’s no entries from before it started. Grab 7.7.5 when it’s released and that shouldn’t happen again.

7

It’s a Kepware server, I don’t use the Ignition OPC-UA server at all. I will upgrade when the 7.5 version comes out. Thank you!