Transaction Groups - unknown, db connections broken

Ignition
1

I had an odd one last night.
Most of my transaction groups suddenly went into an unknown state, 2 of my DB connections broke and wouldn’t reconnect, and some of my OPC-UA servers dropped.

Vision stuff kept working just fine…

whole wack of these error msgs


2013-03-18 9:01:05 PM	GetEndpoints TCPClientChannel UAChannel	Uncaught throwable while processing chunk.
 	
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@12df8ef rejected from java.util.concurrent.ScheduledThreadPoolExecutor@1323350[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 59]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Unknown Source)
	at com.inductiveautomation.xopc.client.stack.TCPClientChannel.scheduleRenew(TCPClientChannel.java:429)
	at com.inductiveautomation.xopc.client.stack.TCPClientChannel.receiveMessage(TCPClientChannel.java:621)
	at com.inductiveautomation.xopc.common.stack.UAChannel.processAsymmetricChunk(UAChannel.java:1340)
	at com.inductiveautomation.xopc.common.stack.UAChannel.processChunk(UAChannel.java:808)
	at com.inductiveautomation.xopc.common.stack.UAChannel.access$000(UAChannel.java:66)
	at com.inductiveautomation.xopc.common.stack.UAChannel$ProcessChunk.run(UAChannel.java:1623)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Restarting Ignition cleared it, but I’d like to know the root cause.

2

System Specs

Windows 2003 R2 SP2
4GB Ram
Java 7.17

Ignition 7.5.6

Also attached are the wrapper.log and logs.bin files.
logs.zip (710 KB)

3

Can you tell me the full version and build number of the UA module you have installed?

My guess is that you have 1.5.6 b520, which means you grabbed the botched release that accidentally got uploaded, which included a UA module cut from a development branch. It had a memory leak, which is probably what caused the problems you’re seeing.

If that’s the case, you can grab the correct UA module from the current downloads page.

4

Kevin:

According to the version number I have 1.5.6(b517)

-Mike

5

I grabbed the one from the website and updated it anyhow, Same version number in Ignition though.

Which brings up a minor point, I also grabbed the SQL bridge module as I noticed the site listed the latest as b252, but that installed as b249(which I had from the upgrade already)?

6

You’ve definitely got something weird going on. Your log is full of these messages:

 WARN  [TCPServerChannel              ] [09:06:44,726]: Connection lost due to IOException.

This is the OPC-UA server indicating that a client TCP connection has closed (uncleanly). It’s happening ~15 seconds in your logs. If your regular “loopback” connection is connected and functioning, this would seem to indicate you’ve got some other UA client attempting to connect regularly and probably failing for whatever reason.

Are you handy with Wireshark?

7

Kevin:

Yeah, Wireshark and I are buddies.

-Mike

edit:
Cleared that error, It was a Kepware install on the Ignition box itself.