Transaction Groups Stopped Execution

Ignition
1

Hi,

We are running Ignition Version 8.1.7 and have just had an issue with all the Transaction Groups failing to update at the ‘Next execution’ time.

Info at time of fault;

Ignition Version: 8.1.7 (Licensed)
Module Version: 10.1.7 (SQL-Bridge)
Uptime: 3 Months
CPU: around 4%
Mem usage: Around 6 Gb
Devices: 193, 141 Running (Normal)​, Others offline (Normal).
Transaction Groups: 440, 411 Running, Others disabled, No Errors
Transaction Status: Running (All that are enabled)
Execution Scheduling: Timer (Most 5 to 10 seconds)
Trigger: Run-Always Expression (expression checking for 2 tags to be true)
Next execution time: 10 or 5 seconds after the ‘Last execution’ time (depending on the group)
Database: Online and operational.
Other: No Clock Drift.

Tried;

Pause / Un-pause - Status changed to pause and back to run, group(s) did not execute.

Disabled / Enabled transaction - Status would then change to ‘Unknown, waiting for execution’, ‘Run-always’ trigger would stop updating, Last execution would not update, Transaction would show the ‘?’ mark.

Restarted Sql-Bridge (web interface) - All transaction changed to ‘Unknown, waiting for execution’ last execution would stay blank, Run-Always Expression not updating. No errors shown, No information in Logs, Next execution would show the expected timer after the module restarted. Thread dump looked good.

Restarted Server (gwcmd --restart) - Transaction Groups restarted without issue.

This is the first time I have found this issue, my team member has had the odd issue and found the disable / enable would fix.

What other options would I have to restart the group(s) with out restarting the server?
Has anyone else had a similar issue?

Regards
Matt.

2

I would try restarting the SQL Bridge module. Probably worth reporting to support. I haven’t seen anything on the forums like it. (Fair warning–an official fix will will almost certainly require you to upgrade to latest.)

3

Thanks, will monitor and notify support if it happens again.

4

Sorry for reviving this topic, but I'm having the same issue.
This issue is happening to our system at least from 8.1.31 until 8.1.37 (which we are currently running). When I tried to restart the SQL Bridge module, it would restart but the groups would simply disappear from the gateway, and would be stuck on unknown on the designer.
@Matthew_Woodroffe was this solved for you at the time? If so, could you share what was done to fix it?

5

Issue was not solved, we do restart our servers more often due policy changes, this may mask the issue for us.

6

I see. Our scenario requires restarts to be very sparse, so we can't really go that route.
The only thing I see that could possibly cause an issue with SQL Bridge is that we load remote data from an older SCADA software using it, using DB -> OPCUA, and sometimes that database disconnects due to our client's less then stellar internet connectivity.
Would you happen to also have used SQL Bridge to load data from the DB to OPCUA as well?

7

Has anyone followed my old advice? (The part about reporting to support.) This sounds like a real bug that needs investigating by IA, with someone who can reproduce it.

8

Not yet, I wanted to first make sure there wasn't a publicly known fix. I'll be reporting this issue soon.

1 Like
9

Don't know if this is at all related, but last week we noticed one of our transaction groups had stopped running and we couldn't get it to execute.

Turns out the trigger was set to 2:00 AM, and on March 10th when we changed the time (EST to EDT) it tried executing and generated an error. We ended up changing the trigger schedule to 3:00 AM and it started running again.

10

It might be... Maybe some kind of issue on the scheduling configuration of the module? I'm currently in contact with support, so I'll give an update here if we find a solution to this problem.

11

It's been a while since the last message, but I'm still working with the support team. So far, it seems to be an issue with a hanging connection to the DB.
They also mentioned that this issue might be fixed in the upcoming 8.1.39, as observed in the RC1 release notes.
If we find anything else out, I'll disclose here.

(edit: fixed the link from RC1 to release, as the former has been removed from the notes)

1 Like
12

Also, my issue might be related to the jTDS driver we are using for MSSQL connections. jTDS - SQL Server and Sybase JDBC driver / Bugs / #621 Hang in SocketRead is an existing issue for that driver, and we are currently testing using the extra connection property socketKeepAlive=true; to see if it will work properly.

(edit: this seems to have done the trick in our case, but 8.1.39 is out and should fix other issues related to the transaction groups freezing)

1 Like