Transaction group stuck 'unable to start' after reboot, no way to monitor or auto-recover?

Ignition
1

We're on Ignition 8.3.7. When SQL Server isn't up yet at gateway reboot, the first transaction group to load fails to start and doesn't retry. Every other group in the project handles the same outage fine and starts degraded, but this one just sits there until we manually disable and re-enable it.

Questions:

  1. Is there a supported way to monitor transaction group status from outside the designer?
  2. Is there a way to restart or re-enable a transaction group programmatically?

What we've tried:

  • Delayed start so Ignition boots after SQL (fixes the root cause, but doesn't help if it happens again for another reason)
  • The status API works in a browser but we can't authenticate to it from a script
  • EAM's agent status/history functions only show agent-level info, not individual groups
  • Error rate alarms on the EAM controller would catch it indirectly, but not the group itself

Has anyone found a way to monitor or auto-recover a stuck group like this?

2

Is there a supported way to monitor transaction group status from outside the designer?

Yes. The Gateway webpage has a Status → Transaction Groups (also under Services) page at /services/transaction-groups, showing per-group running/errored/fatally-errored state.

For scripted/external monitoring, the underlying REST endpoints are:

- GET /data/fsql/api/v1/status/projects — per-project running/non-running/errored counts + throughput

- GET /data/fsql/api/v1/status/project/{name} — per-group path/state/error detail for a single project

Note that these are monitoring only, and don't give you a way to act on the transaction groups.

Is there a way to restart or re-enable a transaction group programmatically?

Not at this time (but this is a great feature request!)

I'll also file a ticket to fix the underlying issue so you'll have less need to restart the groups. Thanks for letting us know!

3

Another point to consider is that it is best practice not to run Ignition and DB within the same physical server/VM. Having the DB running all the time on a separate instance would also prevent this from happening.