Alarm Transitioning in Pipeline for 5 days

I am using Ignition version 7.7.2 on Windows Server 2008 R2.

During installation we were able to test alarms and we did receive callouts. The customer contacted me after several days of not receiving any alarm notifications. I checked the alarm pipeline status and I am seeing maybe 2 dozen alarms that show their status as “transitioning” from a delay block for a little over 5 days. See attached screen shot. EDIT: The time listed under “Time in Block” ranges from 14 hour to over 5 days. They are NOT all the same time.

Has anyone else seen this or have an idea of what could be causing these alarms to hang on this step? The only thing after the delay block is a notification block.

Same problem with 7.7.3_rc1 on debian linux.

I can confirm the same problem exists on 7.7.3 rc1 on Windows too.

Mayon - or anyone else that has experienced this issue - can you check and see if the problem coincides with a system restart? My Ignition logs show a lot of master/backup syncing happening right before the first alarm gets caught in the pipeline. The windows log shows an automatic system restart occurred at that time as well.

Once again, the server restarted last night and this morning notifications stopped working with all alarms hanging in the pipeline.

So i investigated my system logs more thoroughly. The server was reset again around 3:05 this morning. The messages that appear when the Ignition service start seem reasonable. Specifically I see two messages from the Pipeline logger about two pipelines starting.

The next notable log messages reference establishing a connection to the redundant node. These are followed immediately by messages from the Pipeline logger about retiring my current pipelines. The version number to be retired matches what was started up earlier in the log. I do not see any subsequent messages about two more pipelines starting up.

For some reason it appears that the active pipelines are being retired when all of the redundant node stuff is being worked out, but no new versions are taking their place. Log files are attached. The system functions after restarting the gateway (without a system restart).

logs (3).bin.gz (698 KB)

Anyone know anything about this issue in official release of version 7.7.3? I would love to hear that there has been a resolution.

I think this problem, or a problem very similar sounding, has been fixed for 7.7.4.