Hello guys, I'm seeing a very strange problem with the alarm pipeline. I get an alarm that clears in a few seconds. But the pipeline keeps spamming out messages even after that alarm clears. Anyone have a clue as to what is happening?
Some context. This started happening after the server crashed a while ago. And I wasn't around to handle the fallout. So someone else did something to stop the alarms after the crash. I have looked through everything I can think of and fixed everything they did but I thought maybe I was missing some random setting they might have changed. If you have any ideas that would be greatly appreciated.
I'm in the process now of deleting this pipeline and creating a new one to try and solve the problem. I'm hoping that will be enough. But just in case I wanted to post here and pick your brain for anything I might be missing.
Pipeline is pretty basic. Just a loop that sends Email and SMS on 10 min interval. And I only have it set as the Active pipeline in my alarms. Don't currently have pipelines for the clear and ack but that should not matter if I understand correctly.
As I understand it this should be very straight forward. If there is an active alarm on a tag. then that activates the pipeline. and when it clears the pipeline shuts down. So this spamming after alarm is clear is not making any sense to me. Again any help appreciated.
Ok deleting the pipeline did not work. Its like there is a phantom pipeline thread on the server after the crash. that has lost its exit condition.
Next I'm going to try and delete the On Call Roster. To try and stop the spam. If it is not copied and stored in the thread then hopefully that will do it.
Yeah the phantom pipeline never showed up there after the crash.
But the good news is that deleting the On Call Roster. stopped the spam at last.
This is almost certainly a 1 in a million problem, but someone might want to consider adding a way for users to kill threads manually. Ignition must be storing something thread related in memory because restarting the gateway did not kill it. And the gateway already has a way of viewing the threads. just add a button to kill them and weird problems like this would be more easily solvable.
An update to this issue. After much headache and many spam bombs later. I think I have it narrowed down to a network traffic jam. These alarms are checking for Bad Quality. So it’s supposed to catch the event of a network failure or power outage to monitoring equipment. So what I think is happening is that we get a spike in network traffic and that causes a couple of seconds communications loss and we get an alarm. But since the alarms originate on a edge gateway and network is jammed then the main gateway never gets the clear alarm signal or something like that.
This is pure speculation on my part. But I found that setting a few seconds deadband on the alarm solved the issue for me. Now this is only a band aide solution because now I’m potentially missing issues that would tell me if something is failing. But until my network guy figures out what is causing this jam randomly at 6:30 in the morning then this is the only thing I can think of to escape the spam of death.
