Having an issue with Alarming.
Using Version 8.
I am seeing multiple alarms in the alarms journal, even though I only have 1 alarms configured on the tag.
Identical source Path,
Identical event time,
Different Event ID.
You probably have another project on the gateway with the same tag configured with the same alarm to the same journalâŚ
@michael.mccabe were you able to ever figure out the cause of this? We are having the same problem. I have a suspicion that a gateway restart will âfix itâ but thought I would post on here first to see if I can get some ideas on what to check since itâs not a critical issue at the moment. If not Iâll open a support ticket.
Some alerts double, others triple. These alert instances have different event idâs, identical event times, identical priority, identical eventtype, and identical source paths. The alerts activate at the same time and clear at the same time. It doesnât happen for all of our alerts, just a handful of them. These alerts are configured within a UDT definition. All of the tags within the UDT are a combination of memory, expression, and SQL tags - no OPC tags. The active and clear delay on the alerts are both set to 60 seconds, so it shouldnât be a chattering issue. These alerts have been updated slightly in the past couple days, so not sure if somehow the old definition of the alert is not getting cleared out? The tag browser only shows one definition.
There is only one alarming journal setup to a MySQL database. We do have one other test server gateway setup, but it is only on a trial license and has been timed out for weeks now. So nothing should be coming from that gateway, plus we are seeing triple of some alerts (additionally, some of the alerts that we are seeing triple of are only defined on the production gateway). No other gateways are running with these tags/alerts. The only odd thing that just appeared in the logs was a database journal error where it says the memory buffer is full but when looking at the store and forward it is completely empty and there is no way that it was full when the error occurred (error below).
My understanding is that tags are all gateway scoped (excluding vision client tags) so the number of projects on the gateway should have no impact on the number of alert instances generated. Plus we would see all alerts duplicated if that were the case, not just a handful of them.
We are using Ignition 8.0.10 running on an AWS EC2 instance (Linux).
DatabaseJournal 08Apr2020 14:24:59 Error logging alarm event to the database.
java.lang.Exception: Unable to add data to memory buffer, buffer full.
at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.insertInBuffer(MemoryStore.java:177)
at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.syncdStoreData(MemoryStore.java:80)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:233)
at com.inductiveautomation.ignition.gateway.history.stores.MultiStageStore.syncdStoreData(MultiStageStore.java:144)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:233)
at com.inductiveautomation.ignition.gateway.history.DefaultStoreAndForwardEngine.storeData(DefaultStoreAndForwardEngine.java:146)
at com.inductiveautomation.ignition.gateway.history.HistoryManagerImpl.storeHistory(HistoryManagerImpl.java:199)
at com.inductiveautomation.ignition.gateway.alarming.journal.DatabaseAlarmJournal.storeRawEvent(DatabaseAlarmJournal.java:203)
at com.inductiveautomation.ignition.gateway.alarming.journal.AbstractAlarmJournal.storeEvent(AbstractAlarmJournal.java:106)
at com.inductiveautomation.ignition.gateway.alarming.journal.AlarmJournalManagerImpl.storeEvent(AlarmJournalManagerImpl.java:281)
at com.inductiveautomation.ignition.gateway.alarming.AlarmManagerImpl.transitionAlarm(AlarmManagerImpl.java:458)
at com.inductiveautomation.ignition.gateway.alarming.evaluation.Alarm.processTransition(Alarm.java:891)
at com.inductiveautomation.ignition.gateway.alarming.evaluation.Alarm.finishTransitionEvent(Alarm.java:848)
at com.inductiveautomation.ignition.gateway.alarming.evaluation.Alarm.initTransitionEvent(Alarm.java:768)
at com.inductiveautomation.ignition.gateway.alarming.evaluation.Alarm.evaluate(Alarm.java:584)
at com.inductiveautomation.ignition.gateway.alarming.AlarmManagerImpl$AlarmEvaluatorImpl.evaluate(AlarmManagerImpl.java:634)
at com.inductiveautomation.ignition.gateway.tags.actors.factories.alarming.AlarmingActor.processValue(AlarmingActor.java:170)
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.ExecutableTag.valueChanged(ExecutableTag.java:319)
at com.inductiveautomation.ignition.gateway.tags.actors.factories.value.AbstractValueGeneratingActor.fireValueChange(AbstractValueGeneratingActor.java:55)
at com.inductiveautomation.ignition.gateway.tags.actors.factories.value.AbstractValueGeneratingActor.fireValueChange(AbstractValueGeneratingActor.java:42)
at com.inductiveautomation.ignition.gateway.tags.actors.factories.value.AbstractExpressionActor.run(AbstractExpressionActor.java:132)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Hi Will,
@WillMT10
A gateway restart will fix this however certainly not desirable.
Yea most of ours would have been OPC or expression tags.
We had no duplicate journal or anything.
We found the below image setup has fixed our problems in the General Alarm settings.
However a restart is still needed after changing these settings anyway.
Thanks michael. I appreciate sharing the settings but that wonât really work in our case. We want to make sure that any recent unacknowledged alarms are easily visible to the user. Looks like Iâll need to contact support.
With this setup we are still showing All active alarms.
Alarms in our project is setup to auto acknowledge.
We are using the Alarm History to see all cases of alarms, so we will only ever show 1 alarm of its type in the active alarms page.
For anyone else that may come across this post, support was not able to determine the root cause of the issue, but we were able to get it resolved for at least the time being. The fix was to stop the gateway, delete the alarms cache (on linux it is a hidden file in the âvar/lib/ignition/dataâ folder called â.alarms_â), and start the gateway. We are keeping an eye on things to see if starts happening again.
1 Like
We are having the same issue on multiple servers. Can someone tell me the path to the alarm cache on a Windows Server install so I can clear it?
Is Inductive looking into this issue or is this the final fix?
Youâll find it in the C:\Program Files\Inductive Automation\Ignition\data folder. It is a hidden file that starts with .alarms_ and will end with what looks like a random set of numbers (i.e. - .alarms_1585163708133). Make sure you stop your gateway before removing (I moved it to a separate directory as a just in case instead of deleting it).
Just a heads up that we are still having this problem on our gateway. Every time we make an edit to a tag in a UDT, that tag has the duplicate alarms start appearing again. Restarting the gateway seems to âfix itâ. This issue has been an open support ticket since my last post back in AprilâŚ
I forgot to close the loop here. Once we upgraded from 8.0.15 to 8.0.16 we noticed that we had a bunch of new, weird, duplicate UDT instances in our tag browser with a â_duplicate_Xâ suffix (X being replaced with a number to indicate the duplicate count). No idea where these tags came from as they were never there before. Support did some digging and could see that they did exist in our 7.9 gateway backup but were NOT visible in the tag browser and did NOT cause duplicate alarms. Once we deleted these duplicate UDT instances from our 8.0.16 gateway, the duplicate alarms stopped. Havenât had an issue since (about a year ago).
Hi Will,
@WillMT10
A quick question here, as I seem to be experiencing a similar situation, this time upgraded from an 8.0.3 to 8.0.16. Did the duplicate UDT instances just appear after upgrading? How did you manage to find them in your tag browser?
Yup, they just appeared. Found them by just scrolling through the tag browser looking for UDTs that ended in â_duplicate_Xâ (X being replaced with a number to indicate the duplicate count). So if I had a UDT named pump_200, if there were duplicates, I would see pump_200_duplicate_1, pump_200_duplicate_2, etc. If you have too many UDTs, you could create a script to search through for any tags that contain âduplicateâ in their name.
Hi Will,
Thanks for the quick reply. In my case, no duplicates seems to have appeared in the tag browser. Iâll have to follow up with support. Will
update this thread when I have some information to share.
I just wanted to say a sincere thank you for coming back with the updates.
I quickly got to the bottom of a similar issue thanks to your post.
Somehow we had triplicate tags in the tag browser which was smashing our PLC and causing the duplicates.
it looked like this
- Area 1 tags
- Lots of Tags for area 1
- Area 2 Tags
- Lots of Tags for area 2
- Area 1 tags
- Lots of Tags for area 1
- Area 1 tags
- Lots of Tags for area 1
Deleting the Duplicate folder worked a treat.
No idea how it happened (update or human error).
Anywho. Thank you 
1 Like