Yesterday we had an issue with Ignition, for some reason the connection to an OPC stopped working, but everything seemed to be normal, even in the status all OPC connections were reported as “Connected”, when I manually tried to write to a tag I got a notification that IGN was not able to write the tag, after a couple of minutes of looking around (I thought the problem was in the PLC side) I decided to review the logs, and noticed several errors in the log.
Is there a way to programmatically read the logs and trigger some notifications?
I am thinking something like develop a script that somehow reads the logs, detects errors and triggers notifications depending on the errors.
Even better solution would be, if the Ignition had in the settings (on the gateway web page) something like ‘Send Email notification on…’ when record in the wrapper log is of certain type (of your choosing: ERROR, WARNING, INFO,…).
I haven’t tried this out yet, but I did notice that the wrapper that Ignition uses is capable of enabling remote syslog (in addition to the local console and file (wrapper.log) logging). See the wrapper.syslog.xxxx keys in the wrapper docs here. These could be added to the ignition.conf file, presumably to enable these features.
You could then probably look at some of the existing solutions out there (I’m not an expert on these, FWIW). One that I found, for example, is Papertrail. You can read about their Cloud Syslog offering here. There are numerous logging services out there though, and they’re likely designed to be able to add keywords or trigger on known patterns (in various logger types) like you’re looking for. Might be worth a look.
Hope this helps (and please share if you come up with a nice solution!).
We have that, When I saw the notification I began checking other stuff, (Network, PLC, etc...)
What I am trying to find is a way to notify about errors in the logs that can give us more insight on what the problem is. On top of that, there may be problems that not related to equipment and still will show up in the logs.
Keep in mind that some errors in the wrapper log might reflect severe problems that prevent Ignition from passing on these log entries itself. Kinda’ like leaving a sick person alone and telling them to call you when they pass out. A truly reliable monitor has to be outside Ignition.
That's true, I have considered that, but I believe that would be a major issue.
That's one option I was considering, having an external service to read the wrapper logs and trigger notifications if needed, but I have some concerns, I don't want to affect in any form Ignition's log.
Just as a follow-up (and also a partial response in support of @pturmel’s comment on external monitoring), I did mock up a solution to this. I was unable to get the remote syslog functionality of the Java Wrapper to work. I’m not sure why, it did try to make a connection to Papertrail but it didn’t work. I couldn’t get it to produce much usable debug logs. I was successful in using Papertrail’s remote_syslog2 tool to monitor the wrapper.log and do the rsyslog piece independently. This is probably the best solution for the reasons mentioned above, in the event the wrapper dies altogether. Worked pretty well!
