Alarm pipeline failing to send email

Ignition
1

I’m getting an error when trying to send an email via an alarm pipeline. The pipeline uses an SMTP profile, and when I test the profile I can send an email no worries. When I test the pipeline however to the same email (me), it fails with below:

INFO   | jvm 1    | 2020/01/08 11:44:09 | E [a.N.EmailNotificationProfile  ] [01:14:09]: Error sending message to user 'usr-prov:default:/usr:nick'. Remaining retries=0 alarm-pipeline=Developer, alarm-notification-profile=Email, alarm-source=default/_Testing/TEST ALARM/Alarm, alarm-name=Alarm
INFO   | jvm 1    | 2020/01/08 11:44:09 | javax.mail.MessagingException: Exception reading response
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2304)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.smtp.SMTPTransport.issueSendCommand(SMTPTransport.java:2181)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.smtp.SMTPTransport.finishData(SMTPTransport.java:1980)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1197)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at javax.mail.Transport.send0(Transport.java:254)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at javax.mail.Transport.send(Transport.java:124)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.inductiveautomation.ignition.alarming.notification.email.EmailNotificationProfile$EmailTask.sendMessage(EmailNotificationProfile.java:736)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.inductiveautomation.ignition.alarming.notification.email.EmailNotificationProfile$EmailTask.buildAndMaybeSendMessage(EmailNotificationProfile.java:724)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.inductiveautomation.ignition.alarming.notification.email.EmailNotificationProfile$EmailTask.run(EmailNotificationProfile.java:644)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.inductiveautomation.ignition.alarming.notification.email.EmailNotificationProfile.lambda$sendNotification$0(EmailNotificationProfile.java:472)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/08 11:44:09 | Caused by: java.net.SocketTimeoutException: Read timed out
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.net.SocketInputStream.socketRead0(Native Method)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:124)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:89)
INFO   | jvm 1    | 2020/01/08 11:44:09 | 	at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2284)
2

Hello Nick,

Has this error already been solved? Because I have the same issue.

3

Also getting the same thing when our server is trying to send alarm emails:
[Trimmed down snippets from log…]

javax.mail.AuthenticationFailedException: 
Caused by: javax.mail.MessagingException: Exception reading response
Caused by: java.net.SocketTimeoutException: Read timed out

I’m running on Ignition 8.0.12 and this email profile works most of the time. I’d estimate the server sends about 20-30 emails per day and only 1 or 2 are failing this way. Doesn’t seem to be any recurring factor in terms of recipient or time of day. If it matters, we are using smtp.office365.com as our outgoing server.

4

For now I found and doubled the SMTP timeout on the email profile (from 10 seconds to 20). I’m going to run this way for a few days and report back on if it changes anything. (I also adjusted the SMTP timeout on the alarm notification profile in case that one takes precedence over the email profile.)

5

I also discovered that you can adjust the retry count in the notification block of the alarm pipeline in Designer. That had been set to 0 on my pipelines. I increased it to 2.

I haven’t had any emails fail since increasing the timeouts and retries. I suspect I may have resolved my issue, but I won’t feel totally confident until it’s been a week without any failures.

EDIT 4 HOURS LATER:

Just to continue this conversation with myself, I’m much more confident that adjusting the retries on the alarm notification block has solved my issue.

Even with the increased timeout, I just had another error message like above show up in the Ignition log, but this time it said “retries=2” at the end. There was no log message showing that it was retried successfully, but the audit log showed a successful event just after the failed event, for the same recipient and same alarm event ID. There were no retry events in the audit log for my old failures before I increased the retry count.

TO SUMMARIZE: In my case I was getting the mail errors sporadically for unknown issues between my Ignition server and the SMTP server, and increasing the timeouts on the SMTP profile and the retries on the alarm notification pipeline increased reliability enough that I suspect it will not be an issue for me any more.

(I also just learned that the forum software won’t let you be the only person to reply to a thread more than 3 times in a row, and had to put this final comment on as an edit of my prior reply…)

4 Likes