SMS notification limitations with Airlink/sierra wireless modem

Ignition
1

Just noticed that if you try to send too many SMS Notifications at once, they will just fail.
This means sending 10 alarm "cleared" messages to 15 users, will fail. Some users will not get the notification.
Replicated it by sending 150 alarms to my phone at once, only about 100 got trough. Confirmed with telecommunication provider.

This fault shows up in the log.
AirlinkUdpConnection 17Dec2024 15:05:40 Received ack: ACK301ACK - No Buffers available.

I guess the buffer/memory in the Sierra wireless modem cant handle more than approximately 100 messages waiting to be sent.

2

What model are you using?

3

We are using RV50X

4

Ah. We use a different model, but we also don't get that volume of alerts.

5

Twilio SMS notification is probably a better option for you.

6

We are starting to realize that twilio might be the way :sweat_smile:
Im also thinking setting up Email notifications might be more reliable than SMS.

Had some reports of missing SMS notifications a few times in the past, but never got to the bottom of it.
This might not be related to the previous incidents, but at least this one was easy to replicate and confirm. And it gave us a warning in the gateway log.

Took a look at the RV50 logs now and this is how it looks like:

Dec 17 14:05:40 err ALEOS_SERVICES_SMSPROC: SMS Buffer full
Dec 17 14:05:40 info ALEOS_SERVICES_SMSPROC: Sending message index 69. Next Buf Index 69
Dec 17 14:05:40 err ALEOS_SERVICES_SMSPROC: sendMsgToQCon failed (-4)
Dec 17 14:05:40 err ALEOS_SERVICES_SMSPROC: Buffers apparently full when sending
Dec 17 14:05:40 info ALEOS_SERVICES_SMSPROC: sendMsgToQCON: Logging message At 15:05:40, alarm "Alarm" at TEST PIPELINE transitioned to Active. To acknowledge, reply '717619' of size 98 to QCON
Dec 17 14:05:40 info ALEOS_SERVICES_SMSPROC: sendMsgToQCON: Response is At 15:05:40, alarm "Alarm" at TEST PIPELINE transitioned to Active. To acknowledge, reply '717619' destination *PHONENUMBER* sms size 98

The previous incidents from memory:

  1. Customer states one SMS not received on 3/10 phones, all messages marked as success in audit log. Blamed ISP because it looked like they were sent from us with the "success" status in the audit log.
  2. Customer states one SMS not received at all, i can find the alarm event in the alarm journal but no log of any SMS sent to anyone in the audit log. OPC tags that generate this alarm have frequently bad quality for a few seconds due to questionable 4G connection to PLC. But if its registered in the alarm journal it should also have gone trough to the alarm pipeline, right?.. And this would have resulted in something like 14 SMS messages, not 100+ so i don't think its related to this issue.
  3. This one. I notice 14 "AirlinkUdpConnection - No buffer avalible" in the gateway log and start investigating. 132 messages should have been sent, ISP confirms that only 118 was sent, and 107 of the 118 has delivered status.