Historian with MQTT engine

My current setup has an on-premises server and a cloud server. The on-prem server talks to the PLC's and has an MQTT transmitter to transmit all the tag data to the cloud server.

Currently, the cloud server has a historian license, as the historian database is also in the cloud. However, I've just realised that this may not work the way I intended - since the on-prem server has no historian license, none of the "tag history" parameters show up when I'm defining my UDT's or tags on that server.

What's the "correct" way to handle a historian with an MQTT engine? Should the on-prem server have the license, and send the data directly to the historian in the cloud? Or can the cloud server have the historian? If the cloud server has the license, how can I get my UDT's from the on-prem server (or standard tags, for that matter) to have tag history enabled when they arrive in the cloud?

I thought MQTT had store and forward capabilities already built into it. It forwards each packet with a timestamp. As long as the MQTT driver you are using has a buffer built into it, what you have should work.

I maybe wrong tho. Just something to look into.

You are sending MQTT data to the MQTT Engine in the cloud, so whatever configuration you have on your on-prem instance will not transfer to the cloud. No configuration of the tag will transfer. (Alarms / History etc...)

You would need to enable history on your MQTT tags in the cloud. So, the historian in the cloud would need the license.

I hadn't picked up on the alarms yet either. That's frustrating.

Can I at least configure alarms (and tag history) on the UDT definitions in the cloud, and have it auto populate through to each instance? Going through each instance and adding alarms would be an enormous headache.

Take a look at MQTT Transmission - Advanced Settings and specifically the Filtered Properties. You can ship more configuration than what is configured in the default settings here.

I spotted that before and promptly forgot it again. Doesn't solve the problem for the tag history parameters not showing up on the local server to be transmitted in the first place unfortunately, but good to know about!

It also doesn't look like the alarm information is being filtered out, so I'd imagine that's simply a case of not being able to be transmitted, and having to add them back in at the cloud level.

Annoyingly, the string format also doesn't seem to be transmitted, and that's not listed in the filter either. That's a pain - I've been meticulous about making sure my format strings are correct across every analog member of my UDT's so that they always display properly on displays. Guess I'll have to do that manually to the UDT's in the cloud too.

First time with MQTT so there was always going to be some learnings to be had!

It is correct, you cannot transmit every tag property with MQTT Transmission.
This is how we have done our MQTT setup:

  • We do short term history storage on site for the HMI screens onsite.
  • We do long term history storage on the cloud side, you need a few settings right to make sure store and forward is working properly
  • Alarms are not transmittable, but we needed functionality to acknowledge alarms both onsite as in the cloud gateway. The only way to cover this use case currently is having a remote tag provider with remote alarm journal setup. It's confusing at first, but the RTP doesn't actually send tag data to the cloud as long as these tags are not used for storage and building screens (you have the MQTT tags for that). So only the alarms will be send. Ofcourse if your use case only needs alarms in the cloud, you can just create the setup in the cloud without the RTP.
  • We have all of our MQTT tags copied in reference tags in a separate tag provider. This was needed to cover the following use case: Sometimes our site UDT's would change. right now you have to delete the UDT in MQTT Engine before the update can come through. This also deletes all alarm/tag history configuration, which is not what you need ofcourse. This way of setup also enables you to add extra tags in your provider or add extra tags in your udt's. All you need for this is a script that copies all tags to a tag provider.

For those that have cellular/satellite connections on the edge - the RTP is not an option as it would provide high data costs. I would assume in your case you are using the RTP with a WAN connection and not a remote connection?

I am in the same boat right now setting up a client with a remote connection and trying to figure out the best way to get alarms to the AWS Ignition instance. If using MQTT and setting up alarming on the cloud side and the remote connection fails, I have read that the timestamps will all be off when the connection comes back.

If your use case uses cellular data, I think the best you can do is indeed to configure alarms on the cloud instance.

Roughly how many tags are you referencing from your tag provider to the MQTT engine? We are looking at doing this same thing, but unsure how the system will handle this as we scale up.

We actually have 1000s of tags, I have a tag script that when triggered will browse the MQTT Engine and create the Reference tags automatically in the "Default" tag provider. So, there is nothing to do manually besides toggle the script trigger which scales well.

We tested it with 100k tags without any issues.