We have an issue with the MQTT store and forward process being implemented within Ignition.
I have currently 299 tags on a test project, using Ignition’s Sample_SQLiteDatabase. I am using NodeRed to simulate MQTT Sparkplug B module to test out the Store-and-Forward mechanism. I am getting the data displayed in the Perspective session in Designer when I play it. However, when the comms is halted, we just get “No” data. We were expecting the store and forward feature to store the data once the connection is restored but it’s not working.
I can see that you're using the MQTT Sparkplug out node. What is the destination you want to send the data to? If you are publishing data for the MQTT Engine Module in Ignition, then you should use the MQTT Sparkplug Device node.
My second question is; what is the reason why you want to send is_historical data in the first place? is it because you don't have access to realtime data from Node-Red and are pulling historical data at a fixed interval? or is it because you have created you own store-forward buffer system? or something else?
"I intend to send data to the MQTT Engine Module in Ignition. My goal is to use
the Sparkplug Device node, but currently, this node does not recognize the ‘is_historical’ data property.
In my application, I have an edge PC with a mobile connection (GSM, LTE). When there is poor signal strength, I want to save historical data locally and then transmit it to the MQTT broker once the mobile signal becomes available again. To achieve this, I am developing my own store-forward buffer system.
Could you assist me in understanding why the ‘is_historical’ property is not functioning as expected with the Device node?
Currently, I have successfully created a working test flow using the ‘mqtt sparkplug out’ node, specifically with the ‘NBIRTH’ message type. However, I encountered difficulties when attempting to use ‘DDATA’ to create a functional ‘DBIRTH’ on Ignition SCADA. Although I can see ‘DBIRTH’ messages in MQTT Explorer, they do not appear in Ignition SCADA. It seems that I might be making an error in my configuration.
My secondary objective is to become proficient in using all the MQTT-Sparkplug-Plus nodes and successfully create message types such as ‘NDEATH,’ ‘DBIRTH,’ ‘DDEATH,’ ‘NDATA,’ ‘DDATA,’ ‘NCMD,’ ‘DCMD,’ and ‘STATE’ that work seamlessly within Ignition SCADA.
As a newcomer to Node-RED and the Sparkplug B protocol, I am still grappling with some concepts. Your assistance is greatly appreciated. Thank you!"
Step 1: Make sure that you have the latest version of node-red-contrib-mqtt-sparkplug-plus installed (Version 2.1.0+)
Step 2: In Node-Red go to “Import”>”Examples” and select “node-red-contrib-mqtt-sparkplug-plus” > “Simple Device” to create a small sample flow with a MQTT Device.
Step 3: Double click the MQTT Device and go to Advanced settings and select “Store Forward when not connected”.
Now the device will buffer to memory when MQTT broker is not connected and send messages as the connection gets back online (it will automaticly add the is_historical flag to buffered metrics).
Buffer when destination Ignition/MQTT Engine is not connected to the broker:
Step 1: Double click the MQTT Device and edit the broker configuration.
Step 2: Select the “Sparkplug” tab and check the “Store Forward when primary application is offline” and enter the primary application name/Destination (This needs to match what you defined as Primary Host ID under the MQTT Engine. (See example below)
This will setup a memory only store forward. If you want to have a disk backed store forward mechanism, then you need to configure your node-red context store to be disk backed.
I cannot find the “Sets” folder under the MQTT Engine. Could you please guide me on where to locate this folder?
If you are the author of this node plus is there any possibility that in the future MQTT Device node would just except " is_historical" flag without reading state of primary application, because this would work only if I am using Ignition. I would just catch state of Device node and if node is not connected .I would store data in to a folder and send it when node is connected again.
Lastly, I’m interested in learning how to create all MQTT Sparkplug message types using the MQTT Out node. Understanding the Sparkplug specification thoroughly is my goal. If my code encounters any issues, would it be possible to seek advice from you here or in the Node-RED forum?
Sorry, I send a screenshot from the MQTT Transmission module. It's the Primary Host ID in the screenshot you need to set.
You don't need the primary host buffer setup to store forward, it's just an extra "feature" to make sure that you store forward if node-red is connected to the MQTT broker, but the primary application (in this case MQTT Engine) is not. You can just choose to only store forward when node-red is not connected to the broker.
If you want a better understanding of the sparkplug protocol and how the primary application works, then I would recommend that you read the sparkplug specification:
You can send is_historical attribute with all the nodes (but I'm pretty sure you don't want to). It's not really documented anywhere but the attribute needs to be named isHistorical.
Thank you for your answer. I was testing the host buffer setup for store and forward. It is storing data and sending it to the Ignition MQTT server, but Ignition doesn’t recognize the ‘isHistorical’ in your MQTT metric. Did you test that historical metric on Ignition SCADA? If I send my metric with the ‘is_historical’ flag, the history works on Ignition’s ‘Easy Chart’.”Component of a metric should looke like that :
name : Identifier for the metric
alias : A numerical alias for the metric, used to reduce the payload size in repetitive messages
timestamp : Indicates when the metric was sampled or created
datatype : The kind of data the metric holds (e.g., Boolean, Int32, String)
value : The actual data
is_historical : A Boolean flag which denotes whether this metric represents a historical value
is_transient : A Boolean flag which denotes if this metric should not be historized
is_null : A Boolean flag which denotes whether this metric has a null value
metadata : Metadata object associated with the metric
properties : Propertyset object associated with the metrics
In all documentation, even in the Google Protocol Buffer schema, the historical flag is written as ‘is_historical’.”
This is documented on page 64 in specification that you give me.
Hello. Yesterday, I was testing the isHistorical flag with your MQTT Sparkplug B device, the Ignition MQTT Transmission module, and my Node-RED is_historical flag. I was unable to create working historical data with your MQTT Sparkplug Device when forwarding the isHistorical flag( in Ignition SCADA). With the Ignition MQTT Transmission module , historical data is working, but it’s only because the data is packaged into a single metric . However, on the chart, I notice some small gaps. Interestingly, only when using my test flow with the is_historical flag , the historical data appears perfectly on the Ignition chart.
I’ve read the standard you provided twice now, and there is no doubt that the correct historical flag is ‘is_historical’ .
The only place where I find the attribute ‘isHistorical’ is in an example for JavaScript on GitHub.
[In the Sparkplug Specification, it is clearly stated without a doubt that every metric must include a Sparkplug Datatype from the **Sparkplug Protobuf Schema.
I am asking you can you please change attributes based on specification , here are the recommended changes:
Change “isHistorical” to “is_historical”.
Change “isTransient” to “is_transient”.
I am 100% sure that this is what specification are saying and that this changes would work.
