Handling a JSON list of metrics in MQTT

Ignition
1

I am currently working on a system, that will receive data via MQTT engine in JSON format. Some structures are lists of metrics like the following:

   {
    "id": "550e8400-e29b-11d4-a716-446655440000",
    "machineId": "WM703
    "eventType": "machine-data",
    "eventTime": "2024-11-09T19:32Z",
    "eventCounter": 2,
    "data": [
        {
            "name": "temperature",
            "value": 75,
            "measurementUnit": "°C",
            "dataType": "float",
        },
        {
            "name": "pressure",
            "value": 1.5,
            "measurementUnit": "bar",
            "dataType": "float",
        },
        {
            "name": "operational",
            "value": true,
            "measurementUnit": "",
            "dataType": "boolean",
        },
       ...
    ]
}

Usually we would want to publish this list with report by exception - meaning only the array fields, that changed will be transmitted.
I think when using SparkplugB Ignition can handle if the length of the data-Array changes. With JSON however it messes up the Tags in MQTT engine, because array fields will just get replaced and not properly updated. If for example an update would set the "operational" value to false, the structure in the MQTT engine would just replace the temperature and look like this:

 {
    "id": "550e8400-e29b-11d4-a716-446655440000",
    "machineId": "WM703
    "eventType": "machine-data",
    "eventTime": "2024-11-09T19:32Z",
    "eventCounter": 2,
    "data": [
        {
            "name": "operational",
            "value": 0,
            "measurementUnit": "",
            "dataType": "boolean",
        },
        {
            "name": "pressure",
            "value": 1.5,
            "measurementUnit": "bar",
            "dataType": "float",
        },
        {
            "name": "operational",
            "value": true,
            "measurementUnit": "",
            "dataType": "boolean",
        },
       ...
    ]
}

Is there any other option, than manually parsing the data into tags?
Are there any plans to extend the functionalities of the MQTT engine?

2

The MQTT point is the unit of change that gets efficiently handled. If you may an array the published point, any change to the array will send the "new" array.

Publish each element of the array separately to get efficient updating of single array elements.

3

Thank you @pturmel.
I can not change the format as there are other consumers that will work with this structure.

I was just wondering why JSON payloads are treated differently than the Sparkplug metrics. Ignition can handle different length of the SpB metrics array.

4

One more thought: the problem is the that the array is not fully updated in the MQTT engine tag provider. If the first published array has a length of 3 and the second publish just contains 1 element, the array in the tag provider will still have 3 elements after the second publish, bit only the first one will get updated. This makes it quite cumbersome even if I want to parse it manually.

5

The only problem I see is that the MQTT engine doesn't remove the other tags after receiving a subsequent message with a different (shorter) array length, although I suspect if you talk to Cirrus Link that will be something of a "known" issue.

If I'm understanding this right, you've:

  1. chosen a schema that is a fundamentally poor fit for MQTT in the first place
  2. invented your own partial/delta update scheme for these arrays embedded in that structure

I'm not sure you expect a generic Subscriber to know that suddenly receiving an array of a different length means it's really just a partial update to some other element in the original array. How would it know that? How would it locate the elements?

6

Not sure why you think that schema is a bad fit for mqtt. I think it makes way more sense to have a generic data object, that can handle different data points from a variety of producers.

My expectation how the array would be handled is probably influenced by the way SpB is handled.

And you are right, the proper update of the array would solve the problem.

7

Because like Phil already mentioned, the unit of change is the topic. You are publishing a large complex structure and expecting that only isolated parts will change. If this isn't published over a bandwidth sensitive connection then I guess it doesn't matter, but you're missing out on the efficiencies that report by exception actually offers because you have to report the whole structure every time any part of it changes.

1 Like
8

Right, and SpB is a published spec that both sides can agree on the behavior of... but now you're just doing generic MQTT with a JSON structure you invented, right?

What if the array elements didn't have a "name" member? What if that member was called "field" instead? What if it was a different shape? What if it was primitive elements?

1 Like
9

@Kevin.Herron got your point thx!