Handling a JSON list of metrics in MQTT

Ignition
1

I am currently working on a system, that will receive data via MQTT engine in JSON format. Some structures are lists of metrics like the following:

   {
    "id": "550e8400-e29b-11d4-a716-446655440000",
    "machineId": "WM703
    "eventType": "machine-data",
    "eventTime": "2024-11-09T19:32Z",
    "eventCounter": 2,
    "data": [
        {
            "name": "temperature",
            "value": 75,
            "measurementUnit": "°C",
            "dataType": "float",
        },
        {
            "name": "pressure",
            "value": 1.5,
            "measurementUnit": "bar",
            "dataType": "float",
        },
        {
            "name": "operational",
            "value": true,
            "measurementUnit": "",
            "dataType": "boolean",
        },
       ...
    ]
}

Usually we would want to publish this list with report by exception - meaning only the array fields, that changed will be transmitted.
I think when using SparkplugB Ignition can handle if the length of the data-Array changes. With JSON however it messes up the Tags in MQTT engine, because array fields will just get replaced and not properly updated. If for example an update would set the "operational" value to false, the structure in the MQTT engine would just replace the temperature and look like this:

 {
    "id": "550e8400-e29b-11d4-a716-446655440000",
    "machineId": "WM703
    "eventType": "machine-data",
    "eventTime": "2024-11-09T19:32Z",
    "eventCounter": 2,
    "data": [
        {
            "name": "operational",
            "value": 0,
            "measurementUnit": "",
            "dataType": "boolean",
        },
        {
            "name": "pressure",
            "value": 1.5,
            "measurementUnit": "bar",
            "dataType": "float",
        },
        {
            "name": "operational",
            "value": true,
            "measurementUnit": "",
            "dataType": "boolean",
        },
       ...
    ]
}

Is there any other option, than manually parsing the data into tags?
Are there any plans to extend the functionalities of the MQTT engine?

2

The MQTT point is the unit of change that gets efficiently handled. If you may an array the published point, any change to the array will send the "new" array.

Publish each element of the array separately to get efficient updating of single array elements.

3

Thank you @pturmel.
I can not change the format as there are other consumers that will work with this structure.

I was just wondering why JSON payloads are treated differently than the Sparkplug metrics. Ignition can handle different length of the SpB metrics array.

4

One more thought: the problem is the that the array is not fully updated in the MQTT engine tag provider. If the first published array has a length of 3 and the second publish just contains 1 element, the array in the tag provider will still have 3 elements after the second publish, bit only the first one will get updated. This makes it quite cumbersome even if I want to parse it manually.

5

The only problem I see is that the MQTT engine doesn't remove the other tags after receiving a subsequent message with a different (shorter) array length, although I suspect if you talk to Cirrus Link that will be something of a "known" issue.

If I'm understanding this right, you've:

  1. chosen a schema that is a fundamentally poor fit for MQTT in the first place
  2. invented your own partial/delta update scheme for these arrays embedded in that structure

I'm not sure you expect a generic Subscriber to know that suddenly receiving an array of a different length means it's really just a partial update to some other element in the original array. How would it know that? How would it locate the elements?

6

Not sure why you think that schema is a bad fit for mqtt. I think it makes way more sense to have a generic data object, that can handle different data points from a variety of producers.

My expectation how the array would be handled is probably influenced by the way SpB is handled.

And you are right, the proper update of the array would solve the problem.

7

Because like Phil already mentioned, the unit of change is the topic. You are publishing a large complex structure and expecting that only isolated parts will change. If this isn't published over a bandwidth sensitive connection then I guess it doesn't matter, but you're missing out on the efficiencies that report by exception actually offers because you have to report the whole structure every time any part of it changes.

1 Like
8

Right, and SpB is a published spec that both sides can agree on the behavior of... but now you're just doing generic MQTT with a JSON structure you invented, right?

What if the array elements didn't have a "name" member? What if that member was called "field" instead? What if it was a different shape? What if it was primitive elements?

1 Like
9

@Kevin.Herron got your point thx!

10

I have a similar problem. MES system publishes some data on the MQTT server when work on the machine starts. Part of this data is also a bill of material - a list of JSON objects. The problem is, when a new product has a shorter BOM as the previous one. Ignition (or MQTT Engine) keeps the tags that were created before and not overwritten with the new BOM elements. It would seem logical to me that when a new list appears in a new JSON payload, all elements of the old list are deleted and replaced with new ones, even if the new list is shorter! I could add empty elements myself, but BOMs have lengths from a few elements to several hundred. This seems to me to be an unnecessary waste of space. The only workaround we came up with is to send information about the length of the BOM and then only consider the first so many elements.

11

My solution for this is to not let the MQTT-Engine parse the payload as JSON. I take it as string reference it in a document_tag and have a change script on this tag parse it as I need it and write it to other tags or in a database.

12

Mapping a JSON Object to tags when its shape will be continuously expected to change is a bit awkward. The JSON elements have something of an ephemeral nature, where tags are a little more permanent.

Often with tags you configure alarms, history, target them in bindings, make them part of tag paths, etc... which is I think why the MQTT module does not simply remove these tags when JSON changes shape in a way that no longer needs them, and that change may be temporary.

13

I would understand this for object structure, not for a list. If we have a list, it's kind of self-evident that its length can change. That's what lists are for. If I wanted a fixed and immutable structure, I could use multiple named objects.

14

It doesn't really matter if its an Array or Object because the same problem remains: the tags exist, which implies you can do all the things I mentioned with them, but tags would be deleted configuration would be lost if/when the list size changed.

Fully mapping a JSON structure to the Ignition tag system is just a bad fit. What you've done using a Document tag is probably the right approach. It would be nice if our Document tags were more powerful though.

15

What about speed and performance? If I map the JSON BOM to a string tag and manually parse it when it changes? Is such a process any slower or more resource-intensive than what the MQTT Engine does? We process approx. 1000 machines, changes happen mostly simultaneously, within 20-30 minutes at shift changes.

16

I don't really have a good sense for the performance characteristics of each, and the MQTT modules are not IA modules so I can't go take a peek to get a better sense.

17

If you must parse manually, you probably want my Integration Toolkit, so you can do it without jython overhead. (Or less overhead.)

18

Thanks! Can you please direct me to where I could get more information about this?
I'm also curious if you recommend any other way to get the BOM from the MES system into Ignition via an MQTT server? Can I map the JSON from MQTT to a Tag of type Document or Dataset? It is not necessary that the JSON list be mapped to a set of Tags. It is only important that Ignition has information about which raw materials are currently used on the machine.

19

Probably should read this whole topic to get a feel for what it can do:

Either. Given your format, I'd recommend a dataset, with dedicated columns for the possible datatypes. (Similar to IA's historian tables.)

Probably something like this in an expression tag (event triggered):

transform(
	jsonDecode({path/to/source/jsonAsString}),
	transform(
		doDate(value()['eventTime'],
		unionAll(
			asMap(
				'machineId', 'str',
				't_stamp', 'date',
				'varName', 'str',
				'varUnit', 'str',
				'boolValue', 'B',
				'longValue', 'L',
				'doubleValue', 'D'
			),
			forEach(
				value(1)['data'],
				value(1)['machineId'], // All rows get this ID
				value(), // All rows get the parsed event time
				it()['name'],
				it()['measurementUnit'],
				if(
					it()['dataType'] = 'boolean',
					toBoolean(it()['value']),
					null
				),
				if(
					it()['dataType'] = 'int' || it()['dataType'] = 'long',
					toLong(it()['value']),
					null
				),
				if(
					it()['dataType'] = 'float' || it()['dataType'] = 'double',
					toDouble(it()['value']),
					null
				)
			)
		)
	)
)