# Handling a JSON list of metrics in MQTT **URL:** \ **Category:** Ignition\ **Tags:** ignition81\ **Created:** [November 29, 2024, 3:50pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266 "2024-11-29T15:50:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![henning.e.heine](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/henning.e.heine/32/70958_2.png) [@henning.e.heine](https://forum.inductiveautomation.com/u/henning.e.heine)\ **Post date:** [November 29, 2024, 3:50pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/1 "2024-11-29T15:50:32Z")
I am currently working on a system, that will receive data via MQTT engine in JSON format. Some structures are lists of metrics like the following: ```python { "id": "550e8400-e29b-11d4-a716-446655440000", "machineId": "WM703 "eventType": "machine-data", "eventTime": "2024-11-09T19:32Z", "eventCounter": 2, "data": [ { "name": "temperature", "value": 75, "measurementUnit": "°C", "dataType": "float", }, { "name": "pressure", "value": 1.5, "measurementUnit": "bar", "dataType": "float", }, { "name": "operational", "value": true, "measurementUnit": "", "dataType": "boolean", }, ... ] } ``` Usually we would want to publish this list with report by exception - meaning only the array fields, that changed will be transmitted. I think when using SparkplugB Ignition can handle if the length of the data-Array changes. With JSON however it messes up the Tags in MQTT engine, because array fields will just get replaced and not properly updated. If for example an update would set the "operational" value to false, the structure in the MQTT engine would just replace the temperature and look like this: ```python { "id": "550e8400-e29b-11d4-a716-446655440000", "machineId": "WM703 "eventType": "machine-data", "eventTime": "2024-11-09T19:32Z", "eventCounter": 2, "data": [ { "name": "operational", "value": 0, "measurementUnit": "", "dataType": "boolean", }, { "name": "pressure", "value": 1.5, "measurementUnit": "bar", "dataType": "float", }, { "name": "operational", "value": true, "measurementUnit": "", "dataType": "boolean", }, ... ] } ``` Is there any other option, than manually parsing the data into tags? Are there any plans to extend the functionalities of the MQTT engine? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 29, 2024, 5:00pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/2 "2024-11-29T17:00:47Z")
The MQTT point is the unit of change that gets efficiently handled. If you may an array the published point, any change to the array will send the "new" array. Publish each element of the array separately to get efficient updating of single array elements. ---
**Author:** ![henning.e.heine](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/henning.e.heine/32/70958_2.png) [@henning.e.heine](https://forum.inductiveautomation.com/u/henning.e.heine)\ **Post date:** [November 29, 2024, 6:10pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/3 "2024-11-29T18:10:54Z")
Thank you @pturmel. I can not change the format as there are other consumers that will work with this structure. I was just wondering why JSON payloads are treated differently than the Sparkplug metrics. Ignition can handle different length of the SpB metrics array. ---
**Author:** ![henning.e.heine](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/henning.e.heine/32/70958_2.png) [@henning.e.heine](https://forum.inductiveautomation.com/u/henning.e.heine)\ **Post date:** [November 29, 2024, 6:22pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/4 "2024-11-29T18:22:18Z")
One more thought: the problem is the that the array is not fully updated in the MQTT engine tag provider. If the first published array has a length of 3 and the second publish just contains 1 element, the array in the tag provider will still have 3 elements after the second publish, bit only the first one will get updated. This makes it quite cumbersome even if I want to parse it manually. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 30, 2024, 2:39pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/5 "2024-11-30T14:39:36Z")
The only problem I see is that the MQTT engine doesn't remove the other tags after receiving a subsequent message with a different (shorter) array length, although I suspect if you talk to Cirrus Link that will be something of a "known" issue. If I'm understanding this right, you've: 1. chosen a schema that is a fundamentally poor fit for MQTT in the first place 2. invented your own partial/delta update scheme for these arrays embedded in that structure I'm not sure you expect a generic Subscriber to know that suddenly receiving an array of a different length means it's really just a partial update to some other element in the original array. How would it know that? How would it locate the elements? ---
**Author:** ![henning.e.heine](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/henning.e.heine/32/70958_2.png) [@henning.e.heine](https://forum.inductiveautomation.com/u/henning.e.heine)\ **Post date:** [November 30, 2024, 3:01pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/6 "2024-11-30T15:01:36Z")
Not sure why you think that schema is a bad fit for mqtt. I think it makes way more sense to have a generic data object, that can handle different data points from a variety of producers. My expectation how the array would be handled is probably influenced by the way SpB is handled. And you are right, the proper update of the array would solve the problem. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 30, 2024, 3:11pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/7 "2024-11-30T15:11:56Z")
> [@henning.e.heine](#): > > Not sure why you think that schema is a bad fit for mqtt Because like Phil already mentioned, the unit of change is the _topic_. You are publishing a large complex structure and expecting that only isolated parts will change. If this isn't published over a bandwidth sensitive connection then I guess it doesn't matter, but you're missing out on the efficiencies that report by exception actually offers because you have to report the whole structure every time any part of it changes. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 30, 2024, 3:13pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/8 "2024-11-30T15:13:31Z")
> [@henning.e.heine](#): > > My expectation how the array would be handled is probably influenced by the way SpB is handled. Right, and SpB is a published spec that both sides can agree on the behavior of... but now you're just doing generic MQTT with a JSON structure you invented, right? What if the array elements didn't have a "name" member? What if that member was called "field" instead? What if it was a different shape? What if it was primitive elements? ---
**Author:** ![henning.e.heine](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/henning.e.heine/32/70958_2.png) [@henning.e.heine](https://forum.inductiveautomation.com/u/henning.e.heine)\ **Post date:** [November 30, 2024, 3:17pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/9 "2024-11-30T15:17:15Z")
@Kevin.Herron got your point thx! ---
**Author:** ![Jani\_Kleindienst1](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Jani\_Kleindienst1](https://forum.inductiveautomation.com/u/Jani_Kleindienst1)\ **Post date:** [April 7, 2025, 1:09pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/10 "2025-04-07T13:09:22Z")
I have a similar problem. MES system publishes some data on the MQTT server when work on the machine starts. Part of this data is also a bill of material - a list of JSON objects. The problem is, when a new product has a shorter BOM as the previous one. Ignition (or MQTT Engine) keeps the tags that were created before and not overwritten with the new BOM elements. It would seem logical to me that when a new list appears in a new JSON payload, all elements of the old list are deleted and replaced with new ones, even if the new list is shorter! I could add empty elements myself, but BOMs have lengths from a few elements to several hundred. This seems to me to be an unnecessary waste of space. The only workaround we came up with is to send information about the length of the BOM and then only consider the first so many elements. ---
**Author:** ![henning.e.heine](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/henning.e.heine/32/70958_2.png) [@henning.e.heine](https://forum.inductiveautomation.com/u/henning.e.heine)\ **Post date:** [April 7, 2025, 1:33pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/11 "2025-04-07T13:33:10Z")
> [@Jani\_Kleindienst1](#): > > hen a new list appears in a new JSO My solution for this is to not let the MQTT-Engine parse the payload as JSON. I take it as string reference it in a document\_tag and have a change script on this tag parse it as I need it and write it to other tags or in a database. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [April 7, 2025, 1:46pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/12 "2025-04-07T13:46:06Z")
Mapping a JSON Object to tags when its shape will be continuously expected to change is a bit awkward. The JSON elements have something of an ephemeral nature, where tags are a little more permanent. Often with tags you configure alarms, history, target them in bindings, make them part of tag paths, etc... which is I think why the MQTT module does not simply remove these tags when JSON changes shape in a way that no longer needs them, and that change may be temporary. ---
**Author:** ![Jani\_Kleindienst1](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Jani\_Kleindienst1](https://forum.inductiveautomation.com/u/Jani_Kleindienst1)\ **Post date:** [April 7, 2025, 3:02pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/13 "2025-04-07T15:02:55Z")
I would understand this for object structure, not for a list. If we have a list, it's kind of self-evident that its length can change. That's what lists are for. If I wanted a fixed and immutable structure, I could use multiple named objects. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [April 7, 2025, 3:06pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/14 "2025-04-07T15:06:00Z")
It doesn't really matter if its an Array or Object because the same problem remains: the tags exist, which implies you can do all the things I mentioned with them, but tags would be deleted configuration would be lost if/when the list size changed. Fully mapping a JSON structure to the Ignition tag system is just a bad fit. What you've done using a Document tag is probably the right approach. It would be nice if our Document tags were more powerful though. ---
**Author:** ![Jani\_Kleindienst1](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Jani\_Kleindienst1](https://forum.inductiveautomation.com/u/Jani_Kleindienst1)\ **Post date:** [April 7, 2025, 4:20pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/15 "2025-04-07T16:20:27Z")
What about speed and performance? If I map the JSON BOM to a string tag and manually parse it when it changes? Is such a process any slower or more resource-intensive than what the MQTT Engine does? We process approx. 1000 machines, changes happen mostly simultaneously, within 20-30 minutes at shift changes. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [April 7, 2025, 4:21pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/16 "2025-04-07T16:21:54Z")
I don't really have a good sense for the performance characteristics of each, and the MQTT modules are not IA modules so I can't go take a peek to get a better sense. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 7, 2025, 4:25pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/17 "2025-04-07T16:25:51Z")
If you must parse manually, you probably want my Integration Toolkit, so you can do it without jython overhead. (Or **less** overhead.) ---
**Author:** ![Jani\_Kleindienst1](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Jani\_Kleindienst1](https://forum.inductiveautomation.com/u/Jani_Kleindienst1)\ **Post date:** [April 7, 2025, 6:01pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/18 "2025-04-07T18:01:16Z")
Thanks! Can you please direct me to where I could get more information about this? I'm also curious if you recommend any other way to get the BOM from the MES system into Ignition via an MQTT server? Can I map the JSON from MQTT to a Tag of type Document or Dataset? It is not necessary that the JSON list be mapped to a set of Tags. It is only important that Ignition has information about which raw materials are currently used on the machine. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 7, 2025, 6:08pm UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/19 "2025-04-07T18:08:02Z")
> [@Jani\_Kleindienst1](#): > > more information about this? Probably should read this whole topic to get a feel for what it can do: > [@Automation Professionals' Integration Toolkit Module](https://forum.inductiveautomation.com/t/automation-professionals-integration-toolkit-module/74934): > > Automation Professionals is pleased to announce the immediately availability of a major upgrade to its free Simulation Aids Integration Toolkit module. The new feature justifying the "major upgrade" language is the implementation of eleven thirty-one new expression functions which use iteration to accomplish complex transforms of datasets and lists, to or from other datasets and lists, applying filtering, ordering, and computed new columns along the way. While there are certain operations that… > [@Jani\_Kleindienst1](#): > > Can I map the JSON from MQTT to a Tag of type Document or Dataset? Either. Given your format, I'd recommend a dataset, with dedicated columns for the possible datatypes. (Similar to IA's historian tables.) Probably something like this in an expression tag (event triggered): ```python transform( jsonDecode({path/to/source/jsonAsString}), transform( doDate(value()['eventTime'], unionAll( asMap( 'machineId', 'str', 't_stamp', 'date', 'varName', 'str', 'varUnit', 'str', 'boolValue', 'B', 'longValue', 'L', 'doubleValue', 'D' ), forEach( value(1)['data'], value(1)['machineId'], // All rows get this ID value(), // All rows get the parsed event time it()['name'], it()['measurementUnit'], if( it()['dataType'] = 'boolean', toBoolean(it()['value']), null ), if( it()['dataType'] = 'int' || it()['dataType'] = 'long', toLong(it()['value']), null ), if( it()['dataType'] = 'float' || it()['dataType'] = 'double', toDouble(it()['value']), null ) ) ) ) ) ``` ---
**Author:** ![KristijanG](https://avatars.discourse-cdn.com/v4/letter/k/e99b99/32.png) [@KristijanG](https://forum.inductiveautomation.com/u/KristijanG)\ **Post date:** [April 8, 2025, 6:50am UTC](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266/20 "2025-04-08T06:50:21Z")
I have a question regarding the expression tag and the expression example. I can't find the functions like transform, unionAll, asMap, ...? How did you mean it? [Next page](https://forum.inductiveautomation.com/t/handling-a-json-list-of-metrics-in-mqtt/99266.md?page=2)