# \[BUG-15901\] JSON encoding and decoding a "Dataset" datatype property **URL:** \ **Category:** Ignition\ **Tags:** ignition80\ **Created:** [January 24, 2020, 10:47pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617 "2020-01-24T22:47:56Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![haitham.sumrain](https://avatars.discourse-cdn.com/v4/letter/h/cab0a1/32.png) [@haitham.sumrain](https://forum.inductiveautomation.com/u/haitham.sumrain)\ **Post date:** [January 24, 2020, 10:47pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/1 "2020-01-24T22:47:56Z")
I am trying to pass a value of “Dataset” type as a value in a dictionary which I need to store in a “String” format. Assume the following: dataset: variable of type “Dataset” dict = {‘set’ : dataset} To build the string representation for the dictionary, I am using system.util.jsonEncode. When I try to retrieve the dictionary from its string representation using system.util.jsonDecode, I do not get the dataset back in “Dataset” type. Instead, I get a representation of it in some form. Is there a way to get back the dataset from its representation? If not, is there any other way to achieve the following: Dataset --\> Dictionary --\> String --\> Dictionary --\> Dataset? ---
**Author:** ![osolorzano](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/osolorzano/32/6432_2.png) [@osolorzano](https://forum.inductiveautomation.com/u/osolorzano)\ **Post date:** [January 25, 2020, 12:59am UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/2 "2020-01-25T00:59:53Z")
I was able to do so with a few extra steps. It basically goes Dataset -\> Encoded dataset -\> Dictionary -\> String -\> Dictionary -\> Encoded dataset -\> Dataset. In the example here, I am reading a dataset tag, but it should work with any other dataset given you first convert it to a pydataset. ```auto #Original Dataset dataset = system.tag.readBlocking(['asdf'])[0].value #Convert to Pydata pydata = system.dataset.toPyDataSet(dataset) #Encode JUST the dataset encoded_dataset = system.util.jsonEncode(pydata) #Create the Dict some_dict = {'the_data':encoded_dataset} #Encode the Dict with the encoded data inside encoded_dict = system.util.jsonEncode(some_dict) #Decode the dictionary decoded_dict = system.util.jsonDecode(encoded_dict) #Decode the inner dataset self.custom.dataset = system.util.jsonDecode(decoded_dict.get('the_data')) ``` All this said, this should probably be improved on our end. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [January 25, 2020, 1:08am UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/3 "2020-01-25T01:08:42Z")
You can import and use [TypeUtilities](http://files.inductiveautomation.com/sdk/javadoc/ignition79/795/com/inductiveautomation/ignition/common/TypeUtilities.html#datasetToJSON-com.inductiveautomation.ignition.common.Dataset-); something like this should work: ```auto from com.inductiveautomation.ignition.common import TypeUtilities jsonStr = str(TypeUtilities.datasetToJSON()) ds = TypeUtilities.datasetFromJSON(jsonStr) ``` ---
**Author:** ![haitham.sumrain](https://avatars.discourse-cdn.com/v4/letter/h/cab0a1/32.png) [@haitham.sumrain](https://forum.inductiveautomation.com/u/haitham.sumrain)\ **Post date:** [January 29, 2020, 10:06pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/4 "2020-01-29T22:06:54Z")
Thanks @osolorzano. That worked but I agree with you, an improvement here can be quite helpful. ---
**Author:** ![haitham.sumrain](https://avatars.discourse-cdn.com/v4/letter/h/cab0a1/32.png) [@haitham.sumrain](https://forum.inductiveautomation.com/u/haitham.sumrain)\ **Post date:** [January 29, 2020, 10:09pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/5 "2020-01-29T22:09:00Z")
@paul-griffith Thanks for your response but the solution did not work. When trying to get the dataset from JSON string, it errors out as the json string could not be coerced to JSON. ---
**Author:** ![weertske](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/weertske/32/3353_2.png) [@weertske](https://forum.inductiveautomation.com/u/weertske)\ **Post date:** [February 24, 2020, 6:22pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/6 "2020-02-24T18:22:08Z")
The reason Paul's instruction threw an error is the decode command should be: > ds = TypeUtilities.toDataset(jsonStr) The advantage of using TypeUtilities is that you can pass in a "real" dataset which returns a "real" dataset instead of a pythonDataset. ---
**Author:** ![hwbrill](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/hwbrill/32/14377_2.png) [@hwbrill](https://forum.inductiveautomation.com/u/hwbrill)\ **Post date:** [June 7, 2020, 5:09am UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/7 "2020-06-07T05:09:26Z")
how about this? create a json dict in a project library script to be call by the ‘runScript()’ expression function then to use the derived tag to push the json to a dataset memory tag? using jsonGet and Set? ---
**Author:** ![weertske](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/weertske/32/3353_2.png) [@weertske](https://forum.inductiveautomation.com/u/weertske)\ **Post date:** [July 23, 2020, 4:21pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/8 "2020-07-23T16:21:19Z")
@paul-griffith Paul, I am using: ```auto str(TypeUtilities.datasetToJSON(dataset)) ``` to convert a dataset to a string for messaging. This works fine until the message contains unusual characters. For instance, if I try to send the Swedish word “kalenderår” it gets corrupted to “kalenderÃ¥r”. Any ideas? ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [July 23, 2020, 4:40pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/9 "2020-07-23T16:40:45Z")
`str` is collapsing to a lower character encoding. Try using `unicode()` around the TypeUtilities call. ---
**Author:** ![weertske](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/weertske/32/3353_2.png) [@weertske](https://forum.inductiveautomation.com/u/weertske)\ **Post date:** [July 23, 2020, 5:06pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/10 "2020-07-23T17:06:05Z")
Thank you. Thank you. Thank you. I love easy fixes. ---
**Author:** ![richardNZ15](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/richardnz15/32/33373_2.png) [@richardNZ15](https://forum.inductiveautomation.com/u/richardNZ15)\ **Post date:** [March 18, 2022, 3:54am UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/11 "2022-03-18T03:54:18Z")
I’m tying to use the TypeUtils library to encode and decode datasets because i’d like to store the datasets in SQL. I’ve noticed that using TypeUtils will cause me to lose milliseconds on my datetime data in my datasets, as you can see below: ![MillisecondTruncate](https://us1.discourse-cdn.com/inductiveautomation/original/3X/a/6/a60bbf499d6aa68e1003c034297052295cc7cf4f.png) it looks like the milliseconds are stripped out during conversion from dataset to JSON string. See below the first two rows from the dataset, in JSON format: ```auto [[-1.7109261751174927,3.7401225566864014,-0.0666724145412445,3.195615530014038,"Fri Mar 18 16:28:18 NZDT 2022"], [-1.6445746421813965,3.6924030780792236,-0.05308615043759346,3.193087339401245,"Fri Mar 18 16:28:18 NZDT 2022"]` ``` What is determining how datetime objects should be represented as strings? I’m sure milliseconds can be recorded, just not sure if I have power with these particular functions to change the behaviour. ---
**Author:** ![richardNZ15](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/richardnz15/32/33373_2.png) [@richardNZ15](https://forum.inductiveautomation.com/u/richardNZ15)\ **Post date:** [March 18, 2022, 4:58am UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/12 "2022-03-18T04:58:32Z")
Managed to get it working with some hacking of a custom jsonEncodeDataset function i wrote. Any constructive critisicsm of how to better extract the Java “type” for columns would be much appreciated. ```auto ds = self.getSibling("Table RawData").props.data pyDS = system.dataset.toPyDataSet(ds) # Create column keyValues # Example of format required for column list: #// columns = [{"name":"AIN0","type":"java.lang.Double"},{"name":"AIN1","type":"java.lang.Double"},{"name":"AIN2","type":"java.lang.Double"},{"name":"AIN3","type":"java.lang.Double"},{"name":"t_stamp","type":"java.util.Date"}] cols = [] for c in range(ds.getColumnCount()): # NOTE: Type is returned as "class java.lang.Double", but when cast to a string becomes: type = str(ds.getColumnType(c)).replace("","") cols.append({"name":ds.getColumnName(c), "type":type}) # end for # Create row keyValues newRows = [] for row in pyDS: # Couldn't decode using TypeUtils. Maybe due to Timezone? ###dateStr = system.date.format(row["t_stamp"], "yyyy-MM-dd HH:mm:ss.SSS z") dateStr = system.date.format(row["t_stamp"], "yyyy-MM-dd HH:mm:ss.SSS") newRows.append([row["AIN0"], row["AIN1"], row["AIN2"], row["AIN3"], dateStr]) # end for newDict = {"columns":cols, "rows":newRows} encoded_dict = system.util.jsonEncode(newDict) self.parent.custom.jsonDataStr = encoded_dict ``` EDIT: I realize i have hard coded the extraction of data with the **newRows.append(** line. I’ll sort that out but you get the gist of it. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [March 18, 2022, 3:00pm UTC](https://forum.inductiveautomation.com/t/bug-15901-json-encoding-and-decoding-a-dataset-datatype-property/31617/13 "2022-03-18T15:00:15Z")
For `type`, use `ds.getColumnType(c).getCanonicalName()`; canonical name is an attribute of `java.lang.Class`: [Class (Java SE 11 & JDK 11 )](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getCanonicalName()) As for your precision loss question: Yes, unfortunately the way TypeUtilities ends up encoding to JSON uses a method that has no awareness of Date objects, and thus is just calling their default `toString()` implementation, which has a hardcoded pattern: [https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Date.html#toString()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Date.html#toString()) Safer than relying on any string representation would probably be hardcoding Dates to store their time as epoch millis; the existing TypeUtilities method should still be able to reconstruct that into a dataset with the right type.