# Issue converting imported json list of dicts from unicode to a list

**URL:** <https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492>\
**Category:** Ignition\
**Tags:** scripting\
**Created:** [August 5, 2023, 5:09pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492 "2023-08-05T17:09:32Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 5, 2023, 5:09pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/1 "2023-08-05T17:09:32Z")

</div>

Hello all,

I am trying to import json data into my window when within actionPerformed script editor. The json data data I am importing is UTF-8 with BOM. When I print the type, it will say unicode, but I need a list or some type of datatype that I can modify the columns and remove certain keys from my list of dicts. I have already tried using ast.literal\_eval and doing data = data[3:], but each of these give me different issues.

When I do ast.literal\_eval i get the following error:  
Traceback (most recent call last):  
File "event:actionPerformed", line 6, in   
File "C:\Users\jaalcid.ignition\cache\gw10.132.252.23\_8088\C0\pylib\ast.py", line 49, in literal\_eval  
node\_or\_string = parse(node\_or\_string, mode='eval')  
File "C:\Users\jaalcid.ignition\cache\gw10.132.252.23\_8088\C0\pylib\ast.py", line 37, in parse  
return compile(source, filename, mode, PyCF\_ONLY\_AST)  
File "", line 1  
[  
^  
SyntaxError: no viable alternative at character ''

When I do data = data[3:] my list breaks into a bunch of individual Unicode segments.

Here is my json data (A list of dicts):  
[  
{  
"Term" : "Intercept",  
"Estimate" : -3.26667457316861,  
"Std Error" : 4.64402760030829,  
"t Ratio" : -0.703414116865231,  
"Prob\>|t|" : 0.496416819895656,  
"Lower 95%" : -13.4881104045032,  
"Upper 95%" : 6.95476125816592  
},  
{  
"Term" : "Flow Rate",  
"Estimate" : 0.00278164114479191,  
"Std Error" : 0.00361876491577232,  
"t Ratio" : 0.768671414014261,  
"Prob\>|t|" : 0.456950449399681,  
"Lower 95%" : -0.00518320673268325,  
"Upper 95%" : 0.0107464890222671  
},  
{  
"Term" : "Pressure",  
"Estimate" : 0.00572389801864138,  
"Std Error" : 0.00422223047666328,  
"t Ratio" : 1.3556574067375,  
"Prob\>|t|" : 0.200173228869231,  
"Lower 95%" : -0.00356916860298116,  
"Upper 95%" : 0.0150169646402639  
},  
{  
"Term" : "Temperature",  
"Estimate" : 0.00667732381905267,  
"Std Error" : 0.00368811336635803,  
"t Ratio" : 1.81049852750227,  
"Prob\>|t|" : 0.0953115732142802,  
"Lower 95%" : -0.00144015896903698,  
"Upper 95%" : 0.0147948066071423  
}  
]

Here is my script I have been trying to run:  
import ast

idata = system.file.openFile('json')  
data = system.file.readFileAsString(idata, "UTF-8")  
obj = system.util.jsonDecode(data)  
change = ast.literal\_eval(obj)  
print type(change)

I would really appreciate any help with this, I am on a big-time crunch am unsure what to try next. Thanks!

---

<div class="post-metadata">

**Author:** ![Transistor](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/transistor/32/3380_2.png) [@Transistor](https://forum.inductiveautomation.com/u/Transistor)\
**Post date:** [August 5, 2023, 5:23pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/2 "2023-08-05T17:23:12Z")

</div>

Try this in Script Console to see if it clarifies things for you.

```auto
str = """[
	{
		"Term" : "Intercept",
		"Estimate" : -3.26667457316861,
		"Std Error" : 4.64402760030829,
		"t Ratio" : -0.703414116865231,
		"Prob>|t|" : 0.496416819895656,
		"Lower 95%" : -13.4881104045032,
		"Upper 95%" : 6.95476125816592
	},
	{
		"Term" : "Flow Rate",
		"Estimate" : 0.00278164114479191,
		"Std Error" : 0.00361876491577232,
		"t Ratio" : 0.768671414014261,
		"Prob>|t|" : 0.456950449399681,
		"Lower 95%" : -0.00518320673268325,
		"Upper 95%" : 0.0107464890222671
	},
	{
		"Term" : "Pressure",
		"Estimate" : 0.00572389801864138,
		"Std Error" : 0.00422223047666328,
		"t Ratio" : 1.3556574067375,
		"Prob>|t|" : 0.200173228869231,
		"Lower 95%" : -0.00356916860298116,
		"Upper 95%" : 0.0150169646402639
	},
	{
		"Term" : "Temperature",
		"Estimate" : 0.00667732381905267,
		"Std Error" : 0.00368811336635803,
		"t Ratio" : 1.81049852750227,
		"Prob>|t|" : 0.0953115732142802,
		"Lower 95%" : -0.00144015896903698,
		"Upper 95%" : 0.0147948066071423
	}
]"""

dic = system.util.jsonDecode(str)
print type(dic)
print type(dic[0])
print dic
print dic[1]['Term']

```

* * *

Tip: format code as code in your posts.

 ![Formatting code](https://us1.discourse-cdn.com/inductiveautomation/original/3X/7/0/70741738ea3cbbe8a2d249e1194c80035567335c.png)

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [August 5, 2023, 11:51pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/3 "2023-08-05T23:51:56Z")

</div>

`ast.literal_eval` will evaluate datastructures from a string literal (as its name suggests).

If you're decoding your json before with `system.util.jsonDecode` your data is not a string anymore, and `literal_eval` is no longer relevant. Use one or the other, though frankly I'm not sure how you got to `literal_eval` in the first place. It's really not a common thing to use.

What happens if you just use `jsonDecode` ? It should be enough to convert your json string into a list of dicts.  
If for some reason it doesn't work, you can try `json.loads(json_string)`.

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 1:26pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/4 "2023-08-07T13:26:39Z")

</div>

Hello Pascal, thank you for the help. When I run just the jsonDecode, I get the following error: "TypeError: unicode indices must be integers". When I try the json.loads(), I get the following error: "ValueError: No JSON object could be decoded". Do you have any idea why this would be?

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [August 7, 2023, 1:30pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/5 "2023-08-07T13:30:23Z")

</div>

I'll need to see the code. Formatted with either the preformatted text tool:

> ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/7/2/7297903348a842ec0a5b84720b90e8c1791597e9.png)

or with triple back ticks:

> ```python  
> code here  
> ```

There's no way to tell you where the error is if we don't have access to exactly what the code was when the error happened.

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 2:56pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/6 "2023-08-07T14:56:47Z")

</div>

''' idata = system.file.openFile('json')  
data = system.file.readFileAsString(idata, "UTF-8")  
obj = system.util.jsonDecode(data)

keys\_to\_keep = ["Estimate","Term"]

filtered\_list = [  
{key: dct[key] for key in keys\_to\_keep}  
for dct in obj  
]

newlod =   
for dictionary in filtered\_list:  
term = dictionary["Term"]  
estimate = dictionary["Estimate"]  
new\_dict = {term: estimate}  
newlod.append(new\_dict)

print(newlod)

headers =   
info =

for dictionary in newlod:  
for key, value in dictionary.iteritems():  
headers.append(key)  
info.append(value)

lol = [info]

print (headers)  
print (lol)  
ds = system.dataset.toDataSet(headers, lol)  
event.source.parent.getComponent('Power Table').data = ds

# The provider and folder the Tag will be placed at.

baseTagPath = "[default]Coefficients"

# Properties that will be configured on that Tag.

tagName = "MLR"  
valueSource = "memory"  
Value = ds  
sampleMode = "TagGroup"  
tagGroup = "Default"

# Configure the tag.

tag = {  
"name": tagName,  
"value" : Value,  
"valueSource": valueSource,  
"sampleMode" : sampleMode,  
"tagGroup" : tagGroup  
}

# Set the collision policy to Abort. Thus, if a Tag already exists at the base path,

# we will not override the Tag. If you are overwriting an existing Tag, then set this to "o".

collisionPolicy = "o"

# Create the Tag.

system.tag.configure(baseTagPath, [tag], collisionPolicy)  
'''

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [August 7, 2023, 2:58pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/7 "2023-08-07T14:58:05Z")

</div>

> [@pascal.fragnoud](#):
>
> Formatted with either the preformatted text tool:
> 
> > ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/7/2/7297903348a842ec0a5b84720b90e8c1791597e9.png)
> 
> or with triple back ticks:
> 
> > ```python
> > code here
> > 
> > ```

Your code is unreadable as it is.

---

<div class="post-metadata">

**Author:** ![lrose](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/lrose/32/62724_2.png) [@lrose](https://forum.inductiveautomation.com/u/lrose)\
**Post date:** [August 7, 2023, 2:59pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/8 "2023-08-07T14:59:04Z")

</div>

Those weren't back ticks, they were single quotes.

Typically, on a standard QWERTY keyboard layout, anyway the back tick is located on the same key as the `~`. Mine is just above the left Tab Key.

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 3:04pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/9 "2023-08-07T15:04:11Z")

</div>

```auto
idata = system.file.openFile('json')
data = system.file.readFileAsString(idata, "UTF-8")
obj = system.util.jsonDecode(data)

keys_to_keep = ["Estimate","Term"]

filtered_list = [
	{key: dct[key] for key in keys_to_keep}
	for dct in obj
]

newlod = []
for dictionary in filtered_list:
	term = dictionary["Term"]
	estimate = dictionary["Estimate"]
	new_dict = {term: estimate}
	newlod.append(new_dict)

print(newlod)

headers = []
info = []

for dictionary in newlod:
	for key, value in dictionary.iteritems():
		headers.append(key)
		info.append(value)
		
lol = [info]
	
print (headers)
print (lol) 
ds = system.dataset.toDataSet(headers, lol)
event.source.parent.getComponent('Power Table').data = ds

# The provider and folder the Tag will be placed at. 
baseTagPath = "[default]Coefficients"
 
# Properties that will be configured on that Tag.
tagName = "MLR"
valueSource = "memory"
Value = ds
sampleMode = "TagGroup"
tagGroup = "Default"
 
# Configure the tag.
tag = {
            "name": tagName,           
            "value" : Value,
            "valueSource": valueSource,
            "sampleMode" : sampleMode,
            "tagGroup" : tagGroup
        }
 
# Set the collision policy to Abort. Thus, if a Tag already exists at the base path,
# we will not override the Tag. If you are overwriting an existing Tag, then set this to "o".
collisionPolicy = "o"
 
# Create the Tag.
system.tag.configure(baseTagPath, [tag], collisionPolicy)

```

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 3:04pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/10 "2023-08-07T15:04:30Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [August 7, 2023, 3:05pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/11 "2023-08-07T15:05:36Z")

</div>

Okay, and what's the exact error you get with this code? And what line number does the error message correspond to in your sample, if the line numbers don't match up?

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 3:10pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/12 "2023-08-07T15:10:27Z")

</div>

Traceback (most recent call last):  
File "event:actionPerformed", line 8, in   
File "event:actionPerformed", line 8, in   
TypeError: unicode indices must be integers

---

<div class="post-metadata">

**Author:** ![lrose](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/lrose/32/62724_2.png) [@lrose](https://forum.inductiveautomation.com/u/lrose)\
**Post date:** [August 7, 2023, 3:13pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/13 "2023-08-07T15:13:34Z")

</div>

You're expecting `dct` to be a dictionary, but it isn't. Can you provide the JSON from the file that you're reading. It seems that it isn't properly formed.

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [August 7, 2023, 3:14pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/14 "2023-08-07T15:14:58Z")

</div>

Looks like your list contains things that are not dicts. A unicode string, to be accurate.

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 3:34pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/15 "2023-08-07T15:34:12Z")

</div>

[  
{  
"Term" : "Intercept",  
"Estimate" : -3.26667457316861,  
"Std Error" : 4.64402760030829,  
"t Ratio" : -0.703414116865231,  
"Prob\>|t|" : 0.496416819895656,  
"Lower 95%" : -13.4881104045032,  
"Upper 95%" : 6.95476125816592  
},  
{  
"Term" : "Flow Rate",  
"Estimate" : 0.00278164114479191,  
"Std Error" : 0.00361876491577232,  
"t Ratio" : 0.768671414014261,  
"Prob\>|t|" : 0.456950449399681,  
"Lower 95%" : -0.00518320673268325,  
"Upper 95%" : 0.0107464890222671  
},  
{  
"Term" : "Pressure",  
"Estimate" : 0.00572389801864138,  
"Std Error" : 0.00422223047666328,  
"t Ratio" : 1.3556574067375,  
"Prob\>|t|" : 0.200173228869231,  
"Lower 95%" : -0.00356916860298116,  
"Upper 95%" : 0.0150169646402639  
},  
{  
"Term" : "Temperature",  
"Estimate" : 0.00667732381905267,  
"Std Error" : 0.00368811336635803,  
"t Ratio" : 1.81049852750227,  
"Prob\>|t|" : 0.0953115732142802,  
"Lower 95%" : -0.00144015896903698,  
"Upper 95%" : 0.0147948066071423  
}  
]

I believe its because of the fact its encoded with a BOM. Makes the decoding not work properly.

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 3:34pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/16 "2023-08-07T15:34:32Z")

</div>

Yes, would this be because of the BOM decryption?

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [August 7, 2023, 3:47pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/17 "2023-08-07T15:47:16Z")

</div>

I have no idea what BOM means.

That said, If I paste that data as a string then decode it with `jsonDecode`, I have no issue running line 8.  
So I'm not sure what's going on here.

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [August 7, 2023, 3:52pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/18 "2023-08-07T15:52:16Z")

</div>

It's pretty unlikely the BOM matters; it should be getting silently handled by Java's file parsing machinery under the hood. Can you upload **the actual file** you're using, if you're that concerned it's the BOM? I really don't think it is.

---

<div class="post-metadata">

**Author:** ![Joseph\_Alcide](https://avatars.discourse-cdn.com/v4/letter/j/ebca7d/32.png) [@Joseph\_Alcide](https://forum.inductiveautomation.com/u/Joseph_Alcide)\
**Post date:** [August 7, 2023, 3:57pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/19 "2023-08-07T15:57:18Z")

</div>

[Three Parameter Model Data.json](https://forum.inductiveautomation.com/uploads/short-url/p9ZJHIP1cxWrpOasHdiNALgUSat.json) (1010 Bytes)

---

<div class="post-metadata">

**Author:** ![pascal.fragnoud](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pascal.fragnoud/32/24728_2.png) [@pascal.fragnoud](https://forum.inductiveautomation.com/u/pascal.fragnoud)\
**Post date:** [August 7, 2023, 4:20pm UTC](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492/20 "2023-08-07T16:20:38Z")

</div>

I can't help you any further with your decoding issue, but I can suggest a few things to make the script simpler:

```python
filtered_list = [
	{key: dct[key] for key in keys_to_keep}
	for dct in obj
]

```

You don't need that. You're accessing things by name afterward anyway.

* * *

```python
newlod = []
for dictionary in filtered_list:
	term = dictionary["Term"]
	estimate = dictionary["Estimate"]
	new_dict = {term: estimate}
	newlod.append(new_dict)

```

this can be done with a comprehension:

```python
newlod = [{d['Term']: d['Estimate']} for d in filtered_list]

```

* * *

```python
headers = []
info = []
for dictionary in newlod:
	for key, value in dictionary.iteritems():
		headers.append(key)
		info.append(value)

```

this too, though it's a bit more complicated:

```python
headers, info = zip(*[d.items()[0] for d in newlod])

```

* * *

I don't have much to say about the rest of the script.

so this:

```python
keys_to_keep = ["Estimate","Term"]

filtered_list = [
	{key: dct[key] for key in keys_to_keep}
	for dct in obj
]

newlod = []
for dictionary in filtered_list:
	term = dictionary["Term"]
	estimate = dictionary["Estimate"]
	new_dict = {term: estimate}
	newlod.append(new_dict)

print(newlod)

headers = []
info = []

for dictionary in newlod:
	for key, value in dictionary.iteritems():
		headers.append(key)
		info.append(value)
		
lol = [info]
	
print (headers)
print (lol) 
ds = system.dataset.toDataSet(headers, lol)

```

can be simplified to this:

```python
newlod = [{d['Term']: d['Estimate']} for d in obj]
headers, data = zip(*[d.items()[0] for d in newlod])
ds = system.dataset.toDataSet(headers, [data])

```

[Next page](https://forum.inductiveautomation.com/t/issue-converting-imported-json-list-of-dicts-from-unicode-to-a-list/77492.md?page=2)