Issue converting imported json list of dicts from unicode to a list

Ignition
1

Hello all,

I am trying to import json data into my window when within actionPerformed script editor. The json data data I am importing is UTF-8 with BOM. When I print the type, it will say unicode, but I need a list or some type of datatype that I can modify the columns and remove certain keys from my list of dicts. I have already tried using ast.literal_eval and doing data = data[3:], but each of these give me different issues.

When I do ast.literal_eval i get the following error:
Traceback (most recent call last):
File "event:actionPerformed", line 6, in
File "C:\Users\jaalcid.ignition\cache\gw10.132.252.23_8088\C0\pylib\ast.py", line 49, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File "C:\Users\jaalcid.ignition\cache\gw10.132.252.23_8088\C0\pylib\ast.py", line 37, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "", line 1
[
^
SyntaxError: no viable alternative at character ''

When I do data = data[3:] my list breaks into a bunch of individual Unicode segments.

Here is my json data (A list of dicts):
[
{
"Term" : "Intercept",
"Estimate" : -3.26667457316861,
"Std Error" : 4.64402760030829,
"t Ratio" : -0.703414116865231,
"Prob>|t|" : 0.496416819895656,
"Lower 95%" : -13.4881104045032,
"Upper 95%" : 6.95476125816592
},
{
"Term" : "Flow Rate",
"Estimate" : 0.00278164114479191,
"Std Error" : 0.00361876491577232,
"t Ratio" : 0.768671414014261,
"Prob>|t|" : 0.456950449399681,
"Lower 95%" : -0.00518320673268325,
"Upper 95%" : 0.0107464890222671
},
{
"Term" : "Pressure",
"Estimate" : 0.00572389801864138,
"Std Error" : 0.00422223047666328,
"t Ratio" : 1.3556574067375,
"Prob>|t|" : 0.200173228869231,
"Lower 95%" : -0.00356916860298116,
"Upper 95%" : 0.0150169646402639
},
{
"Term" : "Temperature",
"Estimate" : 0.00667732381905267,
"Std Error" : 0.00368811336635803,
"t Ratio" : 1.81049852750227,
"Prob>|t|" : 0.0953115732142802,
"Lower 95%" : -0.00144015896903698,
"Upper 95%" : 0.0147948066071423
}
]

Here is my script I have been trying to run:
import ast

idata = system.file.openFile('json')
data = system.file.readFileAsString(idata, "UTF-8")
obj = system.util.jsonDecode(data)
change = ast.literal_eval(obj)
print type(change)

I would really appreciate any help with this, I am on a big-time crunch am unsure what to try next. Thanks!

2

Try this in Script Console to see if it clarifies things for you.

str = """[
	{
		"Term" : "Intercept",
		"Estimate" : -3.26667457316861,
		"Std Error" : 4.64402760030829,
		"t Ratio" : -0.703414116865231,
		"Prob>|t|" : 0.496416819895656,
		"Lower 95%" : -13.4881104045032,
		"Upper 95%" : 6.95476125816592
	},
	{
		"Term" : "Flow Rate",
		"Estimate" : 0.00278164114479191,
		"Std Error" : 0.00361876491577232,
		"t Ratio" : 0.768671414014261,
		"Prob>|t|" : 0.456950449399681,
		"Lower 95%" : -0.00518320673268325,
		"Upper 95%" : 0.0107464890222671
	},
	{
		"Term" : "Pressure",
		"Estimate" : 0.00572389801864138,
		"Std Error" : 0.00422223047666328,
		"t Ratio" : 1.3556574067375,
		"Prob>|t|" : 0.200173228869231,
		"Lower 95%" : -0.00356916860298116,
		"Upper 95%" : 0.0150169646402639
	},
	{
		"Term" : "Temperature",
		"Estimate" : 0.00667732381905267,
		"Std Error" : 0.00368811336635803,
		"t Ratio" : 1.81049852750227,
		"Prob>|t|" : 0.0953115732142802,
		"Lower 95%" : -0.00144015896903698,
		"Upper 95%" : 0.0147948066071423
	}
]"""

dic = system.util.jsonDecode(str)
print type(dic)
print type(dic[0])
print dic
print dic[1]['Term']

Tip: format code as code in your posts.

Formatting code
Formatting code741×191 17.4 KB

1 Like
3

ast.literal_eval will evaluate datastructures from a string literal (as its name suggests).

If you're decoding your json before with system.util.jsonDecode your data is not a string anymore, and literal_eval is no longer relevant. Use one or the other, though frankly I'm not sure how you got to literal_eval in the first place. It's really not a common thing to use.

What happens if you just use jsonDecode ? It should be enough to convert your json string into a list of dicts.
If for some reason it doesn't work, you can try json.loads(json_string).

2 Likes
4

Hello Pascal, thank you for the help. When I run just the jsonDecode, I get the following error: "TypeError: unicode indices must be integers". When I try the json.loads(), I get the following error: "ValueError: No JSON object could be decoded". Do you have any idea why this would be?

5

I'll need to see the code. Formatted with either the preformatted text tool:

image

or with triple back ticks:

```python
code here
```

There's no way to tell you where the error is if we don't have access to exactly what the code was when the error happened.

6

''' idata = system.file.openFile('json')
data = system.file.readFileAsString(idata, "UTF-8")
obj = system.util.jsonDecode(data)

keys_to_keep = ["Estimate","Term"]

filtered_list = [
{key: dct[key] for key in keys_to_keep}
for dct in obj
]

newlod =
for dictionary in filtered_list:
term = dictionary["Term"]
estimate = dictionary["Estimate"]
new_dict = {term: estimate}
newlod.append(new_dict)

print(newlod)

headers =
info =

for dictionary in newlod:
for key, value in dictionary.iteritems():
headers.append(key)
info.append(value)

lol = [info]

print (headers)
print (lol)
ds = system.dataset.toDataSet(headers, lol)
event.source.parent.getComponent('Power Table').data = ds

The provider and folder the Tag will be placed at.

baseTagPath = "[default]Coefficients"

Properties that will be configured on that Tag.

tagName = "MLR"
valueSource = "memory"
Value = ds
sampleMode = "TagGroup"
tagGroup = "Default"

Configure the tag.

tag = {
"name": tagName,
"value" : Value,
"valueSource": valueSource,
"sampleMode" : sampleMode,
"tagGroup" : tagGroup
}

Set the collision policy to Abort. Thus, if a Tag already exists at the base path,

we will not override the Tag. If you are overwriting an existing Tag, then set this to "o".

collisionPolicy = "o"

Create the Tag.

system.tag.configure(baseTagPath, [tag], collisionPolicy)
'''

7

Your code is unreadable as it is.

2 Likes
8

Those weren't back ticks, they were single quotes.

Typically, on a standard QWERTY keyboard layout, anyway the back tick is located on the same key as the ~. Mine is just above the left Tab Key.

9 
idata = system.file.openFile('json')
data = system.file.readFileAsString(idata, "UTF-8")
obj = system.util.jsonDecode(data)

keys_to_keep = ["Estimate","Term"]

filtered_list = [
	{key: dct[key] for key in keys_to_keep}
	for dct in obj
]

newlod = []
for dictionary in filtered_list:
	term = dictionary["Term"]
	estimate = dictionary["Estimate"]
	new_dict = {term: estimate}
	newlod.append(new_dict)

print(newlod)

headers = []
info = []

for dictionary in newlod:
	for key, value in dictionary.iteritems():
		headers.append(key)
		info.append(value)
		
lol = [info]
	
print (headers)
print (lol) 
ds = system.dataset.toDataSet(headers, lol)
event.source.parent.getComponent('Power Table').data = ds

# The provider and folder the Tag will be placed at. 
baseTagPath = "[default]Coefficients"
 
# Properties that will be configured on that Tag.
tagName = "MLR"
valueSource = "memory"
Value = ds
sampleMode = "TagGroup"
tagGroup = "Default"
 
# Configure the tag.
tag = {
            "name": tagName,           
            "value" : Value,
            "valueSource": valueSource,
            "sampleMode" : sampleMode,
            "tagGroup" : tagGroup
        }
 
# Set the collision policy to Abort. Thus, if a Tag already exists at the base path,
# we will not override the Tag. If you are overwriting an existing Tag, then set this to "o".
collisionPolicy = "o"
 
# Create the Tag.
system.tag.configure(baseTagPath, [tag], collisionPolicy)
10

Thank you!

1 Like
11

Okay, and what's the exact error you get with this code? And what line number does the error message correspond to in your sample, if the line numbers don't match up?

12

Traceback (most recent call last):
File "event:actionPerformed", line 8, in
File "event:actionPerformed", line 8, in
TypeError: unicode indices must be integers

13

You're expecting dct to be a dictionary, but it isn't. Can you provide the JSON from the file that you're reading. It seems that it isn't properly formed.

14

Looks like your list contains things that are not dicts. A unicode string, to be accurate.

15

[
{
"Term" : "Intercept",
"Estimate" : -3.26667457316861,
"Std Error" : 4.64402760030829,
"t Ratio" : -0.703414116865231,
"Prob>|t|" : 0.496416819895656,
"Lower 95%" : -13.4881104045032,
"Upper 95%" : 6.95476125816592
},
{
"Term" : "Flow Rate",
"Estimate" : 0.00278164114479191,
"Std Error" : 0.00361876491577232,
"t Ratio" : 0.768671414014261,
"Prob>|t|" : 0.456950449399681,
"Lower 95%" : -0.00518320673268325,
"Upper 95%" : 0.0107464890222671
},
{
"Term" : "Pressure",
"Estimate" : 0.00572389801864138,
"Std Error" : 0.00422223047666328,
"t Ratio" : 1.3556574067375,
"Prob>|t|" : 0.200173228869231,
"Lower 95%" : -0.00356916860298116,
"Upper 95%" : 0.0150169646402639
},
{
"Term" : "Temperature",
"Estimate" : 0.00667732381905267,
"Std Error" : 0.00368811336635803,
"t Ratio" : 1.81049852750227,
"Prob>|t|" : 0.0953115732142802,
"Lower 95%" : -0.00144015896903698,
"Upper 95%" : 0.0147948066071423
}
]

I believe its because of the fact its encoded with a BOM. Makes the decoding not work properly.

16

Yes, would this be because of the BOM decryption?

17

I have no idea what BOM means.

That said, If I paste that data as a string then decode it with jsonDecode, I have no issue running line 8.
So I'm not sure what's going on here.

18

It's pretty unlikely the BOM matters; it should be getting silently handled by Java's file parsing machinery under the hood. Can you upload the actual file you're using, if you're that concerned it's the BOM? I really don't think it is.

19

Three Parameter Model Data.json (1010 Bytes)

20

I can't help you any further with your decoding issue, but I can suggest a few things to make the script simpler:

filtered_list = [
	{key: dct[key] for key in keys_to_keep}
	for dct in obj
]

You don't need that. You're accessing things by name afterward anyway.

newlod = []
for dictionary in filtered_list:
	term = dictionary["Term"]
	estimate = dictionary["Estimate"]
	new_dict = {term: estimate}
	newlod.append(new_dict)

this can be done with a comprehension:

newlod = [{d['Term']: d['Estimate']} for d in filtered_list]

headers = []
info = []
for dictionary in newlod:
	for key, value in dictionary.iteritems():
		headers.append(key)
		info.append(value)

this too, though it's a bit more complicated:

headers, info = zip(*[d.items()[0] for d in newlod])

I don't have much to say about the rest of the script.

so this:

keys_to_keep = ["Estimate","Term"]

filtered_list = [
	{key: dct[key] for key in keys_to_keep}
	for dct in obj
]

newlod = []
for dictionary in filtered_list:
	term = dictionary["Term"]
	estimate = dictionary["Estimate"]
	new_dict = {term: estimate}
	newlod.append(new_dict)

print(newlod)

headers = []
info = []

for dictionary in newlod:
	for key, value in dictionary.iteritems():
		headers.append(key)
		info.append(value)
		
lol = [info]
	
print (headers)
print (lol) 
ds = system.dataset.toDataSet(headers, lol)

can be simplified to this:

newlod = [{d['Term']: d['Estimate']} for d in obj]
headers, data = zip(*[d.items()[0] for d in newlod])
ds = system.dataset.toDataSet(headers, [data])
2 Likes