[IGN-5231] jsonEncode issue

dandersen · March 17, 2021, 10:04pm

I must be missing something here. I call a function that returns a dictionary that I then encode into JSON. If I use the dictionary object returned by the function and pass it into the JSON encode, I hit a recursion error. If I create a new variable with the same dictionary, it works fine. Does anyone have any ideas?

reqParameters = ""
reqRemainingPath = ""

broken = request(reqParameters, reqRemainingPath)
works = {'racks': [{'sampleCount': 0, 'createdAt': 0, 'trayCode': u'asdfasdf', 'rackType': 1}, {'sampleCount': 0, 'createdAt': 0, 'trayCode': u'qwerqwer', 'rackType': 2}, {'sampleCount': 0, 'createdAt': 0, 'trayCode': u'zxcvzxcv', 'rackType': 1}]}
print type(broken)
print type(works)
print broken
print works
system.util.jsonEncode(works)
system.util.jsonEncode(broken)

Console Output:

<type 'dict'>
<type 'dict'>
{'racks': [{'sampleCount': 0, 'createdAt': 0, 'trayCode': u'asdfasdf', 'rackType': 1}, {'sampleCount': 0, 'createdAt': 0, 'trayCode': u'qwerqwer', 'rackType': 2}, {'sampleCount': 0, 'createdAt': 0, 'trayCode': u'zxcvzxcv', 'rackType': 1}]}
{'racks': [{'rackType': 1, 'sampleCount': 0, 'createdAt': 0, 'trayCode': u'asdfasdf'}, {'rackType': 2, 'sampleCount': 0, 'createdAt': 0, 'trayCode': u'qwerqwer'}, {'rackType': 1, 'sampleCount': 0, 'createdAt': 0, 'trayCode': u'zxcvzxcv'}]}
u'{"racks":[{"sampleCount":0,"createdAt":0,"trayCode":"asdfasdf","rackType":1},{"sampleCount":0,"createdAt":0,"trayCode":"qwerqwer","rackType":2},{"sampleCount":0,"createdAt":0,"trayCode":"zxcvzxcv","rackType":1}]}'
Traceback (most recent call last):
  File "<buffer>", line 34, in <module>
RuntimeError: maximum recursion depth exceeded

PGriffith · March 17, 2021, 10:31pm

What are the types of the objects in broken? Any chance they’re qualified values or quality codes?

dandersen · March 18, 2021, 2:15pm

Broken is just a dictionary with a list of dictionaries. Each dictionary in that list is just integers.

I run this code against broken and I get all integers.

for key in broken:
	print type(broken[key])
	print broken[key]
	for listDic in broken[key]:
		for key2 in listDic:
			print type(listDic[key2])

<type 'list'>
[{'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}, {'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}, {'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}]
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>
<type 'int'>

PGriffith · March 18, 2021, 3:49pm

Does system.util.jsonEncode(dict(broken)) also fail?
It’s gotta be something about the scripting function returning a weird object, but without your code it’s hard to replicate.

dandersen · March 18, 2021, 3:52pm

Yes that type cast also fails. I’ll post the code in anther comment here.

dandersen · March 18, 2021, 3:55pm

Test code run from Script Console to get debug output

from project.app.res import vial, rack
from project.app import logger

def request(parameters, remainingPath):
	if remainingPath == "":
		# get all racks/vials
		logger.apiLog.debug("Get All Racks and Vials")
		return getAllInventory()
		
		
def getAllInventory():
	rackList = []
	inventory = {'racks':[]}
	# get a all racks in inventory
	rackDS = system.db.runNamedQuery("inventory/getAllRacks")
	if rackDS.getRowCount():
		# we have racks in inventory. get all vials for each rack.
		trayDS = system.dataset.toPyDataSet(rackDS)
		for tray in trayDS:
			rackList.append(rack(tray).__dict__)
	inventory['racks'] = rackList
	return inventory

reqParameters = ""
reqRemainingPath = ""

broken = request(reqParameters, reqRemainingPath)
works = {'racks': [{'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}, {'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}, {'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}]}
print type(broken)
print type(works)
print broken
print works
system.util.jsonEncode(works)
system.util.jsonEncode(dict(broken))

Here is Class rack from project.app.res

class rack(object):
	def __init__(self, dbRecord):
		self.trayCode = dbRecord['trayCode']
		self.createdAt = dbRecord['createdAt']
		self.sampleCount = dbRecord['sampleCount']
		self.rackType = dbRecord['rackType']
		self.location = {'carouselCode':dbRecord['carouselCode']}
		self.location['col'] = dbRecord['col']
		self.location['row'] = dbRecord['row']

chasondeshotel · March 18, 2021, 4:38pm

This doesn’t answer the question but there’s some things I would do along the path of trying to fix it

# these will be cached by Ignition, always use the long form of method calls / constructors
#from project.app.res import vial, rack
#from project.app import logger

# set defaults
def request(parameters = None, remainingPath = None):
	if remainingPath:
		# get all racks/vials
		project.app.logger.apiLog.debug("Get All Racks and Vials")
		return getAllInventory()
		
		
def getAllInventory():
	rackList = []
	inventory = {}
	# get a all racks in inventory
	rackDS = system.db.runNamedQuery("inventory/getAllRacks")
	if rackDS.getRowCount():
		# we have racks in inventory. get all vials for each rack.
		#
		# you are turning rack dataset into a py dataset but it's the same data
		# wouldn't this be rackPyDS?
		# 
		trayDS = system.dataset.toPyDataSet(rackDS)
		for tray in trayDS:
			rackList.append(rack(tray).__dict__)
	inventory['racks'] = rackList
	return inventory

broken = request()
works = {'racks': [{'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}, {'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}, {'sampleCount': 3, 'createdAt': 2, 'trayCode': 1, 'rackType': 4}]}

import pprint
pprint.pprint(broken)

chasondeshotel · March 18, 2021, 4:47pm

def getAllInventory():
	rackList = []
	inventory = {}
	# get a all racks in inventory
	# toPyDataSet will return None if the named query comes back empty
	rackPyDS = system.dataset.toPyDataSet(system.db.runNamedQuery("inventory/getAllRacks"))
	# I don't know how you want to get to trays, but this is still rackPyDS
	#trayDS = system.dataset.toPyDataSet(rackDS)
	if rackPyDS:
		inventory['racks'] = [project.app.res.Rack(rack).__dict__ for rack in rackPyDS]
		return inventory

edit:

rack is referring to the rack class not rackDS. By convention classes should be uppercase… Rack. That’s confusing.

You should also use the long form to instantiate that object: project.app.res.Rack(tray)

Why are you instantiating objects to get their dictionary and not passing the actual objects around?

dandersen · March 18, 2021, 5:07pm

I create an instance of an object to be able to use its __dict__ as I need the dictionary representation of that object so it can be encoded in JSON. I realize I could build up the dictionary my self but then as I add more to the class, I would have to update the code that loops over the object.

I’m a little confused about your solution. I don’t have any issue with getAllInventory() method. That works correctly. The issue is when I try to encode the dictionary from that method. If I pass it directly into system.util.jsonEncode(), it fails, where if I create the same exact dictionary, it works. I ran your modified code and I get the same error.

chasondeshotel · March 18, 2021, 5:18pm

So my remaining question is what does each rack in rackPyDS look like because this is going to have to change:

inventory['racks'] = [project.app.res.Rack(rack).__dict__ for rack in rackPyDS]

In response to your edit:

There’s obvious issues with the code. Not entirely sure where your issue lies but if we solve the ones we see then maybe we’ll fix it on the way. If that doesn’t fix it at least we get a concise bug report

Granted I should have spent more time proofreading but the lowercase object instantiation with a name two characters away from a variable used two lines before threw me for a loop

dandersen · March 18, 2021, 5:24pm

I don't think I understand the question below.

So my remaining question is what does each rack in rackPyDS look like because this is going to have to change:

chasondeshotel · March 18, 2021, 5:30pm

I’m presuming racks and trays are different things and racks contain trays?

because this

rackDS = system.db.runNamedQuery("inventory/getAllRacks")

and this

trayDS = system.dataset.toPyDataSet(rackDS)

are the same data except one is in dataSet form and one is in pyDataSet form.

So either we need to get trays from racks or those variable names are muddying the water

If we need to get trays from racks, I would like to know what the data for one rack looks like

dandersen · March 18, 2021, 5:32pm

Ah I see. Racks and Trays are the exact same thing. We just changed the name for debug purposes to make sure it wasn’t causing any issues. The same corrected code is below that includes your suggested changes.

def getAllInventory():
	rackList = []
	inventory = {}
	# get a all racks in inventory
	rackDS = system.db.runNamedQuery("inventory/getAllRacks")
	if rackDS.getRowCount():
		# we have racks in inventory. get all vials for each rack.
		rackPyDS = system.dataset.toPyDataSet(rackDS)
		for rack in rackPyDS:
			rackList.append(project.app.res.Rack(rack).__dict__)
	inventory['racks'] = rackList
	return inventory

chasondeshotel · March 18, 2021, 5:41pm

Aha. Gotcha. I thought for sure you going one level deeper but indexing the parent every time. I was so excited I found the solution lol

I’m going to make a table and named query to see if I can duplicate this

I almost feel bad about being a total distraction but getting that class name uppercased was well worth the effort (oh, and the long form calls, that will bite you eventually)

dandersen · March 18, 2021, 5:44pm

No worries! I appreciate the help and pointers. As much as I like to one line code, this is going to be maintained by some folks that aren’t super Python savvy so that is why I’m really spelling everything out and not condensing the loops and such.

Here is what the output of that query looks like.

Edit:
This is in Ignition 7.9.14 so I’m wondering if its an issue with how Jython 2.5?

chasondeshotel · March 18, 2021, 5:51pm

Yeah that's totally fair. I do think

rackPyDS = system.dataset.toPyDataSet(system.db.runNamedQuery("inventory/getAllRacks"))
if rackPyDS:

is easier because it reads super Englishy and you don't have to carry variables in your head but I totally get that plenty of folks get lost at comprehensions

This is in Ignition 7.9.14 so I’m wondering if its an issue with how Jython 2.5?

I only have access to 8.1.15 at the moment. Maybe we can narrow it down at least. Thanks for the screenshot. Working on this now

edit in case anyone else wants to take a stab:

create table racks (
	carouselCode char(1)
	, col int
	, [row] int
	, createdAt datetime null
	, trayCode nvarchar(20)
	, sampleCount int
	, rackType int
)

insert into backup_getAllRacks (
	carouselCode, col, [row], createdAt, trayCode, sampleCount, rackType
	)
values
	('A',1,1,NULL,'asdfasdf',NULL,1)
	, ('A',1,2,NULL,'qwerqwer',NULL,2)
	, ('A',1,3,NULL,'zxcvzxcv',NULL,1)

WillMT10 · March 18, 2021, 6:26pm

I might be missing something, but what you have in the printout for broken doesn’t match what your named query is showing. Specifically the NULL values for createdAt and sampleCount, and I don’t see anything that is handling those NULLs in your scripts

dandersen · March 18, 2021, 6:32pm

The NULLs in the DB just get converted to a None type. When system.util.jsonEncode() comes across a None Type, it just puts “null” into the JSON output.

WillMT10 · March 18, 2021, 6:47pm

These are getting converted to integers prior to the jsonEncode()…maybe a default behavior of __dict__ that I’m not aware of. One thing I noticed is that the type returned from __dict__ is not actually a true dict, it is apparently a dictproxy, so I’m not sure if that could cause some bad behaviors.

chasondeshotel · March 18, 2021, 6:47pm

@WillMT10 I noticed that too. I’m quite sure (this time) that this is the actual issue at hand:

system.util.jsonEncode(Rack('foo').__dict__)