Saving user configuration property object to SQL nvarchar and retrieving it again

I'm suffering from Engineer's Block. (Well, if writer's can suffer their version then why can't we. Try it on your boss.)

My application has some user configuration checkboxes linked to a `view.custom.mainTableShowColumns' and (the abbreviated version) looks like

{
  "process": true,
  "notes": true,
  "meta": false
}
  1. How do I script to convert that to a string to store in an MSSQL nvarchar column?
  2. How do I get it back to load the user's preferences back into `view.custom.mainTableShowColumns'?

My attempts so far either have <ObjectWrapper> and Unicode 'u' designators attached, or else store the qualified values of each of the object members.

Many thanks.

With MSSQL I used a NVARCHAR(MAX) column and system.util.jsonEncode to save to the db and system.util.jsonDecode when retrieving it out and it works well for me so far. That’s for me saving a python dictionary - I dont’ know if there are any special gotchas in your situation here, I know perspective objects are all wrapped up in stuff but if you’re able to get it into a dictionary and go from dictionary to the perspective object - I think the above should be fine.

If it is static ie always just three column process/notes/meta - I would opt for a column per field though. Always nice to ensure data integritywhen possible. This would also make probably make retrieval simpler.

3 Likes

What Brian said.
You may need to un-qualify the Perspective prop to a jython dict to feed to .jsonEncode().

1 Like

Thanks, Phil. Unfortunately your Integration Toolkit is not available on this customer's gateway. What's the long-hand approach to achieve the same result?

I understand, but in this case I'm trying to make it a generic user preference storage so there could be all sorts of stuff in there.

Thanks.

Recursive function, similar to this:

from java.util import Map
from com.inductiveautomation.ignition.common.model.values import QualifiedValue
try:
	pHook = system.util.toolkitCtx().getModule('com.inductiveautomation.perspective')
	objWrapper = pHook.class.classLoader.loadClass("com.inductiveautomation.perspective.gateway.script.PropertyTreeScriptWrapper$ObjectWrapper")
	mappings = (dict, Map, objWrapper)
except:
	mappings = (dict, Map)

# Strip qualified values out of hierarchical entities.
def deep(x):
	if x is None:
		return None
	if isinstance(x, QualifiedValue):
		return deep(x.value)
	if isinstance(x, basestring):
		return x
	if isinstance(x, mappings):
		x = dict(x)
		retv = dict()
		for k, v in x.items():
			retv[k] = deep(v)
		return retv
	try:
		return [deep(v) for v in x]
	except:
		return x

You'll need to use another method to get the gateway context.

(I wrote the above for use in the tag report utility in the Exchange, before I added the scripting and expression unQualify() functions.)

I'm doing something similar to that for storing some trend related properties per user. You're welcome to try and decipher and see if this helps you. This packages the config into a message handler, sends it to a remote gateway, and executes a namedquery to save/load it on the other gateway's database.

Fair warning - AI supported coding, so may not be best practices but it currently works well for me.

def savequickpoints(UserName, Data):
	"""
	Saves the quickpoints dataset to the database as a JSON string.
	"""
	# 1. Build the list from the dataset
	points_list = []
	
	if Data is not None:
		py_data = system.dataset.toPyDataSet(Data)
		
		for row in py_data:
			points_list.append({
				"Pad": row["Pad"],
				"Pen Name": row["Pen Name"],
				"Description": row["Description"],
				"Path": row["Path"]
			})

	# 2. Encode to JSON
	# THIS IS THE CRITICAL STEP. 
	# We must convert the list of objects into a text string.
	json_string = system.util.jsonEncode(points_list)

	# DEBUG: Print this to the console to confirm it looks like text, not "Dataset [...]"
	print "Saving the following string to SQL:", json_string

	# 3. Call the remote data handler
	return get_remote_data(
		query_path="custom_trend/savequickpoints",
		database="-redacted-",
		UserName = UserName,
		Data = json_string 
	)	

def loadquickpoints(UserName):
	"""
	Fetches the JSON string from SQL and converts it to a Dataset.
	"""
	# 1. Fetch the data
	result = get_remote_data(
		query_path="custom_trend/loadquickpoints",
		database="-redacted-",
		UserName = UserName
	)

	# 2. Sanitize the Result (Handle Dataset vs String)
	json_string = ""
	
	if result is None:
		return None
	
	# Check if result is an Ignition Dataset (common if Query Type is 'Select')
	if hasattr(result, 'getValueAt'):
		if result.getRowCount() > 0:
			json_string = result.getValueAt(0, 0) 
		else:
			return None
	else:
		# Assume it is already the scalar string
		json_string = result

	# 3. Decode JSON
	if not json_string:
		return None

	try:
		data_structure = system.util.jsonDecode(json_string)
	except:
		# Log the error if needed, or just return None
		return None

	# 4. Normalize to List
	# This prevents the 'unicode' error by ensuring we are looping over a list of objects,
	# not iterating over the keys of a single dictionary.
	if isinstance(data_structure, dict):
		data_list = [data_structure]
	elif isinstance(data_structure, list):
		data_list = data_structure
	else:
		return None # Unknown format

	# 5. Convert to Dataset
	headers = ["Pad", "Pen Name", "Description", "Path"]
	rows = []
	
	for item in data_list:
		# Critical Check: ensure the item is a dictionary before calling .get()
		if isinstance(item, dict):
			rows.append([
				item.get("Pad"),
				item.get("Pen Name"),
				item.get("Description"),
				item.get("Path")
			])
			
	return system.dataset.toDataSet(headers, rows)

The named queries are just:

loadquickpoints:
SELECT Data
FROM custom_quickpoints
WHERE UserName=:UserName
savequickpoints:
IF EXISTS (SELECT 1 FROM custom_quickpoints WHERE UserName = :UserName)
BEGIN
    -- User exists: Update their existing record
    UPDATE custom_quickpoints
    SET Data = :Data
    WHERE UserName = :UserName
END
ELSE
BEGIN
    -- New User: Insert a new record
    INSERT INTO custom_quickpoints (UserName, Data)
    VALUES (:UserName, :Data)
END

Edit: The on change script that is attached to session.custom.quickpoints:

def valueChanged(self, previousValue, currentValue, origin, missedEvents):
	UserName = self.props.auth.user.userName
	Data = self.custom.quickpoints
	if UserName:
	    remote_utils.savequickpoints(UserName, Data)

nvarchar(max) will work perfectly fine. Since you're using MSSQL however, you can consider using its JSON datatype which is optimized in different ways.

2 Likes

When you do a insert with a named query do you use a String param type? With a prep query is it still just doing system.util.jsonEncode? Also retrieval do you still have to decode? Just curious how this plays with Ignition

That was my plan.

I'm rather discouraged with the complexity of what I thought would be a simple task - to convert a single layer deep dictionary into a string - only to find that the values are all qualified. At this stage I'm thinking that something like,

concat(
	'{',
	'"process": ', {value}['process'], ', ',
	'"notes": ', {value}['notes'], ', ',
	'"meta": ', {value}['meta'],
	'}'
)

would be much simpler but at this stage I'm wondering why bother saving it as a JSON string at all if it's so complicated to pass it in and out of the database table. I might as well just do

concat(
	'process: ', {value}['process'], '; ',
	'notes: ', {value}['notes'], '; ',
	'meta: ', {value}['meta'], ';'
)

to return a string of the form,
process: true; notes: true; meta: false;
and store that instead. It would be easy to split on retrieval.

I do it with a

system.db.runPrepUpdate(“INSERT INTO table (pythonDict) VALUES (?)”, [system.util.jsonEncode(myDict)])

but you would still need to handle converting it to a regular dict first without the qualified values and then getting it out convert back to one with qualified values presumably to make it work right.

My question was actually towards @djs about the JSON type I didn’t know how nicley the JBDC worked with that and if it that had extra considerations though this would have little to do with your current problem of nested qualified values.

I’d opt for @pturmel’s script above, use it to convert programatically to a dict and then you can use a query like I just wrote. Going back the otherway though :person_shrugging: Go back through your dictionary casting everything as a QualifiedValue I guess?

Yeah, you'd just use a String param type. The database would handle parsing the JSON and validating it, but in terms of IO it's a varchar. Here's the basic flow:

  1. Encode your dict with system.util.jsonEncode() into a json string.
  2. Pass to a named query as a string or to a prep query to store it in the DB.
  3. Retrieve the json string from the DB (do modifications on it in SQL if you want).
  4. Use system.util.jsonDecode() to decode the retrieved json string back into a dict.
2 Likes

Well that was easy! It looks like I don't need to un-qualify the Perspective prop.


Figure 1. JSON encoding of the dictionary.

And jsonDecode() seems to work fine on the way back out too.

Thanks, guys.

3 Likes

So it works just like the nvarchar(max) from the ignition side its just the db optimizes on its end, nice! Good tidbit.

1 Like

Next problem:


Figure 1. View in Designer
(1) is a query binding to what's stored in the database.


Figure 2. View in the browser.

Can anyone think of a reason the data is returned as an escaped string in the browser?

It's the same on Chrome and Edge browsers.

I don’t know that you can do a query binding since you will need to do a system.util.jsonDecode on the column

Is your query binding returning JSON?

1 Like