Json Diffing Discussion

FWIW, I made a script recently that achieves this with JSON tag exports, and could be tweaked for many other cases (currently uses the "name" key for sorting lists of dictionaries).

# Functions designed to be called via objectScript() in UI expressions.

from java.io import File
from java.lang import System, Throwable
from java.util import TreeMap
from com.inductiveautomation.ignition.common.util import Comparators

logger = system.util.getLogger(system.util.getProjectName() + '.' + system.reflect.getModulePath())
tkCtx = system.util.toolkitCtx()
ciAlnumCmp = Comparators.alphaNumeric(True)

def ordering(subject):
	if hasattr(subject, 'items'):
		subst = TreeMap(ciAlnumCmp)
		for k, v in subject.items():
			subst[k] = ordering(v)
		return subst
	first = None
	try:
		first = subject[0]
	except:
		pass
	try:
		if first and hasattr(first, 'items'):
			subst = TreeMap(ciAlnumCmp)
			subst.update([(x.get('name', str(i)), ordering(x)) for (i, x) in enumerate(subject)])
			return [x for x in subst.values()]
	except Throwable, t:
		logger.info("Java failure in ordering:", t)
	except Exception, e:
		logger.info("Jython failure in ordering:", system.reflect.asThrowable(e, None))
	return subject
#

#
def orderedJson(source):
	json = system.util.jsonDecode(source)
	ordered = ordering(json)
#	print repr(ordered)
	return system.util.jsonEncode(dict(_=ordered), 2)[6:-1]
#

Note that ordering() is a recursive deep copy operation, where dictionary-like types are replaced with a key-ordered dictionary, and lists of dictionaries are conditionally re-ordered by their inner name keys.

The use of name for lists of dictionaries could be made an argument to make this more widely applicable.

Consider this a plea for this algorithm or similar to become the default behavior whenever Ignition writes JSON to disk or for download/export.

The client who prompted this script's creation found it effective for re-synchronizing UDT definitions from dev to prod.

8 Likes

I should mention that side-by-side or inline diff views of related tag JSON that has been ordered this way is simply beautiful. If looking for an abbreviated form to show scattered small changes, I recommend diff -U20 file1 file2 so that the context around differences tends to include plenty of unique identifying information.

This script is a simple yet critical quality of life upgrade for our DevOps workflows. Here's a screenshot of diffing two ordered JSON outputs in Notepad++, something impossible to do before:

I urge IA to adopt this ordering for all JSON exports.

  • Imagine an upgrade to Designer where Export JSON creates these ordered, diffable exports by default rather than having to route the current exports through a reordering tool.
  • Imagine in 8.3 once things like our tag providers are in the filesystem natively, we can see meaningful changes inside our source code management tools.
2 Likes

I haven't tried this yet, but will this enable diffing of tag structures that have had tags added, eg new tags added to the top of a folder for example? If you try to diff this with the default json export, it'll end up trying to diff the new tags against old tags (assuming the tags end up sorting in the order of representation insode the folder) since diff tools don't understand the names are the primary keys.

Looked like it worked correctly for my client. When using the command line, the -U20 is probably required.

Diff tools are pretty aggressive at finding large blocks of unchanged text to "synchronize" against.

You should try it. (:

You can test in the script console with readFileAsString() and writeFile().

1 Like

I'll give it a go for sure when I get in the office. it's a tad late/early at the moment :sweat_smile:

1 Like

You're worse than @Kevin.Herron. :roll_eyes:

1 Like

When I'm online at that hour it's because I'm up early, never because I'm up late.

1 Like

Same for me these days. This is me waking up at an abominable time rather than staying up

1 Like

Perfect adjective. If I'm on the forum before ~7am (breakfast with the forum), it's because I'm visiting Europe.

1 Like

This part works as expected, yielding ordered dictionaries and lists. But....

So I've been poking around with org.json.JSONObject and discovered that IA isn't shipping the version that is publicly posted on Github. Which is good, because the public code that generates the pretty-printed indented format is ugly and non-extensible and won't actually output dictionaries with ordered keys, and can't easily be made to do so. (The author has strong opinions on this.)

The version included with Ignition has override-able methods offering sorted keys out of the box (but not human-friendly). Which are, in fact, what is coming out of the above jsonEncode(). :frowning_face:

Any of the usual suspects care to point at the modifications?

I'd like to override to actually get the encoding in my desired key order, but it will be pointless if this is IA custom and v8.3 changes it.

The org.json stuff is pretty much only used in old/legacy code. We use gson in anything written in the last decade?

Your SystemUtils bytecode suggests you are still using it for scripted encode/decode.

But I'm inclined to abandon it too, if that's what you've done.

Of course, those scripting functions have been around forever, and we're not going to swap out the underlying implementation because it will 100% cause a regression for somebody.

I think the plan is to add new system.json functions in 8.3 or 8.3.x?

4 Likes

Ok. I'll patch up something that'll hold until the new ("reference") implementations appear. (Encode/Decode in expression functions has become very useful.)

Do any methods/constructors in the GSON implementations cooperate with ordered maps?

(I suppose I should just research that... But not today. My brain is scorched.)

GSON's JsonObject uses a LinkedTreeMap internally.

For maximal confusion you'll find that there is both google-namespaced GSON and an internal IA GSON on the class path.

Any chance the JavaDoc for com.inductiveautomation.ignition.common.gson could be included in the public SDK docs?

I'll probably want the artifact IDs before too long...

Mmm, probably not in the SDK docs, this is a separately published artifact that doesn't appear to be published with javadoc JARs. We did publish source JARs, but I think those get stripped away when accessing the public nexus proxy.

We should probably publish it to the thirdparty repo instead. @Brian_Ray might know more.

1 Like

Ah! I see it here:

Thanks, this helps.

Hopefully that lines up... that may be an older version. We have another one on our internal GitHub that is newer.