[IGN-4488]Is tags compare between 2 versions of the project impossible?

Ignition
1

I have 2 versions of the project and i need to compare the tags to find out if there are missing tags/configuration in the master project. I have exported the tags (json file type) and when trying to compare the files using Notepad ++ i noticed that json re-shuffles the order of tags and comparison impossible which made me very sad :frowning:

Is there any other way to compare the tags to find what is missing in the master project?

General thought:
is using json file for exports not limiting ignition scada a lot as we cannot use excel for heavy duty manipulation of the tags?

Please help/advise

2

If you have a lot of tags you should do that with a program xd

Something like this? should output all the names that are not in the others list…

this should give you alteast some names to know where to look next xd

import json
def json_extract(obj, key):
	    """Recursively fetch values from nested JSON."""
	    arr = []
	
	    def extract(obj, arr, key):
	        """Recursively search for values of key in JSON tree."""
	        if isinstance(obj, dict):
	            for k, v in obj.items():
	                if isinstance(v, (dict, list)):
	                    extract(v, arr, key)
	                elif k == key:
	                    arr.append(v)
	        elif isinstance(obj, list):
	            for item in obj:
	                extract(item, arr, key)
	        return arr
	
	    values = extract(obj, arr, key)
	    return values


list1 = json_extract(json.loads(open('C:/Path/to/tags.json').read()), 'name')
list2 = json_extract(json.loads(open('C:/Path/to/tagsVersion2.json').read()), 'name')

compare = list(set(list1) - set(list2)) + list(set(list2) - set(list1))
print(compare)
1 Like
3

Use a tool that understands the semantics of JSON, e.g. http://www.jsondiff.com/
JSON is a human-readable format, but is not plain text.

On the contrary,

  1. Exporting to Excel would be dramatically limiting. Excel's strict row/column model, by definition, cannot fit nested data. If each row is a tag, where do you put properties that can have multiple values, such as alarms?
  2. Modern Excel versions can directly read JSON files, although you'll still ultimately have the same problem of shoving multi-dimensional nested data into a strict R/C format: Power Query JSON connector - Power Query | Microsoft Learn

Using a simple, but structured format, allows for a huge variety of tooling; JSON is a 'de-facto' standard for interchange between APIs and has an ecosystem to match. For instance, there's an incredibly powerful tool call jq that can make very complex operations absolutely trivial; see this post for an example:

1 Like
4

Ignition’s JSON output (everywhere) should still be sorted and repeatable for plain text diffs. For VCS compatibility, in addition to human eye-strain reduction. jsondiff and similar tools are a work-around, not a solution.

2 Likes
5

Views and other Perspective resources that land on disk are sorted, but that’s a good point, we should do that for other areas.

3 Likes
6

Where JSON compare tools like jsondiff fail though is where they don’t understand the tag structure built up of name keys and hence is unable to know that these are the keys and their props to compare with each other. A custom compare tool needs to be written which understands this inbuilt structure.

I wrote a tool in python, but it would be great if a compare tool would be added into ignition itself. I think there’s already an idea for it from memory

1 Like
7

While you are at it, let me plug the inclusion of defaults for properties. Yes, that makes files bigger. But it makes movement across versions more reliable (defaults can and have changed at times). Omitting them is a hangover from Vision resource optimization that shouldn’t be propagated.

3 Likes
8

Or make it an option to copy with defaults or without. Sometimes it’s useful not to have defaults, but for the majority of the time it would be better with, although maybe we’re talking about different things… I’m talking about copying json from the context menu

1 Like
9

For the JSON representation of UDT definitions the member tag configurations are stored as a list of dictionaries (sorry, using python terms to reference JSON equivalents), changes to a UDT definition tend to result in a change to the order of the of member tag configurations in the "Copy JSON" representation of that list.

Unfortunately all the semantic diff tools I have found do not provide means to treat a list as unsorted or to presort it by a specific key for each dictionary in the list of dictionaries.... making the compare results somewhat useless.

Realizing that I am reviving a three year old thread, has anyone come across a solution/tool that will do a semantic diff on JSON text and allow for the sorting or compare of a list of dictionaries based on a specified dictionary key (or keys) (e.g. sort and match based on the "name" key in each dictionary)?

My specific use case is doing a "Copy JSON" of a UDT definition on two gateways that "should" match, however with many hands in the pot there is inevitably drift and I want an easy way to compare the two UDT definitions, see how they semantically differ so I can decide how to bring them back to alignment. Similar issues exist when attempting to compare the JSON export of "standardized" views across multiple gateways.

10

I'll attempt switching to proper JSON terminology...

I found JSON Editor Online which does allow me to select any list of dictionaries (array of objects) in a JSON tree and sort the list (array) by a specified dictionary (object) key but I'd then have to do that for each and every list of dictionaries (array of objects) on each side of the compare. Still hoping there is a tool that can do this to the whole JSON tree with a few pre-defined rules (e.g. sort all lists of dictionaries (arrays of objects) using the value of the 'name' key in each dictionary (object). Any suggestions?
image
image

11

You simply won't be able to use a standard diff tool for this, regardless of the sorting of your json, since tags could be added or removed in different versions.

See here

12

Instead of looking further for an elusive tool that could apply sorting to my JSON exports I instead made scripts that sort the JSON string.

I can then export the "sorted" JSON version of a UDT definition from two different Ignition gateways and am able to compare using any text 'standard' text comparison tool that I like. A semantics aware text compare is not strictly needed because I sort the keys within each object and sort each array of objects.

Working like a charm so far. Using script console on two designer sessions (two gateways) I run the exportTags_sorted() function on the same tag path (recursive or not) dumping each output to its own file. I then have the diff tool in VS Code monitoring both files and can easily spot meaningful differences.

import json
from collections import OrderedDict


def exportTags_sorted(file_path=None, tag_paths=[], recursive=False):
	"""
	Export tags as JSON... with sorting applied to make text comparisons work better
	"""
	
	# Set the sort order for each JSON object, key names not listed with be sorted alphabetically
	jsonObject_keyOrder = ['name', 'tagType', 'typeId', 'typeColor',  'tooltip', 'documentation', 'parameters']
	
	# Set the JSON object key used to sort each JSON array of objects 
	jsonArray_keyOrder = ["name"]
	
	# Export tags as a JSON string
	json_string = system.tag.exportTags(
	#	  filePath   = file_path # if omitted, causes the function to return the tag export as a string.
		  tagPaths   = tag_paths
		, recursive  = recursive
		, exportType = 'json'
	)
	
	# Sort the JSON string
	sorted_json_string = sort_json_string(json_string, jsonObject_keyOrder, jsonArray_keyOrder)
	
	if file_path is not None:
		print "Writing to '{}'".format(file_path)
		with open(file_path, 'w') as file:
			file.write(sorted_json_string)
	
	return sorted_json_string


def sort_json_string(json_string, jsonObject_keyOrder=["name"], jsonArray_keyOrder=["name"], indent=2):
	"""
	Read a JSON string and return a sorted version of it.
	This allows better comparion using text comaprison tools.
	
	Object attributes will be sorted by key name. In the specified key name order, then by key name alphabetically.
	Arrays of Objects will be sorted by the value of the attribute having a specified key name.
	
	Args:
		json_string (str): The JSON string to be sorted.
		jsonObject_keyOrder (list): A list of key names, in order of priority, used to sort JSON objects (based on key name).
			Key names not included in the list will be sorted alphabetically after the key names that are included. 
		jsonArray_keyOrder (list): A list of keys that can be used to sort a JSON array of objects, the first key that exists in all objects is used.
	"""
	
	# Load (decode) the JSON string into a Python object
	pyObj = json.loads(json_string)
#	pyObj = system.util.jsonDecode(json_string)
	
	# Create a dictionary to assign priorities to the JSON object attribute keys
	jsonObject_keyPriority = {key: i for i, key in enumerate(jsonObject_keyOrder)}
	
	# Custom sorting function for JSON Object, assign priority by object attribute key names
	def jsonObject_key_priority(key):
		return (jsonObject_keyPriority.get(key, len(jsonObject_keyOrder)), key)
	
	# Recursive function to sort dictionary keys using the custom sort
	def sortObject(obj):
		if isinstance(obj, dict):
			# Sort the dictionary keys using the custom sorting function
			# Using an ordered dictionary allows order to be retained when encoding back to JSON string.
			sorted_dict = OrderedDict()
			for key in sorted(obj.keys(), key=jsonObject_key_priority):
				sorted_dict[key] = sortObject(obj[key])
			return sorted_dict
		elif isinstance(obj, list):
			# Sort list elements if they are dictionaries and each contains a key listed in jsonArray_keyOrder
			if all(isinstance(item, dict) and any(k in item for k in jsonArray_keyOrder) for item in obj):
				for key in jsonArray_keyOrder:
					if all(key in item for item in obj):
					#x	obj = list(obj) # Shallow copy, just so the sorting does not affect the original
					#x	obj.sort(key=lambda x: x[key])
						obj = sorted(obj, key=lambda x: x[key])
						break
			return [sortObject(item) for item in obj]
		else:
			return obj
	
	# Sort the JSON structure
	sorted_data = sortObject(pyObj)
	
	# Convert the Python object back to a JSON string
	sorted_json_string = json.dumps(sorted_data, indent=indent)
#	sorted_json_string = system.util.jsonEncode(sorted_data, indent)

	return sorted_json_string

def exportTags_sorted_example_usage(tag_paths=[], recursive = True):
	if not len(tag_paths):
		tag_paths = ['[default]_types_/Diagnostics/Disk Partition Info']
	
	project_name = system.util.getProjectName()
	dateString = system.date.format(system.date.now(), "YYYY-MM-dd_HHmmss")

	file_path = 'c:/x/{}_Test.json'.format(project_name)

	sorted_json = exportTags_sorted(file_path=file_path, tag_paths=tag_paths, recursive=recursive)
	print sorted_json
13

@nminchin, I'm not sure I'm follow your argument here. If the JSON is consistently sorted before the compare than one should easily be able to find added and removed tags while doing a compare. Likely depends on the aptitude of your compare tool as to whether it will recognize added/removed lines associated with added/removed tags) or just mismatch the comparison from that point on. Even if it creates a mismatch in the compare results, as long as everything else is consistently sorted I can manually compensate for that without much headache.

14

Standard diff tools don't know that the "primary keys" of tag json are the "name" keys, so they don't know what they're comparing.

You will get incorrect diff results if you try to compare tag json with added or removed tags using standard tools. If the tag names match, it will be ok

15

Exactly why I was looking for a tool that can apply sorting to each JSON array of objects by telling it to sort based on the 'name' key in each object. Per my last message above I then went the route of applying my own sorting to the JSON string prior to the compare and that is working well.
Semantic diff tools can handle the fact that keys on JSON objects may appear in different order but always assume the order of an array is significant while Ignition seemingly arbitrarily changes the order of those on almost any configuration change.

16

If the built-in JSON export tools (Designer GUI and scripting) would sort by key name within each JSON object and sort each JSON array of objects by the 'name' key that would go a long way towards making the JSON exports more directly comparable. I have yet to find a JSON export with an array of objects that does not use the 'name' key (though that may exist).

There may (or may not) be cases where the order of a list of objects has significance but Ignition should know when it is not significant and apply sorting ('name' key) for ease of direct text based comparison.

While semantic compare tools can handle unordered JSON object keys I suggest default sorting of object attributes by key name as well.

1 Like
17

Example:

image

Original Tags:

Original Tags 
{
  "tags": [
    {
      "name": "Folder A",
      "tagType": "Folder",
      "tags": [
        {
          "valueSource": "memory",
          "dataType": "Float8",
          "name": "Tag AA",
          "value": 12,
          "tagType": "AtomicTag",
          "engUnit": "°C"
        },
        {
          "valueSource": "memory",
          "dataType": "Boolean",
          "name": "Tag AB",
          "value": true,
          "tagType": "AtomicTag",
          "engUnit": ""
        }
      ]
    },
    {
      "valueSource": "memory",
      "dataType": "Float8",
      "name": "Tag A",
      "value": 12,
      "tagType": "AtomicTag",
      "engUnit": "°C"
    },
    {
      "valueSource": "memory",
      "dataType": "Boolean",
      "name": "Tag B",
      "value": true,
      "tagType": "AtomicTag",
      "engUnit": ""
    }
  ]
}

Changed/New Tags:

Changed Tags 
{
  "tags": [
    {
      "name": "Folder A",
      "tagType": "Folder",
      "tags": [
        {
          "valueSource": "memory",
          "dataType": "Float8",
          "name": "Tag AA",
          "value": 12,
          "tagType": "AtomicTag",
          "engUnit": "°C"
        },
        {
          "valueSource": "memory",
          "dataType": "Boolean",
          "name": "Tag AC",
          "value": true,
          "tagType": "AtomicTag",
          "engUnit": ""
        }
      ]
    },
    {
      "name": "Folder B",
      "tagType": "Folder",
      "tags": [
        {
          "valueSource": "memory",
          "dataType": "Float8",
          "name": "Tag BA",
          "value": 12,
          "tagType": "AtomicTag",
          "engUnit": "°C"
        },
        {
          "valueSource": "memory",
          "dataType": "Boolean",
          "name": "Tag BC",
          "value": true,
          "tagType": "AtomicTag",
          "engUnit": ""
        }
      ]
    },
    {
      "valueSource": "memory",
      "dataType": "String",
      "name": "Tag B",
      "value": "Hello World!",
      "tagType": "AtomicTag",
      "engUnit": ""
    }
  ]
}

What I'm expecting to be able to see from a diff is:
Folder A/Tag BB: Removed
Folder A/Tag CC: Added
Folder B: Added
Folder B/Tag BA: Added
Folder B/Tag BB: Added
Tag A: Removed
Tag B.dataType: Boolean -> String

Results from a semantic diff (note I've sorted the tag json above):

image
image1256×1138 137 KB

This output is garbage, and this is a simple example.
What you get is:

  • Folder A/Tag AB renamed to Tag AC (wrong)
  • Folder B was added (correct)
  • Tag A was moved into? and renamed to Folder B/Tag BA (wrong)
  • Folder B/Tag BC added (correct)
  • Tag B was removed (wrong)
  • Tag B was added again?? (wrong)

With my script, you get:

tagpath tagtype change from to
root/Folder A/Tag BB AtomicTag Tag removed None None
root/Folder A/Tag CC AtomicTag Tag added None None
root/Tag B.dataType AtomicTag Property modified Boolean String
root/Tag B.value AtomicTag Property modified TRUE Hello World!
root/Tag A AtomicTag Tag removed None None
root/Folder B Folder Tag added None None
root/Folder B/Tag BA AtomicTag Tag added None None
root/Folder B/Tag BC AtomicTag Tag added None None

Which is exactly what i'm expecting to see (and I can see that I missed a change of the Tag B value in my expectations list)

But even if a diff DID work, it would be so hard to actually tell what's changed due to the nature of JSON. If you had lots of tags in deeply nested folders, in order to know the tagpath of the tags you will be constantly scrolling up trying to work out the folders each nest level is, concatenating mentally as you go, then you have to go and find the tag again to find the differences.
Personally, I'd prefer a table that just listed the tagpaths and their changes :man_shrugging:

1 Like
18

It is always significant in JSON. You cannot reorder lists.

Order of keys in objects is explicitly not significant in JSON. You can always present keys in order, whether in a UI, or in a copy to clipboard, or a file save.

I would hope IA chooses to use java.util.TreeMap(com.inductiveautomation.ignition.common.util.Comparators.alphaNumeric(false)) or similar when assembling objects for JSON conversion.

19

Except that for Ignition the order of individual tag configurations in a JSON array of objects is not functionally significant. One could export the JSON string, reorder the array(list) and reimport with no functional difference in Ignition behavior.

I agree that JSON arrays (lists) are inherently ordered, just saying for Ignition behavior that order often has no significance and you can reorder the list without functional impact.

20

Agreed, for this specific case. Only Ignition itself can determine where this would apply, and order them when generating the JSON.

(Pssst! IA! : use the same TreeMap(alphaNumeric) to assemble lists of tags and then output the TreeMap's .values ....)

1 Like