Perspective Ad-Hoc Trends with Remote Tag Providers

I was working with the Perspective AdHoc trends view created by Matt Raybourn on the exchange, and noticed that when working with remote tag providers it was excruciatingly slow to load the tag tree. The reason for this is that the browsing is recursive, and a separate system.tag.browse call is executed for every single tag. In a case where you have hundreds or thousands of tags, this can start to take much longer than acceptable to load in the client.

The system.tag.browse and system.tag.getConfiguration functions are very similar, except that the getConfiguration function has a built in “recursive” parameter, that allows you to send one remote call and collect every tag and all of its children, instead of needing to execute multiple remote calls to build the page.

I rewrote the binding transform on the items property of the tag browse tree to the following, and this allows remote providers to be scanned and rendered much more quickly than using system.tag.browse, it is a little more complicated, and system.tag.getConfiguration doesnt always return things in the same order they exist in the tag folder so I had to sort them, but this can make the page much more usable in cases where previously it would take up to 30 minutes to load a remote tag provider with 500 tags. This may not be the absolute fastest way to query large remote tag providers, but its the best one I have come up with.

Hope this helps someone, and if Matt Raybourn sees it than it may be valuable to be on the actual exchange item itself!

	#This could come from a property
	tagProvider = "[Texas North]"
	
	#[{label, expanded, data, items[]}]
	#The replacement function for browseConfiguration (Using the full dictionary of tags)
	def browseTagConfiguration(tagConfiguration, startPath):
		tags = []
		
		for result in tagConfiguration:
			tagPath = startPath + "/" + str(result["path"])
			
			#If there is a number as the last item in the path, use that as the sorting index, comment this out for alphabetical sorting
	#		try:
	#			nameArray = tagPath.split(' ')
	#			index = int(nameArray[len(nameArray) - 1])
	#		except:
	#			index = 9999
			#Leave this for alphabetical sorting
			index = 0
			
			
			#Check the tagType to determine the item structure
			if str(result["tagType"]) in ["Folder", "UdtInstance", "Provider"]:
				tag = {'label': result["path"], 'expanded': False, 'data': {'folder':tagPath, 'index':index}}
				
				#An empty folder will not contain the "tags" key in its configuration
				if result.has_key("tags"):
	
					#Numerical sorting
					#tag['items'] = sorted(browseTagConfiguration(result["tags"],tagPath ), key = lambda i: i["data"]['index'])
					
					#Alphabetical sorting
					tag['items'] = sorted(browseTagConfiguration(result["tags"],tagPath ), key = lambda i: i['label'])
					
					if len(tag['items']) > 0:
						tags.append(tag)
			else:
				#A tag without history enabled will not contain the "historyEnabled" key in its configuration
				if result.has_key("historyEnabled"):
					#I only wanted to return tags that I was collecting history on
					if result["historyEnabled"]:
						tags.append({'label': result["path"], 'data': {'tag': tagPath, 'index':index}, 'items': [], 'expanded': False})
	
		return tags
	
	
	def browseHistoricalTags(path):
			tags = []
			#results = system.tag.browse(path).getResults()
			results = system.tag.browseHistoricalTags(path).getResults()
			
			if results != None:
				for result in results:
					if result.hasChildren() == True:
						tag = {'label': result.path.lastPathComponent, 'expanded': False, 'data': {'folder': result.path}}
						tag['items'] = browseHistoricalTags(result.path)
						if len(tag['items']) > 0:
							tags.append(tag)
					else:
						tags.append({'label': result.path.lastPathComponent, 'data': {'tag': result.path}})
			return tags
	
	if value:
		tags = browseHistoricalTags(tagProvider)
	else:
		tagConfiguration = system.tag.getConfiguration(tagProvider, 1)[0]

		#Numerical sorting
		#tags = sorted(browseTagConfiguration(tagConfiguration["tags"],''), key = lambda i: i["data"]['index'])
		#Alphabetical Sorting
		tags = sorted(browseTagConfiguration(tagConfiguration["tags"],tagProvider ), key = lambda i: i['label'])
		
	
	return tags

EDIT: Originally I accidentally left the incorrect function in for browseHistoricalTags! I have replaced it with the correct one.

6 Likes

Well done!

1 Like

Very cool, and kudos for sharing your solution! @mraybourn :slight_smile:

1 Like

Yeah, that is awesome, thanks for posting it!

1 Like

I was a bit sad that from 7 to 8, the tag browse function lost its recursive option. Do you know why that was?

As a follow up to this,

If you have a system with a huge number of tags, this can start to become a bit slow as it tries to recursively iterate through a huge number of tags.

So I built another version that browses one folder at a time progressively

3 Likes