Exploring Efficient Methods to Build an OPC Server Address Tree

[mkowalk]

Hello,
I am currently working on finding the fastest method to generate/obtain a complete OPC server node/address model/tree/object while preserving its hierarchy for a given OPC Server. Any recommendations or insights would be greatly appreciated.

Context
The goal of capturing the entire tree is to store OPC addresses as records in a database table, maintaining the hierarchy of the addresses. This allows me to query the database and retrieve all child tag addresses for selected nodes efficiently. My plan is to leverage this table for additional features that will rely on this hierarchical structure. Below is a partial screenshot of what I achieved in my initial attempt to generate the OPC address tree:

Initial Attempt
In my first attempt, I used system.opc.browseServer recursively to generate a nested Python dictionary. This approach worked well for smaller systems, such as a PLC with approximately 1,000 tags, taking about 15–30 seconds to complete. However, when running the same script on a system with over 3,000 tags, it failed to generate results even after 5–10 minutes.

For context, these scripts were run from the project library and tested using the scripting console. Although the console did not freeze, the execution time appeared to increase exponentially with the number of tags, making it impossible to determine if or when the process would complete.

For me to consider this a success, I am personally aiming to keep execution time under 1 minute understanding that adding more tags will of course increase execution time.

Second Approach
Due to the performance issues in the first attempt, I started exploring a second approach (shown below). This involved investigating whether the complete OPC address tree already existed in the background and could be retrieved directly using references from the Java API. Observing how the designer's OPC Browser retrieves data raised questions about whether it performs on-demand browsing of child nodes —similar to what I attempted in my first approach, or if it relies on a fully constructed model. I stopped before going farther because I feared I was getting to far off in the weeds for something that probably doesn't exist after my initial investigation.

Questions
For those with experience in this area, I would appreciate your thoughts on the following:
1. Is it possible to retrieve a full OPC server list of PyOPCTagEx or OPCBrowseElement objects, does a method exist to retrieve this without a recursive browse?
2. Is there a better way to construct the full object, or is recursive browsing the best option?
3. Would you recommend breaking the browse operations into smaller chunks for database inserts rather than building a single, large object in one go?
4. Are there significant performance differences between system.opc.browseServer and the methods in the Java API? Would investing time in leveraging the Java API / Java Core Library offer any tangible speed benefits in this case?
5. Are the methods and classes in the java API for OPC browsing thread safe and/or ok to call concurrently. wondering if It might be worth while investigating if I can split up generating my nested object into parallel tasks instead of one sequential task.

Thank you in advance for any guidance or suggestions! I'm hoping to determine if my first approach would be effective with additional tweaks or if my efforts would be better focused on another route.

[Kevin.Herron]

@mkowalk

1. Is it possible to retrieve a full OPC server list of PyOPCTagEx or OPCBrowseElement objects, does a method exist to retrieve this without a recursive browse?
2. Is there a better way to construct the full object, or is recursive browsing the best option?

No, it's not held in memory, it's always browsed "live".

The address space of an OPC server can be very large and change at any time.

You could gain a little performance by executing your script in the gateway scope (message handler, tag change script, etc...), which would remove the overhead of designer-gateway comms, but this is still a fundamentally slow operation you're proposing.

[Kevin.Herron]

@mkowalk

In my first attempt, I used system.opc.browseServer recursively to generate a nested Python dictionary. This approach worked well for smaller systems, such as a PLC with approximately 1,000 tags, taking about 15–30 seconds to complete. However, when running the same script on a system with over 3,000 tags, it failed to generate results even after 5–10 minutes.

Maybe you made a mistake?

Testing with both the designer-gateway overhead and the overhead of my PLC being located over a VPN connection, I managed:

browsed 25563 tags in 149.759000063
>>> 

with:

def browse(server, nodeId, tags, depth = 0):
	children = system.opc.browseServer(server, nodeId)
	
	for child in children:
		childType = str(child.getElementType())
		childNodeId = child.getServerNodeId().getNodeId()
		
		#print '%s%s' % (depth * '  ', childNodeId)
		tags[childNodeId] = child
		
		if (childType == 'FOLDER' or childType == 'OBJECT'):
			browse(server, childNodeId, tags, depth + 1)

script console:

import time

server = 'Ignition OPC UA Server'
root = '[logix57]'
tags = {}

start = time.time()
browse.browse(server, root, tags)
finish = time.time()

print 'browsed %s tags in %s' % (len(tags), finish - start)