Browsable OPC UA Address Space Help

I'm trying to implement browsing for my dynamic OPC UA address space, and I'm kinda lost.
I feel like I'm close but I'm missing some core idea.

Using this as a reference:

Here's what I'm trying to do:

  1. My OidAddressSpace handles reads/writes to the end device.

    • As successful reads happen, an OidModel is built that knows the addresses and their corresponding types.
    • Eventually I'd like to support walking the end device's entire address tree, but that's not implemented yet.
    • This address space is dynamic, no Node objects are created.
  2. I have a second address space fragment, BrowsableOidModelAddressSpace, that uses the details from the OidModel to support browsing.

    • I'm adding a folder under my device's node [device]OIDs. I can browse and see this folder.
    • When this folder is browsed, I am returning reference's built based on information from the OidModel.
    • The NodeIds I'm returning in my references look correct, I can read/subscribe to them, but I can't see anything in the browser :confused:

I'm confused on whether this is a problem with my OidAddressSpace, my BrowsableOidModelAddressSpace, or my approach in general.

OidAddressSpace
package com.mussonindustrial.embr.snmp.agents.opc

import com.mussonindustrial.embr.snmp.agents.context.OidModel
import com.mussonindustrial.embr.snmp.agents.devices.SnmpAgentDevice
import com.mussonindustrial.embr.snmp.model.OidValue
import com.mussonindustrial.embr.snmp.model.Snmp4jOid
import com.mussonindustrial.embr.snmp.opc.DeviceContextManagedAddressSpaceFragment
import com.mussonindustrial.embr.snmp.utils.isOid
import kotlin.jvm.optionals.getOrNull
import org.eclipse.milo.opcua.sdk.core.AccessLevel
import org.eclipse.milo.opcua.sdk.core.ValueRank
import org.eclipse.milo.opcua.sdk.server.AddressSpace
import org.eclipse.milo.opcua.sdk.server.AddressSpace.ReferenceResult
import org.eclipse.milo.opcua.sdk.server.AddressSpaceComposite
import org.eclipse.milo.opcua.sdk.server.AddressSpaceFilter
import org.eclipse.milo.opcua.sdk.server.SimpleAddressSpaceFilter
import org.eclipse.milo.opcua.stack.core.*
import org.eclipse.milo.opcua.stack.core.types.builtin.*
import org.eclipse.milo.opcua.stack.core.types.builtin.unsigned.UInteger
import org.eclipse.milo.opcua.stack.core.types.enumerated.NodeClass
import org.eclipse.milo.opcua.stack.core.types.enumerated.TimestampsToReturn
import org.eclipse.milo.opcua.stack.core.types.structured.ReadValueId
import org.eclipse.milo.opcua.stack.core.types.structured.ViewDescription
import org.eclipse.milo.opcua.stack.core.types.structured.WriteValue

class OidAddressSpace(val device: SnmpAgentDevice, composite: AddressSpaceComposite) :
    DeviceContextManagedAddressSpaceFragment(device.context.deviceContext, composite) {

    private val filter = SimpleAddressSpaceFilter.create { it.getPath().isOid() }
    private val model = device.model

    override fun read(
        context: AddressSpace.ReadContext,
        maxAge: Double,
        timestamps: TimestampsToReturn,
        readValueIds: List<ReadValueId>,
    ): List<DataValue> {
        val requests = readValueIds.map { ReadRequest(it) }
        val toProcess = requests.filter { it.value == null }

        val valueReads =
            toProcess.filter {
                AttributeId.from(it.readValueId.attributeId).get() == AttributeId.Value
            }
        model.read(valueReads.map { it.oid }).zip(valueReads).forEach { (value, result) ->
            result.value = value.value
        }

        val nonValueReads =
            toProcess.filter {
                AttributeId.from(it.readValueId.attributeId).get() != AttributeId.Value
            }
        readNonValueAttributes(nonValueReads).zip(nonValueReads).forEach { (value, result) ->
            result.value = value
        }

        return requests.map { it.value ?: DataValue(Variant.of(null)) }
    }

    fun readNonValueAttributes(results: List<ReadRequest>): List<DataValue> {
        val descriptors = model.getDescriptors(results.map { it.oid })

        return results.zip(descriptors).map { (it, descriptor) ->
            val nodeId = it.readValueId.nodeId
            val attributeId = AttributeId.from(it.readValueId.attributeId).getOrNull()

            try {
                val result =
                    when (attributeId) {
                        AttributeId.NodeId -> nodeId

                        AttributeId.NodeClass -> NodeClass.Variable

                        AttributeId.BrowseName ->
                            device.context.deviceContext.qualifiedName(nodeId.getPath())

                        AttributeId.DisplayName -> LocalizedText.english(descriptor.oid.numeric)
                        AttributeId.Description -> LocalizedText.english(descriptor.oid.numeric)

                        AttributeId.WriteMask,
                        AttributeId.UserWriteMask -> UInteger.valueOf(0)

                        AttributeId.DataType ->
                            when (descriptor) {
                                is OidModel.ValueDescriptor -> descriptor.snmpType.uaDataType
                                else -> OpcUaDataType.String.nodeId
                            }
                        AttributeId.ValueRank ->
                            when (descriptor) {
                                is OidModel.ValueDescriptor -> ValueRank.Scalar.value
                                else -> ValueRank.Scalar.value
                            }
                        AttributeId.ArrayDimensions ->
                            when (descriptor) {
                                is OidModel.ValueDescriptor -> intArrayOf()
                                else -> intArrayOf()
                            }

                        AttributeId.AccessLevel,
                        AttributeId.UserAccessLevel -> AccessLevel.toValue(AccessLevel.READ_WRITE)

                        AttributeId.Value ->
                            throw UaException(
                                StatusCodes.Bad_InternalError,
                                "attributeId: $attributeId",
                            )

                        else ->
                            throw UaException(
                                StatusCodes.Bad_AttributeIdInvalid,
                                "attributeId: $attributeId",
                            )
                    }!!

                DataValue(Variant(result))
            } catch (e: UaException) {
                DataValue(e.statusCode)
            }
        }
    }

    override fun write(
        context: AddressSpace.WriteContext,
        writeValues: List<WriteValue>,
    ): List<StatusCode?> {
        val results = writeValues.map { WriteRequest(it) }

        results.forEach {
            if (it.writeValue.attributeId == null) {
                it.value = StatusCode(StatusCodes.Bad_AttributeIdInvalid)
            }
            if (it.writeValue.indexRange != null && it.writeValue.indexRange.isNotEmpty()) {
                it.value = StatusCode(StatusCodes.Bad_NotImplemented)
            }
            if (AttributeId.from(it.writeValue.attributeId).getOrNull() != AttributeId.Value) {
                it.value = StatusCode(StatusCodes.Bad_NotImplemented)
            }
        }

        val valueWrites = results.filter { it.value == null }
        model
            .write(valueWrites.map { it.oid to it.writeValue.value.value.value })
            .zip(valueWrites)
            .forEach { (value, result) -> result.value = value.value }

        return results.map { it.value }
    }

    override fun browse(
        context: AddressSpace.BrowseContext,
        view: ViewDescription,
        nodeIds: List<NodeId>,
    ): List<ReferenceResult> {
        return emptyList()
    }

    override fun gather(
        context: AddressSpace.BrowseContext,
        view: ViewDescription,
        nodeId: NodeId,
    ): ReferenceResult.ReferenceList {
        return ReferenceResult.ReferenceList(emptyList())
    }

    override fun getFilter(): AddressSpaceFilter {
        return filter
    }

    fun NodeId.getPath(): String {
        return device.stripDeviceName(this)
    }

    inner class ReadRequest(val readValueId: ReadValueId) : OidValue<DataValue?> {
        override val oid = Snmp4jOid(readValueId.nodeId.getPath())
        override var value: DataValue? = null
    }

    inner class WriteRequest(val writeValue: WriteValue) : OidValue<StatusCode?> {
        override val oid = Snmp4jOid(writeValue.nodeId.getPath())
        override var value: StatusCode? = null
    }
}

BrowsableOidModelAddressSpace
package com.mussonindustrial.embr.snmp.agents.opc

import com.mussonindustrial.embr.common.logging.getLogger
import com.mussonindustrial.embr.snmp.agents.devices.SnmpAgentDevice
import com.mussonindustrial.embr.snmp.opc.DeviceContextManagedAddressSpaceFragment
import com.mussonindustrial.embr.snmp.utils.isOid
import com.mussonindustrial.embr.snmp.utils.removeAllNodes
import java.util.function.Function
import java.util.function.Supplier
import org.eclipse.milo.opcua.sdk.core.Reference
import org.eclipse.milo.opcua.sdk.server.*
import org.eclipse.milo.opcua.sdk.server.AddressSpace.ReferenceResult
import org.eclipse.milo.opcua.sdk.server.nodes.UaFolderNode
import org.eclipse.milo.opcua.stack.core.AttributeId
import org.eclipse.milo.opcua.stack.core.NodeIds
import org.eclipse.milo.opcua.stack.core.StatusCodes
import org.eclipse.milo.opcua.stack.core.types.builtin.DataValue
import org.eclipse.milo.opcua.stack.core.types.builtin.LocalizedText
import org.eclipse.milo.opcua.stack.core.types.builtin.NodeId
import org.eclipse.milo.opcua.stack.core.types.builtin.Variant
import org.eclipse.milo.opcua.stack.core.types.enumerated.NodeClass
import org.eclipse.milo.opcua.stack.core.types.enumerated.TimestampsToReturn
import org.eclipse.milo.opcua.stack.core.types.structured.ReadValueId
import org.eclipse.milo.opcua.stack.core.types.structured.ViewDescription

class BrowsableOidModelAddressSpace(val device: SnmpAgentDevice, composite: AddressSpaceComposite) :
    DeviceContextManagedAddressSpaceFragment(device.context.deviceContext, composite) {

    private val root = "OIDs"
    private val model = device.model
    private val logger = getLogger()
    private val filter =
        SimpleAddressSpaceFilter.create {
            logger.info("Filtering path $it: path=${it.getPath()}")
            if (nodeManager.containsNode(it)) return@create true
            return@create it.getPath().isOid()
        }

    init {
        lifecycleManager.addLifecycle(
            object : Lifecycle {
                override fun startup() {
                    addNodes()
                }

                override fun shutdown() {
                    nodeManager.removeAllNodes()
                }
            }
        )
    }

    fun addNodes() {
        val oidFolder =
            UaFolderNode(
                nodeContext,
                nodeId(root),
                qualifiedName(root),
                LocalizedText.english(root),
            )
        nodeManager.addNode(oidFolder)

        oidFolder.addReference(
            Reference(
                oidFolder.nodeId,
                NodeIds.Organizes,
                deviceNodeId.expanded(),
                Reference.Direction.INVERSE,
            )
        )
    }

    override fun browse(
        context: AddressSpace.BrowseContext,
        view: ViewDescription,
        nodeIds: List<NodeId>,
    ): List<ReferenceResult> {

        val results = ArrayList<ReferenceResult>()

        nodeIds.forEach { nodeId ->
            val references = ArrayList<Reference>()

            val path = nodeId.getPath()
            logger.info("Browsing path=${path}")
            if (path != root) {
                results.add(super.browse(context, view, listOf(nodeId))[0])
                return@forEach
            }

            val oids = model.listOids()
            val descriptors = model.getDescriptors(oids)

            descriptors.forEach { descriptor ->
                logger.info("Browsed descriptor for OID: ${descriptor.oid.numeric}")
                references.add(
                    Reference(
                        nodeId(descriptor.oid.numeric),
                        NodeIds.HasComponent,
                        nodeId.expanded(),
                        Reference.Direction.INVERSE,
                    )
                )
            }
            results.add(ReferenceResult.of(references))
            logger.info("Browse results: ${references.map { it.sourceNodeId }}")
        }

        return results
    }

    override fun read(
        context: AddressSpace.ReadContext,
        maxAge: Double,
        timestamps: TimestampsToReturn,
        readValueIds: MutableList<ReadValueId>,
    ): MutableList<DataValue?> {
        val values = ArrayList<DataValue?>()

        for (readValueId in readValueIds) {
            val node = nodeManager.get(readValueId.nodeId)
            if (node != null) {
                logger.info("Reading node=${node.nodeId}")
                val value =
                    AttributeReader.readAttribute(
                        context,
                        node,
                        readValueId.attributeId,
                        timestamps,
                        readValueId.indexRange,
                        readValueId.dataEncoding,
                    )

                values.add(value)
            } else {
                val path = readValueId.nodeId.getPath()
                logger.info("Reading path=${path}")
                when (path) {
                    root -> {
                        logger.info("Reading Root attribute: ${readValueId.attributeId}")
                        values.add(
                            AttributeId.from(readValueId.attributeId)
                                .map { DataValue(readAttribute(readValueId.nodeId, it)) }
                                .orElseGet { DataValue(StatusCodes.Bad_AttributeIdInvalid) }
                        )
                    }

                    else -> {
                        if (path.isOid()) {
                            logger.info("Reading OID attribute: ${readValueId.attributeId}")
                            values.add(
                                AttributeId.from(readValueId.attributeId)
                                    .map(
                                        Function { attributeId: AttributeId? ->
                                            val variant =
                                                readAttribute(readValueId.nodeId, attributeId!!)
                                            DataValue(variant)
                                        }
                                    )
                                    .orElseGet(
                                        Supplier { DataValue(StatusCodes.Bad_AttributeIdInvalid) }
                                    )
                            )
                        } else {
                            values.add(DataValue(StatusCodes.Bad_NodeIdUnknown))
                        }
                    }
                }
            }
        }

        return values
    }

    private fun readAttribute(nodeId: NodeId, attributeId: AttributeId): Variant {
        val result =
            when (attributeId) {
                AttributeId.NodeId -> nodeId
                AttributeId.NodeClass -> NodeClass.Object
                AttributeId.BrowseName -> {
                    deviceContext.qualifiedName(nodeId.getPath())
                }

                AttributeId.DisplayName,
                AttributeId.Description -> {
                    LocalizedText.english(nodeId.getPath())
                }

                else -> null
            }

        return if (result == null) Variant.NULL_VALUE else Variant(result)
    }

    override fun getFilter(): AddressSpaceFilter {
        return filter
    }

    fun NodeId.getPath(): String {
        return device.stripDeviceName(this)
    }
}


Any pointers would be appreciated :slightly_smiling_face:
(Also don't judge the OidAddressSpace's attribute reading too harshly, I still have more descriptor types to implement).

It's a little silly, but sometimes when debugging the browse stuff I find it helps to look at Wireshark. Often you'll see the browse happen, that it returns some references, and then a subsequent read for attributes of those NodeIds is where things go wrong.

Also add logging to the filter in both your address spaces and make sure one isn't claiming nodes the other should handle.

1 Like

Ooh that's a good idea. The BrowseResponse results are definitely empty.

BrowseRequest

Frame 23256: 194 bytes on wire (1552 bits), 194 bytes captured (1552 bits) on interface \Device\NPF_{37217669-42DA-4657-A55B-0D995D328250}, id 0
Raw packet data
Internet Protocol Version 4, Src: 100.68.72.71, Dst: 100.70.98.88
Transmission Control Protocol, Src Port: 59383, Dst Port: 62541, Seq: 11585, Ack: 8042, Len: 154
OpcUa Binary Protocol
    Message Type: MSG
    Chunk Type: F
    Message Size: 154
    SecureChannelId: 441419179
    Security Token Id: 3
    Sequence Number: 57
    RequestId: 57
    Message: Encodeable Object
        TypeId: ExpandedNodeId
        BrowseRequest
            RequestHeader: RequestHeader
            View: ViewDescription
            RequestedMaxReferencesPerNode: 100
            NodesToBrowse: Array of BrowseDescription
                ArraySize: 1
                [0]: BrowseDescription
                    NodeId: NodeId
                        .... 0011 = EncodingMask: String (0x3)
                        Namespace Index: 1
                        Identifier String: [snmp_agent]OIDs
                    BrowseDirection: Forward (0x00000000)
                    ReferenceTypeId: NodeId
                        .... 0000 = EncodingMask: Two byte encoded Numeric (0x0)
                        Identifier Numeric: 33
                    IncludeSubtypes: True
                    Node Class Mask: All (0x00000000)
                    Result Mask: All (0x0000003f)

BrowseResponse

Frame 23258: 112 bytes on wire (896 bits), 112 bytes captured (896 bits) on interface \Device\NPF_{37217669-42DA-4657-A55B-0D995D328250}, id 0
Raw packet data
Internet Protocol Version 4, Src: 100.70.98.88, Dst: 100.68.72.71
Transmission Control Protocol, Src Port: 62541, Dst Port: 59383, Seq: 8042, Ack: 11739, Len: 72
OpcUa Binary Protocol
    Message Type: MSG
    Chunk Type: F
    Message Size: 72
    SecureChannelId: 441419179
    Security Token Id: 3
    Sequence Number: 55
    RequestId: 57
    Message: Encodeable Object
        TypeId: ExpandedNodeId
        BrowseResponse
            ResponseHeader: ResponseHeader
            Results: Array of BrowseResult
                ArraySize: 1
                [0]: BrowseResult
                    StatusCode: 0x00000000 [Good]
                    ContinuationPoint: <MISSING>[OpcUa Null ByteString]
                    References: Array of ReferenceDescription
                        ArraySize: 0
            DiagnosticInfos: Array of DiagnosticInfo
                ArraySize: 0

Based on my logging, it's hitting the right address space:

Ok, so there are some "internal" reads that happen as part of browsing.

You should be able to breakpoint inside Milo classes if you're in an IDE.

Start here-ish: milo/opc-ua-sdk/sdk-server/src/main/java/org/eclipse/milo/opcua/sdk/server/servicesets/impl/helpers/BrowseHelper.java at ad47a4c64e658b91bdb12656d91c94216ef09f11 · eclipse-milo/milo · GitHub

Note here we read "browse attributes": milo/opc-ua-sdk/sdk-server/src/main/java/org/eclipse/milo/opcua/sdk/server/servicesets/impl/helpers/BrowseHelper.java at ad47a4c64e658b91bdb12656d91c94216ef09f11 · eclipse-milo/milo · GitHub

Make sure these are returning correctly: milo/opc-ua-sdk/sdk-server/src/main/java/org/eclipse/milo/opcua/sdk/server/servicesets/impl/helpers/BrowseHelper.java at ad47a4c64e658b91bdb12656d91c94216ef09f11 · eclipse-milo/milo · GitHub

1 Like

Found it! Looks like only forward references are included in BrowseResults.

I needed to change:

// Not working
references.add(
    Reference(
        nodeId(descriptor.oid.numeric),
        NodeIds.HasComponent,
        nodeId.expanded(),
        Reference.Direction.INVERSE,
    )
)

// Working
references.add(
    Reference(
        nodeId,
        NodeIds.HasComponent,
        nodeId(descriptor.oid.numeric).expanded(),
        Reference.Direction.FORWARD,
    )
)

I ended up changing to Organizes instead of HasComponent to fit my model better.

I was also able to prune a lot out of my browsable model implementation; it's basically just a ManagedAddressSpaceFragment with a single node :man_shrugging:
The main trick was to include all the references tracked in the NodeManager along with the dynamic references.

Working address space:

package com.mussonindustrial.embr.snmp.agents.opc

import com.mussonindustrial.embr.common.logging.getLogger
import com.mussonindustrial.embr.snmp.agents.devices.SnmpAgentDevice
import com.mussonindustrial.embr.snmp.opc.DeviceContextManagedAddressSpaceFragment
import com.mussonindustrial.embr.snmp.utils.removeAllNodes
import org.eclipse.milo.opcua.sdk.core.Reference
import org.eclipse.milo.opcua.sdk.server.*
import org.eclipse.milo.opcua.sdk.server.nodes.UaFolderNode
import org.eclipse.milo.opcua.stack.core.NodeIds
import org.eclipse.milo.opcua.stack.core.types.builtin.LocalizedText
import org.eclipse.milo.opcua.stack.core.types.builtin.NodeId
import org.eclipse.milo.opcua.stack.core.types.structured.ViewDescription

class BrowsableOidModelAddressSpace(val device: SnmpAgentDevice, composite: AddressSpaceComposite) :
    DeviceContextManagedAddressSpaceFragment(device.context.deviceContext, composite) {

    private val root = "OIDs"
    private val model = device.model
    private val logger = getLogger()
    
    init {
        lifecycleManager.addLifecycle(
            object : Lifecycle {
                override fun startup() {
                    addNodes()
                }

                override fun shutdown() {
                    nodeManager.removeAllNodes()
                }
            }
        )
    }

    fun addNodes() {
        val oidFolder =
            UaFolderNode(
                nodeContext,
                nodeId(root),
                qualifiedName(root),
                LocalizedText.english(root),
            )
        nodeManager.addNode(oidFolder)

        oidFolder.addReference(
            Reference(
                oidFolder.nodeId,
                NodeIds.Organizes,
                deviceNodeId.expanded(),
                Reference.Direction.INVERSE,
            )
        )
    }

    override fun browse(
        context: AddressSpace.BrowseContext,
        view: ViewDescription,
        nodeIds: List<NodeId>,
    ): List<AddressSpace.ReferenceResult> {

        val results = ArrayList<AddressSpace.ReferenceResult>()

        nodeIds.forEach { nodeId ->
            val references = ArrayList<Reference>()

            references.addAll(nodeManager.getReferences(nodeId))

            val path = nodeId.getPath()
            logger.info("Browsing path=${path}")

            model.getDescriptors(model.listOids()).forEach { descriptor ->
                logger.info("Browsed descriptor for OID: ${descriptor.oid.numeric}")
                references.add(
                    Reference(
                        nodeId,
                        NodeIds.Organizes,
                        nodeId(descriptor.oid.numeric).expanded(),
                        Reference.Direction.FORWARD,
                    )
                )
            }
            results.add(AddressSpace.ReferenceResult.of(references))
            logger.info("Browse results: ${references.map { it.targetNodeId }}")
        }

        return results
    }

    fun NodeId.getPath(): String {
        return device.stripDeviceName(this)
    }
}
1 Like

Okay, next question :grin:

How can I return browse results asynchronously or batch-wise?

I think another thing to be aware of when you're doing dynamic references, and might be related to why your inverse reference didn't work, is that there are two methods you implement for browsing: browse and gather.

You can't.

Is Gather an OPC UA concept or a Milo concept?
What’s the difference?

It's part of the way Milo implements browsing.

browse is "this NodeId belongs to you, give me all its references".

gather is "references for which NodeId is the source are being gathered from all address spaces, give me anything you may have to contribute".

Which is how you can add an inverse reference to a node you don't own, and browsing forward references from that node can still end up pointing to your node. In the "managed" address spaces this calls into nodeManager.getReferences, which will have created the inverse of the inverse reference you registered with it (i.e. a forward reference from some node you don't own) at the time you registered the inverse reference.

  @Override
  public synchronized void addReferences(Reference reference, NamespaceTable namespaceTable) {
    addReference(reference);

    reference.invert(namespaceTable).ifPresent(this::addReference);
  }

This all just works if you use UaNode instances and "managed" things, and the more you stray away from that the more you need to handle yourself.

1 Like

I suppose another way to think about it is that gather is what makes cross-AddressSpace browsing possible, especially when you don't "own" the other AddresSpace, because you can't reach into that other AddressSpace to actually add a forward reference to one of your nodes, so instead you get a chance to contribute that reference during gather.

2 Likes

Got it, that makes sense. Thanks!