Creating tag groups with the openapi endpoints in the gateway?

[epaulson]

I am brand-new to Ignition (I set up my first gateway last week) so I apologize if this is a dumb question, but is it possible to create and assign tag groups through the API the gateway exposes (which I think is new in 8.3?)

My use case is similar to what was discussed about a year ago here: Ignition Scripting of new Devices without Designer - #8 by Kevin.Herron - I want to be able to add Modbus devices to a gateway programmatically, and it looks like I can get a lot of the way there with the new API - there’s a ‘Create Devices’ endpoint that looks like it takes enough configuration that I can set the ModbusTCP device and port and other configuration information with just a POST call, and then even better, I think I can upload an ‘address-mapping.txt’ for it and get registers mapped, and then they’ll hopefully show up in the internal OPC-UA server for Ignition, which is awesome. (I haven’t tried this yet, a fully automated run is tomorrow morning’s project)

However, in just experimenting with it on the Gateway only, it looks like the default Modbus polling is 1 second, and the way to fix that is with Tag groups and adjusting the polling setting on the group, but in poking around in the OpenAPI documentation page on the gateway I can’t figure out how/if I can create a new tag group in the gateway and assign my newly mapped registers to that tag group.

I would like to avoid the Designer if possible, in part because at scale pointing and clicking is not ideal, and just more practically for the moment I’m experimenting on my Macbook with an ARM Ubuntu VM in UTM, and the Gateway works great, but Designer doesn’t run here.

[paul-griffith]

@epaulson

is it possible to create and assign tag groups through the API the gateway exposes

No, not through the rest API nor directly via scripting at present. It's very poorly documented at present because it's brand new to 8.3, but there is another possible solution to your issue; more on that later.

@epaulson

(which I think is new in 8.3?)

Correct, the entire many file based configuration storage is new to 8.3; prior versions used a SQLite "internal database" to store all configuration and we did not support editing that database directly.

@epaulson

then even better, I think I can upload an ‘address-mapping.txt’ for it and get registers mapped

For posterity, while you can do this, the address mapping feature is generally recommended against as an antipattern. While the ability to "know" which addresses you have when configuring tags later is nice, the added indirection only serves to make things more confusing when you're debugging. Further, it's a non-standard item that's Ignition specific instead of a more direct mapping of holding registers/etc that users of other SCADA/PLC software will probably find more readable.

@epaulson

it looks like the default Modbus polling is 1 second, and the way to fix that is with Tag groups and adjusting the polling setting on the group

There's an important distinction to pull out of this statement that will help you understand things.
When you create a device connection in Ignition, ~nothing happens other than a device connection being made. Devices (mostly; some have different protocols that require this) do not poll by themselves. It is the act of creating an OPC UA subscription or read, usually by creating an OPC tag, that actually tells Ignition's built in OPC UA server to issue a read call to a device.
That is, you can create a Modbus device connection, and whether populated with an address map or not, nothing is being read from your device until you create Ignition tags that consume those OPC UA addresses. It is the tags you create that use the tag group settings, and that in turn means that your Modbus device is being polled at whatever your tag group rates are, but until those tags are made nothing is being polled.

In other words: you cannot assign registers to a tag group.
You can give particular tags a tag group, and those tags in turn may be reading registers, which may sound like a distinction without a difference, but understanding that fundamental fact now will make things easier to understand down the road.

@epaulson

I would like to avoid the Designer if possible, in part because at scale pointing and clicking is not ideal

This isn't the worst goal, but part of using Ignition is going to be using the designer, and there's no reason to hamstring yourself now. In particular, "at scale" you're going to have a much easier time creating UDTs to make setup en-masse easier and less error prone, and there are tools like the multi-instance wizard built in to the Designer that just don't work without a GUI.

@epaulson

Macbook with an ARM Ubuntu VM in UTM, and the Gateway works great, but Designer doesn’t run here.

Run the gateway virtualized, but run the designer launcher "locally" on your Macbook.
For one, performance is going to be much better.
For another, that's how you should run things in production - you shouldn't run the designer on the local environment where Ignition is installed.
For a third thing, that's how you're going to end up running things when you're connected to 'real' systems - any number of gateways you connect to are going to end up being physically remote from your laptop so you're going to have to run the designer launcher somewhere to get connected.

If you're worried about it, while I'm obviously not a neutral party, Ignition's installed applications are good citizens and won't cause a bunch of annoyances across your system if they're installed 'natively' - you can happily install and uninstall them as regular OS applications at will.

---

All that said, like I said at the top it's not currently possible to create tag groups dynamically via the REST API or scripting.
But I'll posit that truly dynamic creation is probably not what you really need, and you could probably get by with creating a set of 'standard' tag groups for all your systems. The way to do that (immutably) in 8.3 is via the external resource collection.
In the installation directory, in the data/ folder, you'll see a series of folders in this directory:
data/config/resources/core/ignition/tag-group/
Each of those folders will correspond 1:1 with the 'realtime tag providers' defined on your system (probably just one named default/).
In that folder you'll see a folder for each tag group you have defined; by default Default/ and Default Historical/.
In each of these folders will be two files, config.json and resource.json. You'll notice config.json has all the settings for any particular tag group.

If you go up a few levels you'll also see a
data/config/resources/external/ folder.
Any Ignition resources you create in that external/ folder are guaranteed to be loaded by Ignition, and guaranteed not to be written back to by Ignition - they become immutable system configuration delivered by the filesystem. So if you're setting up a hundred gateways, and you've created Ignition resources in the external/ directory of each, you can guarantee those resources are available (and cannot be removed).
Also note that you can just create resources via the filesystem in core/ if you want "regular" resources that can be modified/removed via the designer.

"Creating" an Ignition resource via the filesystem is easy - just duplicate the folder structure, so that e.g.
data/config/resources/core/ignition/tag-group/default/Default/
Moves to
data/config/resources/external/ignition/tag-group/default/Default/

At that point, you can either hit the 'scan filesystem' button or restart your gateway, and we'll pick up those new resources.

Some more context on the way Ignition "resources" are built up via disparate files:

Ignition internal Folder Structure

I thought we had some kind of reference page for this in the manual, but all I found was this, which is not quite the full story: This old post of mine goes into more detail, though is a bit out of date: Basically: projects/ has an ignorable .resources/ folder. All other folders in projects/ will be scanned - if they contain a valid project.json manifest, then they make up an Ignition project. The name of the project will be the literal name of the folder on disk. Below each project direc…

[epaulson]

Paul - thank you very much for the detailed response, it helped me better understand some things (and showed me some things that I am still confused about)

@paul-griffith

@epaulson

then even better, I think I can upload an ‘address-mapping.txt’ for it and get registers mapped

For posterity, while you can do this, the address mapping feature is generally recommended against as an antipattern. While the ability to "know" which addresses you have when configuring tags later is nice, the added indirection only serves to make things more confusing when you're debugging. Further, it's a non-standard item that's Ignition specific instead of a more direct mapping of holding registers/etc that users of other SCADA/PLC software will probably find more readable.

Thanks, that’s good to know - however, I do think there’s some magic happening in the address-mapping.txt with Modbus that I don’t quite understand. I spun up an Azure x86 Linux VM so I could launch Designer, and set up my first simulated Modbus device using the address mapping feature. (My simulated device is just updating a few holding registers every 2 seconds with the current number of seconds since the minute rolled over, plus an offset so each register has a different value). With the registers I mapped with address mapping, just using a prefix of V_, I can see the values change in the Designer tag value every two seconds as I expected. Since the Ignition Quick Connect OPC client is not in 8.3, I downloaded opcua-commander for Node to quickly take a look at the values as well, and I see them updating from outside of the Ignition ecosystem.

However, when I run a second copy of my simulator on a different port, I can use Ignition Gateway to connect to that device successfully, and I can manually create a tag in Designer using the [DeviceName2]HR# syntax in Designer for OpcItemPath and see the correct value being updated every two seconds in Designer (with a different offset so I know it’s from the 2nd device) - however, I never see this new tag in opcua-commander. I’ve tried Saving the project in Designer, selecting the ‘Update Project’ item in the Designer menu, restarting opcua-commander, even restarting the Gateway, and it never shows up. I’ve tried setting the OpcItemPath to be ns=1;s=[DeviceName2]HR# syntax to exactly match what shows up from the V_# registers in the tag browser for their OpcItemPath, that doesn’t make a different either in opcua-commander - though I still see the values updating in Designer.

So somewhere there’s something else happening between what’s happening with a Modbus address mapping and creating a tag manually and I’m just not sure what - there’s a lot of “tags” in Ignition and so I might be missing a step or a concept, or how tags go from Designer back into the tree in the OPCUA server exposed from the Gateway. I’m not sure where the ‘UnitId 0’ folder that shows up in opcua-commander looking at the Gateway’s OPCUA server under my first device that was created with address mapping comes from - it doesn’t show up in the Tag Browser in Designer as a folder.

I don’t mind using Designer to create types as necessary and copying those around, but my goal, like the poster from a year ago, is to use Ignition to bridge from protocols like Modbus over to OPC-UA and then pull data out of OPC-UA, so most of Designer (building UIs and such) is not that relevant to me, hence my desire to avoid it and program right against the Gateway.

I think I have a few more questions but I feel like I’m missing something between tags in the Designer and tags in the Gateway OPCUA server so I’m going to hold off on the rest of your response for now. Thank you again for your detailed first response, I feel like I’m getting closer!

[epaulson]

I did figure out why my new hand-created Modbus tags were updating in Designer but were not visible in the external opcua-commander tool: I did not have the ‘Expose tag providers’ enabled on the Ignition OPC-UA server, turning that on made them show up.

I’m a little bit bummed that I don’t seem to have any control over how/where tags show up in the Ignition OPA-UA server. If I use Modbus address mapping, direct in the Modbus driver on the Gateway, I get my data exposed under the Devices tree in OPC-UA when looking at the Ignition server from a 3rd party client. If I create tags for the Modbus registers in Designer in a Tag provider, then when I look at the Ignition OPC-UA server the tags show up under the Tag Provider tree, and not the Device part of the tree. It’s not the end of the world but it would be nice to have more control over how the Ignition OPC-UA server exposes its data.

A few more questions:

• There is a new API in 8.3 to create devices, which is nice - but it does not look like there’s an API to create tags? I can create them via scripting so it doesn’t really matter too much, and using Designer to add the scripts is workable.

• I can run arbitrary scripts by just embedding them into a WebDev endpoint - is there a REST API in 8.3 that is equally generic and is just “run this script right now?” I don’t think this is important since I can just put whatever script I want into a WebDev endpoint, it’s more curiosity.

• Is there an API to add a script/WebDev endpoint? Using Designer (or the file system out-of–band) will work for me, again I’m just wondering.

• With WebDev, it does look like virtually everything the built-in 8.3 API does was possible before with the scripting interface. This is certainly not meant to say “the new 8.3 API isn’t cool” and it’s nice to have a lot of functionality just available automatically without having to write a script and create a WebDev setup to expose it, but I’m wondering if there are things the new API does that weren’t possible in the previous scripting/WebDev world?

• Finally, it looks like maybe a lot of what I want to do (setup Gateways to monitor Modbus devices and expose that data through the Ignition gateway OPA-UA server) is a lot of creating and configuring Devices and Tags (and Tag groups for some metadata) - for deployment, I was going to probably use Docker containers. If I’m just creating devices/tags/tag groups for gateways, can I just do this all through the filesystem that I mount inside the containers, and manage the Ignition configuration files and deployments outside using whatever devops/deployment/config tooling makes sense?

[paul-griffith]

@epaulson

It’s not the end of the world but it would be nice to have more control over how the Ignition OPC-UA server exposes its data.

:shrug: Ultimately the OPC UA node path to use is a property of each particular driver.

@epaulson

There is a new API in 8.3 to create devices, which is nice - but it does not look like there’s an API to create tags? I can create them via scripting so it doesn’t really matter too much, and using Designer to add the scripts is workable.

Correct, for now. In 8.3.2, there's going to be new REST API endpoints that work essentially the same as the import/export tags scripting functions - not quite the same as fully generic CRUD, but possibly good enough in the short term.

@epaulson

I can run arbitrary scripts by just embedding them into a WebDev endpoint - is there a REST API in 8.3 that is equally generic and is just “run this script right now?” I don’t think this is important since I can just put whatever script I want into a WebDev endpoint, it’s more curiosity.

No (by design). For one, it would undercut the product offering of Webdev. For another (more substantial) it would be a colossal security risk. If you really want to do this, you can do fun (safe) generic scripting endpoint stuff using the getattr Python builtin. You'll quickly run into issues with argument coercion, though.

@epaulson

Is there an API to add a script/WebDev endpoint? Using Designer (or the file system out-of–band) will work for me, again I’m just wondering.

No, there's currently no supported means to add project level resources, by design. We may still end up changing that over time, but for now knowing that legitimate edits came in from the designer (or arbitrarily via the filesystem) makes things easier on us.

@epaulson

With WebDev, it does look like virtually everything the built-in 8.3 API does was possible before with the scripting interface. This is certainly not meant to say “the new 8.3 API isn’t cool” and it’s nice to have a lot of functionality just available automatically without having to write a script and create a WebDev setup to expose it, but I’m wondering if there are things the new API does that weren’t possible in the previous scripting/WebDev world?

In your particular case for devices, yes, the capability set is roughly the same.
What's you're perhaps not appreciating is the much larger portion of the rest of gateway configuration (which you happen to not need) where the only way to do things previously was via undocumented, unsupported scripting hacks - there was never a scripting function for every single piece of gateway config you could previously only C/R/U/D via the web interface.

@epaulson

If I’m just creating devices/tags/tag groups for gateways, can I just do this all through the filesystem that I mount inside the containers, and manage the Ignition configuration files and deployments outside using whatever devops/deployment/config tooling makes sense?

In a word, yes, you absolutely can (and likely should) "layer" configuration from different layers into your deployed setups using deployment modes. Taking advantage of external vs core (vs dev/test/prod/whatever else you want) is a huge part of the advantage of 8.3 that, because it's so bleeding edge, there's a lot less guidance about.

[pturmel]

@epaulson

were updating in Designer but were not visible

If your external OPC client allows you to enter addresses directly, instead of browsing, you can get those same manual addresses externally. Without an address map.

You seem to be hung up on browsing for OPC items. Some protocols expose information about available tags/addresses, like most Rockwell protocols. Others, like Modbus, do not. You have to know what addresses you want, and what you expect to find there.

IA's driver's address mapping is a convenience for browsing, that is not needed for actual data access.

If you'd like to browse Modbus addresses without having to use address mapping, consider trying my alternate driver. That automatic browsability is one of several advanced features.

[epaulson]

@paul-griffith

@epaulson

If I’m just creating devices/tags/tag groups for gateways, can I just do this all through the filesystem that I mount inside the containers, and manage the Ignition configuration files and deployments outside using whatever devops/deployment/config tooling makes sense?

In a word, yes, you absolutely can (and likely should) "layer" configuration from different layers into your deployed setups using deployment modes. Taking advantage of external vs core (vs dev/test/prod/whatever else you want) is a huge part of the advantage of 8.3 that, because it's so bleeding edge, there's a lot less guidance about.

Thanks for this entire message, it was very helpful. Because I’m brand-new to Ignition and I’m landing right after 8.3 came out, I lack the perspective of not knowing what wasn’t always available so as I read through things and think to myself “well why would you do this any other way” it’s not always apparent to me that the answer is “because that way didn’t exist until a hot minute ago”

@pturmel

If your external OPC client allows you to enter addresses directly, instead of browsing, you can get those same manual addresses externally. Without an address map.

You seem to be hung up on browsing for OPC items. Some protocols expose information about available tags/addresses, like most Rockwell protocols. Others, like Modbus, do not. You have to know what addresses you want, and what you expect to find there.

I don’t think I’d say I’m hung up on browsing - more to the point, how I see using Ignition is all of my users are software devs downstream from Ignition and the only way they interact with it is through writing integration code that speaks OPC-UA , so I care - mostly for aesthetic reasons - what the presentation of the OPC-UA tree the Ignition server is handing out looks like. Ignition is abstracting away Modbus, so the Modbus protocol lack of browsing isn’t a big deal, because I’d map whatever I cared about ahead of time and make it available through OPC-UA. It’s just that if you look at an Ignition server through an OPC-UA client, and don’t know anything about Ignition, it’s a little strange that you can see a tree of Devices, and for some of them you can see data points, but for others if you want to find the data from that device you go looking elsewhere in the OPC-UA tree. Again, not a big deal, just wondered if I could control what that OPC-UA tree looked like.

I have looked a bit at your advanced driver and it does some helpful things, I will look at it more deeply as I explore more of Ignition.

[Kevin.Herron]

@epaulson

and for some of them you can see data points, but for others if you want to find the data from that device you go looking elsewhere in the OPC-UA tree.

It's not that you're looking elsewhere exactly.

The Exposed Tag OPC UA Nodes are different Nodes than the ones from e.g. the Modbus driver that are not available when browsing the address space but are available when dynamically addressed.

Exposed Tags expose all Ignition tags in the OPC UA address space. This is separate from, and unrelated, to the fact that Ignition also has an OPC UA server with drivers, except that it's common for many of your Ignition OPC tags to come from the Ignition OPC UA server/drivers.

Some drivers just don't expose their Nodes for browsing (Modbus). Ignition's OPC UA server and drivers are primarily there to serve Ignition as the OPC UA client, not to compete as some kind of Kepware-ish product.

In most cases external systems are going to be interested in the Ignition tags exposed in the OPC UA address space and less interested in the devices, but maybe your case is different.

[pturmel]

@epaulson

Ignition is abstracting away Modbus

Unfortunately, no. The OPC server in Ignition isn't magically adding structure to what its Modbus driver presents. I wouldn't call it an abstraction.

It is also important to understand that in OPC UA, browsing for nodes (the "exposed" tree) is not guaranteed to deliver all possible node addresses. None of the Ignition drivers do so, though most will expose the information available to auto-discovery (within the target device's protocol). My alternate driver simply exposes all possible addresses within the Modbus protocol for a default node, where IA's driver exposes nothing (without config). Many legacy PLC protocols simply do not offer enumeration of the PLC content. At all.

[epaulson]

@paul-griffith

In a word, yes, you absolutely can (and likely should) "layer" configuration from different layers into your deployed setups using deployment modes. Taking advantage of external vs core (vs dev/test/prod/whatever else you want) is a huge part of the advantage of 8.3 that, because it's so bleeding edge, there's a lot less guidance about.

I’ve been going down this route and have been having good luck, but I’m stuck on creating tags via the filesystem - I think I’m missing a file or a permission somewhere, but I’m not sure. I have a working Ignition setup with the Docker image, bind-mounting external/ into the container and have adjusted the server-config and created a Modbus device by editing files there, and that’s working OK.

I have a Modbus process that Ignition is able to read from - I can directly address it via the built-in OPC-UA server with a little util using the node-opcua toolkit, this works and I see my holding register value change:

node object-examine.js "ns=1;s=[TestDev1]HR18"

(the modbus “server” a slightly hacked up version of server_updating.py from pymodbus, it has 6 holding registers that I update a value in every 2 seconds so it’s easy to see if something’s changing)

I have "exposedTagsEnabled": true in my server-config/config.json - the external directory is bind-mounted into the Ignition gateway container like so with docker-compose:

volumes:
- ./volumes/gw-external:/usr/local/bin/ignition/data/config/resources/external

I started with a VM with Ignition installed without Docker, so a regular install and the Modbus device added through the gateway’s web interface, the server configured through the web interface, and tags created with Designer. I tried to copy the files from tag-definition/ over the working setup to another VM that I am configuring entirely through the filesystem, and it doesn’t come out quite the same. Here’s what my files look like inside my container:

ignition@6623a59970e0:/usr/local/bin/ignition/data/config/resources/external/ignition/tag-definition/default$ ls
tags.json  TestFolder1	unary-resource.json
ignition@6623a59970e0:/usr/local/bin/ignition/data/config/resources/external/ignition/tag-definition/default$ cat unary-resource.json 
{
  "scope": "G",
  "version": 1,
  "restricted": false,
  "overridable": true,
  "files": [
    "tags.json"
  ],
  "attributes": {
    "config": {}
  }
}

ignition@6623a59970e0:/usr/local/bin/ignition/data/config/resources/external/ignition/tag-definition/default$ cat tags.json
[
  {
    "valueSource": "opc",
    "opcItemPath": "[TestDev1]HR18",
    "dataType": "Int2",
    "name": "NativeTagTestTag1",
    "tagType": "AtomicTag",
    "opcServer": "Ignition OPC UA Server"
  }
]
ignition@6623a59970e0:/usr/local/bin/ignition/data/config/resources/external/ignition/tag-definition/default$ cat TestFolder1/tags.json 
[
  {
    "valueSource": "opc",
    "opcItemPath": "ns\u003d1;s\u003d[TestDev1]HR21",
    "dataType": "Int2",
    "name": "IgntionTagsTestTag2",
    "tagType": "AtomicTag",
    "opcServer": "Ignition OPC UA Server"
  }
]
ignition@6623a59970e0:/usr/local/bin/ignition/data/config/resources/external/ignition/tag-definition/default$ cat TestFolder1/unary-resource.json 
{
  "scope": "G",
  "version": 1,
  "restricted": false,
  "overridable": true,
  "files": [
    "tags.json"
  ],
  "attributes": {
    "config": {}
  }
}
ignition@6623a59970e0:/usr/local/bin/ignition/data/config/resources/external/ignition/tag-definition/default$ 

But I am having no luck with getting these exported with updating values - I should see something like this, which is from my configured-with-the-web-interface VM:

But what I get on my setup where I created the tags via the filesystem I am missing the first tag entirely, and the second tag (in the folder) does show up but the OPCUA variable object has no Value property:

So I’m just not quite sure what to try - I suspect I’m missing something simple, like I need to copy another few files over, like maybe the tag-provider tree (I’m assuming - but maybe I’m wrong - that if I don’t have tag provider definitions in overlaid in external/ then the gateway just uses what it has in core/ ). I’ve tried different OPC paths -one with the ns=1;s= prefix stuff and one without, and that doesn’t seem to matter on either my working setup or my not-working setup. Or maybe there’s a permission I clicked on when setting up the VM with the web interface that I’ve missed copying over to the external/ tree on my Docker test, but I’ve been scouring all of the settings and I just don’t see anything.

I think I am close, the file system feels like exactly what I wanted to do, and the IA team did a nice job with the Docker image and documentation!

[epaulson]

@epaulson

So I’m just not quite sure what to try - I suspect I’m missing something simple

So sometimes what you need is to walk away from the problem and get a good night’s sleep, and this morning when looking at it again, I finally noticed in the ‘Platform Overview’ that I had an unhealthy OPC Connection - digging into it, the default ‘Loopback’ connection to the internal OPC-UA server was faulted with a status of bad_useraccessdenied and seemingly couldn’t connect to itself, which would of course explain what was going on.

I have no idea what I did, and the whole fun of containers is that it’s easy to start over, so I started with a fresh container and using a named volume for data/ and configured things only using the web interface to get my Modbus device attached again, and then went in and created the tags by editing files in the named volume while the container was shut down, and that worked - my new Ignition tag is refreshing with the latest data from the Modbus driver.

Sorry for anyone who read through my very long previous message. I am going to leave it up just in case someone else ever gets a “why do I have an ignition tag showing up in Tag Providers but it has no value” so they can see to check that OPCUA loopback connection health.

I will now go through and see if I can find a working configuration with a bind mount for /external, which is the end goal so I can track my config changes and tags easier in git.