Remote Tag Provider History Queries

Ignition
,
1

I'm working on a couple tasks related to historians and history queries, and have encountered some behaviors of remote tag providers (RTPs) that have me stymied:

  • There's no public way to retrieve the gateway name that a RTP is pointing at, nor any public way to retrieve the far end's real tag provider name.

(Reflection has exposed com.inductiveautomation.ignition.gateway.tags.distributed.client.RemoteTagProvider and com.inductiveautomation.metro.api.ServerId to me, and a number of private fields in the former that look promising, so all is not lost.)

  • The default behavior of RTPs when history is queried for paths of the form [rtpName]some/path is to pass the query to the remote end and make it do all of the work. (Not sure if the local end does any caching.)

  • An optional behavior of RTPs when history is queried, given appropriate configuration settings for the RTP, is to divert all queries to a local database connection's history provider using a full historical (qualified) path of the form:

histprov:localDB:/drv:remoteGWName:realTagProvider:/tag:some/path

This latter behavior allows a front-end gateway (presumably a UI gateway doing charting and data analysis tasks) to get history directly from the correct database, by having its own connection. Or, perhaps, having a connection to a live database replica that is co-located with the front-end gateway. Since Ignition's historian places source gateway and tag provider metadata into sqlt_* tables, a full historical tagpath constructed as above will deliver the same result as a query over the GAN, without the GAN traffic. And if via a co-located replica, without any extra WAN traffic to the source database at all.

(I strongly recommend that my clients using front-end/back-end configurations use database replicas, so that heavy query loads for UIs and data mining do not impact back-end history storage and other close-coupled productions tasks.)

But there's a doozy of a catch. If the source gateway uses multiple database connections for storing different kinds of history, or non-database history providers (3rd parties), you have to choose:

  • Use the GAN for all history, unconditionally, or

  • Use one local DB to handle one DB storage provider in that remote gateway's tag provider, breaking all of the others.

  • Or, if willing to dive under the hood and use reflection to extract the remote gateway name and real tag provider name, manually construct full historical tagpaths using each tag's historian properties. (If you do this, you can leave the RTP configured for the GAN, and only manually construct some local diversions--the ones you've created local connections for.)

That last approach also works for 3rd party historians, modulo possible alternate historical path syntax.

And that sets up my questions:

  • Has anyone else bumped into this and worked out any solutions?

  • Has IA noticed this and made a so-far private plan to mitigate? v8.3?

  • Am I missing any public methods for constructing the :/drv:gateway:realProvider fragment for target tags in a remote tag provider?

  • Has anyone subclassed IA's RTP in a java module to offer better options? (Perhaps a mapping for tags' storageProvider names to local history provider names? Perhaps with a fallback to GAN when a particular provider isn't mapped?)

Other suggestions welcome, of course.

3 Likes
2

Hmmm. I thought, for sure, this would nerd-snipe the usual suspects. :frowning_face:

2 Likes
3

While I can't help in this situation, I think I'm understanding what you're getting at....am I correct in thinking a TLDR of this would be:

You're trying to access historical data from a RTP but instead of passing the traffic over the GAN you want to query a local replicated copy of the historical data by constructing a historical path (although would this historical path need to be present on the local gateway to bypass the GAN at all?)

4

Yes, but in the case where the real tag provider uses multiple storage providers. RTPs provide the database mode option, but that only works if the real tag provider only uses a single storage provider.

History queries for the problem case can be diverted intelligently without that help from the RTP if scripts can access the remote gateway name and the real tag provider name in that remote gateway.

5

What we have done is have an enterprise gateway with Historical Databases setup for each remote location named appropriately.
Site1Hist
Site2Hist
Site3Hist
etc.

Then on the remote gateways we utilize the history splitter that stores it to both the local gateway for 6 months, and to the enterprise history store for x years.

The splitter has the Limit First Query option on the local gateway that we set to 6 months, so if the local users request data past the 6 month prune, it will then redirect out to the enterprise history provider to get that data.

Then on the enterprise side we have custom interfaces developed both in webDev and Perspective to query against the Enterprise History stores.

Most of our GAN traffic is just incoming history records to the enterprise store.
Currently have 58 gateways tied into it.

2 Likes
6

Sounds like you aren't using the database diversion option in your front end gateway, but manually constructing qualified paths that hit the local DB connections. How are you getting the remote gateway names? Configuration constants somewhere?

7

Yes exactly that.
We use the tooltip fields from the tags for a user friendly display name of the tags.
So we build database tables of all the tags for each location that includes the tag paths and other relevant information. Scheduled and triggered scripting to refresh the database when needed.

We then use either tree views or table views of the tags for the users to select from.

2 Likes
8

This is far enough out (and deep enough in the weeds) that I think both Kevin and I don't have good answers. I tagged in the team that's in charge of the overall tag system, we'll see if anyone takes the bait.

1 Like
9

One of the things that we have done to get around this is to set up a GatewayMessageHandler which essentially reads "[System]Gateway/SystemName" and returns the result.

We have also named our RTP with the DNS name of the server (maybe a bit of a security risk but it works).

Since, Historical Paths are basically QualifiedPaths, you can use:

com.inductiveautomation.ignition.common.QualifiedPath.Builder to construct a Historical Path.

You can use the helper functions in QualifiedPathUtils() to go back and forth to TapPaths and Strings, though not directly to a HistoricalTagPath object.

With a GatewayContext you can get a reference to the GatewayTagManager or GatewayTagHistoryManager which has a getTagProviders() or getTagHistoryProviders() function, though I'm not sure that is what you would consider "public".

10

But that doesn't help for remote gateways. You need that name to send them a message to get that name?

Yeah, no problem with that, or its utils. Well familiar with all of the various tag path and qualified path manipulations.

Yes, those are documented in the SDK API, so I consider them public, and I use my Integration Toolkit's system.util.toolkitCtx() to not have to trust an undocumented import. That's how I got to the RTP objects that I reflected.

Also the DNS name of a server is not, typically, the name of the gateway itself, and it is the latter that is needed in the :/drv: section of a qualified path.

11

The "remoteServer" parameter of system.util.sendRequest is optional, if not provided then it broadcast the message across the GAN, all you need is the project name and the messageHandler name.

Yes, what I meant was they are all purposefully the same.

12

Say what? sendRequest waits for a reply. What happens to all the other replies?

13

Good question. I've never needed to try it with multiple remote servers having exactly the same message handlers. It is just how we have done it, so I thought it was worth mentioning.

Nope, turns out, I'm wrong.

When I tested this, the messageHandler was also on the local gateway, which is why it worked. It definately doesn't work across the GAN.

ERRRR... Sorry Phil

2 Likes
14

So, could you use that message handler though with the remoteServer to trigger a messageHandler on the remote server to collect all necessary local data and send it back to the requesting server?

Somewhat of a handshake, but ServerA sends a message to ServerB with it's own name, and some payload of what values it needs from the tag provider config (is this available via local methods?) or a fixed set of parameters to return, then sends those back to ServerA with another message handler to record those settings for future use?

15

One of the items needed in a script is the backend gateway name associated with a particular RTP on the frontend side. I don't see how any possible messaging will get that for you. (You would already have that gateway name if you could send a message to it.)

16

Don't you somehow need to establish 2-way communications in the first place when setting up the GAN, so you would have to be on the remote server when setting it up and just document it at that time? Then once you get that 2-way comms going with message handlers, could automate collecting the data you need from the remote side.

17

That's precisely what I'm trying to avoid. Having to document something that the system already knows, for use in other scripts. Which risks documentation getting out of sync and the scripts making errors.

18

Yeah, I get that part. I thought your end goal in all of this was bypassing the remote history provider and if you would just need to "document" a list/dataset of servers then if you could set up messaging between them to dynamically pull the rest of the info you need so that if anything is changed later, it's not statically documented and could self-update, but if a static gateway name is all that's needed, it's minimal compared to having to document it all.

Even just being able to get the gateway name would probably help you out tremendously.

19

As noted in my OP, the two key pieces of information are

  • Remote gateway name.
  • Real tag provider name in that gateway.

An RTP has both of these in its non-public fields. I can't find any public way to grab those two pieces.

A third key piece of information, not otherwise automatically available, is the mapping from multiple storage provider names in the remote gateway to storage provider names in the local gateway. An RTP in database mode provides a mapping for one storage provider.

20

I haven't tested anything, but would this URL give you item #1 of remote gateway name?

http://<GatewayIP>:8088/system/gwinfo