Store and Forward engine for HTTP calls

Module Development
1

Hello,
I am developing a module that will monitor tags and publish their values into an HTTP API. Is the Store and Forward engine capable of buffering the tags data in case of the HTTP endpoint is down and then forward the data once is back? Or should I develop the buffer myself?
Any suggestions would be greatly appreciated.

2

This.

The store and forward system only works with databases and historians.

Theoretically, you could define a new "history flavor" that targets an API, but I suspect that would be dramatically more work than just making your own queue.

2 Likes
3

Thanks again for your prompt response.
Does anyone have any suggestions on how to tackle this? Should I read the tags and store in the Persistent Records until I'm able to send to the API and then delete them? Should I use a separate Database? In that case, I know I can create the database connection, but is it possible to auto create a Database, like SQLite for instance? I want to minimize the user configuration and also don't want to write to any of our clients database.
Last question, is a Dataset tag a better option to store the queue?

4

Does your queue need to survive a gateway restart? How long are you willing to tolerate the API being down and the queue growing?

If this is really a module as the topic suggests, then no, any kind of tag is the wrong choice. You'd have an implicit requirement that some tag of some kind just exists in some provider. It would be weird.

5

Thanks for your reply.
Yes, I want the queue to survive a restart. In terms of size I'd say at least a day but I will make it configurable.
The tag option, I was more considering the module creating the the Provider and some status tags including the queue like the MQTT Engine module does.

6

I am still not sure how to proceed with this task. I have been evaluating my options and I am leaning towards creating a Persistent record table as my queue but I couldn't find anywhere how to delete records from this table as I don't want this to grow indefinitely. Is this a good idea at all or should I consider other options?

7

Create your own SQLite instance (the driver will be on the general gateway classpath and should be usable from your gateway module hook without issue).

Don't try to shoe-horn things into our PersistentRecord system:

  1. It relies on a crusty, not that good in the first place ORM
  2. It's going away in 8.3
  3. What you're attempting to store is persistent data, but not configuration, which is what the config.idb PersistentRecords are stored in is for.

Point to a file in a well known directory (unfortunately, these are going to change from 8.1 -> 8.3) you can obtain from GatewayContext.getSystemManager().

Figure out your own serialization scheme such that you can store your persistent queue data into SQLite, with its limited set of possible types (numeric, text, binary, essentially).