Potential pitfalls with a Hub & Spoke architecture with multiple independent hubs using a shared spoke

[Bjornar_Alvestad]

I am using terminology from more usual Hub & Spokes architectures in an attempt describe what I want to achieve. However, this use-case may be a bit odd, so the terms I am using may be inaccurate. Apologies in advance for that.

Summary

We have multiple independent systems using Ignition (called hub in description below) which will communicate to a shared Ignition sub-system (called spoke in description below). The spoke system acts as a shared system the hubs interact with. It is not collecting data from the individual hubs.

Any potential pitfalls with an architecture like this?
Any recommendations for how to show alarm journal for the spoke onboard the hubs?

More detailed system description and questions below.

Detailed description and questions

Description of our system

We have a use case where we end up having some spoke Ignition gateways that multiple other hub gateways connect to, but where these hubs are completely independent systems.

This spoke system will act as a sub-system to the different hubs. The individual hubs can go in and out of communication range with the spoke system. The hubs will use the spokes as a remote tag provider.

Multiple hubs can talk to the shared spoke system at the same time, but only one hub is "in control" of the spoke system at a time.

The security of the spoke system is considered as weaker than the hub systems.

We want to be able to view alarms and alarm journal for the spoke system on the hub systems, and to be able to acknowledge the spoke systems alarms from there. We will also use the tags to show the perspective views of the spoke system on the hub systems.

Currently we are going for full Ignition on the shared spoke, but we may end up changing that to Edge later on, if that can cover our needs.

Figure above shows three independent hub systems, and two spoke systems they communicate with via WiFi. The servers in each hub system has the same IP, so it is used masquerading on the network going out on WiFi.

Pitfalls?

Is there any known pitfalls with this architecture, or is this something that is already in use by others?

I have done some initial testing using full Ignition version for the spoke system, and it seems to work fine so far.

Alarm journal

We need be able to see view the alarm journal for the spoke system at the hub systems. Here I see two possible options.

1. Remote query-only DB connection from the hub to the spoke.

2. Add additional Journal on each hub system, and have the spoke connect to this as a remote journal. The Store and Forward system on the spoke system will then ensure that each of the hub systems get their copy of the spoke systems journal updated when they are back in communication range.

Option 1 at first glance seems like the cleaner option, but there are some concerns that Ignition would implicitly trust a DB connection as safe, and therefore have less protections in place than if we go indirectly via e.g. the Journal system instead.

Any thoughts regarding potential security concurs with Option 1?

Option 2 has been tested and seems to work fine. But it seems a bit round about having to try and make a copy of spoke systems alarm list on the hub systems just using store and forward.

Potential minor disadvantage I see here is we are keeping multiple copies of the alarm journal for the spoke system, and we may run into situation where the different hub systems have a different journal to represent the spoke system.

Potential advantage could be that network usage would be less when browsing alarm journal (not tested how much data it actually is yet), and also the possibility to view journal when outside communication range.

Is there a third option where we could e.g. just connect a journal in the hub system to the journal system on the spoke?

Keep in mind that the spoke system is considered less secure (and less important), and we do not want this to become a potential attack vector for the hub systems.

[pturmel]

You've broken the common use of the terms "hub" and "spoke". It's just a mesh. It also isn't clear why you can't have just one gateway pair, period, since you appear to be on a LAN with some WiFi.

Or, if you really do need all of those, consider adding one more pair to be the "coordinator" gateway, that manages permissions and source of control. Then all of the other gateways can point at that pair as the actual hub.

That may be extra important if you try to use Edge, as I think Sync Services only points at one upstream gateway.

[Bjornar_Alvestad]

Thank you for your quick reply. I will try to clear up a few of the confusion points here.

What I have called independent hubs/systems are individual vessels.

These are each their own system, and have no communication with each other. Nor any communication to a "coordinator".

The drawing does not show a mesh network, but I see that it could be confusing. The drawing was just meant to show that the different vessels may have a network connection to the same shore station at the same time.

There is no communication between the different vessels. They are each their own independent system. The shore stations only makes its tags, alarms and history available to the vessels. It does not have any more access to them than that.

The shared "spokes" are shore station which the a vessel can dock at to charge.

There is normally not people at the shore station, except during service, and the process is semi automatic.

The control of the shore station goes via PLC onboard vessel to PLC on shore station.

The reason for wanting the Ignition system onboard to also connect to the Ignition system on the shore station is to see alarms for the store station itself, status on it and alarm history, without having to pass this via the PLCs. So the PLCs only need to transfer the data that they actually need to get the job done.

Note that the drawing and system I describe here is just a simplified system setup, and omits a lot of the system that I did not consider relevant for the discussion (e.g. full network setup).

[pturmel]

Ok. I would not use Ignition's gateway network for temporary connections. Full stop.

Put the WebDev module on the shore stations and design an API, with desired security, to expose what you need to any temporarily connected vessel.

[Bjornar_Alvestad]

Thank you once again for your reply. But could you please be a bit more concrete regarding what specifically is the issue with the Gateway Network? And what would we gain by using WebDev module instead of just using what the Gateway Network provides? And conversely what would we loose by not doing so?

E.g. getting alarms transferred, and acknowledging alarms is very easy via the Gateway Network. Also you get the tags nicely mirrored, and can easily set access levels on them via Service Security.
With the WebDev is that equally easy to achieve? If not, then what do we gain which would justify the additional development time spent to be able to achieve this?

It could also be more helpful for us if you instead of giving a go/no go answer without any reasoning, if you instead would challenge us with questions like:
What happens if this or that... , Have you considered this..., Would/could this be a simpler/better setup for your use case? Etc.

I also noticed you used the term "temporary connection". Just to make sure we are on the same page here:
In what way do you consider it a "temporary connection"? Is it just due to the network connection may drop for a little while if it goes our of range? Or did you mean something more with this term?

[plarochelle]

Have you considered using an MQTT broker as a central repository? All other nodes would just subscribe to the data being published from the original source.

[PGriffith]

@Bjornar_Alvestad

In what way do you consider it a "temporary connection"? Is it just due to the network connection may drop for a little while if it goes our of range? Or did you mean something more with this term?

As you described it, the gateway network will only be connected when the vehicles are at the shore station. Presumably, as vessels, they spend a statistically significant amount of time not at shore, and out of network communication with your shore station.

Ignition's gateway network really isn't designed for 'transitory' or even 'periodic' connections. It's structured around persistent, stable connections, ideally in datacenters; the majority of communication happens over a persistent websocket.

Phil's suggesting the use of the WebDev module, specifically a "Python endpoint", for you to program the exact pieces of data pushing and pulling you need to do. You would use Webdev to create a web API (on the shore station, probably), and whenever the vessel gets in range/knows it's at shore, it would then use system.net.httpClient to push or pull whatever data it needs to/from the shore station.

This is (a lot) more engineering effort. But, you have absolute control over the network payloads in both directions, you're not (ab)using the gateway network mechanism in a way that it's not really intended to be used, and you're able to make guarantees about the system that you otherwise couldn't.

All that said, my only official recommendation as an IA employee would be to talk to our sales engineering department. You're absolutely not the first customer with onshore/offshore, transitory connection requirements, and they're going to have directly useful advice.

[pturmel]

Sorry, was traveling today. But, what Paul said.

Anyways, if you configure the WiFi+router infra at each shore station to resolve a particular http://something.local/ url to your WebDev API, the vessels no longer need to care at connection time which shore station it is. Identities and further security can be established within the API, instead of relying on the gateway network to "figure it out" (which it probably won't).

[Bjornar_Alvestad]

Thank you very much for all your answers, suggestions with WebDev module and MQTT, and for the clarification on what the gateway network is designed for.

We are currently considering multiple different options both on software and hardware side, and one of them is also using 5G/4G (via VPN) instead of WiFi so that we can manage to keep connection with the shore station for a greater/full range. But there could of course still be situations where the vessel loose connection (e.g. away for service etc.).

If the gateway network were still used despite potential for unstable connection, and connection was lost for a period of time. Could this affect the vessels system (if so, in what way), or would any potential consequences of this only be for the shore station system / loss of functionality for that part on the vessel?
Understanding this for us would also be useful with regards to the more normal edge panels, since also these are connected via gateway network. And while under normal conditions it should not loose network connection, it could, and it would be good to know potential consequences of failure. My past impression was that this (for edge) could loose connection for up to 7 days without loosing history.

I will also take your suggestion regarding contacting sales to get some discussion going via that channel as well.