Redundant MQTT setup

We use a primary and backup (warm) Ignition server to capture production data. Because data is of critical importance to us, we use two servers. A situation has arisen where a device needs to communicate via the MQTT protocol. For the continuous provision of the MQTT server, we came up with the following setup, which we tested on our test servers: we installed the Cirrus Link MQTT Distributor and MQTT Engine modules on both servers (primary and backup) and configured them according to this document. The MQTT client connects to one of the MQTT distributor servers. However, we have some problems with the operation. The MQTT Engine is configured to convert JSON messages into a tag structure in the "MQTT Engine" tag provider. Everything was working fine until we initiated a failover to the backup server. When switching to the backup server occurs, the tag content in the "MQTT Engine" tag provider is not synchronized. We said OK, we probably have to put the data in the default tag provider. We did this with both the reference tag and the expression tag in the default tag provider, where reference and expression pointed to a tag of the "MQTT Engine" tag provider. When we tested the failover again, we found that the tags in the default tag provider are not synchronized with the backup server either (memory tags are synchronized). Then we came up with the idea of adding a valueChange script to the tag in the "MQTT Engine" tag provider, which would write to the memory tag on the default tag provider every time the value changes. In this case, it worked a little better, but still not as we expected: when the MQTT message was received, the value of the tag was set in "MQTT Engine" tag provider, where the script was executed and the value was copied to the memory tag of the default tag privider. When we switched to the backup server again, the value of the new memory tag was correctly synchronized and the client started using the backup MQTT server. But... While the redundant backup server was running, the valueChange script was not executed! We even added logging and other methods to the script to check if it was running, but it wasn't!
In the end, we used the following solution:

  • An MQTT message arrives from the client to the MQTT Distributor.
  • The MQTT Distributor forwards the message to the MQTT Engine.
  • MQTT Engine converts the message into a tag structure on its tag provider.
  • This structure is duplicated on the default tag provider with reference tags.
  • Reference tags have set valueChange scripts that overwrite the values in memory tags (same structure again!).

The thing works, but we don't like having to triple the tag structure for each device! I am wondering what is the good practice in such cases? Does anyone have experience with similar setup?