I am planning implementation of Ignition redundancy on an existing system that has the Ignition Gateway and a PostgreSQL historical database on the same server hardware (co-located).
The database connection string Ignition uses localhost to reference the database running on the same hardware (e.g. jdbc:postgresql://localhost:5432/ignition).
Of course when the redundant pair of Ignition servers has a failover event and the Standby server is active it will be trying to make a database connection using that same connection string and unless I take some extra measures there will be no database available on the standby server.
As such I am looking at options to continuously replicate the database from the master server to the standby server such that historical data will be available to the standby server even when the master server is offline (Ignition and PostgreSQL).
I have been able to find information on implementing that PostgreSQL replication but that has also resulted in some questions.
I would have thought logical but it seems there is far more examples for using physical/streaming which suggest to me that it is the option more often used (in general).
My assumption is that asynchronous would be better due to performance concerns.
If I were to use Partial as the history mode on the Standby Ignition gateway (history will be cached until the Master is available again) will that cached data be available for historical trends while the Standby server is active?
My thought is to use master-slave replication, as such the database instance on the standby server will not propagate changes back to the master server.
It seems there are multi-master (bi-directional) replication options for PostgreSQL but those all seem to require add-ins by third parties and seem extra complicated. Are they worth it?
I'm interested in advice on this plan and thoughts from anyone who has experience setting up redundancy while use PostgreSQL for the database. Thanks!
Don't do it. You will suffer and eventually throw up your hands in disgust. (And this is not specific to PostgreSQL. Any brand of DB.)
You will need to set up redundancy on the database side completely separately from the gateway hardware. If you need a redundant DB at all. You might not, if your gateway store and forward system can be trusted.
Whatever redundancy technique you use, on whatever brand, you need it to be "hidden" from Ignition. Ignition's master and backup gateway simply connect to a magical target URL/address that hides all of the DB redundancy details.
@pturmel, thanks for the response, always appreciated
Are you advising against...
Having Ignition Gateway the database and on the same server hardware at all.
Attempting database replication as outlined above.. with Ignition Gateway and database on the same server hardware for both the master and the standby server in an Ignition redundant pair?
All of the above.
Something else?
I know the architecture examples by IA always show the database physically separated from the Ignition gateway but find it hard to believe that it is often implemented that way except for large implementations (I could be way off on that). Doing so just introduces another set of hardware redundancy concerns.
It is my thinking that the use of localhost in the connection string in this scenario provides (as you say) "magical target URL/address that hides all of the DB redundancy details" as the connection string would be valid on both servers so long as I implemented database replication.
My concerns were really only:
Guidance on technical detail for implementing redundancy with PostgreSQL
Whether I should invest extra effort into finding a way to do multi-master DB replication.
If using one-way replication, and "partial" history mode on the standby gateway would cached data on the standby gateway be available to history function/charts for trending.
My approach above is intended to...
provide Ignition gateway redundancy
provide database redundancy (in the case of gateway failover)
I'm interested in your suggestions/recommendations for an alternative approach.
Slightly more than mild general recommendation against gateway and database in same machine. Including separate VMs in same hypervisor, in most cases. CPU utilization patterns are substantially different and the DB steals CPU from the gateway.
Strong recommendation against having the DB on either master or backup gateway's hardware. Whether the DB is also redundant or not.
Ewww! No! Then you must use multi-master DB redundancy, which is notoriously difficult to manage reliably. (And not available with PostgreSQL anyways.)
Buy enough hardware for three systems, at minimum, if Ignition redundancy is used.
Good:
Primary Gateway on bare metal
Backup Gateway on bare metal
Database on bare metal
Better:
Beefy hardware for Hypervisor w/ Primary gateway VM & Primary DB VM
Beefy hardware for Hypervisor w/ Backup gateway VM & Replica DB VM
Third, but lighter-duty, hardware to provide traffic cop/split brain protection, proxied DB address, et cetera.
More Better:
Beefy hardware for Hypervisor w/ Primary gateway VM & Primary DB VM
Beefy hardware for Hypervisor w/ Backup gateway VM & Replica DB VM
Beefy hardware for Hypervisor w/ Ignition cold standby images of Primary and Backup gateways, plus 2nd Replica DB VM
Best
Beefy hardware for Hypervisor w/ Primary DB VM and Ignition cold standby images of Primary and Backup VMs
Beefy hardware for Hypervisor w/ Replica DB VM, traffic cop, and Ignition cold standby images of Primary and Backup VMs
Beefy hardware for Hypervisor w/ Primary gateway VM & Replica DB #2 VM
Beefy hardware for Hypervisor w/ Backup gateway VM & Replica DB #3 VM and/or DB archiver.
Our current architecture is more closely aligned to this:
Beefy hardware for Primary Gateway and Primary DB on bare metal (Linux OS).
Beefy hardware for Standby Gateway on bare metal (Linux OS).
Looking at adding replica DB to this one.
I'm unlikely to get buy-in for additional server hardware (beefy or not) at each of several Ignition implementation locations. So, I'm looking to do the best I can with two "beefy hardware" bare metal servers at each location.
I assume your use of a hypervisor in your "#2 Better" configuration is to enable management/isolation of CPU/Memory utilization patterns. Do you have any opinion on the use of docker containers to achieve similar (light-weight alternative to VMs). There seems to be lots of options for implementing CPU/Memory resource constraints in docker.
Gateway on bare metal, DB in docker container (with DB data volume)
Or, Gateway in docker container, DB in docker container (each with data volume)
Do you have any preference/recommendations into specific applications/services you might use to implement these features (I'm looking for breadcrumbs to facilitate my own research)? I have a basic understanding of database proxies and/or DNS proxies, less so with the concepts of "traffic cop/split brain protection".
Do you have a preference/recommendation in database platform for use with Ignition? Does your preferred option have better multi-master replication support than PostgreSQL?
That wasn't one of my combinations. If you can only have two pieces of beefy hardware, put an independent gateway on one, and a DB on the other. Skip the redundancy.
Under no circumstances should gateway redundancy be implemented on the same bare metal hardware as the database. Under no circumstances should a redundant DB be established without split-brain protections (a third hardware device).
Adding redundancy without the hardware to run it is like sticking ten pounds of [expletive] in a five pound bag. Doesn't matter to me what buy-in you might or might not have. I don't want you pointing back at this topic later and saying that any of your partial solutions had even a glimmer of approval from me.
{/rant}
Yes, VMs permit isolation of both RAM and CPU in mixed environments. This is particularly true with Ignition versus Database because they have such different utilitization patterns. In my scattered experience with commercial hypervisors, it was difficult (and in one older case, impossible) to prevent CPU cycle stealing away from Ignition to benefit the Database. Ignition is a "choppy" workload that suffers poor latency if you take away its CPU idle time.
The options where I recommend more than two hypervisors enable migration of any particular VM for maintenance purposes, including maintenance of the hypervisor hardware itself.
Containerization technology is the equal of VMs in many respects. Not sure if the live migration options are there yet. But containers can certainly do the isolation parts.
Replication Manager is key to PostgreSQL redundancy/hot standbys/replication/traffic cop tooling.
Barman is an accelerator for standing up replacement replicas after an outage, and for implementing backups (distinct from redundancy). Integrates tightly with Repmgr.
HAProxy may eliminate the need for floating IP addresses--usage depends on Repmgr implementation choices.
FWIW: I have not done a full kit of the above solo (yet). It is complicated. Consider hiring EnterpriseDB to set it up and train your people. (Similar would be true for other brands, too.)
Oh, and I do not prefer any other DB brand ahead of PostgreSQL. I don't trust any multi-master technology.
@pturmel I am curious what hypervisor and VM software you would recommend? I am looking to start virtualizing our systems as you mentioned in the "Better" option above and starting to look at our options. Currently we just run Linux servers with a Ignition gateway on one and a DB on a separate Linux server.
I use the Linux native "Kernel Virtual Machine" technology via libvirt (libvirtd + qemu-kvm + virt-manager). For a more data-center ish approach, consider using oVirt in place of virt-manager.
In ubuntu, and I suspect fedora, installing virt-manager pulls in the whole stack.