# How to Implement Ignition Redundancy with PostgreSQL Database on the Same Server **URL:** \ **Category:** Ignition\ **Tags:** ignition81\ **Created:** [October 24, 2023, 2:51pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908 "2023-10-24T14:51:57Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![tyler.bennett](https://avatars.discourse-cdn.com/v4/letter/t/f04885/32.png) [@tyler.bennett](https://forum.inductiveautomation.com/u/tyler.bennett)\ **Post date:** [October 24, 2023, 2:51pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/1 "2023-10-24T14:51:57Z")
I am planning implementation of Ignition redundancy on an existing system that has the Ignition Gateway and a PostgreSQL historical database on the same server hardware (co-located). The database connection string Ignition uses `localhost` to reference the database running on the same hardware (e.g. `jdbc:postgresql://localhost:5432/ignition`). Of course when the redundant pair of Ignition servers has a failover event and the Standby server is active it will be trying to make a database connection using that same connection string and unless I take some extra measures there will be no database available on the standby server. As such I am looking at options to continuously replicate the database from the master server to the standby server such that historical data will be available to the standby server even when the master server is offline (Ignition and PostgreSQL). I have been able to find information on implementing that PostgreSQL replication but that has also resulted in some questions. 1. Of the [two types PostgreSQL replication, logical and physical](https://kinsta.com/blog/postgresql-replication/#types-of-postgresql-replication), which is most appropriate for this scenario. 2. Of the [two modes of PostgreSQL replication, synchronous and asynchronous,](https://kinsta.com/blog/postgresql-replication/#replication-modes) which is most appropriate for the scenario? 3. If I were to use `Partial` as the history mode on the Standby Ignition gateway (history will be cached until the Master is available again) will that cached data be available for historical trends while the Standby server is active? It seems there are multi-master (bi-directional) replication options for PostgreSQL but those all seem to require add-ins by third parties and seem extra complicated. Are they worth it? I'm interested in advice on this plan and thoughts from anyone who has experience setting up redundancy while use PostgreSQL for the database. Thanks! Some seemingly related posts: [postgresql-replication-and-ignition-redundancy](https://forum.inductiveautomation.com/t/12298) ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [October 24, 2023, 3:25pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/2 "2023-10-24T15:25:58Z")
> [@tyler.bennett](#): > > on the same server hardware Don't do it. You will suffer and eventually throw up your hands in disgust. (And this is not specific to PostgreSQL. Any brand of DB.) You will need to set up redundancy on the database side completely separately from the gateway hardware. If you need a redundant DB at all. You might not, if your gateway store and forward system can be trusted. Whatever redundancy technique you use, on whatever brand, you need it to be "hidden" from Ignition. Ignition's master and backup gateway simply connect to a magical target URL/address that hides all of the DB redundancy details. ---
**Author:** ![tyler.bennett](https://avatars.discourse-cdn.com/v4/letter/t/f04885/32.png) [@tyler.bennett](https://forum.inductiveautomation.com/u/tyler.bennett)\ **Post date:** [October 24, 2023, 5:19pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/3 "2023-10-24T17:19:26Z")
@pturmel, thanks for the response, always appreciated > [@pturmel](#): > > Don't do it. You will suffer and eventually throw up your hands in disgust. Are you advising against... 1. Having Ignition Gateway the database and on the same server hardware at all. 2. Attempting database replication as outlined above.. with Ignition Gateway and database on the same server hardware for both the master and the standby server in an Ignition redundant pair? 3. All of the above. 4. Something else? I know the architecture examples by IA always show the database physically separated from the Ignition gateway but find it hard to believe that it is often implemented that way except for large implementations (I could be way off on that). Doing so just introduces another set of hardware redundancy concerns. > [@pturmel](#): > > Whatever redundancy technique you use, on whatever brand, you need it to be "hidden" from Ignition. Ignition's master and backup gateway simply connect to a magical target URL/address that hides all of the DB redundancy details. It is my thinking that the use of `localhost` in the connection string in this scenario provides (as you say) "magical target URL/address that hides all of the DB redundancy details" as the connection string would be valid on both servers so long as I implemented database replication. My concerns were really only: - Guidance on technical detail for implementing redundancy with PostgreSQL - Whether I should invest extra effort into finding a way to do multi-master DB replication. - If using one-way replication, and "partial" history mode on the standby gateway would cached data on the standby gateway be available to history function/charts for trending. My approach above is intended to... - provide Ignition gateway redundancy - provide database redundancy (in the case of gateway failover) I'm interested in your suggestions/recommendations for an alternative approach. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [October 24, 2023, 6:39pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/4 "2023-10-24T18:39:33Z")
> [@tyler.bennett](#): > > Are you advising against... 1. Slightly more than mild general recommendation against gateway and database in same machine. Including separate VMs in same hypervisor, in most cases. CPU utilization patterns are substantially different and the DB steals CPU from the gateway. 2. Strong recommendation against having the DB on either master or backup gateway's hardware. Whether the DB is also redundant or not. > [@tyler.bennett](#): > > that the use of `localhost` in the connection string Ewww! No! Then you **must** use multi-master DB redundancy, which is notoriously difficult to manage reliably. (And not available with PostgreSQL anyways.) > [@tyler.bennett](#): > > alternative approach. Buy enough hardware for three systems, at minimum, if Ignition redundancy is used. 1. Good: - Primary Gateway on bare metal - Backup Gateway on bare metal - Database on bare metal 1. Better: - Beefy hardware for Hypervisor w/ Primary gateway VM & Primary DB VM - Beefy hardware for Hypervisor w/ Backup gateway VM & Replica DB VM - Third, but lighter-duty, hardware to provide traffic cop/split brain protection, proxied DB address, et cetera. 1. More Better: - Beefy hardware for Hypervisor w/ Primary gateway VM & Primary DB VM - Beefy hardware for Hypervisor w/ Backup gateway VM & Replica DB VM - Beefy hardware for Hypervisor w/ Ignition cold standby images of Primary and Backup gateways, plus 2nd Replica DB VM 1. Best - Beefy hardware for Hypervisor w/ Primary DB VM and Ignition cold standby images of Primary and Backup VMs - Beefy hardware for Hypervisor w/ Replica DB VM, traffic cop, and Ignition cold standby images of Primary and Backup VMs - Beefy hardware for Hypervisor w/ Primary gateway VM & Replica DB #2 VM - Beefy hardware for Hypervisor w/ Backup gateway VM & Replica DB #3 VM and/or DB archiver. ---
**Author:** ![tyler.bennett](https://avatars.discourse-cdn.com/v4/letter/t/f04885/32.png) [@tyler.bennett](https://forum.inductiveautomation.com/u/tyler.bennett)\ **Post date:** [October 24, 2023, 9:43pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/5 "2023-10-24T21:43:50Z")
@pturmel, thanks for the detailed response. Our current architecture is ~~more closely aligned to~~ this: - Beefy hardware for Primary Gateway and Primary DB on bare metal (Linux OS). - Beefy hardware for Standby Gateway on bare metal (Linux OS). - Looking at adding replica DB to this one. I'm unlikely to get buy-in for additional server hardware (beefy or not) at each of several Ignition implementation locations. So, I'm looking to do the best I can with two "beefy hardware" bare metal servers at each location. I assume your use of a hypervisor in your "#2 Better" configuration is to enable management/isolation of CPU/Memory utilization patterns. Do you have any opinion on the use of docker containers to achieve similar (light-weight alternative to VMs). There seems to be lots of options for [implementing CPU/Memory resource constraints in docker](https://docs.docker.com/config/containers/resource_constraints/). - Gateway on bare metal, DB in docker container (with DB data volume) - Or, Gateway in docker container, DB in docker container (each with data volume) > [@pturmel](#): > > ...hardware to provide traffic cop/split brain protection, proxied DB address, et cetera. Do you have any preference/recommendations into specific applications/services you might use to implement these features (I'm looking for breadcrumbs to facilitate my own research)? I have a basic understanding of database proxies and/or DNS proxies, less so with the concepts of "traffic cop/split brain protection". Do you have a preference/recommendation in database platform for use with Ignition? Does your preferred option have better multi-master replication support than PostgreSQL? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [October 24, 2023, 10:05pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/6 "2023-10-24T22:05:40Z")
> [@tyler.bennett](#): > > more closely aligned to this That wasn't one of my combinations. If you can only have two pieces of beefy hardware, put an independent gateway on one, and a DB on the other. Skip the redundancy. Under no circumstances should gateway redundancy be implemented on the same **bare metal** hardware as the database. Under no circumstances should a redundant DB be established without split-brain protections (a third hardware device). Adding redundancy without the hardware to run it is like sticking ten pounds of [expletive] in a five pound bag. Doesn't matter **to me** what buy-in you might or might not have. I don't want you pointing back at this topic later and saying that any of your partial solutions had even a glimmer of approval from me. {/rant} Yes, VMs permit isolation of both RAM and CPU in mixed environments. This is particularly true with Ignition versus Database because they have such different utilitization patterns. In my scattered experience with commercial hypervisors, it was difficult (and in one older case, impossible) to prevent CPU cycle stealing away from Ignition to benefit the Database. Ignition is a "choppy" workload that suffers poor latency if you take away its CPU idle time. The options where I recommend more than two hypervisors enable **migration** of any particular VM for maintenance purposes, including maintenance of the hypervisor hardware itself. Containerization technology is the equal of VMs in many respects. Not sure if the live migration options are there yet. But containers can certainly do the isolation parts. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [October 24, 2023, 10:12pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/7 "2023-10-24T22:12:15Z")
> [@tyler.bennett](#): > > into specific applications/services Three recommendations in the PostgreSQL world: - [Replication Manager](https://www.repmgr.org/) - [Backup and Recovery Manager](https://pgbarman.org/index.html) - [High Availability Proxy](https://www.haproxy.com/) Replication Manager is key to PostgreSQL redundancy/hot standbys/replication/traffic cop tooling. Barman is an accelerator for standing up replacement replicas after an outage, and for implementing **backups** (distinct from redundancy). Integrates tightly with Repmgr. HAProxy may eliminate the need for floating IP addresses--usage depends on Repmgr implementation choices. FWIW: I have not done a full kit of the above solo (yet). It is complicated. Consider hiring EnterpriseDB to set it up and train your people. (Similar would be true for other brands, too.) Oh, and I do not prefer any other DB brand ahead of PostgreSQL. I don't trust **any** multi-master technology. ---
**Author:** ![elijah.roberts](https://avatars.discourse-cdn.com/v4/letter/e/a6a055/32.png) [@elijah.roberts](https://forum.inductiveautomation.com/u/elijah.roberts)\ **Post date:** [April 20, 2024, 5:02pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/8 "2024-04-20T17:02:16Z")
@pturmel I am curious what hypervisor and VM software you would recommend? I am looking to start virtualizing our systems as you mentioned in the "Better" option above and starting to look at our options. Currently we just run Linux servers with a Ignition gateway on one and a DB on a separate Linux server. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 20, 2024, 6:37pm UTC](https://forum.inductiveautomation.com/t/how-to-implement-ignition-redundancy-with-postgresql-database-on-the-same-server/80908/9 "2024-04-20T18:37:06Z")
I use the Linux native "Kernel Virtual Machine" technology via `libvirt` (`libvirtd` + `qemu-kvm` + `virt-manager`). For a more data-center ish approach, consider using `oVirt` in place of `virt-manager`. In ubuntu, and I suspect fedora, installing virt-manager pulls in the whole stack.