That is a design error. The redundant gateway needs to be on the exact same networks as the primary gateway. And both primary and redundant gateways need to be set to use their names as their public addresses.
If you are connecting remotely, put entries in your own /etc/hosts file with the appropriate names if you cannot make your local DNS provide them.
(In most cases, there should not be a redundancy network. It can lead to "split brain" situations.)