My controller only accepts one modbusTCP connection, and more connections will fail to connect. Normally, Ignition only uses one connection, so there is no problem. This problem I'm having is that the connection to the device failed from a few seconds to a few minutes. I believe Ignition is using more than one connection because other modbus clients are also unable to connect to the controller, unless disable the network.
The solution was that I freshly installed Ignition in a different path and restored the gateway backup.
The following didn't work:
Disable and re-enable the device.
Stop and restart the Ignition service.
Restart the computer.
Upgrade Ignition with the same version(8.1.33).
Any network issue that breaks the TCP connection will cause Ignition to attempt to reconnect. If the issue was breakage that the PLC didn't also detect, then it can take over 90 seconds (standard TCP timeout, not specific to ModbusTCP) to clear out. A device that won't accept another connection while that timeout is running is doomed to occasional 90+ second outages.
Complain to the controller maker for screwing that up.
Consider also putting an external firewall in front of the PLC and only permit the Ignition IP address to make a connection to port 502.
You can also use wireshark on the Ignition gateway to document its behavior. (Use tshark in the background and ring buffer mode to capture for long durations leading up to a problem.)
Thank you for your reply, pturmel. I did use a traffic monitoring software and found that only Ignition was connecting to the PLC's IP, but I forgot to take a screenshot or video, so not mentioned before.