A Modbus TCP to Modbus serial gateway is responsible for managing all of the traffic on the serial cable. There is probably a timeout setting within that gateway that is longer than your stale timeout in your scan classes. I can imagine better handling of serial timeouts on the RS485 side, but all of that code lives in the protocol gateway, not in Ignition.
Note that multiple disconnected devices will create even longer delays in the RS485 scan time.
Increase the “Communication Timeout” setting on the device connection.
If your Modbus gateway supports multiple connections you could make one device connection per unit/slave id, but I’m not sure this will help because the Modbus gateway is still going to be gating access to the serial line and when a timeout occurs the other connections will still have to wait.