Gateway Network Problems with Duplicate Connection IDs

[hgarner]

We are working to set up Gateway Network Connections between several Ignition Gateways and a Central Ignition Server to allow us to gather tag information from each Gateway to build real-time Dashboards. We have a mixture of 32-bit Ignition installations with v7.9.1 and a few 64-bit Ignition installations with v7.9.5. We are having an issue with 3 of the 64-bit Ignition installations with v7.9.5 in that they will not all stay connected to the central server. All 3 were made from the same drive image, so perhaps this may be part of the issue. I have included an excerpt of the wrapper.log file from the Central Ignition Gateway below that shows continuous messages that appear related to these 3 machines. It seems the gan-remote-connection-id associated with each of these 3 machines is the same which I expect could be part of the problem. Perhaps this is related to each of these machines using the same drive image.

INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: <- incoming local='ignition-nordevengr02_centralgateway' remote='ignition-gae870rd1drw371' method=shutdown: [100] Shutting down connection 'ignition-nordevengr02_centralgateway' to http://10.1.51.71:8088/main/system: Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw371 connection status has been updated from Running to Shutdown: shutdown() has been called gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.M.Routing                   ] [06:47:32]: Route disconnected between server '_0:0:Ignition-gae870rd1drw371' and connection 'ignition-gae870rd1drw371|9598565b-85e5-4281-8e94-7ed78aa415b3' 
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw371 connection status has been updated from Shutdown to Faulted: onClose has been called on web socket:Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw372 connection status has been updated from Unknown to Initialized gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.CentralManager              ] [06:47:32]: Registering connection: ignition-gae870rd1drw372 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw372 connection status has been updated from Initialized to Running: Web socket session established for ignition-gae870rd1drw372|9598565b-85e5-4281-8e94-7ed78aa415b3 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: <- incoming local='ignition-nordevengr02_centralgateway' remote='ignition-gae870rd1drw372' method=shutdown: [100] Shutting down connection 'ignition-nordevengr02_centralgateway' to http://10.1.51.72:8088/main/system: Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw372 connection status has been updated from Running to Shutdown: shutdown() has been called gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw372 connection status has been updated from Shutdown to Faulted: onClose has been called on web socket:Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw369 connection status has been updated from Unknown to Initialized gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.CentralManager              ] [06:47:32]: Registering connection: ignition-gae870rd1drw369 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw369 connection status has been updated from Initialized to Running: Web socket session established for ignition-gae870rd1drw369|9598565b-85e5-4281-8e94-7ed78aa415b3 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: <- incoming local='ignition-nordevengr02_centralgateway' remote='ignition-gae870rd1drw369' method=shutdown: [100] Shutting down connection 'ignition-nordevengr02_centralgateway' to http://10.1.51.69:8088/main/system: Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw369 connection status has been updated from Running to Shutdown: shutdown() has been called gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.M.Routing                   ] [06:47:32]: Route disconnected between server '_0:0:Ignition-gae870rd1drw369' and connection 'ignition-gae870rd1drw369|9598565b-85e5-4281-8e94-7ed78aa415b3' 
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw369 connection status has been updated from Shutdown to Faulted: onClose has been called on web socket:Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw371 connection status has been updated from Unknown to Initialized gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.CentralManager              ] [06:47:32]: Registering connection: ignition-gae870rd1drw371 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:32 | I [m.T.W.WSConnection            ] [06:47:32]: [100] ignition-gae870rd1drw371 connection status has been updated from Initialized to Running: Web socket session established for ignition-gae870rd1drw371|9598565b-85e5-4281-8e94-7ed78aa415b3 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: <- incoming local='ignition-nordevengr02_centralgateway' remote='ignition-gae870rd1drw371' method=shutdown: [100] Shutting down connection 'ignition-nordevengr02_centralgateway' to http://10.1.51.71:8088/main/system: Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw371 connection status has been updated from Running to Shutdown: shutdown() has been called gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.M.Routing                   ] [06:47:33]: Route disconnected between server '_0:0:Ignition-gae870rd1drw371' and connection 'ignition-gae870rd1drw371|9598565b-85e5-4281-8e94-7ed78aa415b3' 
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw371 connection status has been updated from Shutdown to Faulted: onClose has been called on web socket:Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw372 connection status has been updated from Unknown to Initialized gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.CentralManager              ] [06:47:33]: Registering connection: ignition-gae870rd1drw372 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw372 connection status has been updated from Initialized to Running: Web socket session established for ignition-gae870rd1drw372|9598565b-85e5-4281-8e94-7ed78aa415b3 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: <- incoming local='ignition-nordevengr02_centralgateway' remote='ignition-gae870rd1drw372' method=shutdown: [100] Shutting down connection 'ignition-nordevengr02_centralgateway' to http://10.1.51.72:8088/main/system: Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw372 connection status has been updated from Running to Shutdown: shutdown() has been called gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw372 connection status has been updated from Shutdown to Faulted: onClose has been called on web socket:Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw369 connection status has been updated from Unknown to Initialized gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.CentralManager              ] [06:47:33]: Registering connection: ignition-gae870rd1drw369 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw369 connection status has been updated from Initialized to Running: Web socket session established for ignition-gae870rd1drw369|9598565b-85e5-4281-8e94-7ed78aa415b3 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: <- incoming local='ignition-nordevengr02_centralgateway' remote='ignition-gae870rd1drw369' method=shutdown: [100] Shutting down connection 'ignition-nordevengr02_centralgateway' to http://10.1.51.69:8088/main/system: Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw369 connection status has been updated from Running to Shutdown: shutdown() has been called gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.M.Routing                   ] [06:47:33]: Route disconnected between server '_0:0:Ignition-gae870rd1drw369' and connection 'ignition-gae870rd1drw369|9598565b-85e5-4281-8e94-7ed78aa415b3' 
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw369 connection status has been updated from Shutdown to Faulted: onClose has been called on web socket:Incoming: AbstractHttpConnectionFactory.destroyConnection() gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw371 connection status has been updated from Unknown to Initialized gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.CentralManager              ] [06:47:33]: Registering connection: ignition-gae870rd1drw371 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3
INFO   | jvm 1    | 2018/06/06 02:47:33 | I [m.T.W.WSConnection            ] [06:47:33]: [100] ignition-gae870rd1drw371 connection status has been updated from Initialized to Running: Web socket session established for ignition-gae870rd1drw371|9598565b-85e5-4281-8e94-7ed78aa415b3 gan-remote-connection-id=9598565b-85e5-4281-8e94-7ed78aa415b3

Below is an excerpt from the Central Gateway Gateway Network Settings Configuration Page for Incoming Connections. The first entry is associated with the 3 machines that are giving the issue and the value in the brackets matches the gan-remote-connection-id shown in the log above. Looking at the page the Gateway Name will cycle through the 3 machine names but the gan-remote-connection-id in the brackets does not change. The values in the brackets for the rest of the machines appear to be URLs. I think if we could change the value in brackets for the 3 machines showing issues to something like the URL shown for the other machines this problem might be solved and we would have separate connections, but I am not sure how to accomplish this. Any help with this issue would be very much appreciated.

[pturmel]

@hgarner

All 3 were made from the same drive image

I'd be shocked if that's not the issue. It may be a simple as changing the partition/filesystem UUIDs. Or perhaps the virtual drive's emulated serial number. (IA doesn't document what goes into their licensing system ID, but I suspect those are part of it.) Regardless, you'll need to unlicense the two systems before making the changes, then relicense afterwards.

This definitely should be done privately via a support ticket.

[hgarner]

The 11 or so machines with 32-bit Ignition were set up with a drive image with Windows 7 and do not have any problems. The image is unlicensed, and we install the license afterwards. I will create a ticket for this.

Thanks

[mgross]

The Gateway Network really doesn’t like identical machines created from drive images. What you need to do is delete the .uuid file in the /data folder then restart the gateway. This will rebuild a .uuid file specifically for the machine, and then the central gateway should be able to figure things out.

[hgarner]

This seems to have done the trick on one of the machines I was able to delete the .uuid file and restart the gateway. We will do this on one of the other 3 machines soon which should take care of this issue, and we can either uninstall Ignition from our drive image as it only takes 3 minutes to install or just follow this procedure to delete the old .uuid file.

Thanks for the quick response that seems to have taken care of this problem.

[isaias.berhe]

@mgross

he Gateway Network really doesn’t like identical machines created from drive images. What you need to do is delete the .uuid file in the /data

I just also was able to apply this recommended solution to fix the same issue i had with a second Front end gateway created from an image of an existing one. Thanks a lot @mgross