We have a remote Ignition Gateway that if we lose internet connectivity to it and then regain connection, then the tags will have stale quality in the Vision Client and cause overlays on Components binded to OPC Tags even though the Gateway shows good quality with these tags.
The steps that can taken to recreate the issue are below:
1. Launch a Vision Client on a machine that is remote from the Ignition Gateway.
2. Disconnect Internet on the machine with the Vision Client in order to lose connection to the Ignition Gateway.
3. Reconnect Internet on the machine with the Vision Client.
4. Overlays for stale tag quality do not go away.
However, it does work as expected from my VM, and the Tags resume good quality after reconnecting to the Gateway. I’m not sure why it only works from my VM and never from my Base Machine, but I have logs from the client console from both the VM and my Base Machine.
After analyzing the logs, I found that there is one line that really sticks out in the client console on my VM (works) that isn’t on the one from my Base Machine (doesn’t work):
15:27:48.253 [ClientExecEngine-4] WARN tags.subscriptions - Subcription crc mismatch, will resubscribe.
I am not sure if missing that is the meat of the issue so I will paste the logs from the Client Console from both Vision Clients and the information about my base machine, my VM, and the Ignition Server.
Ignition Server Info:
OS - Microsoft Windows Server 2016 Datacenter
Ignition Version - 8.0.2 (b2019060511)
Base Machine Info:
OS - Microsoft Windows 10 Pro
VM Info:
OS - Microsoft Windows Server 2012 R2 Standard
VM Client Console (works):
15:27:27.855 [GatewayConnection-1] WARN com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Connection to Gateway lost, due to exception.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: connect timed out
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:347)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.ping(GatewayInterface.java:975)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.publicPing(GatewayInterface.java:968)
at com.inductiveautomation.ignition.client.gateway_interface.AbstractGatewayConnection$GatewayMonitorTask.run(AbstractGatewayConnection.java:1060)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:394)
... 10 common frames omitted
15:27:27.859 [GatewayConnection-1] INFO tags.subscriptions - Changing connected quality to 'Bad_Stale'
15:27:27.964 [GatewayConnection-1] INFO Scripting.ScriptManager.NC21_HMI - Pausing scripts...
15:27:27.968 [GatewayConnection-1] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Starting reconnect thread.
15:27:29.760 [ClientExecEngine-3] ERROR tags.subscriptions - Error updating tag subscriptions on gateway.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: connect timed out
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:905)
at com.inductiveautomation.ignition.client.tags.impl.ClientTagSubscriptionManager.syncSubscriptions(ClientTagSubscriptionManager.java:288)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:394)
... 11 common frames omitted
15:27:29.807 [ClientExecEngine-3] WARN tags.subscriptions - Subcription crc mismatch, will resubscribe.
15:27:39.868 [ClientExecEngine-3] ERROR tags.subscriptions - Error sending complete subscription set to gateway.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: connect timed out
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:905)
at com.inductiveautomation.ignition.client.tags.impl.ClientTagSubscriptionManager.syncSubscriptions(ClientTagSubscriptionManager.java:307)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:394)
... 11 common frames omitted
15:27:45.059 [ClientExecEngine-4] ERROR Vision.Components.AlarmStatusTable - Error when running query alarm status function.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: connect timed out
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:905)
at com.inductiveautomation.factorypmi.application.components.AlarmStatusTable$AlarmStatusPoller.runFunction(AlarmStatusTable.java:1304)
at com.inductiveautomation.factorypmi.application.components.AlarmStatusTable$AlarmStatusPoller.runFunction(AlarmStatusTable.java:1240)
at com.inductiveautomation.factorypmi.application.components.util.GatewayFunctionPoller$FunctionRunner.run(GatewayFunctionPoller.java:117)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:421)
... 13 common frames omitted
15:27:48.111 [GatewayConnection-1] INFO tags.subscriptions - Changing connected quality to 'Good'
15:27:48.116 [GatewayConnection-1] INFO Scripting.ScriptManager.NC21_HMI - Resuming scripts...
15:27:48.149 [GatewayConnection-1] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Stopping reconnect thread.
15:27:48.253 [ClientExecEngine-4] WARN tags.subscriptions - Subcription crc mismatch, will resubscribe.
Base Machine Client Console (doesn’t work):
14:56:27.915 [ClientExecEngine-4] WARN com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Connection to Gateway lost, due to exception.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: connect timed out
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:905)
at com.inductiveautomation.factorypmi.application.components.AlarmStatusTable$AlarmStatusPoller.runFunction(AlarmStatusTable.java:1304)
at com.inductiveautomation.factorypmi.application.components.AlarmStatusTable$AlarmStatusPoller.runFunction(AlarmStatusTable.java:1240)
at com.inductiveautomation.factorypmi.application.components.util.GatewayFunctionPoller$FunctionRunner.run(GatewayFunctionPoller.java:117)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:394)
... 13 common frames omitted
14:56:27.916 [ClientExecEngine-4] INFO tags.subscriptions - Changing connected quality to 'Bad_Stale'
14:56:29.746 [ClientExecEngine-4] INFO Scripting.ScriptManager.NC21_HMI - Pausing scripts...
14:56:29.747 [ClientExecEngine-4] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Starting reconnect thread.
14:56:29.747 [ClientExecEngine-4] ERROR Vision.Components.AlarmStatusTable - Error when running query alarm status function.
com.inductiveautomation.ignition.client.gateway_interface.GatewayException: connect timed out
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.newGatewayException(GatewayInterface.java:341)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:501)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:273)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.sendMessage(GatewayInterface.java:268)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.invoke(GatewayInterface.java:905)
at com.inductiveautomation.factorypmi.application.components.AlarmStatusTable$AlarmStatusPoller.runFunction(AlarmStatusTable.java:1304)
at com.inductiveautomation.factorypmi.application.components.AlarmStatusTable$AlarmStatusPoller.runFunction(AlarmStatusTable.java:1240)
at com.inductiveautomation.factorypmi.application.components.util.GatewayFunctionPoller$FunctionRunner.run(GatewayFunctionPoller.java:117)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at com.inductiveautomation.ignition.client.gateway_interface.GatewayInterface.getResponse(GatewayInterface.java:394)
... 13 common frames omitted
14:56:41.002 [GatewayConnection-1] INFO tags.subscriptions - Changing connected quality to 'Good'
14:56:41.009 [GatewayConnection-1] INFO Scripting.ScriptManager.NC21_HMI - Resuming scripts...
14:56:41.043 [GatewayConnection-1] INFO com.inductiveautomation.ignition.client.gateway_interface.GatewayConnectionManager - Stopping reconnect thread.