PLC connection to Ignition server lost after power cycle

Ignition
1

PLC was power cycled after routine PM. Then the network connection to the Ignition server installed on a Windows Server 2019 failed. Status on the SCADA was offline (grayed out). Performed soft reset on Ignition server remotely but upon reboot, all status blacked out and the device connections under Config were “ReconnectWait”. Unable to ping the PLC IP address from Ignition server but can ping it from my laptop. Error log is "LogixDriver - Browse failed.io.netty.channel.ConnectTimeoutException:…

image
image1073×454 73.6 KB

IT confirmed there is no issue with firewall. Any suggestions on what or where to check?

2

That's a network or firewall error. Ignition itself cannot reach the PLC. Could be a local firewall at the server, or along the path to the PLC. IT may not be wrong about a particular firewall, but if they are responsible for the network, then they are wrong in the larger sense.

3

I would check this first, I've seen windows, seemingly randomly, re-enable the firewall. More anti windows ammo for @pturmel !

4

The supply seems to be endless.

1 Like
5

The Automation Integrator tried to disable the Windows Defender firewall on Windows Server 2019 that hosts Ignition but to no avail. Our local IT team ran a pcap and was not seeing any intraVLAN traffic in the PCAP pulled from the server from the last hour. Their next step is to pull a PCAP from the switch connected to the server. It’s really odd because the lines are running and PLC is reporting data to the metric reporting system except Ignition. Is/are there a specific TCP/UDP port(s) that Ignition uses that we can toggle to see if the traffic flows through the PLC? I am not an IT Engineer but a Controls technician responsible for site automation. Our IT team has been troubleshooting for quite a while but no luck.

6

IT confirmed there is no config issue on Cisco ISA3K firewall and Cisco IE2K switch that’s connected to the Allen-Bradley PLC.

7

The modern Logix driver (noted in the stacktrace) uses port 44818.

8

Yes, Ignition is configured for port 44818. Nothing had been changed except the power cycle to the control cabinet/PLC. It was working before the power cycle.

image

9

I meant that your IT group needs to ensure that port is unblocked. It clearly is a problem with your server’s firewall, your server’s network permissions, or a router along the path from your server to the PLC.

Your IT group needs to look closer.

10

I totally agree. IT is still checking at the moment but they also wanted the vendor to come out and do a fresh install of the Ignition server just to rule out. The only changed they made was a config push to the ISA3K firewall to fix Windows Server 2019 critical vulnerabilities and it would not have affected the ports including port 44818.

11

I checked Gateway scripts and both EnetStatus and LaneStatus had errors shown below:

EnetStatus:
Traceback (most recent call last):
File “<TimerScript:XXXXXX/EnetStatus @4,000ms >”, line 5, in
at com.inductiveautomation.ignition.gateway.script.LegacyGatewayTagUtilities.processTagBrowse(LegacyGatewayTagUtilities.java:353)
at com.inductiveautomation.ignition.gateway.script.LegacyGatewayTagUtilities._browseTags(LegacyGatewayTagUtilities.java:328)
at com.inductiveautomation.ignition.gateway.script.LegacyGatewayTagUtilities.browseTagsImpl(LegacyGatewayTagUtilities.java:268)
at com.inductiveautomation.ignition.common.script.builtin.LegacyTagUtilities.browseTags(LegacyTagUtilities.java:117)
at jdk.internal.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
java.lang.Exception: java.lang.Exception: Error in browse results: Bad_NotFound

8.1.5 (b2021042810)
Azul Systems, Inc. 11.0.10

LaneStatus:
Traceback (most recent call last):
File “<TimerScript:XXXXXX/LaneStatus @4,000ms >”, line 5, in
at com.inductiveautomation.ignition.gateway.script.LegacyGatewayTagUtilities.processTagBrowse(LegacyGatewayTagUtilities.java:353)
at com.inductiveautomation.ignition.gateway.script.LegacyGatewayTagUtilities._browseTags(LegacyGatewayTagUtilities.java:328)
at com.inductiveautomation.ignition.gateway.script.LegacyGatewayTagUtilities.browseTagsImpl(LegacyGatewayTagUtilities.java:268)
at com.inductiveautomation.ignition.common.script.builtin.LegacyTagUtilities.browseTags(LegacyTagUtilities.java:117)
at jdk.internal.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
java.lang.Exception: java.lang.Exception: Error in browse results: Bad_NotFound

8.1.5 (b2021042810)
Azul Systems, Inc. 11.0.10

12

If you cannot ping the PLC from the Ignition server, but can from elsewhere, the problem is not Ignition. Can another computer/laptop plugged into the same Ethernet port your Ignition server uses ping the PLC? If so, the problem is probably a firewall on the Ignition server. If not, the problem is somewhere on the network between the Ignition server and PLC.

13

In my opinion this is nonsense. At this point we know the device was restarted, as was the server, and IT did something to one of your firewalls. That's not necessarily all they did, they may have also pushed changes and or patches to your server. The fact that you cannot disable the windows firewall on the server, just to test, is concerning. I would be escalating this since it sounds like you are getting the run around at the moment.

2 Likes