Intermittent Connection & Odd ControlLogix legacy driver behavior on L18s over NAT router

Ignition
1

I have 30 devices going through a MikroTik NAT router. And it works pretty well. I have the same rules set up on all the interfaces of the router. I’ll post the export from that switch below.

But the devices are named 1-30 and they go through the associated interface on the Nat router.

But, on devices 1-7 and – none of the others. They’ll go from connected → browse → to disconnected, downloading, or back online. Like in the video down below. I also have some thread dumps from around this time. But, I was wondering if anyone else has seen behavior like this? I am going to check the timeslice on these devices. As they’re the oldest in the fleet.

Ignition-IS-DP-IGNITION_thread_dump20210423-102659.txt (1.8 MB) Ignition-IS-DP-IGNITION_thread_dump20210423-102655.txt (1.1 MB) Ignition-IS-DP-IGNITION_thread_dump20210422-130302.txt (973.6 KB) Ignition-IS-DP-IGNITION_thread_dump20210422-130259.txt (817.8 KB) Ignition-IS-DP-IGNITION_thread_dump20210421-154255.txt (794.6 KB) Ignition-IS-DP-IGNITION_thread_dump20210421-154251.txt (720.7 KB)

obfuscated router export

# apr/19/2021 16:47:27 by RouterOS 6.47
# software id = G5MK-UT30
#
# model = CRS354-48G-4S+2Q+
# serial number = *********
/interface bridge
add admin-mac=******* auto-mac=no comment=defconf fast-forward=no name=bridge
/interface ethernet
set [ find default-name=ether48 ] arp=proxy-arp
set [ find default-name=sfp-sfpplus1 ] rx-flow-control=auto tx-flow-control=auto
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/user group
set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge port
add bridge=bridge comment=defconf interface=ether48 learn=yes
add bridge=bridge comment=defconf interface=ether49
add bridge=bridge comment=defconf interface=qsfpplus1-1
add bridge=bridge comment=defconf interface=qsfpplus1-2
add bridge=bridge comment=defconf interface=qsfpplus1-3
add bridge=bridge comment=defconf interface=qsfpplus1-4
add bridge=bridge comment=defconf interface=qsfpplus2-1
add bridge=bridge comment=defconf interface=qsfpplus2-2
add bridge=bridge comment=defconf interface=qsfpplus2-3
add bridge=bridge comment=defconf interface=qsfpplus2-4
add bridge=bridge comment=defconf hw=no interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=ether47
add bridge=bridge comment=defconf interface=ether46
add bridge=bridge comment=defconf interface=ether45
add bridge=bridge comment=defconf interface=ether44
add bridge=bridge comment=defconf interface=ether43
/interface list member
add interface=bridge list=LAN
add interface=qsfpplus1-1 list=WAN
/ip address
add address=210.152.80.220/18 interface=sfp-sfpplus1 network=210.152.64.0
add address=210.152.79.11/24 interface=bridge network=210.152.79.0
add address=210.152.79.12/24 interface=bridge network=210.152.79.0
add address=210.152.79.13/18 interface=bridge network=210.152.64.0
add address=210.152.79.14/18 interface=bridge network=210.152.64.0
add address=210.152.79.15/18 interface=bridge network=210.152.64.0
add address=210.152.79.16/18 interface=bridge network=210.152.64.0
add address=210.152.79.17/18 interface=bridge network=210.152.64.0
add address=210.152.79.18/18 interface=bridge network=210.152.64.0
add address=210.152.79.19/18 interface=bridge network=210.152.64.0
add address=210.152.79.20/18 interface=bridge network=210.152.64.0
add address=210.152.79.21/18 interface=bridge network=210.152.64.0
add address=210.152.79.22/18 interface=bridge network=210.152.64.0
add address=210.152.79.23/18 interface=bridge network=210.152.64.0
add address=210.152.79.24/18 interface=bridge network=210.152.64.0
add address=210.152.79.25/18 interface=bridge network=210.152.64.0
add address=210.152.79.26/18 interface=bridge network=210.152.64.0
add address=210.152.79.27/18 interface=bridge network=210.152.64.0
add address=210.152.79.28/18 interface=bridge network=210.152.64.0
add address=210.152.79.29/18 interface=bridge network=210.152.64.0
add address=210.152.79.30/18 interface=bridge network=210.152.64.0
add address=210.152.79.31/18 interface=bridge network=210.152.64.0
add address=210.152.79.32/18 interface=bridge network=210.152.64.0
add address=210.152.79.33/18 interface=bridge network=210.152.64.0
add address=210.152.79.34/18 interface=bridge network=210.152.64.0
add address=210.152.79.35/18 interface=bridge network=210.152.64.0
add address=210.152.79.36/18 interface=bridge network=210.152.64.0
add address=210.152.79.37/18 interface=bridge network=210.152.64.0
add address=210.152.79.38/18 interface=bridge network=210.152.64.0
add address=210.152.79.39/18 interface=bridge network=210.152.64.0
add address=210.152.79.40/18 interface=bridge network=210.152.64.0
add address=210.152.79.41/18 interface=bridge network=210.152.64.0
add address=210.152.79.42/18 interface=bridge network=210.152.64.0
add address=210.152.79.43/18 interface=bridge network=210.152.64.0
add address=210.152.79.44/18 interface=bridge network=210.152.64.0
add address=210.152.79.45/18 interface=bridge network=210.152.64.0
add address=210.152.79.46/18 interface=bridge network=210.152.64.0
add address=210.152.79.47/18 interface=bridge network=210.152.64.0
add address=210.152.79.48/18 interface=bridge network=210.152.64.0
add address=210.152.79.49/18 interface=bridge network=210.152.64.0
add address=210.152.79.50/18 interface=bridge network=210.152.64.0
add address=210.152.79.51/18 interface=bridge network=210.152.64.0
add address=210.152.79.52/18 interface=bridge network=210.152.64.0
add address=210.152.79.53/18 interface=bridge network=210.152.64.0
add address=210.152.79.54/18 interface=bridge network=210.152.64.0
add address=210.152.79.55/18 interface=bridge network=210.152.64.0
add address=210.152.79.56/18 interface=bridge network=210.152.64.0
add address=210.152.79.57/18 interface=bridge network=210.152.64.0
add address=210.152.79.58/18 interface=bridge network=210.152.64.0
add address=210.152.80.221/18 interface=bridge network=210.152.64.0
add address=192.168.0.75/24 interface=ether1 network=192.168.0.0
add address=192.168.2.75/24 interface=ether2 network=192.168.2.0
add address=192.168.3.75/24 interface=ether3 network=192.168.3.0
add address=192.168.4.75/24 interface=ether4 network=192.168.4.0
add address=192.168.5.75/24 interface=ether5 network=192.168.5.0
add address=192.168.6.75/24 interface=ether6 network=192.168.6.0
add address=192.168.7.75/24 interface=ether7 network=192.168.7.0
add address=192.168.8.75/24 interface=ether8 network=192.168.8.0
add address=192.168.9.75/24 interface=ether9 network=192.168.9.0
add address=192.168.10.75/24 interface=ether10 network=192.168.10.0
add address=192.168.11.75/24 interface=ether11 network=192.168.11.0
add address=192.168.12.75/24 interface=ether12 network=192.168.12.0
add address=192.168.13.75/24 interface=ether13 network=192.168.13.0
add address=192.168.14.75/24 interface=ether14 network=192.168.14.0
add address=192.168.15.75/24 interface=ether15 network=192.168.15.0
add address=192.168.16.75/24 interface=ether16 network=192.168.16.0
add address=192.168.17.75/24 interface=ether17 network=192.168.17.0
add address=192.168.18.75/24 interface=ether18 network=192.168.18.0
add address=192.168.19.75/24 interface=ether19 network=192.168.19.0
add address=192.168.20.75/24 interface=ether20 network=192.168.20.0
add address=192.168.21.75/24 interface=ether21 network=192.168.21.0
add address=192.168.22.75/24 interface=ether22 network=192.168.22.0
add address=192.168.23.75/24 interface=ether23 network=192.168.23.0
add address=192.168.24.75/24 interface=ether24 network=192.168.24.0
add address=192.168.25.75/24 interface=ether25 network=192.168.25.0
add address=192.168.26.75/24 interface=ether26 network=192.168.26.0
add address=192.168.27.75/24 interface=ether27 network=192.168.27.0
add address=192.168.28.75/24 interface=ether28 network=192.168.28.0
add address=192.168.29.75/24 interface=ether29 network=192.168.29.0
add address=192.168.30.75/24 interface=ether30 network=192.168.30.0
add address=192.168.31.75/24 interface=ether31 network=192.168.31.0
add address=192.168.32.75/24 interface=ether32 network=192.168.32.0
add address=192.168.33.75/24 interface=ether33 network=192.168.33.0
add address=192.168.34.75/24 interface=ether34 network=192.168.34.0
add address=192.168.35.75/24 interface=ether35 network=192.168.35.0
add address=192.168.36.75/24 interface=ether36 network=192.168.36.0
add address=192.168.37.75/24 interface=ether37 network=192.168.37.0
add address=192.168.38.75/24 interface=ether38 network=192.168.38.0
add address=192.168.39.75/24 interface=ether39 network=192.168.39.0
add address=192.168.40.75/24 interface=ether40 network=192.168.40.0
add address=192.168.41.75/24 interface=ether41 network=192.168.41.0
add address=192.168.142.227/24 comment="Epic II - D23" interface=ether42 network=192.168.142.0
add address=210.152.79.104/18 interface=bridge network=210.152.64.0
/ip dhcp-client
add disabled=no interface=bridge
/ip firewall filter
add action=fasttrack-connection chain=forward src-address=210.152.80.1
/ip firewall mangle
add action=accept chain=prerouting src-address=210.152.80.1
add action=accept chain=postrouting dst-address=210.152.80.1
add action=accept chain=forward src-address=210.152.80.1
add action=accept chain=postrouting dst-address=210.152.79.104 src-address=210.152.80.1
/ip firewall nat
add action=accept chain=srcnat comment="DST NAT BEGIN" disabled=yes
add action=dst-nat chain=dstnat dst-address=210.152.79.104 to-addresses=192.168.25.76
add action=dst-nat chain=dstnat dst-address=210.152.79.11 to-addresses=192.168.0.1
add action=dst-nat chain=dstnat dst-address=210.152.79.12 to-addresses=192.168.2.1
add action=dst-nat chain=dstnat dst-address=210.152.79.13 to-addresses=192.168.3.1
add action=dst-nat chain=dstnat dst-address=210.152.79.14 to-addresses=192.168.4.1
add action=dst-nat chain=dstnat dst-address=210.152.79.15 to-addresses=192.168.5.1
add action=dst-nat chain=dstnat dst-address=210.152.79.16 to-addresses=192.168.6.1
add action=dst-nat chain=dstnat dst-address=210.152.79.17 to-addresses=192.168.7.1
add action=dst-nat chain=dstnat dst-address=210.152.79.18 to-addresses=192.168.8.1
add action=dst-nat chain=dstnat dst-address=210.152.79.19 to-addresses=192.168.9.1
add action=dst-nat chain=dstnat dst-address=210.152.79.20 to-addresses=192.168.10.1
add action=dst-nat chain=dstnat dst-address=210.152.79.21 to-addresses=192.168.11.1
add action=dst-nat chain=dstnat dst-address=210.152.79.22 to-addresses=192.168.12.1
add action=dst-nat chain=dstnat dst-address=210.152.79.23 to-addresses=192.168.13.1
add action=dst-nat chain=dstnat dst-address=210.152.79.24 to-addresses=192.168.14.1
add action=dst-nat chain=dstnat dst-address=210.152.79.25 to-addresses=192.168.15.1
add action=dst-nat chain=dstnat dst-address=210.152.79.26 to-addresses=192.168.16.1
add action=dst-nat chain=dstnat dst-address=210.152.79.27 to-addresses=192.168.17.1
add action=dst-nat chain=dstnat dst-address=210.152.79.28 to-addresses=192.168.18.1
add action=dst-nat chain=dstnat dst-address=210.152.79.29 to-addresses=192.168.19.1
add action=dst-nat chain=dstnat dst-address=210.152.79.30 to-addresses=192.168.20.1
add action=dst-nat chain=dstnat dst-address=210.152.79.31 to-addresses=192.168.21.1
add action=dst-nat chain=dstnat dst-address=210.152.79.32 to-addresses=192.168.22.1
add action=dst-nat chain=dstnat dst-address=210.152.79.33 to-addresses=192.168.23.1
add action=dst-nat chain=dstnat dst-address=210.152.79.34 to-addresses=192.168.24.1
add action=dst-nat chain=dstnat dst-address=210.152.79.35 to-addresses=192.168.25.1
add action=dst-nat chain=dstnat dst-address=210.152.79.36 to-addresses=192.168.26.1
add action=dst-nat chain=dstnat dst-address=210.152.79.37 to-addresses=192.168.27.1
add action=dst-nat chain=dstnat dst-address=210.152.79.38 to-addresses=192.168.28.1
add action=dst-nat chain=dstnat dst-address=210.152.79.39 to-addresses=192.168.29.1
add action=dst-nat chain=dstnat dst-address=210.152.79.40 to-addresses=192.168.30.1
add action=dst-nat chain=dstnat dst-address=210.152.79.41 to-addresses=192.168.31.1
add action=dst-nat chain=dstnat dst-address=210.152.79.42 to-addresses=192.168.32.1
add action=dst-nat chain=dstnat dst-address=210.152.79.43 to-addresses=192.168.33.1
add action=dst-nat chain=dstnat dst-address=210.152.79.44 to-addresses=192.168.34.1
add action=dst-nat chain=dstnat dst-address=210.152.79.45 to-addresses=192.168.35.1
add action=dst-nat chain=dstnat dst-address=210.152.79.46 to-addresses=192.168.36.1
add action=dst-nat chain=dstnat dst-address=210.152.79.47 to-addresses=192.168.37.1
add action=dst-nat chain=dstnat dst-address=210.152.79.48 to-addresses=192.168.38.1
add action=dst-nat chain=dstnat dst-address=210.152.79.49 to-addresses=192.168.39.1
add action=dst-nat chain=dstnat dst-address=210.152.79.50 to-addresses=192.168.40.1
add action=dst-nat chain=dstnat dst-address=210.152.79.51 to-addresses=192.168.41.1
add action=dst-nat chain=dstnat dst-address=210.152.79.52 to-addresses=192.168.142.250
add action=accept chain=srcnat comment="BASIC NAT BEGIN" disabled=yes
add action=masquerade chain=srcnat dst-address-type=unicast
add action=dst-nat chain=dstnat dst-address=210.152.79.150 to-addresses=192.168.25.162
/ip firewall raw
add action=accept chain=prerouting src-address=210.152.80.1
add action=accept chain=output dst-address=210.152.80.1
/ip route
add distance=1 gateway=210.152.80.1
/system identity
set name="Device Translator"
/system routerboard settings
set boot-os=router-os
2

image
image1154×472 42.3 KB

image
image1106×675 32.8 KB

3

We had the same problem , did not pinpoint the issue, what I can say is not on PLC side.
We have a VPN maintained by mobile provider. Several sites have VPN access as well as broadband fiber optic. For SCADA we mainly use the VPN for security reasons.
At some moment, some PLC started to have these strange behaviour, disconnect-connect at random. Changed one PLC with newer type/firmware, no result. Not sure if the moment is related with the installation of a CRS switch in one location, to help connect with a remote location by fiber as well as suplement port count. The PLC on that location was started disconnecting so often was extremely annoying.
Anyway, after I routed the PLC through a protected forward rule by the internet router all disconnection stopped, including other sites.
I believe the VPN routing plus some bad rules in my switches are at fault but no time to check for sure.
As info system consisting of: one Ignition production server located on central dispatcher. To development server on engineering. Remote location have Mikrotik or Teltonika 3G routers for VPN, and Mikrotik routers on broadband.
PLC are CMX V17, V19(good low firmware).
If you can try to bypass the CRS, put an RB in parallel for the most troublesome PLC ones.

4

L18 PLCs only have one IP and can only ever have one. You can't extra network card on them. So they're being NAT routed to bring their public IPs forward from their private IPs. They're all just separated by one subnet.

What is odd, and leading me to believe plc problem over router problem, was that they pretty much go in age order. Where machine 1 is the oldest. It just seems odd that its just the first seven. lol

But, I have tired everything my limited networking knowledge knows what to do. On the router

5

I did not mean an extra card or a second IP, just if possible to bypass the CRS switch and to connect to your network nat-ing or forward it by another router.
In my case originally the connection from that L32E to Ignition was like this: L32E->CRS112->LHG 4G->VPN->LHG 4G->unmanaged->Ignition.
My (temporary…) fix is L32E->unmanaged->RB3011->internet->RB4011->unmanaged->Ignition.
Meaning in same panel I could connect the L32E to another switch an totally bypass the CRS and VPN( and the snake pit of rules from CRS LHG 4G VPN)

At start thought also must be the PLC, but new one had same behaviour in same spot/IP as the old. Your case may be different though.
Did you try to swap any IP or two PLC old/new for testing?

6

Can’t get rid of the CRS unfortunately. Its what prepares all theses subnets to go into our corp WAN. Through 1 to 1 NAT translation. And like I said its a good device. Or has been for me. All the other devices have very stable and solid connection.

If they could get another network card. The PLCs. I would just do that and hook the WAN to that.

7

Only difference machine to machines.

image
image1452×648 147 KB