Bacnet on Ignition running Debian more than on BBMD and diffrent IP ranges

ekb · March 24, 2024, 12:37am

I need several network cards that are on the same IP range to talk to services on other IP ranges. I need it since I will connect to bacnet BBMD in different ip ranges.

I do this on a other site with kepware on a windows server. There I have 100 virtual network cards that I can set different BBMD hosts on. Or I can connect with the IP address of the PLC

From the ignition server I can ping every site and have connection.
But when I try to make more network cards on the ignition server (Debian) I can't ping from every network card. And I only get BBMD to work on one card.

I think it's has something to do with the routing table, but I'm not sure.
I made a test server to try to get the right network setup.

Anybody have any tips on what I can do?

caesar@testVMdebian:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether aa:67:27:eb:cc:a9 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 10.240.1.206/24 brd 10.240.1.255 scope global dynamic ens18
       valid_lft 79869sec preferred_lft 79869sec
    inet6 fe80::a867:27ff:feeb:cca9/64 scope link 
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fe:24:26:77:48:94 brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet 10.240.1.212/24 brd 10.240.1.255 scope global dynamic ens19
       valid_lft 79869sec preferred_lft 79869sec
    inet6 fe80::fc24:26ff:fe77:4894/64 scope link 
       valid_lft forever preferred_lft forever
4: ens20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 4e:7e:d5:a3:6d:44 brd ff:ff:ff:ff:ff:ff
    altname enp0s20
    inet 192.168.2.105/24 brd 192.168.2.255 scope global dynamic ens20
       valid_lft 36736sec preferred_lft 36736sec
    inet6 fe80::4c7e:d5ff:fea3:6d44/64 scope link 
       valid_lft forever preferred_lft forever

I tride pinging from the two cards on the same ip range.

caesar@testVMdebian:~$ ping -c 2 10.240.5.10 && ping -I ens18 -c 2 10.240.5.10 && ping -I ens19 -c 2 10.240.5.10
PING 10.240.5.10 (10.240.5.10) 56(84) bytes of data.
64 bytes from 10.240.5.10: icmp_seq=1 ttl=62 time=82.6 ms
64 bytes from 10.240.5.10: icmp_seq=2 ttl=62 time=53.3 ms

--- 10.240.5.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 53.330/67.981/82.632/14.651 ms
PING 10.240.5.10 (10.240.5.10) from 10.240.1.206 ens18: 56(84) bytes of data.
64 bytes from 10.240.5.10: icmp_seq=1 ttl=62 time=60.5 ms
64 bytes from 10.240.5.10: icmp_seq=2 ttl=62 time=63.7 ms

--- 10.240.5.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 60.473/62.103/63.733/1.630 ms
PING 10.240.5.10 (10.240.5.10) from 10.240.1.212 ens19: 56(84) bytes of data.

--- 10.240.5.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1021ms
pipe 2
caesar@testVMdebian:~$

tshark looking at the packages

caesar@testVMdebian:~$ sudo tshark -i ens19 -f "host 10.240.5.10" --color -c 30
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens19'
    1 0.000000000 fe:24:26:77:48:94 → Broadcast    ARP 42 Who has 10.240.5.10? Tell 10.240.1.212
    2 1.021185048 fe:24:26:77:48:94 → Broadcast    ARP 42 Who has 10.240.5.10? Tell 10.240.1.212
    3 2.045175810 fe:24:26:77:48:94 → Broadcast    ARP 42 Who has 10.240.5.10? Tell 10.240.1.212

caesar@testVMdebian:~$ sudo tshark -i ens18 -f "host 10.240.5.10" --color -c 30
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens18'
    1 0.000000000 10.240.1.206 → 10.240.5.10  ICMP 98 Echo (ping) request  id=0xd1eb, seq=1/256, ttl=64
    2 0.082620835  10.240.5.10 → 10.240.1.206 ICMP 98 Echo (ping) reply    id=0xd1eb, seq=1/256, ttl=62 (request in 1)
    3 1.001739487 10.240.1.206 → 10.240.5.10  ICMP 98 Echo (ping) request  id=0xd1eb, seq=2/512, ttl=64
    4 1.055058252  10.240.5.10 → 10.240.1.206 ICMP 98 Echo (ping) reply    id=0xd1eb, seq=2/512, ttl=62 (request in 3)
    5 1.056409916 10.240.1.206 → 10.240.5.10  ICMP 98 Echo (ping) request  id=0xe5ea, seq=1/256, ttl=64
    6 1.116872579  10.240.5.10 → 10.240.1.206 ICMP 98 Echo (ping) reply    id=0xe5ea, seq=1/256, ttl=62 (request in 5)
    7 2.058043729 10.240.1.206 → 10.240.5.10  ICMP 98 Echo (ping) request  id=0xe5ea, seq=2/512, ttl=64
    8 2.121756731  10.240.5.10 → 10.240.1.206 ICMP 98 Echo (ping) reply    id=0xe5ea, seq=2/512, ttl=62 (request in 7)
    9 2.123445020 fe:24:26:77:48:94 → Broadcast    ARP 42 Who has 10.240.5.10? Tell 10.240.1.212
   10 3.144559986 fe:24:26:77:48:94 → Broadcast    ARP 42 Who has 10.240.5.10? Tell 10.240.1.212
   11 4.168562798 fe:24:26:77:48:94 → Broadcast    ARP 42 Who has 10.240.5.10? Tell 10.240

The router info

caesar@testVMdebian:~$ routel 
         target            gateway          source    proto    scope    dev tbl
        default         10.240.1.1                                    ens18 
    10.240.1.0/ 24                    10.240.1.206   kernel     link  ens18 
   192.168.2.0/ 24                   192.168.2.105   kernel     link  ens20 
     10.240.1.0          broadcast    10.240.1.206   kernel     link  ens18 local
     10.240.1.0          broadcast    10.240.1.212   kernel     link  ens19 local
   10.240.1.206              local    10.240.1.206   kernel     host  ens18 local
   10.240.1.212              local    10.240.1.212   kernel     host  ens19 local
   10.240.1.255          broadcast    10.240.1.206   kernel     link  ens18 local
   10.240.1.255          broadcast    10.240.1.212   kernel     link  ens19 local
  100.68.234.98              local   100.68.234.98   kernel     hosttailscale0 local
      127.0.0.0          broadcast       127.0.0.1   kernel     link     lo local
     127.0.0.0/ 8            local       127.0.0.1   kernel     host     lo local
      127.0.0.1              local       127.0.0.1   kernel     host     lo local
127.255.255.255          broadcast       127.0.0.1   kernel     link     lo local
    192.168.2.0          broadcast   192.168.2.105   kernel     link  ens20 local
  192.168.2.105              local   192.168.2.105   kernel     host  ens20 local
  192.168.2.255          broadcast   192.168.2.105   kernel     link  ens20 local
            ::1                                      kernel              lo 
        fe80::/ 64                                   kernel           ens18 
        fe80::/ 64                                   kernel           ens19 
        fe80::/ 64                                   kernel           ens20 
            ::1              local                   kernel              lo local
fe80::4c7e:d5ff:fea3:6d44              local                   kernel           ens20 local
fe80::a867:27ff:feeb:cca9              local                   kernel           ens18 local
fe80::fc24:26ff:fe77:4894              local                   kernel           ens19 local
caesar@testVMdebian:~$

arp info

caesar@testVMdebian:~$ sudo arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.240.5.10                      (incomplete)                              ens19
10.240.1.1               ether   68:54:ed:00:58:b8   C                     ens18
192.168.2.1              ether   20:97:27:04:aa:21   C                     ens20
10.240.1.1               ether   68:54:ed:00:58:b8   C                     ens19
caesar@testVMdebian:~$

pturmel · March 24, 2024, 1:10am

It looks like you have some overlapping subnets (multiple IPs in the same subnet, but on different interfaces). Most outgoing packets targeting an external address in such a subnet will come from a single source IP on just one of those interfaces, unless you take steps to force the sender to select a particular one. Some IA drivers have an option for this purpose.

ekb · March 28, 2024, 12:10am

I get the bacnet device to work when I use
IP adresse 10.240.1.6

It tride adding on more IP adresse to the nettwork card
IP adresse 10.240.1.20

storm@StormCloud:~$ sudo ip addr add 10.240.1.20/24 dev eth0 brd 10.240.1.255
storm@StormCloud:~$ ip addr show dev eth0
2: eth0@if309: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:22:16:6d:3c:65 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.240.1.6/24 brd 10.240.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.240.1.20/24 brd 10.240.1.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f422:16ff:fe6d:3c65/64 scope link 
       valid_lft forever preferred_lft forever

Error from log

LocalDeviceManager	28Mar2024 00:48:21
	Error initializing LocalDevice "BacnetOLD1C". bindAddress=10.240.1.20, bindPort=47808, networkNumber=1, deviceNumber=40169

com.serotonin.bacnet4j.exception.BACnetException: Foreign registration timeout
at com.serotonin.bacnet4j.npdu.ip.IpNetwork.sendForeignDeviceRegistration(IpNetwork.java:964)
at com.serotonin.bacnet4j.npdu.ip.IpNetwork.registerAsForeignDevice(IpNetwork.java:231)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager.addLocalDevice(LocalDeviceManager.kt:190)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager.access$addLocalDevice(LocalDeviceManager.kt:37)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager$LocalDeviceRecordListener.recordUpdated(LocalDeviceManager.kt:233)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager$LocalDeviceRecordListener.recordUpdated(LocalDeviceManager.kt:206)
at com.inductiveautomation.ignition.gateway.localdb.PersistenceInterfaceImpl.notifyRecordUpdated(PersistenceInterfaceImpl.java:179)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl.doNotifyRecordUpdated(RedundantPersistenceInterfaceImpl.java:111)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl$RecordUpdateListener.recordUpdated(RedundantPersistenceInterfaceImpl.java:407)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl$RecordUpdateMessage.notify(RedundantPersistenceInterfaceImpl.java:457)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl$RecordUpdateListener.receiveCall(RedundantPersistenceInterfaceImpl.java:390)
at com.inductiveautomation.ignition.gateway.redundancy.QueueableMessageReceiver.receiveCall(QueueableMessageReceiver.java:47)
at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.dispatchMessage(RedundancyManagerImpl.java:1042)
at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl$ExecuteTask.run(RedundancyManagerImpl.java:1110)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:544)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Then I tride a new networkcard

storm@StormCloud:~$ sudo ip addr add 10.240.1.20/24 dev eth4 brd 10.240.1.255
storm@StormCloud:~$ ip addr show dev eth4
458: eth4@if459: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e6:68:99:53:e4:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.240.1.20/24 brd 10.240.1.255 scope global eth4
       valid_lft forever preferred_lft forever
    inet6 fe80::e468:99ff:fe53:e4fb/64 scope link 
       valid_lft forever preferred_lft forever
storm@StormCloud:~$

Error from log

LocalDeviceManager	28Mar2024 00:53:48
	Error initializing LocalDevice "BacnetOLD1C". bindAddress=10.240.1.20, bindPort=47808, networkNumber=1, deviceNumber=40169

com.serotonin.bacnet4j.exception.BACnetException: Foreign registration timeout
at com.serotonin.bacnet4j.npdu.ip.IpNetwork.sendForeignDeviceRegistration(IpNetwork.java:964)
at com.serotonin.bacnet4j.npdu.ip.IpNetwork.registerAsForeignDevice(IpNetwork.java:231)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager.addLocalDevice(LocalDeviceManager.kt:190)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager.access$addLocalDevice(LocalDeviceManager.kt:37)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager$LocalDeviceRecordListener.recordUpdated(LocalDeviceManager.kt:233)
at com.inductiveautomation.ignition.drivers.bacnet.LocalDeviceManager$LocalDeviceRecordListener.recordUpdated(LocalDeviceManager.kt:206)
at com.inductiveautomation.ignition.gateway.localdb.PersistenceInterfaceImpl.notifyRecordUpdated(PersistenceInterfaceImpl.java:179)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl.doNotifyRecordUpdated(RedundantPersistenceInterfaceImpl.java:111)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl$RecordUpdateListener.recordUpdated(RedundantPersistenceInterfaceImpl.java:407)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl$RecordUpdateMessage.notify(RedundantPersistenceInterfaceImpl.java:457)
at com.inductiveautomation.ignition.gateway.redundancy.RedundantPersistenceInterfaceImpl$RecordUpdateListener.receiveCall(RedundantPersistenceInterfaceImpl.java:390)
at com.inductiveautomation.ignition.gateway.redundancy.QueueableMessageReceiver.receiveCall(QueueableMessageReceiver.java:47)
at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.dispatchMessage(RedundancyManagerImpl.java:1042)
at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl$ExecuteTask.run(RedundancyManagerImpl.java:1110)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:544)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

If I use 10.240.1.6/24 on eth0 I get no error in the log, and it works.

Nothing else is using that IP adress. 10.240.1.20. I even tride other IP adresses.

Error initializing LocalDevice "BacnetOLD1C". bindAddress=10.240.1.69, bindPort=47808, networkNumber=1, deviceNumber=40169

As you can see It's many nettworks, but I think the setup on the bacnet driver should bind to the ip adresse I pick, and use that one to connect to the BBMD adress.

We don't have any issus with the devices running modbus or opc/ua on the diffrent subnets.

I'm running ignition on a proxmox server in a lxc debian image.

ekb · March 28, 2024, 9:47pm

I used tshark to see what was sent on the nettworkcards when I change between 10.240.1.6 and 10.240.1.69.
First I tried 10.240.1.6 then I change to 10.240.1.69. Red mark

When I change back to 10.240.1.6 It works again.