OPC Dropout Error On Ignition Gateway (Opto22 Groov Epic Running CodeSys program)

Ignition
1

We have an Ignition Gateway that lost its OPC Connections to 3 separate Opto-22 Groov Epic Controllers, all running the CodeSys platform. The following was the log error message on the gateway. The OPC interfaces are used to bring the CodeSys program tags into Ignition. As far as we know, there was nothing weird about the network setup. There are about 100 devices on the static IP network. Most of the "Traffic" is ethernet/IP data between the 3 Groov Epic controllers and the 85 VSD/starters (Eaton Power Equipment). The network has been fine for a few months with no issues. We did have some phoenix contact wireless access equipment that was technically plugged in but not configured on the same subnet. That equipment may have caused the blip, but I am unsure what caused this, and am curious if this is an Opto-22 issue or an Ignition issue. We had to restart the Ignition server and re-compile/re-download the codesys programs into each controller to resolve the issue. All of the drives faulted out on field bus error so the Ethernet/IP data definitely dropped out. None of the controllers faulted but the ethernet/IP connections did. Everything looks good now after restart/re-download. Was it just a blip or is there some type of code or gateway configuration fix we could try? Has anyone else had issues with OPC tag connections dropping out on the Ignition gateway with the following messages?

We are running Version: 8.1.32 (b2023091211)

UascClientMessageHandler 30Aug2024 10:01:34 [remote=/172.168.1.200:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientAcknowledgeHandler 30Aug2024 09:56:26 [remote=/172.168.1.200:4840] Exception caught: UaException: status=Bad_TcpMessageTypeInvalid, message=unknown message type: 262245
UascClientAcknowledgeHandler 30Aug2024 09:56:26 [remote=/172.168.1.200:4840] Exception caught: UaException: status=Bad_TcpMessageTypeInvalid, message=unknown message type: 262245
...drivers.TCPDriver[AugustaCP2].TCPConnect 30Aug2024 09:20:25 Error connecting to 172.168.1.202:4840. connection timed out: /172.168.1.202:4840
...drivers.TCPDriver[AugustaGH].TCPConnecti 30Aug2024 09:20:25 Error connecting to 172.168.1.201:4840. connection timed out: /172.168.1.201:4840
...drivers.TCPDriver[AugustaCP1].TCPConnect 30Aug2024 09:20:25 Error connecting to 172.168.1.200:4840. connection timed out: /172.168.1.200:4840
DatasourceAuditProfile 30Aug2024 09:19:58 Error verifying audit log table.
Gateway 30Aug2024 09:13:20 ERROR(200): Unable to parse message.
Gateway 30Aug2024 09:13:09 ERROR(200): Unable to parse message.
UascClientMessageHandler 30Aug2024 09:12:15 [remote=/172.168.1.201:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientMessageHandler 30Aug2024 09:10:56 [remote=/172.168.1.201:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientMessageHandler 30Aug2024 09:10:07 [remote=/172.168.1.201:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientMessageHandler 30Aug2024 09:09:37 [remote=/172.168.1.202:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientMessageHandler 30Aug2024 09:08:11 [remote=/172.168.1.200:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientMessageHandler 30Aug2024 09:07:21 [remote=/172.168.1.202:4840] Exception caught: UaException: status=Bad_TcpMessageTypeInvalid, message=unknown message type: 262245
UascClientMessageHandler 30Aug2024 09:07:21 [remote=/172.168.1.202:4840] Exception caught: UaException: status=Bad_TcpMessageTypeInvalid, message=unknown message type: 262245
UascClientMessageHandler 30Aug2024 09:07:15 [remote=/172.168.1.202:4840] Exception caught: An existing connection was forcibly closed by the remote host
Gateway 30Aug2024 09:04:39 ERROR(200): Unable to parse message.
UascClientMessageHandler 30Aug2024 09:02:44 [remote=/172.168.1.200:4840] Exception caught: An existing connection was forcibly closed by the remote host
UascClientMessageHandler 30Aug2024 09:01:34
2

There's a few possibilities for this:

  • Something (router/firewall/managed switch) between Ignition and Codesys killed the connection for unknown reasons, or

  • An OS firewall issue on the Ignition gateway briefly blocked the connections, or

  • Codesys deliberately closed the connections for unknown reasons.

You report that the EtherNet/IP drives under Codesys control all died. And needing to recompile/download any Codesys items tells me it isn't anything to do with Ignition.

Restarting the Ignition gateway was probably unnecessary. I suspect the connections would have recovered just fine after fixing Codesys.

3

We are still not 100% certain what happened on this, but once we configured the codesys ethernet/IP instances with a default gateway in Codesys program (they were blank before) that matched the default gateway of the processor running the codesys code (this was also blank before, but modified to match the remote access gateway that was installed) we have not had any more OPC drop outs. There was one additional instance of this happening prior to the remote gateway setup.

2 Likes