Ignition to Siemens SICAM A8000 tag value drop issues

Ignition
1

We have been having issues where tag values from our RTU drop to 0 at random intervals. Certain tags drop at the same time, and it is for a split second. We have engaged Inductive Automation on this and troubleshooting, but since it seems like a complex issue, I am reaching out to the wider community to see if anyone has come across this before.

The drops are very quick and basically a flicker on the screen, but trends reveal the drops:

image
image706×641 28.2 KB

We have:

  • about 400 tags
  • monitored with Wireshark and can see the packet come with a 0 value
  • set poll rates to 5s
  • two servers both with a connection to an RTUs and behaviour is seen on both sets,
  • split tag groups for analogs, binaries, inputs and outputs one one server - still have the behaviour.
  • performed firmware updates on RTU
  • tried to use class based polling with AnalogInputx BinaryInputx addressing but the OPC UA Server does not list all the data points and ‘loses’ some points at times - this was done with the idea to rule out the possibility that gvi address mismatches could cause weird behaviour
  • tried using gvi addressing but problem persists
  • tried disabling most tags and monitoring for ‘drops’ and still got drops even with 4 tags enabled
  • also disabled class based polling while using gvi addressing, slowed down heartbeat tag frequency to every 4s

Planning to try next:

  • Try connecting to RTU on site eliminating as much of the network as possible
  • Try old DNP3 driver and observe
  • Connect to RTU via Kepware and observe
  • Disabling all but one tag and observe and add tags gradually again to see if a specific tag causes this

Hoping someone has seen this before and can provide other suggestions.

Thanks in advance.

2

This is definitive proof that the flaw is in the RTU, not Ignition.

3

What does Siemens support have to say? Show them the Wireshark capture.

4

Even though packets point to RTU, I wanted to be sure that Ignition is not causing this somehow if GVI addresses are incorrect - similar to how addressing affects a Modbus connection - just in case. I guess I just want to rule out as much as I can.

The company who did the RTU programming etc. did contact Siemens and suggested firmware updates which were done, but have since suggested a new test firmware which will collect logs. The plant is in operation at the moment and it is not that easy to get an outage window yet to try the test firmware and collect better logs. But we are considering setting up a A8000 with the same config and Ignition as a test environment with the updated firmware etc.