Seeing significant delay in the read back value of a setpoint after a user inputs a new setpoint

Seeing significant delay in the read back value of a setpoint after a user inputs a new setpoint often in the range of 10-15 seconds. We have two setups to Rockwell controllers - one goes through and FT Linx server (a solution Rockwell had suggested) another through the logix driver. In both cases lag is barely noticeable with a test set of tags(around 10 or so). When we put in the full set of tags (around 150k) we see it slow down, revert to the old value at the fallback time, and go to the correct value 10-15 seconds later. We were able to see that the controller is receiving and updating the value quickly using Studio 5000, but that value is not getting back to the HMI. Our final design system will have around 750k tags

Across how many PLCs?

How many tag providers?

How many devices?

What poll rate for your tag groups?

Are you doing direct poll or leased tags?

Is there a reason you need Read after Write, which is what it sounds like you have configured.

Have you increased the connections and CIP settings on the logix driver?

Are you reading AOI tags?

What model/firmware of logix?

What’s the load % on each devices status?

Still very new to Ignition and design side of SCADA

We have multiple gateways across a few sites but I will pick the laggiest:

PLCs: we want to run off a redundant pair

Tag Providers: 8 Realtime Providers, 20 Historical

Devices: I just see the Logix Driver. We have 2 OPC connections, one of which is the AWS hosted Linx Gateway

Subscribed/Leased for the most part but we put individual tags in polling

CIP Size and Connections: Since we are running through the FT Linx Gateway, we didn’t change these

We had disabled Read after Write since we read a about a race condition occurring

We are reading AOI tags

ControlLogix L85EP

Logix Driver is at 12% but we are not really using it

Reading AOI tags is a big no-no. Hugely inefficient and will drastically affect the performance of the logix direct driver. I’m not sure of its affect on using the Linx server, tho. I’ve always used the Logix driver. Current install is 100k tags and loads are at 0% with direct driver. Not sure why Rockwell would have recommended using Linx server instead.

Money/to make a sale. But also their driver should be more efficient in theory but I've personally never used it.

Make sure all tags, UDTs, and parameters have external access sweet to either read only or read/write. This includes local tags inside the AOI. A single tag set to None will break efficient comms to that entire AOI structure, so just change anything that is none to read only.

Are you trying to read all the PLC tags via a cloud OPC server? If so, I wouldn't expect this to be fast either. Add long as you don't need faster than 1 sec updates, use a local Ignition instance or instances depending on redundancy and load balancing requirements to read data locally and use a gateway connection with a remote tag provider to get all the data to your cloud instance or wherever your main gateway lives.

You'd be better off using the Ignition drivers as they provide a tremendous about if diagnostics and metrics data that you're missing out on by using FTLinx. If you are reading a lot of AOIs, but @pturmel's Logix driver to get even more efficient comms than the built in driver and it's a fraction of the price of the FTLinx gateway.

1 Like

Is there a way to collect round trip times? We have multiple different setups and can connect in multiple ways: (Local Ignition Gateway OR Remote Gateway)→(Cloud Linx OR Local Linx OR Logix)→ Controller. Ideally I would want a timestamp at each point it hits, but for initial testing even something completely contained in Ignition would give me some metrics to compare.

One thing you can do that I did to prove out some communications efficiencies (I’ll have to look and see if I still have the expression I used somewhere) was to read the timestamp on a continuously changing value with a fast update rate on the view/window with a difference in milliseconds shown on the screen between the last update and the current time. This also helps to have time synchronized on all servers/PCs, but will show you how long or how often values are actually updating even when using 3rd party OPC servers that don’t have these diagnostics.

I used this to compare KepServer to Ignition’s built-in Logix drivers about 6 months ago, which ended up exposing a bug in KepServer which was only patched in the last month or so by Kepware as they never officially supported Logix firmware v36 which was breaking their driver (Ignition fared very well and was chosen to use the Ignition Logix driver by our client).