Tags periodically go stale regardless of device load or scan class

I made a post about this issue a few months back but it has been persistent and a solution is elusive. I could really use some insight from those more experienced.

I’m having connection issues with a Controllogix 1756-L72, with only a 1756-EN2TR in it’s chassis. This is running firmware v21.11 and thus uses the v21 Logix driver. The primary symptom is that ignition will update values for about 5 seconds, then the tags go stale (red overlay) and the controls freeze up. After a few more seconds, the tags begin to update again. This cycles continuously. I have two other controllers that are running v20 and use the compact/controllogix drivers. These are working perfectly with 80,000 tags but the faulty controller has only 6,000 tags on scan. I’m running Ignition Gateway version 7.7.3

On the software side of things, this is what I’ve tried:

  1. Moved all v21 tags to their own scanclass and tried different timings
  2. Tried changing the overhead slice time from 20-50%, though this controller is running only periodic tasks, and the diagnostics page shows comms are barely taxed. My v20 controller is running 80k tags without much issue, so I don’t think this is a load factor problem.
  3. Played with maximum number of concurrent CIP connections
  4. Changed timeout settings
  5. Changed stale threshhold
  6. Created a new instance of the driver, freezing occurred even with just one tag.
  7. Logix loggers do not show any errors (My ignition version does not have a diagnostics page for these).
  8. Settings DriverVariableNode to DEBUG shows errors as the tags freeze:[DriverVariableNode[PLC3_AoA] ] [09:45:37,897]: Setting [PLC3_AoA]Program_Control[23].Program_Softkeys.Softkey_12@1000ms to stale. Last update was 12169ms ago. Max age is 5000.0ms.
  9. CIPConnectionManager shows connections taken/released fairly rapidly. Is this normal?

I worked with Ignition support over the phone/remote-in a while back, but they were unable to find anything unusual in my configuration or project. I started looking for possible hardware issues.

The connection path looks like this:
1756-EN2TR >>> 9300-ENA (NAT device) >>> Unmanaged N-Tron 1080X swtich >>> Server

What I have tried with the hardware:

  1. Checked the port diagnostics, definitely showing errors. If I swap to the other port on the EN2TR, the errors begin to accumulate there. I swapped the EN2TR out for a new one, but the problems remain.
    Capture

  2. Research into FCS errors suggests an auto negotiation failure or EM interference, or even bad cabling. I rebuilt the cable ends. I ran two entirely new cables (Not in the wire rack to rule out EM).

  3. I bypassed the unmanaged switch and connected directly to a different controllogix with a EN2TR. I explicitly set both sides of the connection to 100 full duplex. I tried all possible manual combinations. I tried another unmanaged switch too. (Other devices are successfully using this switch)

  4. I bypassed the 9300-ENA NAT device by addressing the EN2TR on our SCADA IP range.

  5. Thinking perhaps it was signal degradation due to the distance of the run, I tried putting a switch in the middle of the cable.

  6. I ran a wireshark capture. I’m a novice, but there is definitely an issue due to the number of retransmissions/out of orders. Here is an example, I can post the entire capture if that would be helpful:

I am fairly confident the hardware is working as it should - some of the hardware is shared by other devices that are working fine on Ignition. I don’t have any connection issues while using RSLogix5000 for instance, so I’m leaning towards Ignition being the problem. I feel like I am getting close to the cause, but I’m at the limits of my expertise. Is there anywhere else that I can check that might explain why this is going on? I’m kind of at the point where I think I need to upgrade ignition and Java, just to rule out bugs in the driver, but I want some evidence the driver is the culprit before I take that risk.

Any help or suggestions would be greatly appreciated.

I don’t have any suggestions but thought I’d let you know your last two screenshots don’t seem to have made it (at least they don’t show in my browser, even after page reload).

Thanks, they were externally linked, I re-uploaded them here so they should be visible now

Have you swapped out the EN2TR?

Out-of-order packets in TCP mean your network is dropping packets. Your other diagnostics suggest it is not congestion. So I’d suspect a bad cable or switch. It’s also possible that something is exhausting the connections on the EN2TR. Try an EN3TR.

So a few things.

First, if the PLC is all periodic tasks adjusting the time slice won’t do anything. You should use the Rockwell task monitor tool to understand what the CPU is doing. You should also check to see if you have any task overlaps occurring, right click on the task and select the monitor tab.

Make sure the scan class is set to driven, I’ve had problems using leased in the past.

If you’re making a lot of PLC change disable the ‘automatic browse’ option of the driver to false and restart the driver manually to control re-browsing. I’ve experienced communication issues like this during commissioning of projects, the number of plc edits and subsequent rebrowsing impacted communication performance.

Does the project pass a string tagpath to the template objects and use indirection for the tag referencing? Or is it passing a UDT instance? I’m sure that would have been something IA would have pointed out.

It’s definately time to update Ignition, not saying it will solve all your problems, but it’s time.

Make sure everything is auto-negotiate. I doubt you have anything that can't auto-negotiate so don't introduce a conflict less you forget to change things back.

Dryhops,

I dug through some of my post histories and it jogged my memory, you probably already have already realized I did your original Ignition application.

Back when I commented to Laura: AB CompactLogix firmware 21 issue:

I now recall trying v21 firmware during the development stages and had a number of problems with it. I even remember setting up Kewpare to compare the results, and Kepware worked great. This led me to roll-back to v20 for the original system. Given that you’re still on v7.7, change the PLC firmware to v20. This will solve your problem and function like your other PLCs. You can then evaluate your path forward to update Ignition to a newer version, then you could evaluate updating your PLC firmware to somthing newer.

Thanks for the responses.

I did swap the EN2TR and triple checked my config. Unfortunately the same behavior occurred.

In regards to the connections being exhausted, I think I’ve been able to rule it out using the task monitor and diagnostics screen. Connections seem barely taxed, then suddenly the runtime spikes:

Paullys50, in my troubleshooting I perused all of your past posts for some clues. I made sure the scanclass was set to driven and I only manually rebrowse. Everything is back to auto-negotiate after manually overriding it didn’t help. I went pretty deep down that rabbit hole when I saw FCS errors, since this is often the cause.

If I understand your question correctly, the project uses indirect tag referencing with parameters (PLCNo, ProgNo) passed to the respective UDT. These UDTs are nearly identical to the original project using the v20 driver. However, the v21 driver tagpath syntax is different than the prior version, so I inherited the UDT tag structure, but overrode the tagpath and scanclass. Here’s an example for clarity:

The issues you experienced during development are definitely making me lean towards upgrading ignition. Seems like this firmware and Ignition version don’t play nicely. Rolling back the firmware on the PLC seems more daunting since I would need to reconfigure my UDTs and PLC program. Updating is always preferred over going backwards, but coordinating downtime can be difficult and the ‘one-way’ internal database updates make me concerned that, if things go wrong, recovery would be messy. (I’m referring to this: https://support.inductiveautomation.com/index.php?/Knowledgebase/Article/View/101/0/important-information-concerning-upgrading-to-777-and-782)

Like you said though, it’s probably time.

Thanks again for all of the help, I appreciate it!

1 Like

I’d recommend upgrading to the soon-to-be-released 7.9.11 when you can.

Actually, click on one of the valves on a mimic screen, and look at the template properties.

In the example below I have two properties "valve" and "valveTagpath". The valve is passing the entire UDT to the valve object on the mimic (which I believe is how your project is might be setup?), and the valveTagpath is passing a string to the valve object.
image

It isn't best practice to pass the entire UDT. It puts all tags on scan within the UDT and can cause performance issues. For the most part, the valve object only requires a few of the tags within the UDT to handle animations and control functions. It's better to pass the valveTagpath as a string, and use indirect bindings to get the specific tag data needed.

image

I can't remember when I learned that lesson so I'm curious if the change made it into your system. I believe you should be passing a PLC Number and a Device Number

image

And using indirect bindings like
image

The valve was just a general example, the principle should be applied everywhere including your program control objects.

Yes it seems like the entire UDT is passed to templates. The UDT itself is defined by indirect tagging, but the template property is bound to the UDT.

While I don’t have performance issues with v20, perhaps since v21 no longer permits reading directly from memory, the performance impact is dramatic. Still, wouldn’t this inefficiency manifest itself as high communications % usage on the controller?

Changing all of the templates is definitely possible, but I would need to rebind each template instance right? The fact that the UDT is a template property makes all of it’s member tags on scan, so I couldn’t, say, derive the tagpath from the UDT and then just change the template to use indirect tagging. I would actually have to unbind each UDT from each template instance, correct?

There are a lot of them :grimacing:.

Yeah, it would be tedious as you would need to change your templates then go through the mimics to pass the new string property. Would take a few days for someone familiar with the project. I personally think the v21+ driver in v7.7 just wasn't mature so how much of an impact my template binding suggestion would have on your current v7.7 v21+ system isn't known. You could create a small test project to see if you notice an improvement.

I think you're on the right path, hit me up in a PM if you want to go over additional details.