After a whole bunch of issues I have gone down the rabbit hole of what is wrong with our setup on site.
I will preface this by saying I a relatively new to all of this… so be gentle
Based on much reading(such wow) I believe I have a list of things our site could take away for a better system.
Some of them are easy to implement, whilst some of them need… significant work to realize. Especially the AOI approach in point 5
Before going too far and upsetting apple carts. I was hoping to get some oversight from the IA gods. specifically @pturmel and @Kevin.Herron - as it was some of your previous posts that have helped me form some of this picture. The last thing I want to do is find out I misunderstood something and led others astray
I hope that my list (once confirmed as correct) is beneficial to others who may end up in a similar situation with “bad coms” specifically when running Allen Bradley gear as the majority on site.
What do I mean by bad coms
High PLC MSG (Class 3) task loads
Overload situations on Ignition Device connections
General network instability
Levers to pull when it comes to bad comms
Consider utilizing multiple Ignition Device Connections to 1 PLC when using a large number of tags
Increase the number of device CIP Connections (CAREFUL - do not go too high - Processor dependent!!!)
Increase Scan Mode Class Times in Ignition (OPC Polling rate)
Delete Continuous Task in the PLC - always go periodic
Do NOT read/write from an Allen Bradley AOI (Add on instruction) - instead use a separate UDT
Prime Number scan times - in the PLC (not ignition)
I am happy to share my much longer winded document if these points are correct - but for now I just want to make sure I have things right.
I appreciate any time anyone can give to this post.
#1 and #2 are functionally identical in the modern Logix driver.
#3 is always helpful.
#4 is usually helpful. If you do this, you do not need to mess with “Communications time slice” setting. Note, the latest L8x processors have a dedicated comms processor for their front ethernet port, mitigating the problem separately.
#5 You can still read/write AOIs for light duty items, like leased tags for configuration UIs.
#6 What? (Prime number poll rates is utterly useless in Ignition. I don’t understand where you’d apply it in the PLC itself.)
Items to also consider:
#7a: Never, ever, ever access aliases. Always directly read from the corresponding base tag.
#7b: Consolidate scattered tag instances into arrays or nest them in structures. The fundamental unit of optimization when polling is the tag. Multiple items won’t be in consecutive memory locations, and therefore be retrievable together, unless they are part of the same tag. If you must, you can use aliases in the PLC to retain the prior logical structure, but access from Ignition via the consolidated tag(s).
#8: Never poll across a WAN. Latency crushes polling protocols. Place your OPC/UA server local to the PLC and let OPC deliver updates across the WAN. (Or use MQTT to deliver across the WAN.)
Thank you so much for the prompt response.
If I could unpack a couple of things (just to make sure I have this right)
#1 & #2 - are identical so for the Logix driver therefore #1 is moot - the max CIP is what it is. Are there other drivers in which multiple connections are recommended?
We are currently running an L82 Processor - and have had a heap of issues
The dedicated port you speak of is going into a managed switch and straight to our ignition server
Time slice settings do not exist in this project (unless they hid them real well)
We have a EN4TR ethernet bridge module out to all field Devices (which is spec’d for the number of devices plus a comfortable margin)
Pretty much everything in the PLC a continuous task(s)
We have 99% MSG 3 load on the PLC CPU
We have a 102% overload on the Ignition Device
Disabling the device in ignition drops the PLC CPU load to 3%
We currently use a significant amount of AOI Read writing
(140+ AOI Blocks x 100 Tags per block )
We have had a ton of communication faults (specifically with some AB E300 Overloads)
I was hoping the hail mary of the continuous task → periodic task will resolve some of this… but I am not liking my chances… at this point
Because if not I am led to conclude the site standard code and UDT’s need to be revisited?
Things have worked fine on site up until this last project - but this was the biggest to date at 7000+ tags which I am thinking was a bridge too far with some of the methodologies
Thank you again so much for taking the time to review this and adding more points.
Once the conversation dies down I will finalize the list to help others
I’ve had success with the legacy ControlLogix doing so. Was helpful with an Omron NJ processor more recently.
If you are referring to the bombshell in this topic, then yes:
If you are going to be changing things in the PLC, you might also consider switching your fastest tag group (or part of it) to “Class 1” Ethernet/IP. This module was created to deal with large numbers of tags at a 100ms pace:
You would probably have even more issues with the older L7x. The L8s are not better because of a dedicated port but because of a dedicated processor that handles comms. There is no time slice in L8s to my knowledge.
Because they can’t be combined with anything else for an optimized read. To external readers, they look like a completely independent tag. Now, go review #7b. (There’s a reason those are #7a and #7b.)
Question in a similar vein - I do a lot of tag reads from UDTs. Currently I have the tag groups optimized by how often I need the data. E.g. Within one UDT, IO values might be 1 sec tag groups, configuration tags that shouldn’t change might be 60 sec groups, etc.
Does this cause a big performance hit? Might it be better to poll the whole UDT at a single fast rate (1 sec) vs this scattered approach? If I’m reading noncontiguous tags within my UDT, does it become a moot point?
So quick update for those of you playing from home.
A colleague has just pointed out something I myself missed completely.
Max Concurrent Requests (OPC Device Settings) - by default is 2.
We just upped ours to 4
The problems we were having went away. Load went right down in ignition by 80%, and by about 30% on our PLC
I see this has been mentioned before back in the day by @pturmel
Before I go screaming it’s fixed and summarizing these findings, is there any healthy Convo to be had around this?
What to do or what to check before you play with this number? Times when it is not a good idea and could have negative outcomes?
On newer firmware/models you can view the comms load from the PLC’s perspective right on the webpage hosted by the device, on some others you have use the Logix5000 status monitor tool (I forget what it’s called).
You can push concurrent requests up as long as the PLC isn’t indicating its comms load is 100%. Once it reaches 100%, or if it’s already there, there’s not much left to be gained.
