Overload with leased tags

In your Logix driver, that is never less than the desired poll rate. (I don't recall where, but you have shown its calculation from the queue dwell time.)

In my driver, that stat is the time from poll cycle start to last subscription item delivery.

I really wanted to remove Device::getSamplingMetrics and let each driver implementation decide how to best do these diagnostics, but didn't really get around to it in time. Most of ours would probably still use the SamplingMetrics concept but I'd rather not force it on third parties (it is an Optional, though...)

Plus that would put Device even closer to being nothing but a Milo AddressSpace :slight_smile:

I switched to SamplingMetrics when I dropped the driver adapter, but it doesn't expose the actual sampling interval as a value I can supply. The Web GUI does its thing to fabricate it. My actual OPC diagnostic nodes supply precisely what I want.

I'm also not yet competent enough in react and with your gateway web libraries to make my own stats display with histograms. (That would be a nice example....)

I have been evolving the Device interface such that if you are mindful about your implementation, you could build something that was ~90% common code, 5% Ignition Device module code, and 5% "plugin to some Milo-based OPC UA server" code...

2 Likes

Oh, yes! I'm paying attention.

2 Likes

Looks like the programmer embedded almost EVERYTHING in every section in one giant UDT. The chances of getting him to change that are close to zero at this point.

For reference, this is a conveyor system, broken out into identical sections. Each section has it's own program, and there's a main program-scoped UDT in each section.

So from what I'm reading, it looks like the best thing for me to do at this point is to make a tag group for each section (kind of like what I'm doing now), and either make each one a direct or a leased at a specific rate.

I only have a few tags in the direct group at this point. I used that group for alarms only. If I put everything in a leased tag group, will the alarms still "work"? I guess what I'm asking is does having an alarm set up on a tag automatically "lease" that tag at that rate? OR would my alarms just not work because there's no other bindings on them?

Or, do I need to just put every section in its own direct tag group for the sake of the alarms?

:+1:
Outside of the UDT access suggestions above (I.e. for AOI & base tag reads):

Try to set all of your tags to poll as slow as possible / as needed. If history is not enabled, no alarms, and tag values are not displayed on an active screen, then there’s little reason to poll them. Those ā€˜scaling and configuration’ tags (that are only shown within a popup) are good candidates for leased tags with a 0s base rate.

If there are periods of maintenance (or when certain processes are running) where 250ms is needed, you can look in to Driven Tag Groups for those periods.

According to Leased Tag Groups:

the driving mechanism is whether a Tag is being displayed on an open window or view: generally via a Tag Binding of some sort

So, keep all alarm tags on scan, with a reasonable approach to selecting scan rates for those.

It won't be zero at FAT. This system will have 197 ethernet nodes.

The issue is that everything being wrapped up in a big UDT means polling elements of it at different rates cancels out any gains I might make by doing this.

Sounds like your FAT will be a challenge.

Perhaps your PLC programmer needs to read this topic and the linked topics.

1 Like

That's one thing I've never understood. I work for a smaller integrator and I program both the PLC and HMI, but it feels like when you move to larger integrators, they start splitting this up and have no coordination of best practices between the 2 groups. So the PLC guy does whatever he wants but doesn't keep in mind how the front end works and you end up with a horrible experience. There needs to be tight integration/coordination between the 2 groups and not the front end guys just trying to make things work that the back end guys did.

3 Likes

I don't want to throw my programmer under the bus. He didn't know. I'm gold certified. I went through Ignition training and have spent a lot of time in the manuals, and I didn't even know any of this. I mean, don't link to Aliased tags? Don't look in AOIs directly? These are hoops I've never had to jump through before. They're not in the documentation. Unless you know specifically where to look in these forum posts, there's no way you'd have any idea IA's driver has this much difficulty with Logix processors.

These problems don't pop up with other brands of PLCs (typically) because most don't have the data structure flexibility that Logix has. It is a blessing, usually, but can also shoot you in the foot. (I don't know of any other brand that has alias tags at all.)

Rockwell's silence about some of its optimization methods doesn't help, but screwed up architectures will also hurt FT View and friends. The private optimizations only help for small numbers of tags.

I have test cases where a well-organized Rockwell PLC will deliver tens of thousands of tags on a one-second poll without overload, and cases where a few thousand tags will crush the PLC with a five-second poll. :man_shrugging:

(Good luck getting tens of thousands of tags out of any other brand's PLCs on a one-second poll.)

You really should have your PLC programmer read these topics.

Regardless of the Ignition based complications of having a single large UDT, there’s reasons that it’s bad practice in just Rockwell systems.
The first one that comes to mind is that it removes your protection from bad copy blocks. If you are doing online changes on a Logix PLC and you typo a Copy command, you can accidentally copy an entire array instead of a single element. The protection in this is that copy cannot cross out of a udt, but if you put all of your tags in one udt, it will overwrite everything with no limits. Especially if you were working with strings like barcodes, arrays of strings will wipe massive chunks of tags very fast.

2 Likes

I wouldn't call it throwing him under the bus but more of a lesson that needs learned. If it were me, I'd rather know that I need to fix things now rather than at the FAT or on site when you're getting horrible performance and it becomes a necessity that you're rewriting a program in front of your customer.

1 Like

For sure, you don't find out about these kinds of things until you step in it in typically. I learned about this on a large scale project that was having similar problems to what you're experiencing. It's one of the biggest hurdles that divides successful large scale Ignition projects from smaller scale ones. You don't need to know how to optimize communications on a small scale project.

This project forced your team to learn about this scalability hurdle and now you will be able to build more optimized projects in the future.

Good points.

I was thinking about it, and really, the only thing I need to read separate from anything else are the alarm tags. Everything else can be leased and read at the same interval. Currently I think my biggest problems are:

Fault bits embedded all over the place in the Mega UDT we have in each program, so to get the faults to work I'm basically forced to scan every single tag in a direct tag group.

There are SOME I/O points in each section (outside the mega UDT) that are aliased.

Do you think it would be advantageous for me to create separate fault UDTs just for Ignition, where I have one or two UDTs that are JUST fault bits and nothing else, and then map the existing faults from the logic in an "Ignition mapping" routine?

Regarding the aliasing, I can probably have that replaced with logic mapping as well, pretty easily.

Yes, this would be ideal.

If there are bunches of the alarms in various places, you could also use one or more aliases point at them. Then scanning those aliases separately won't trigger bulk UDT reads at that pace.

But keep in mind that leasing many scattered UDT members can trigger reading the whole UDT. I suspect you will have to break up the overly large UDTs before you even get the leasing to work well.

I think I'm going to do some hacking and slashing of my programmer's program today and check the results. I don't think I can totally get rid of the UDT but there are some areas where it could be split up and then fixed in the logic with a few strategic search/replace commands.

I think I'm on the right track so far.

I've been going through the PLC program and created several UDTs filled with just collections of Booleans. I created mapping routines to point to the Faults inside the main "big" UDT. Ignition looks at the "simpler" Fault UDTs, which are just contiguous booleans in memory now.

Also, part of the Fault data type in the PLC has a string containing the Device ID, which I'm using as External Data for my alarm messages. Because of this, the strings resided in mega UDT as part of the fault. I changed my equivalent Ignition UDT to make the Device ID a memory tag, so that cuts out ~120 82-character strings we are no longer polling for every second. The Device ID's are "set in stone" I am assured (we are building this system very cookie-cutter). If something does have to change, it'll just have to be changed in Ignition instead of the PLC program. Probably where it should have been in the first place.

There were a few other strings that provided some system IPs and informational stuff they want in the PLC for...reasons. I created a Driven tag with a condition of 1=1, and one-shot. My theory (which so far seems to be correct) is that these will only read those string values ONCE whenever the tag provider starts/refreshes. At any rate, it is not very often, and I don't need it to be. My values last updated around 1.5 hours ago, which is fine.

I've also reduced the number of sections in the system to 2 and once I get those working again, I'll start expanding out and keeping an eye on the capacity. When I began the changes to Ignition, my PLC Class 3 communications was pretty much pegged at 100% all the time, and my Overload on my main tag group was sitting at about 10%. I was polling everything at 2000 ms on account of everything being in the big UDT.

Now, I have two tag groups per section, a direct tag group for alarms, and a leased tag group for everything else. The leased tag group has a base rate of 30 seconds, and a leased rate of 2 seconds. I created a set of these for each section and one for the "main" stuff.

After spending the day going through and pointing the Ignition alarm tags to the "ignition friendly" plc tags, I'm happy to report that my Class 3 comm load is currently bouncing at around 6-25%, and my overload on all tag groups and rates is 0% most of the time. I'm not done changing things over, but I have noticed that as the day went on, my comms loading gradually got better.

I'm waiting for the other shoe to drop and have this all come crashing down, but this is by far the best performance I've been able to get from this app so far. I'm sure I'll look back and be glad I put the time in now, and not at FAT, and we have some very valuable info, now. If this works.

5 Likes