Massive latency issues on HMI

Good afternoon,
This may be a little repetitive since this topic has been covered quite extensively in a post by @A_Power but maybe someone has a fresh perspective for me. Here is a little bit of the background information before we get into the issue and approaches taken.

We are using an AB 1756-L83ES processor with north of 220K tags for a robotic welding cell. The cell itself has 7 HMI each equipped with Ignition Edge on a Dynics HMI. Each HMI is independent with its own license and is connected to the single PLC.

We started noticing substantial latency issues early on with PP which was resolved with the addition of various tag groups to lease certain groups of tags and then directly polled groups to tags which provided values from an AOI for scripting. It's my understanding that the Ignition AB driver has issues with AOIs in the sense of how data is presented. Especially probably with AOIs that are using aliased tags that are not enableIn/EnableOut to UDTs. Well, the program standard we were given has 35 AOIs driven by aliased tags which then is passed into Ignition. The base tags are rarely referenced in the HMI due to their standard.

As soon as we started going through our operation sequences to build parts, the overloads on multiple screens range from 400%-4000% simply from changing screens. We first addressed the PLC program changing from continuous to periodic 50 MS which dropped the overload from 4,000% to around 200%. Each of the 7 HMIs were still experiencing overloads of between 200%-400% so we changed the max concurrent requests to 12 starting on 2 of the 7 HMIs. Through some kind of black magic, it dropped the overload to around 20% on those 2 HMIs. I continued to tinker with adding them into at one point 5 HMI and adjusting the number of concurrent requests since we were starting to experience lagging in the PLC to the point it made programming difficult. Ultimately, the PLC was struggling with any changes to the concurrent requests, so we put it back to the default of 2.

Somewhere along the way, there was mention of the possibility of multiple connections to the PLC. Would this be a viable option to try to help eliminate the latency issues. There was also a shameless plug for the ethernet/IP class 1 scan module (@pturmel), could this also be an option? Completely open to suggestions as I continue to dig through the forums searching so answers.

Thank you in advance!

Does this mean that each of the 7 HMIs is its own Ignition Gateway with its own connection(s) to the PLC?

Yes, that is correct. They are each essentially independent from the other HMIs outside of sharing a common PLC.

Mmm, not sure if that's a great approach, though it does help the keep leasing changes from each HMI from affecting the others.

It's no surprise that your PLC struggled at 7 * 12 = 96 CIP Connections, that's probably beyond its max capacity.

edit: or maybe not beyond the max, but I'm guessing your comms usage is 100%

We tried using a leased polled tag group where we the rate was 0 ms and the leased/driven rate was 1,000 ms so that we were only updating tags on the page and nothing in the background. It has helped in certain instances but the more we try to manually use the screens, the overload percentage grows exponentially and never returns to a percentage below 100%

Where Phil's module might help is not Class 1 comms, it's in regards to reading AOIs. He reverse engineered the AOI format and his driver can read them more efficiently than Ignition's.

If you can't change things up so the data needed by Ignition is available in standard UDTs this might be an option to explore.

Their PLC standard is non-negotiable so I have to figure out a way to incorporate in these AOIs and endless aliased tags into Ignition somehow with a single processor.

Try Phil's module. It has a trial and is basically a drop in replacement, all the OPC Item Paths are the same as Ignition's driver.

At some point here you have to come to terms that your PLC has a very limited and finite amount of comms bandwidth, and you are asking more than it can deliver. Your only options in the end are ask for less and/or ask for it slower.

Note that Phil's module won't/cannot work on Edge.

1 Like

Are you using the comm port on the processor itself or a separate 1756-EN* card?

I'm not sure this is correct.

My understanding is that the main issue with AOI tag types being read is the EnableIn/EnableOut tags are locked to "Hidden" from external access and can not be changed from that.
Edit: This was not correct, but the link is still pertinent to the conversation.

I believe these two things are contradictory unless you're going to pull some big plugs for @pturmel

Oops, missed that these were all Edge...

You'd have to use a standard, stand-alone Ignition Gateway with Phil's module and then connect all the other Edge Gateways to that one.

I will give Phil's module a shot to see if it helps. Am I correct to believe that there is a way to enable multiple connections from gateway to a PLC or is that too burdensome? I agree with you 100% that there needs to be atleast 1 more PLC incorporated. Other cells on the project have 5+ so they are not seeing the issues I am having or understand how it is possible.

For the Logix driver,

concurrent requests = multiple logical connections
multiple configured devices = multiple connections

There is little reason to configure multiple devices except to isolate subscription thrash from lease changes. Your HMIs are already isolated from each other by virtue of being on their own Gateways with their own device connection.

@lrose we are using the comms port on the processor itself.

So with the module not being available with edge and little reason to configure multiple devices since they are isolated, really the best solution here would be to add another processor.

Yeah, probably.

You could also try the FactoryTalk OPC UA server (FT Linx Gateway? I don't know what it's called)... if you happen to have a license already.

They have some proprietary performance tricks. The downside is that it's unreliable crap and would require what could be significant re-work to your projects for new OPC Item Paths, all just to see if it's better.

You would run that on some other server that all the HMIs could make an OPC UA connection to.

Alright, I will run it past the project managers to see what they have to say. I will keep the OPC UA server in my back pocket for the other cell that is 6 weeks behind. Thank you for your helpfulness.

I'd recommend a full standard Ignition gateway for this application anyways. There's (almost) no excuse for putting more than one Edge gateway in a single facility. Use a redundant pair of standard gateways if a 24/7 process.

There are probably many tags that are being read by multiple Edge gateways, pushing traffic even higher.

Ditch the Edge gateways.

Yeah, I was operating under the assumption that each HMI was "ideal" and only reading tags for that HMI, tags that are not shared by any of the other HMIs... but probably should have clarified this.

I should also mention that my driver handles lease/unlease operations much differently from the IA native driver--it doesn't re-optimize everything as items come and go, so non-leased items aren't disturbed.

It should be mentioned that any member of an array or structure that is leased, while other members of the same array/structure are not leased, is likely adding more traffic than relieving it. A detailed examination of the PLC code would be needed to provide proper advice.