Very Concerning Read/Write Issues. Important information for all users

Ignition
1

Hello Ignition Members,

Disclaimer - All of our senior designers/programmers are qualified for their jobs where I work. These problems are described as I see them, they may not be exactly what happened. Therefor they are to be used for troubleshooting the issue only. Thank you.

We are in contact with some ignition developers and are desperately trying to fix this problem. This is a GLOBAL problem therefore it won’t be affecting just us. It may be affecting anyone who is using ignition well. Time for the community to pull together to fix this. Any probable solutions will be tested and greatly appreciated. This problem has become a HAZARD and someone in our plant broke their finger yesterday because a RUD came down on his finger due to the condition of a bit sticking in the PLC after the HMI was interacted with. It could become a lot worse than just a broken finger.

Temporary/Unreliable solution - Go to your Ignition Developer and change your Project>Properties>Client>General>Update Mode to NOTIFY rather than PUSH. This will make life a little more difficult because you will have to go to the HMI to update it. But it could save a life, so do it.
How this temporarily fixes the problem - People will no longer be holding down buttons while the project is updated.

Today our team experienced a problem with the state of a bit in a portion of our Roll Up Door (RUD) logic. The bit was sticking to one state in the PLC after the HMI button was pressed and released

With problem number #1 we thought we figured out the cause. We came to the conclusion that because the HMI was being updated, a MouseReleased script was not being run (when you update the HMI by Push rather than Notify, when an operator has their finger on the button and it updates, no release of the button is registered to the PLC from the HMI script). Therefore the bit in the PLC remains ON until manually turned off, or until the HMI button is pressed again.

List of problems (growing unfortunately, stick around for updates).

1 - We’ve noticed if you have a momentary push button with Mouse/Press and Release scripts, if the button is being held down while a live update occurs. The MouseRelease script will not play and the state of the bit in the PLC program will not change until it is either manually reset in the PLC or if the button on the HMI is pressed again after the update.

2 - (Nightmare Problem) After a lot of trouble shooting and thinking that the Button-sticking situation was related to the OPC-path being cut. We found out that buttons/indicators were sticking to a single state even if the HMI was not updated. We have buttons that send big, metal clamps from HOME to WORK positions and vise-versa. We had an operator working/adjusting some tooling, he tried to send a clamp to WORK position, and as soon as he released the WORK button the clamp flew back to HOME position. Concluding that the HOME_PB bit in the PLC was sticking even if the OPC-path wasn’t updated. (Communication, unfortunately, is poor in our plant. We are not entirely sure yet if the projects weren’t being updated when this particular problem came up). With this problem we did notice one thing significantly different, there are no mouse pressed/released scripts for the clamp buttons.
This really pushed us to think that the problem revolves around the stability of the network itself.

3 - A possibility we’ve noticed that if you exceed the MEMORY on the projects server (Project Website/SYSTEMS/Performance), tags will begin to fault out on the HMI’s. We believe this is because the OPC-server resets itself to allow new connections to be made or to free-up room in the memory (educated-guesstimation). I personally think this causes the same problem as a project update. The OPC Path is cut momentarily so the PLC doesn’t know what to do with the condition of the bit.

Frequently Updated Conclusion.
As of 6/22/2018 @1:47PM problem still not fully understood - no fix
As of 6/22/2018 @2:58PM we believe the problem to be the server hop from HMI to PLC. There needs to be a way to create fluidity between HMI->SERVER->PLC and vise-versa.

As of 6/25/2018 @9:37AM - Thank you for all of your responses. Let me make myself clear, I am not ‘blaming’ ignition. This problem does not have to do with the ignition application itself. There are no issues with dirt sticking on the screens etc. This problem revolves simply around the connectivity and fluidity between the server hop between the HMI and PLC.

No updates from Developers - we will keep you guys posted :+1:

If anyone can make heads or tails of this problem it will be everyone’s favor. This is a problem that has been persistent through out multiple plants we’re in contact with and could begin crippling the face and reliability of ignition. ANY help is greatly appreciated.

I would be thankful if an ignition forum moderator could PM me the conditions for bumping a post to the top of the list (how many times I can do it a day, etc). I would like this pinned but I know thats a lot to ask for. Thank you.

Thank you Ignition forums and Best Regards,
Thomas Shank

2 Likes
2

There’s some documentation(and many forum posts) surrounding momentary buttons – for a variety of reasons there can be no guarantee that the release event will fire in a timely manner or even that it’ll fire at all. You’re sending IO from a client, to a server, to a PLC, to a piece of hardware and any communication disruption between any of those systems will short-circuit whatever operational logic you’ve implemented if you’re designing it with the assumption that all of those events will succeed(and succeed in the correct order).

I would strongly strongly recommend shutting down any systems that are hooked up in this manner until you can do a full audit of your HMI design to ensure it is fault tolerant against the possible modes of failure endemic to networked systems.

8 Likes
3

Thank you for the response. We’ve concluded so far that the server is the underlying problem. We previously used FactoryTalk and PanelViews for these lines and we never had a problem with them. PanelView HMI’s talk directly to the PLC’s in our setups.

The only new thing we’ve implemented into this phase of the project is the server hop. Its acting like a wall that, when the connection is disrupted and information is lost, there is no attempt from the server to correct and correlate the PLC and HMI. Is there anyway to create fluidity and make this connection from HMI->Server->PLC less of a brick wall?

4

How were you publishing changes? Were you pushing updates while users were holding buttons down?

5

When you update a PanelView HMI from the FactoryTalk application, it instantly sends the update request to the HMI stopping the operator in the same way ignition does. We weren’t carrying around USB sticks and physically updating the HMI’s. However PanelViews all used a notify-like update rather than an instant push-like update from the PC.

The only difference again these updates from PanelView shoot directly to the PLC rather than hoping over a server.

6

Any chance this is occurring on a touchscreen? Years ago we came across a reliability issue with momentary buttons (FT View SE application) and the root cause was the touchscreen. IIRC if you held a momentary button on the touchscreen for too long it would time out and send a release even if your finger was still on the screen. There were other issues with sensitivity as well, so if dirt got on the touch screen it could activate something as if it were a press by an operator. I can’t remember the specific details as it was ~ 8 years ago but the solution was registry edits in Windows and/or messing around with the touchscreen settings.

I’d be curious if you have drawn the same conclusion using a touch screen and non-touch (keyboard & mouse).

7

I can assure you that a PanelView, configured with standard tag access, that loses its network connection or suffers a power problem while a momentary button is pressed, will absolutely leave the target bit in the pressed state. This is also true for every single touchscreen product on the market that uses Allen-Bradley's tag data access protocols. And true for any other request/response protocol (Modbus, GE SRTP, Siemens pg). This topic has a pretty extensive discussion of the issue.

The bottom line is that momentary buttons that cannot be allowed to suffer this kind of failure must use an I/O protocol or have some heartbeat from client all the way to the PLC. I/O protocols repeat the bit status on a schedule, and the PLC itself notices when the schedule is broken. Modern Panelviews offer DeviceNet and ControlNet with scheduled I/O, and Classic Panelviews also offer AB RIO and scheduled Ethernet/IP. Other brands (like Siemens and C-More) also offer scheduled Ethernet/IP (aka Class 1 connections).

11 Likes
8

This issue most certainly does affect PanelViews, and FTView ME/SE, and WonderWare, and just about every HMI application out there. Momentary buttons just need to be treated with absolute care. We code all of our momentary buttons on HMI’s so that the PLC is responsible for releasing them once an action is performed. This is of course a problem with “Jog” type operations, but even those are limited to some duration if they come from an HMI. The physical PB’s do not have a timed limitation like the HMI ones do.

No matter what you do on the HMI side of things, if there is any communications interruption or glitch, or even some other background application on the computer causing it to hang or pause, there is simply no way the HMI can fix it.

5 Likes
9

This can happen with any HMI system and it's unfair to blame Ignition. As a person who had worked in a hazardous smelter plant, witnessed many accidents and loss of life due to poor quality products and safety systems but luckily survived, i can fully understand and appreciate your concerns.

I recommend the following strategy which can significantly reduce plant accidents:

  1. Formulate and implement a strict safety protocol for each plant area.
  2. Implement "safety logic validation test" as part of regular PM schedule. An inexperienced PLC programmer can be absolutely blind to such issues and might have not implemented it. Experienced plant operators can easily identify those grey areas.
1 Like
10

I'll add to this that sometimes it's wise to put a time limit on physical push buttons for momentary operations too. They don't always release either (physical failure/jam).

1 Like
11

On most machines we build that involve starting moving parts we use physical buttons, sometimes with double NO contacts and 2 handed operation.

Our HMI’s are for primarily info and parameters only.

Sometimes we do use the Siemens hand held mobile panels over profinet. You can program in to detect hardware failure and again we operate on a dead man button principle.

I like the idea as using the HMI button as a pulse to trigger say a 1second timer. Once timed out it would need to see the button change state to operate again.

1 Like
12

Today, all high level programming languages have unit testing algorithms and software developers have to submit unit test reports as part of their contract. Unfortunately, such integrity test mechanism is not existing in PLC/SCADA programming where a malfunction or failure of logic can lead to accidents and loss of lives. Just wondering, is it not possible for PLC/SCADA vendors, experienced SI's and control system experts to build and supply "PLC/SCADA program integrity test packages" which can become a mandatory part of FAT?.

http://softwaretestingfundamentals.com/unit-testing/

13

Sorry to hear about this, but the HMI isn't the problem here. Every HMI can experience "stuck" bits. I've seen it happen with FT, WW, Siemens... etc. Once the PLC "sees" the command, clear it. That should become your best practice. I would also recommend a safety analysis on your "RUD", sounds like more safeguards need in place.

3 Likes
14

Its strange, we never experience any problems with buttons sticking in the PLC logic when we used PanelView. We must've gotten lucky or something with connection stability etc.
We've come to the conclusion that we're going to stop using instant-push updates with our HMI's so we're no longer disrupting operators while they're pushing buttons. As well, my dad is going to be implementing some sort of PLC logic into our PLC safety code that will reset or check the states of buttons after 2-3 seconds of being pressed.
We've got our fingers crossed that this will decrease the chances of this problem occurring again.

We will update you guys letting you know if we've reduced the problem at all.

Best regards,
Thomas Shank

15

Why wait 2-3 seconds? Once the PLC receives the command just clear, unlatch it.

3 Likes
16

[quote=“jlandwerlen, post:15, topic:18583, full:true”]
Why wait 2-3 seconds? Once the PLC receives the command just clear, unlatch it.
[/quote]Not all the of applications we use execute instantly :stuck_out_tongue:. For example, when we press a button for a pneumatic cylinder to go to Work position, it takes roughly 1-2 seconds for the clamp to close entirely. Therefore it is required that the bit stays on for at least 2-3 seconds so all of the cylinders reach their positions.

If the bit turned off instantly after release we would have problems with cylinders jigging back and forth and a bunch of other ugly scenarios with sensors not being reached.

17

So you’re using momentary buttons to do an automatic machine movement? That doesn’t make much sense… You should have HMI bits independent of the rest of the code to actually move the equipment. So, if you press the work position button, the PLC acks that by setting another bit that will actually take care of the movement and reset when either the tool gets to the correct position or times out (or a safety/alarm is active) and then resets the HMI bit instantly. An automatic movement should not require the operator to hold a button unless it’s safety related or a jog; I’m not a big fan of using soft buttons for jogs anyway and safety should ALWAYS trump an HMI. If the button is required to be held for safety reasons then it needs to be physical, hardwired and WILL stop the machine regardless of what the HMI or any other memory bits in the PLC are telling it to do.

Also, regarding the RUD that you were talking about earlier that someone got their finger in. Usually, any door that opens to allow operator access should have some kind of safety on it, either magnetic or RFID. The other option is some kind of safety curtain (would have to be small enough resolution to be “finger safe”) that would disable the door from closing if there is an obstruction in the way. Again, safety should always trump HMI or button conditions.

6 Likes
18

You are confusing command and actions. It really sounds like you guys need to evaluate things in your process. It sounds like you have potential for disaster.

I’m just trying to help, so please don’t think that I’m being critical. I was shown this stuff along the way as I really didn’t know either.

2 Likes
19

Unfortunately I'm no expert, but it doesn't make much sense to me either. However I'm not the one who developed these HMI's, I'm just the editor.

The person who designed our PLC's is far more knowledgeable than myself and I'm sure he understands the difference between actions and commands. Hes been in the business for 30 years.

To my understanding the cylinders we're actuating are not automatic. they require a momentarily energized bit to complete their Home/Work motions. This is why having a command that unlocks the bit as soon as the action is finished on the HMI would end up Jigging the clamp back and forth. For the clamp to make it to WORK position from HOME, the button must be HELD.

@jlandwerlen Please correct me if my understanding of Actions/Commands are wrong. Would help me in the long run.

20

This program structure is unsuitable for modern HMIs' momentary buttons for the reasons discussed. Either supply hardwired buttons through supported I/O modules, or explore the handshaking/heartbeat/scheduled I/O protocol options described. Consider having your PLC guy read all of this.

4 Likes