Very Concerning Read/Write Issues. Important information for all users

Ignition
21

The first question to be answered in any HMI/SCADA command control design is:

"What if the HMI/SCADA crashes or communication with PLC fails?".

Above all, there must be an emergency stop button near the operator control panel.

I would like to mention an incident where a senseless SI implemented certain command control interlocks for a huge crane system on the SCADA station. One day the SCADA station froze, the crane grab suddenly opened up and dropped a heavy block weighing 600kg from a height of 15ft. Luckily nobody was underneath and no loss of life. The SI coughed up a big penalty and was blacklisted from the plant for ever.

I can say that there's a major flaw in your RUD hardware safety design which is the root cause of software misbehaviour. I truly admire your honest efforts to seek an answer and save human lives. You must propose a plantwide safety audit to your management. Well done Thomas Shank.

1 Like
22

@pturmel Our PLC guy and I both agree. We're talking about designing and implementing a type of heartbeat system as we speak. We're hitting the dead-line for this project and the customer insists we use ignition...

Fortunately, we've found that after disabling Push-to-update and changing it to Notify-to-update, as well as putting in some PLC code that uses OTU's with a reset/timer configuration to unlock all buttons after 4 seconds. We have had no reports of buttons sticking in the PLC or on the HMI. However due to the complications of the server hop, and lack a fluidity between HMI->server<-PLC we can't say we'll be using ignition for similar jobs until the server problem is dealt with. Which is a shame, they're beautiful HMI's and the software is a bunch of fun to work with.

My fairly uneducated question now is, why are we unable to use third-party software for creating the comms between HMI and PLC with ignition. Such as RSLinx for example. I feel like having an actual server is only relevant when you're collecting information from various HMI's and storing it.

We have plenty of E-stops, light-curtains, Safety gates, LOTO, etc. All around the line. If anybody is in a cell working with the robots, nothing will move.

The problem with the RUD was an example for what was happening in the HMI/PLC. We managed to talk to the guy who hurt his finger. He noticed it opened on his finger while he was working on the placement of the magnetic strip at the bottom of the door, and that it retracted as soon as he adjusted the magnetic strip. My guesstimate for this incident is that the door had a safety on keeping it from opening when someone previously pressed it, and when he began to move the mag-strip it broke the safety and the door retracted. Which is really inconvenient because we have light-curtains, that once they're broken, everything stops. He was working in a really odd position just outside of our many safety measures and the coincidences lined up. Which is dangerous because this still wouldn't have happened if things in the HMI or PLC weren't sticking.

Thanks for all the responses guys we appreciate the input.

2 Likes
23

I agree with your customer.

I guess you'll never use it again, because it isn't a server problem. FactoryTalk ViewME's user manual documents that a PanelView's Momentary Pushbutton shall send one value to its connection when pressed, and another value when released. Typically "1" and "0" for a normally open pushbutton. (See page 286 of ViewME's user manual.) Ignition's momentary pushbutton works exactly the same way. In either product, any disruption to the touchscreen or communication path when released will cause the release value to never reach the PLC.
It's a shame that someone chooses not to understand this.

You can use RSLinx, via the OPC-COM module. But since it isn't a server problem, that won't fix anything.

Sorry if this sounds harsh, but whoever designed a circuit/actuator combination that caused motion when a safety interlock opened should be fired. So too with a system architect that doesn't account for communication failure in their designs.

11 Likes
24

If the operator is working in a danger zone, nothing should move, regardless of HMI comms! You need to do a full risk assessment of this machine and make sure everything is safe, it clearly is not right now. This thread is extremely worrying as it sounds like people are doing jobs that they are not qualified to do!
Maybe you should hire someone with safety experience to come and go over your system design, it’ll work out cheaper if you do this now, than if you send a machine out and seriously hurt someone.

4 Likes
25

We didn’t design these roll-up doors. You’re right, it doesn’t make much sense as to why it would, this is the first time its happened. Which is why I have been guessing and not stating facts, I’m not intentionally misunderstanding you… I simply don’t because of what we’ve experienced.
By breaking the safety, I mean taking it out of the picture in general, the safety wasn’t open or closed it just didn’t exist. Sorry for misrepresenting that information. I still feel like if the safety isn’t seen the door shouldn’t open but again, we didn’t design them. and trust me it is very hard to get to that mag-strip without hitting a light-curtain because you have to remove a cover right beside it, so the chances of this even happening were microscopic.

We’re troubleshooting this problem based off of past experiences. Now to me, its strange you say its not the server because with our FactoryTalk setup we never experienced this problem. Let me break down our troubleshooting.

We compared the differences between phase 1 and 2 and the only significant change is; the HMI talks through some sort of server that was setup with Windows 10 and then to the PLC (HMI -> Server -> Switch -> PLC). Compared to phase 1 with FactoryTalk (HMI -> Switch -> PLC).

We then noticed that while a button was pressed, when the OPC path was cut due to either a push-HMI update, or a Server Memory overload, bits in the PLC would remain on, or buttons on the HMI would stay pressed after release. We presumed from what we gathered that the issue revolved around the lack of fluidity between HMI and PLC, caused by this new server.

This is why we changed the update style from Push to Notify and worked some magic in the PLC logic, so far nothing has gone wrong. Once again pushing our bias in favor of the server being a factor in this problem.

Whether its the server or not, there were never any handshaking/heartbeat/scheduled I/O schemes configured in the last phase. And only in this phase, information has been unable to correlate itself between the HMI and PLC. This problem was only recognized after the server was implemented.

Its not harsh, its the truth, however questioning peoples positions on a post where the problem isn’t crystal clear should be disregarded altogether. We’re trying to find a fix not find who we should fire for being unaware. Especially in the Roll-up door scenario where any type of wiring/mechanical problems could be underlying. This is why I called it a guesstimate, you have to remember reputation is hard to gain and easily tarnished. Mistakes will always happen when new technology is introduced.

I appreciate all of your time and input.

Thank you,
Thomas Shank

26

Accyroy - he was just outside of the danger-zone. The whole cell is designed for LOTO and nothing can be in the cell unless it passes through the safety-gate or the operators light curtain. He was working just outside the light curtain in an odd-spot and removed a cover to adjust the mag-strip. He wasn’t required to be inside the cell to do the job so technically he didn’t need to LOTO. It was just one of those - one in a million mishaps, no one has seen it happen before. Safety is huge here where I work and I can guarantee all the senior designers are qualified.

Edit - The vertical light-curtain is fixed just outside of the roll-up door so if anyone enters everything stops. There is also another horizontal light-curtain just inside at the bottom that keeps the operator safe while loading parts, the door wont come down on you if you’re standing anywhere in the light-curtain. The curtains are roughly a foot and a bit off the ground and in order to access the mag-strip cover on the door you need to reach under the light curtain.

27

I think what everyone is saying (myself included) is that you've just been lucky that you have not had the same problem with FactoryTalk. Maybe just pushing out more updates with Ignition, than you did with FactoryTalk, which would up the odds of this scenario occurring? In the setup you outlined, if you were to hold down the momentary button on the FactoryTalk HMI and then unplug the network switch while holding down that button, you would end up in the same scenario.

Forgive me if I'm misunderstanding your earlier posts, but is the momentary button control really necessary? As Duffanator pointed out, it sounds like the RUD has set positions (WORK and HOME) that have sensors to detect when the door is in the proper position. If those are the only positions for the door, there should be no momentary button needed for this control. You should just have one (possibly two) boolean tag in your PLC (tag values: 1 = WORK Position, 0 = HOME Position) that the HMI interacts with via a standard or multi-state button. The PLC would then operate the door to the correct position according to these boolean tags without further input from the operator/HMI. As soon as you press the button the PLC would move the door to the proper position. No need to hold anything down. This would be the same as open/close logic on a pneumatic valve (I've never seen anyone use momentary buttons on a pneumatic valve). If a jog is necessary, maybe instead of a momentary button, you do some timing logic in the PLC that the operator can enter the number of seconds the door should move and a separate button to initiate the movement. Just a thought.

Not to be a stickler, but you can't really say he was out of the "danger-zone" if he was able to get his finger into a moving part. You should probably suggest a re-evaluation of the LOTO procedure for the RUD's if he was able to get his finger crushed by the door while working on it and was not required to LOTO. No matter what you do as a "fix" on the HMI side (Ignition or FactoryTalk) - or even the PLC - this will always be a potential safety hazard and should be addressed with a proper LOTO. You should never rely on software (HMI or PLC) as your safety (except for the rarest of circumstances). What happens if a second operator didn't know the first operator was there working on the door and pressed the button to move the door? Or you accidentally triggered the door bit to move while remotely working on the HMI? Or a million other possibilities (wire breaks, PLC failure, etc). Should always have a physical lockout device that will prevent equipment from moving/running if working on anything that could potentially hurt you (except for rare circumstances, and for all I know this plant is one of those circumstances).

Just my 2 cents,
-Will

2 Likes
28

Will, Thank you for your passive thoughts and input :slight_smile:

The roll up door does have Home/Work positions, I will talk to my senior about it.
To my understanding (not fact), momentary buttons were requested so the mechanical guys could adjust the speed and distance the door rolled up and down. Though it sounds unnecessary, my senior gave it to them to make their lives easier. He also seems to really like momentary buttons.

I understand that we must have been VERY lucky in Phase 1, which is maybe why our troubleshooting hasn't been the most accurate. We can only make presumptions with the information at our table, to my knowledge that's how you troubleshoot.

I am beginning to believe myself that this is one of those rare occasions. My PLC senior has never seen it happen in his 30 years, and there are a couple other seniors that have never seen it happen either. On top of that, the old guy who has been doing this for 20 years has never had it happen to him.

You are correct though, we were updating them more frequently due to it being easier.

One fact that I want to push across. I am aware that if you disconnect comms on FactoryTalk the same scenario will happen. I'm just confused as to why there is no effort from this Wall-like-server to correlate information from the HMI to the PLC.
Does this happen in FactoryTalk to? I would think because the only thing separating the HMI from the PLC being a switch they are able to update and talk to each other directly.

Thanks again Will,
Thomas Shank

29

I totally disagree with this statement. Safety is not an option. "It didn't happen so far.. so, it won't happen in future" attitude encourages disregard and violation of safety norms.[quote="thomasdouglasshank, post:28, topic:18583"]
One fact that I want to push across. I am aware that if you disconnect comms on FactoryTalk the same scenario will happen. I'm just confused as to why there is no effort from this Wall-like-server to correlate information from the HMI to the PLC.
[/quote]

The job of a server is to serve the payload. The payload consumer, PLC In this case, must know how to correlate the information to accomplish a task.

Despite the absence of PLC safety logic, the sticky bit issue didn't pop up in panel view because it's an embedded HMI, talking to the PLC directly in native protocol without any OPC server latency in between. Hence the polling rate is very high which has significantly reduced the sticky bit issue. But it didn't eliminate the possibility in future.

I think, Ignition Edge Panel is the best alternative for Panel view. Ignition team must look into this issue seriously and may even deploy few IE panels and do the needful without any further delay.

30

This thread is really alarmist, and has nothing to do with any bugs or failures in Ignition (as previously stated in spades). This is fully a failure of the SI team and their design. Proper PLC logic and fail-safes need to be put in place to account for any potential miscommunication between the HMI and the PLC. If you need momentary buttons to move actuators, you should hard-wire them, not rely on comms between the various layers in your system, period. And as you finally got around to discovering, sending project updates while systems are in use in a system like this is absolutely insane. People are being very PC with their critiques here, but there can be no sugar-coating this, if you don’t properly design your system from the ground up, you can kill one of your employees. Sticking up a big post alarming the whole community about a non-issue in Ignition is very irresponsible. You should ask for this post to be deleted, and start a proper post discussing the design theory for fail-safe HMI->PLC->Actuator interactions, and likely do it on a different forum, since this really has nothing to do with Ignition…did I already say that?

3 Likes
31

I wouldn't go that far. This post is valuable as an exercise in cleaning up after and/or dealing with SI's who don't understand the communications technology they are using. Such SIs are indeed dinosaurs who should only be trusted with hardwired operator devices.

9 Likes
32

This is not a new issue, Ignition has always had problems with momentary push buttons since the beginning. NO OTHER HMI (Wonderware, Intellution, RsView, FactoryTalk, Cimplicity) has this issues. Inductive’s reply has always been add logic to PLC to release momentary push buttons, I can’t tell you how many PLC codes I had to modify when switching to Ignition to ensure the problem doesn’t happen. I have been waiting foe a fix since factory pmi.

33

You may not have run into issues with momentary push buttons on these other HMIs for a variety of reasons. However, unless they use a different communications protocol, they are not immune to this issue. I have seen this issue on other HMIs and have programmed accordingly for years. This post above sums it up well and links to another with more details:

1 Like
34

Are you using SCIPTS in "standard button" component to write values to PLCs? Ignition supports various types of button components (like "one shot", "momentarily", etc) to directly write and reset button tags in PLCs! Are you using right type of button or you are using "standard push button" for navigation which is to be programmed using scripts? Little more clarity will help. I don't think Ignition guys are so naive as to leave such an obvious issue unnoticed for so many years that could be such a potential risk in plant controls!

I got this doubt as you have written "MouseReleased script was not being run" and "from the HMI script". Where is the question of scripts running unless you have used "standard pushbutton" on HMI to write on to a PLC Tag. The other buttons don't need scripting to bind them selves to PLC tags. Am I missing something here?

35

Great point…always wise words from you sir!

36

Simply not true. I've complained about Panelview Plus since it was introduced, due to the abandonment of scheduled I/O over ethernet, long before IA itself existed. (The old Panelview1000 product has class1 Ethernet/IP support.) A direct connection from a touchscreen to a PLC reduces the chances of a lost button release, but doesn't eliminate it.

1 Like
37

Not even kind of true...

1 Like
38

in my experience its true, I never have had an issue with other HMI’s in 21yrs except Ignition when it comes to momentary push buttons.

39

Ignition must take up this matter seriously. Rockwell, SIEMENS and Schneider have spent billions of dollars and millions of manhours to build the foundations of automation industry. Their PLCs, drives and embedded HMIs are working rocksolid since decades. Definitely, they know something better about embedded HMI’s than others. Ignition has a team of veterans who grew up with these giants. They must keep an open mind and address this issue.

If you conduct a SCADA hackathon, i will standup for Ignition. It will emerge as a clear winner, miles ahead of nearest competitors. I don’t have this kind of confidence on Ignition HMI’s.

40

Hi Thomas,

On a safety note if one of your employee put their hand in a machine they should lock-out that machine
and do a machine start procedure to make sure he lock the proper machine he need to access.

That doesn’t fix your ignition / plc issue but you might not have the HAZARD.

Something to think about…

regards