Very Concerning Read/Write Issues. Important information for all users

Accyroy - he was just outside of the danger-zone. The whole cell is designed for LOTO and nothing can be in the cell unless it passes through the safety-gate or the operators light curtain. He was working just outside the light curtain in an odd-spot and removed a cover to adjust the mag-strip. He wasn’t required to be inside the cell to do the job so technically he didn’t need to LOTO. It was just one of those - one in a million mishaps, no one has seen it happen before. Safety is huge here where I work and I can guarantee all the senior designers are qualified.

Edit - The vertical light-curtain is fixed just outside of the roll-up door so if anyone enters everything stops. There is also another horizontal light-curtain just inside at the bottom that keeps the operator safe while loading parts, the door wont come down on you if you’re standing anywhere in the light-curtain. The curtains are roughly a foot and a bit off the ground and in order to access the mag-strip cover on the door you need to reach under the light curtain.

I think what everyone is saying (myself included) is that you've just been lucky that you have not had the same problem with FactoryTalk. Maybe just pushing out more updates with Ignition, than you did with FactoryTalk, which would up the odds of this scenario occurring? In the setup you outlined, if you were to hold down the momentary button on the FactoryTalk HMI and then unplug the network switch while holding down that button, you would end up in the same scenario.

Forgive me if I'm misunderstanding your earlier posts, but is the momentary button control really necessary? As Duffanator pointed out, it sounds like the RUD has set positions (WORK and HOME) that have sensors to detect when the door is in the proper position. If those are the only positions for the door, there should be no momentary button needed for this control. You should just have one (possibly two) boolean tag in your PLC (tag values: 1 = WORK Position, 0 = HOME Position) that the HMI interacts with via a standard or multi-state button. The PLC would then operate the door to the correct position according to these boolean tags without further input from the operator/HMI. As soon as you press the button the PLC would move the door to the proper position. No need to hold anything down. This would be the same as open/close logic on a pneumatic valve (I've never seen anyone use momentary buttons on a pneumatic valve). If a jog is necessary, maybe instead of a momentary button, you do some timing logic in the PLC that the operator can enter the number of seconds the door should move and a separate button to initiate the movement. Just a thought.

Not to be a stickler, but you can't really say he was out of the "danger-zone" if he was able to get his finger into a moving part. You should probably suggest a re-evaluation of the LOTO procedure for the RUD's if he was able to get his finger crushed by the door while working on it and was not required to LOTO. No matter what you do as a "fix" on the HMI side (Ignition or FactoryTalk) - or even the PLC - this will always be a potential safety hazard and should be addressed with a proper LOTO. You should never rely on software (HMI or PLC) as your safety (except for the rarest of circumstances). What happens if a second operator didn't know the first operator was there working on the door and pressed the button to move the door? Or you accidentally triggered the door bit to move while remotely working on the HMI? Or a million other possibilities (wire breaks, PLC failure, etc). Should always have a physical lockout device that will prevent equipment from moving/running if working on anything that could potentially hurt you (except for rare circumstances, and for all I know this plant is one of those circumstances).

Just my 2 cents,
-Will

2 Likes

Will, Thank you for your passive thoughts and input :slight_smile:

The roll up door does have Home/Work positions, I will talk to my senior about it.
To my understanding (not fact), momentary buttons were requested so the mechanical guys could adjust the speed and distance the door rolled up and down. Though it sounds unnecessary, my senior gave it to them to make their lives easier. He also seems to really like momentary buttons.

I understand that we must have been VERY lucky in Phase 1, which is maybe why our troubleshooting hasn't been the most accurate. We can only make presumptions with the information at our table, to my knowledge that's how you troubleshoot.

I am beginning to believe myself that this is one of those rare occasions. My PLC senior has never seen it happen in his 30 years, and there are a couple other seniors that have never seen it happen either. On top of that, the old guy who has been doing this for 20 years has never had it happen to him.

You are correct though, we were updating them more frequently due to it being easier.

One fact that I want to push across. I am aware that if you disconnect comms on FactoryTalk the same scenario will happen. I'm just confused as to why there is no effort from this Wall-like-server to correlate information from the HMI to the PLC.
Does this happen in FactoryTalk to? I would think because the only thing separating the HMI from the PLC being a switch they are able to update and talk to each other directly.

Thanks again Will,
Thomas Shank

I totally disagree with this statement. Safety is not an option. "It didn't happen so far.. so, it won't happen in future" attitude encourages disregard and violation of safety norms.[quote="thomasdouglasshank, post:28, topic:18583"]
One fact that I want to push across. I am aware that if you disconnect comms on FactoryTalk the same scenario will happen. I'm just confused as to why there is no effort from this Wall-like-server to correlate information from the HMI to the PLC.
[/quote]

The job of a server is to serve the payload. The payload consumer, PLC In this case, must know how to correlate the information to accomplish a task.

Despite the absence of PLC safety logic, the sticky bit issue didn't pop up in panel view because it's an embedded HMI, talking to the PLC directly in native protocol without any OPC server latency in between. Hence the polling rate is very high which has significantly reduced the sticky bit issue. But it didn't eliminate the possibility in future.

I think, Ignition Edge Panel is the best alternative for Panel view. Ignition team must look into this issue seriously and may even deploy few IE panels and do the needful without any further delay.

This thread is really alarmist, and has nothing to do with any bugs or failures in Ignition (as previously stated in spades). This is fully a failure of the SI team and their design. Proper PLC logic and fail-safes need to be put in place to account for any potential miscommunication between the HMI and the PLC. If you need momentary buttons to move actuators, you should hard-wire them, not rely on comms between the various layers in your system, period. And as you finally got around to discovering, sending project updates while systems are in use in a system like this is absolutely insane. People are being very PC with their critiques here, but there can be no sugar-coating this, if you don’t properly design your system from the ground up, you can kill one of your employees. Sticking up a big post alarming the whole community about a non-issue in Ignition is very irresponsible. You should ask for this post to be deleted, and start a proper post discussing the design theory for fail-safe HMI->PLC->Actuator interactions, and likely do it on a different forum, since this really has nothing to do with Ignition…did I already say that?

3 Likes

I wouldn't go that far. This post is valuable as an exercise in cleaning up after and/or dealing with SI's who don't understand the communications technology they are using. Such SIs are indeed dinosaurs who should only be trusted with hardwired operator devices.

9 Likes

This is not a new issue, Ignition has always had problems with momentary push buttons since the beginning. NO OTHER HMI (Wonderware, Intellution, RsView, FactoryTalk, Cimplicity) has this issues. Inductive’s reply has always been add logic to PLC to release momentary push buttons, I can’t tell you how many PLC codes I had to modify when switching to Ignition to ensure the problem doesn’t happen. I have been waiting foe a fix since factory pmi.

You may not have run into issues with momentary push buttons on these other HMIs for a variety of reasons. However, unless they use a different communications protocol, they are not immune to this issue. I have seen this issue on other HMIs and have programmed accordingly for years. This post above sums it up well and links to another with more details:

1 Like

Are you using SCIPTS in "standard button" component to write values to PLCs? Ignition supports various types of button components (like "one shot", "momentarily", etc) to directly write and reset button tags in PLCs! Are you using right type of button or you are using "standard push button" for navigation which is to be programmed using scripts? Little more clarity will help. I don't think Ignition guys are so naive as to leave such an obvious issue unnoticed for so many years that could be such a potential risk in plant controls!

I got this doubt as you have written "MouseReleased script was not being run" and "from the HMI script". Where is the question of scripts running unless you have used "standard pushbutton" on HMI to write on to a PLC Tag. The other buttons don't need scripting to bind them selves to PLC tags. Am I missing something here?

Great point…always wise words from you sir!

Simply not true. I've complained about Panelview Plus since it was introduced, due to the abandonment of scheduled I/O over ethernet, long before IA itself existed. (The old Panelview1000 product has class1 Ethernet/IP support.) A direct connection from a touchscreen to a PLC reduces the chances of a lost button release, but doesn't eliminate it.

1 Like

Not even kind of true...

1 Like

in my experience its true, I never have had an issue with other HMI’s in 21yrs except Ignition when it comes to momentary push buttons.

Ignition must take up this matter seriously. Rockwell, SIEMENS and Schneider have spent billions of dollars and millions of manhours to build the foundations of automation industry. Their PLCs, drives and embedded HMIs are working rocksolid since decades. Definitely, they know something better about embedded HMI’s than others. Ignition has a team of veterans who grew up with these giants. They must keep an open mind and address this issue.

If you conduct a SCADA hackathon, i will standup for Ignition. It will emerge as a clear winner, miles ahead of nearest competitors. I don’t have this kind of confidence on Ignition HMI’s.

Hi Thomas,

On a safety note if one of your employee put their hand in a machine they should lock-out that machine
and do a machine start procedure to make sure he lock the proper machine he need to access.

That doesn’t fix your ignition / plc issue but you might not have the HAZARD.

Something to think about…

regards

FactoryTalk is mentioned quite a bit here, it is a very ambiguous reference as it does not refer to any particular software or hardware. PanelView is also no longer an assumed platform. I assume in this case most often FactoryTalk View ME on a PanelView Plus 6 is what is being compared. A push is not possible on this platform, the project file is replaced over a network and the unit rebooted. With SE an update can be made, but it does not take effect until the applicable screen is reopened.

In any case HMI jogging (any platform) is for machine movements outside of the protected safety zone. Jogging inside the protected area should be performed with a safety dead-man switch in most cases. Or certainly with physical I/O in others. Properly configured local PLC code is also required - the HMI is not what is controlling movements.

I’m also going to go out on a limb and assume you’re not using some custom VBA tied to PanelView buttons to control movements in your comparison case. Why would you use custom scripting on the jog buttons in this case? When you do something like this you take it upon yourself to ensure you account for all possibilities.

Even in a properly designed motion control system it is bad practice to make updates
to HMI or PLC logic while personnel are inside of a risk area.

The fact that you did not originally design/build the equipment is no excuse for improperly implementing/modifying these controls.

5 Likes

Ignoring the safety system issues others have already covered, while the reset timers will indeed fix persistent 'sticky states', 4 seconds is a long time to still be doing potential damage if the button does indeed become stuck again. This is only a bandaid 'solution' applied to dirty skin. IMO, removing all momentary PLC actions from SCADA - where SCADA is setting and resetting PLC values - is a critical step forward. Momentary buttons are a terrible hangover that unfortunately made it into SCADA after hardwired push button panels started being replaced by intelligent HMI/SCADA control systems. This type of jogging action controlled by a SCADA system relying on a comms link is fraught with trouble...

Hi All,

I may be covering ground that has already been covered, but I very rarely if ever us momentary push buttons in any SCADA, HMI application.  I am no way an expert but have had experience with Allen Bradley RSLogix 500 / 5000, Panelview Plus, Panel Builder, DTAM's (for those old enough to remember), Omron and the NB / ND screens

Not sure why that message got sent, but I will continue it now.

Basically I have had some experience with Siemens, AB, Omron, Mitsubishi, Red Lion, Citect, Wonderware and others. In the majority of cases, I use a "Latch" Push Button and turn the bit off within the PLC code. The button is only on for 1 scan. This ensures that if 'communication' is lost between the PLC and the HMI / SCADA, the device will not stay on unless the PLC code allows it. I use this method even when jogging motors etc. The bit is turned on from the screen. it stays on for 1 scan and then it it is reset. If the Push Button is still pressed, it will immediately turn back on for another scan.
This will depend on the scan direction but the Push Button from the screen turns on Bit1 and resets the Push Button then on the line above the "normally Open" of bit1 unlatches / resets Bit1. Bit1 is on for 1 scan cycle and the button is turned off.
I hope I have had a chance to help someone with my method

4 Likes

Writing bits (or anything else) from two directions is inherently racy. After writing, SCADA systems like Ignition expect to see what was written come back on the next polling cycle. There are tweaks to minimize the impact of this (optimistic writes) but they really just open other races.

So, I strongly recommend you NOT set a bit from an HMI and reset that bit in the PLC. At least, not without a relatively long dwell time. That is an anti-pattern. Instead, increment a small integer, letting the change be the equivalent of the pulse from a bit. Keep incrementing (quickly) while the button is pressed. Use a Time-Delay-OFF timer in the PLC with a short timeout to hold the pulse between increments.

(I’m not sure there’s complete round-trip way to do this reliably in Perspective.)

3 Likes