I’ve got a system configured where a machine has a partReady flag and there seems to be a rare race condition where my event doesn’t fire. The tag is on an OPC server running on an Omron PLC and the event/script is setup as a Gateway Tag Change event.
At first I thought it was cause we missed the value of the tag dropping to 0 since that would occur fairly quickly, which it could still be the case, but if it had raised back high I would have expected the timestamp on the OPC tag to have updated. Is that a correct assumption?
If this isn’t the case what else could potentially be done to ensure that this event fires? I am already subscribed to the tag at 250ms. I’m thinking I could potentially push that a little further since we are only monitoring 2 tags that quickly and only 70 tags total from the PLC. Or are there other knobs to turn on the OPC server to make this more reliable? Unfortunately I have no control over the Omron side, so hoping we can improve the reliability on the ignition side.
Probably will reach out to support in the morning to pick their brains too. Unfortunately its really difficult to reproduce and only has happened a couple times in the many hours the machine has ran.
This could be what’s happening, but it depends on implementation details of the OPC UA server that only Omron could answer.
The the server implementation of OPC UA monitored item sampling is truly sampling based internally then it’s possible that it never sees the transition to 0 and back, no value change delivered, no timestamp to update.
One thing that might help is that in 8.1.10 a Tag Group setting called “Include Timestamp-Only Changes” was introduced, which when set will tell a server that we’re interested in changes not just in value or quality but also timestamp. Again, though, whether this works depends on the implementation of the server…
So… in conclusion, a few recommendations:
enable this timestamp setting on the tag group
historize or log the tag value to see if you’re getting the transitions to 0
make sure you check the missedEvents flag if this is a tag event script
Just to emphasize this: That signal must stay off long enough for the OPC implementation to see that it is off. Since timing can vary, the only certain way to capture such events is with a handshake, with code on both ends to enforce it.
The next most reliable method is to not use a boolean. Use an increment-only small integer that changes to signal part ready. (I would have it skip zero when it rolls over, too.) That kind of change is much less likely to be missed on the OPC side.
I think you need to insist on more robust programming on the Omron side.
Increasing the sampling rate works great until it doesn’t. You are just making the problem more rare.
I’ve done a handshake with integrators where they turn on a boolean but I am in charge of turning it off. If the Ignition Gateway is down they don’t get to proceed.
That is almost as bad. There's a race window after you turn it off where it turns back on and your polling never sees it off. Don't write any particular tag from two directions.
If the PLC is waiting for you to turn it off, and that is the only logic remaining, the PLC can turn it back on before OPC polling sees that you succeeded turning it off.
If the process in the PLC is complex, natural delays might have the race only show up when comms are disturbed, so you might have never experienced it. And then it happens and you can’t explain it, because you don’t believe what you are doing is “wrong”.
Yeah, definitely would have preferred to have a more robust handshake. There is some level of handshaking there, but the speed and timing aren't really setup for OPC UA even though they swear this works for all their other customers...
At some point the sampling rate should get less than the speed of the machine and hopefully eliminate it. I just don't know how much further I can push it before I start failing to reach that rate from the OPC server.
Definitely agree, and thats how we usually set it up, unfortunately we had a third party involved who provided a sorta 'Off the shelf' product this time and getting any modifications was worse than pulling teeth
Its not really an Ignition problem, it falls more on OPC-UA and the device programmers. If the PLC writes back before a sample occurs you basically get the issue I'm running into.
Thanks for the help everyone, gonna poke around at some of the settings called out earlier and see if we can reduce it. We also brainstormed maybe pulling in the proc sensor into the handshake logic since that should be a much slower pulse when the new part arrives.
That would violate the specifications for OPC subscriptions. Writes are required to be separate. So no, they cannot. This phenomenon happens with all OPC servers I'm familiar with, not just Ignition's internal server. (@grietveld is using a PLC native server with this problem.)
If a PLC doesn't hold a signal long enough for OPC polling to ensure changes are seen, it is not Ignition's fault. There's no magic wand to wave, and ignoring this kind of specification problem doesn't make it go away.
I have seen interaction between Ignition and the PLC’s take up to 5 seconds. Between switch, server and Ignition delays along with plant wifi networks you have to expect this situation will arise. That is why I have delays built into the PLC’s and or handshakes to verify that Ignition is still online and the transactions are complete to satisfy traceability requirements. It does not happen all the time but when it does the PLC knows there is a delay and will wait.
