Monday at 4:51 PM we upgraded from 8.0.12 to 8.0.13. The next day we noticed a lot of tags being written from Ignition that we never saw before. Today we checked our audit logs and noticed the minute we upgraded to 8.0.13, client machines started writing lots of tags. Many of these tags the audit logs show have never been written to before. For example, our tag “[default]Enterprise/Burleson1/Extrusion/Line1/Ext125/PLC/Dosing/RateChecksInPLC/BL4/RateRatio” is intended to display a measured value. But this morning every client that displays that value writes it back to the PLC every time it changes (every second). We have not made any changes to the components that display the tag (other than to upgrade to 8.0.13). This is just one tag of hundreds that we are having written by client machines that used to only read these tags. In some cases the clients are writing zeros to all the tags and it is wiping out of PID scaling and tuning parameter and taking the line down.
If there is not a quick fix for this we will need to roll back. How hard is it to roll back to 8.0.12 from 8.0.13? Can a 13 gateway backup be restored on a 12 gateway?
Do you have bidirectional enabled on all those tag bindings?
Yes, the bindings are bidirectional, but just to the UDT, not to the subtags that are being written to.
For example, I have a UDT called PIDE (for the Allen Bradley PIDE instruction). And I have a template that shows a few things (like PV, SP, and CV) via a bidirectional binding for the template to the UDT. The template uses a bidirectional binding to the SP (setpiont) so the operator can change it. But when the template loads (the screen that has the template is opened) it writes zeros to all the subtags of the UDT.
Would it be possible to attach an export of your Template to this post or send it to me directly via PM? I would like to duplicate this issue and there are multiple ways to set a template like this up.
Thanks,
Garth
I assume PM is for private message. I sent you the files via messaging on this board.
Thank you for the provided info. I still haven’t been able to replicate this issue, and have provided you a backup via private message that I am hoping you might be able to replicate the issue on (more info provided in the PM).
Thanks,
Garth
It’s not of any help now, but when you upgrade, it’s best to test it first on a different machine (see Best Practices when Upgrading)
We have had issues when upgrading a few times. Especially one time when all tag values were zeroed (still in the 7.8 series IIRC). But also where popups would work at first, but stop working when you make a small modification and save it. So we always make sure we have a backup.
Is there a tool or script example to backup all tag values? Now that I know the damage a bug can do (write zeros to all tags) I think I need to backup all tag values too.
@ggross, I will work on replicating the error in a controlled test setup, but it may be sometime next week before I get to it. for now we have disabled the bidirectional nature of the bindings to the templates and that has stopped the issue. The operators just have to open the popup windows to do do any controls (the popups have bidirectional bindings without issues).
We had made a script to browse over all tags and store the values we know to be settings (frequency, PID parameters, measurement calibrations, …) in a JSON file.
And off course, the opposite script to read the JSON file and write it all back.
Alas, with Ignition 8, we’re still busy transforming scripts like thise to the new version. And this script will need a complete rewrite.
So if you’re willing to write a script for it, I’d like to take a look at it 
I don’t know if this will help but might get you all started in parsing through tags. This is the script I use to create the CSV to replicate existing OPC Node addresses in the Programmable Simulator. While this specific script is meant to run in the Script Console, the browseTags and getAllTags are written so that they work in a Gateway Script. If you are wondering why I don’t use the system.tag.browseTag example in the help, it doesn’t work in a Gateway Script.
To use it, just set the tag_path and opc_split_string based on the description in the file and run it.
@cezipper, this would be useful with the reproduction effort to setup the values for the device in backup I sent you. You will want to take the output of this script from a production Gateway, paste it into a file and import it into the existing simulator.
GenerateSimulatorCSV.py (2.8 KB)
Export tags to a json file. Unless I’m misunderstanding; you should have the values saved in the json code. So you could restore those tags into a folder to copy/paste them in a pinch if your values get blown away.
Test it to make sure before you do the upgrade. I’m not in a place today where I have a live Ignition system to confirm but I’m pretty sure the current value in the tags is backed up when you do a tag export. I tested this on an offline system (my laptop) and it appears to do that.
I had this same issue and have submitted it as a case to Ignition. I had to roll back to 8.0.6. Popups for PID control loops were writing default values back to the tags linked through a UDT to the popup.
We have it replicated in-house and are looking at the cause. Currently it looks like attempting to open a vision window multiple times (double clicking on whatever opens it) might be part of the cause. When it occurs it looks to affect everyone though.
Thanks,
Garth
I tried to replicate it on our test setup (test PLC, MySQL server, and Ignition), I have not been able to do so. I am glad you were able to, as it is easy for us to do in our production environment. Does 8.0.14 have the same issue? We need to roll back or forward this weekend to get out of this. At this point we plan to go to 8.0.12 (as we ran on that for a day without issue, but saw the audit logs light up with writes the minute 8.0.13 gateway came on line). Let me know when you will have a test release for us to try.
8.0.14 does have the same issue as we weren’t able to replicate before release. We are working on a fix that will make it into the nightly builds in the next couple of days. The first version the fix will be part of is 8.0.15.
Thanks,
Garth
We were planning to update from 8.0.7 to 8.0.14 on the 3rd of July. Can you tell me the latest release version that does not have the issue? Can it be replicated in 8.0.12?
Thanks,
Patrick
The change that caused this issue was introduced in 8.0.13. 8.0.12 would be a safe version to go to.
Garth
Did this change make it into 8.0.15 ? My client is experiencing this issue in 8.0.14.
We found that one of the nightly releases of 8.0.15 did have this issue fixed. It looked good in our test environment, but then when we rolled it out in our live system with tens of thousands of tags, it was not making good connection to them all. We are currently on 8.0.12 and are waiting for 8.0.15 to be the recommended download before we try again.
This fix is in 8.0.15-rc1.
@cezipper, can you elaborate on the issues you were seeing with the tags? By any chance were the tags with an issue Expression tags that contained UDT Parameters in them? When adding the datatypes back into parameters, we missed that specific use case and it was broken in nightly builds until late last week.
Thanks,
Garth