What does this mean . Part 1

we have an out of town Master/backup application that had an ongoing issue requiring us to shutdown and restart the Ignition gateway service . This happens about once or twice a month and we need to get this resolved .
This error occurs hourly even when the plant is shut down from 11:00pm to 8:00am.


This occured yesterday.


And today before the issue, this occured:


When a master fails and the backup takes over, is the takeover a hot swap or does the backup start fresh. What I mean is the status of any Gateway DB tags or internal tags used by transaction groups sychronized between the master and the backup?

Hi,

I’ll have to have someone take a look at the first error.

The other two are problems of some sort with redundancy syncing on the backup. Does this happen regularly, or did you just notice it? Are you filtering out other messages in those screen shots?

The 2nd screen shot shows values arriving for tags that either don’t exist, or whose ids weren’t sent properly. This is strange, but not something that would bring the system down.

The 3rd screen shows a more significant problem, where an update arrived for a project record that likely didn’t exist. This would signify a more significant synchronization issue, and might trigger the system to perform a restore from the master. Since the next error 12 seconds later is the result of a system restart, I’m guessing that that’s what happened.

So, the main question is: do these redundancy errors keep happening, or is the main issue around the contrologix driver? Also, what is the exact version of Ignition?

As for your other question, only “static” values (that is, static db tags, or “memory tags” as they’ve been renamed in 7.4), and alert status, is synchronized between the nodes. The OPC & expression tags will evaluate fresh when the backup takes over.

Regards,

Thanks Colby.
We are using V7.3.1 on this install.

As far as the first error, that happens constantly one or twice an hour every day , 24 hours a day.

The other two only occurred on Monday after a power outage at the plant. We were unable to get the master running and had to shut down the master service and run the application using the backup.

We support the plant remotely using LogMeIn as we have one PC at the plant with a Mobile 4G USB stick.

This is an out of town job and I may need to head out to the plant to attempt to get to the bottom of the issues out there.
Main issue we are having is a transaction group hangs on batch completion, preventing a new batch from starting without our intervention. I beleive this is an application problem and not an Ignition or hardware problem.

First, it is always good to upgrade to the latest version of AB drive (currently 1.3.5.184).

If it still occurs, please send us the logs. The images above are cut off and I can decode the bytes to narrow in what is happening.

If i have a 7.3 install how do I get the latest AB driver without invoking the 15% upgrade protection fee ?

As it stands I have already booked myself to make a return trip to this customer on our coin . Last thing I need to hear it may be an Ignition AB driver problem.

[quote=“Curlyandshemp”]If i have a 7.3 install how do I get the latest AB driver without invoking the 15% upgrade protection fee ?

As it stands I have already booked myself to make a return trip to this customer on our coin . Last thing I need to hear it may be an Ignition AB driver problem.[/quote]

Upgrading to the latest version of 7.3 from another version of 7.3 dosen’t cost anything. Just grab 7.3.5 from the archived downloads area and install it.

That being said, make sure you’ve got more planned for your trip than just an AB driver upgrade. I don’t think that log message is going to disappear after you upgrade.

Thanks Kevin.
i beleive this install is V7.3.1. I will upgrade to V7.3.5. I realize that is no cost , but what I was getting at, if a driver in V7.4 solves an issue that V7.3 had, is seems very Rockwell like to charge for the fix.

As far as my trip, i plan on putting event traps on every key pressed and every event trigger by logging to a SQL table the event and time stamp using scripting. My feeling is operators are pressing keys in some sort of weird sequence that is causing a transaction group to hang, that eventually leads to us having to reset the Master via LogMeIn and then most of the above errors occur.

[quote=“Curlyandshemp”]Thanks Kevin.
i beleive this install is V7.3.1. I will upgrade to V7.3.5. I realize that is no cost , but what I was getting at, if a driver in V7.4 solves an issue that V7.3 had, is seems very Rockwell like to charge for the fix.

As far as my trip, i plan on putting event traps on every key pressed and every event trigger by logging to a SQL table the event and time stamp using scripting. My feeling is operators are pressing keys in some sort of weird sequence that is causing a transaction group to hang, that eventually leads to us having to reset the Master via LogMeIn and then most of the above errors occur.[/quote]

While technically that’s how the licensing works, if there were a known fix for a driver that got released in a 7.4 version we would backport it to 7.3 as well.

Also, looking at the ChangeLog for V7.3.5, there have been fixes to the AB Contrologix driver for problems accessing bits of UDTs and DINTs and other UDT related tag issues.
We constantly use UDTs and bits in DINTs , so hopefully upgarding the driver will make the first error on post Part 1 disappear.

[quote=“thechtman”]First, it is always good to upgrade to the latest version of AB drive (currently 1.3.5.184).

If it still occurs, please send us the logs. The images above are cut off and I can decode the bytes to narrow in what is happening.[/quote]

Onsite now and upgraded to latest versions and problem still persists.





attached is the log.bin.gz file from the console.
Same hex dump occured throughout the night , eventhough the plant is shut down and no activity or PLC activity is occurring till Monday morning
When I am at the plant tomorrow, i will look into the master’s PC Windows event viewer and see if any windows events are occuring at the same time as the hex dump.
I have the Internet Time Service shutdown on both PCs as I have found in the past some PCs will hang when this service triggers to get time updates.

I must make note though, that I don’t believe the error that causes the hex dump is the cause of the issues as to why I had to return to this job site. But , there is a ‘ghost in the machine’ somewhere, and I need to eliminate all possibilities . I have a theory that if I cannot find what is causing a problem, I eliminate what is NOT causing the problem. So I need to eliminate this hex dump.
logs.bin.gz (445 KB)

Just noticed there is a V7.3.6 update released the day i travelled here.
Downloaded and will install that before production starts tomorrow (Monday) morning

[quote=“Curlyandshemp”]Just noticed there is a V7.3.6 update released the day i travelled here.
Downloaded and will install that before production starts tomorrow (Monday) morning[/quote]

On the site again, upgraded to V7.3.6, issue still there:



Sorry to resurrect an old post, what is the proper etiquette?

but I never had a resolution to this problem, and this particular job site is having issues again with transaction groups having errors on triggers

Hi,

Looking back over this thread, there were a number of issues that were involved, but it looked like the main one might have been issues with the device driver, right? This would seem to make the most sense for problems with the transaction groups: if the quality of the trigger item is bad, it is not used. Does that seem to be what’s happening? The group should have messages indicating that when you look at its status.

Perhaps you could post the latest wrapper.log file from the master, so that we can re-evaluate the current state of which errors are happening, since some of the other errors in this thread should have been fixed by the last update that you did.

Regards,

[quote=“Curlyandshemp”]Just noticed there is a V7.3.6 update released the day i travelled here.
Downloaded and will install that before production starts tomorrow (Monday) morning[/quote]

At the customer’s site this weekend. Installed V7.5.7 and Errors are gone.