Gateway Start with Active Vision Clients = Bad Tag Values

We have an Ignition Gateway that also runs a Vision client for a local touch screen. The PC is set to auto boot on return of power and auto logon to Windows with a local operator account, and Ignition is also set to auto logon with the intent that the HMI will be ready to run when an operator comes to it. The PC runs Windows 7 Pro 32-bit and Ignition 7.6.4 (b2013112117) | 32-bit. We created a shortcut to start the Vision client in full screen exclusive mode. So far so good.

Then we added the Vision client shortcut created above to the Startup folder in Windows and the wheels fell off the cart. Ignition started asking for a username and password (auto logon was working before). And the Gateway started giving wrong tag values (as in most–I think all, but did not confirm–digital values were inversed–ones became zeros, and zeros became ones). This was seen on other clients connected to the gateway. Restarting the Ignition Gateway from the Gateway Control Utility resolved the incorrect tag values, which otherwise never did resolve (we left it for hours overnight and it continued to display opposite of the correct values). This behavior is repeatable, and unaffected by whether Vision clients on other PCs connecting to this Ignition Gateway are running or not at the time the Gateway PC is starting up.

Watching it boot up, logon to Windows, and launch Ignition Vision client, it looks like the problem may relate to the fact the Vision client is trying to startup on the Gateway PC before the Gateway services have finished starting up (Vision client’s initial attempt to connect to Gateway fails). If, when prompted for a username and password (failed Ignition auto logon) we just close the client, wait a minute (for Gateway services to reach running state) and re-launch the Vision client, auto-logon succeeds.

In short, we have two problems when trying to auto start the Vision client on Windows logon:

  1. Gateway tags get messed up, reading wrong values until Gateway restart
  2. Vision client auto logon fails, and it asks for a username and password

Problem #1 is especially disturbing. How can we auto launch the Vision client on the Ignition Gateway PC without triggering these two errors? I’m guessing we could do this with a batch file with some delay in it. Is there a better way?

Witman,

You hit the nail on the head. You will need to create a batch file with a delay. This will allow the Ignition Gateway to come up and be running before clients are being launched. This should correct the issues you are seeing.

Although I don’t see how this can affect or reverse tag values… that sounds very strange. Maybe you should call in and demonstrate that for support.

Thank you Greg and Kevin. On Friday when I first posted this response (which I’ve edited to remove innaccurate info posted then) I had moved the client start-up command to a *.cmd file that checked for Ignition service running status before starting clients, restarted the Ignition Server once, and all was well for that one restart so I thought we had it.

On restart Sunday we were back to the same old—sometimes. The behaviour is repeatable, but doesn’t always happen. It happened more consistently when I made the initial post above. The only things I can think of that have changed since the initial post I made above are that we’ve switched from Aero to Basic theme (thought I’d already done this… no use for Aero on an OIT), and possibly Windows re-optimized its start-up order a bit with all the restarts (seem to have as our *.cmd file lets us know while it is waiting for the Ignition service to reach running state, which it always did for a few moments initially, but now it finds the service already running by the time it gets there). At any rate, the failed auto logon and tag value havoc are not as consistent now. I even tried going back to client launcher links in the Windows Startup folder to see if that made the problems happen more (it had no affect vs. *.cmd file waiting for Ignition service running, but seemed a little worse than a *.cmd file with a large delay AFTER Ignition service running).

There are two separate issues we are encountering:

1 – Vision Client fails to auto logon. This first issue appears to be resolved if client startup is delayed till after the Gateway (not just the Ignition service) is running—and possibly till other already running clients have successfully reconnected to the Gateway (I’m not sure on this last part).

2 – Gateway tag values corruption – appears to affect all Booleans and many string values, but have not noticed any issues with real or integer values. This second issue happen when clients are trying to reconnect to the Gateway while it is starting up. It happens even if the client launchers/clients on the Gateway PC are never run (in other words, only two remote clients and one designer are connecting—FYI they are each subscribing to about 60 tags). In other words, delaying start of local clients on the Ignition Gateway PC does not resolve this issue if other clients are also connecting. When tag values are corrupted, restarting the Ignition Gateway with the Gateway Control Utility usually fixes them, though one time that did not but restoring a Gateway backup did. Also, restarting Windows on the Gateway PC without clients trying to connect resolves the tag issues. I am guessing the common thread here is that a Gateway restart does fix the issue, but it may recur again during restart under the conditions that caused it in the first place (clients trying to connect during Gateway startup). I will call support regarding this issue.

Bad Tag Values Issue Update
Some relevant details after spending some time looking at this with Robert in support (thanks Robert!):
1 - The affected tags are from a CompactLogix L33ER revision 20 PLC
2 - The fastest way to correct the tag values is to click [color=#BF8000]Refresh Browse[/color] next to the affect PLC device in Gateway’s web interface under Configuration->OPC-UA\Devices (see screenshot below). This is a much easier workaround while waiting for the root issue to be sorted out.
3 - We found the digital values reversing were coincidental; the consistent pattern observed between bits in an array of DINTS is that the bits were moved one array index (one DINT) forward when values were bad. This is in Ignition’s display; PLC values are NOT affected.
4 - Other tags (strings, integers, and likely floats are also affected when this occurs).
5 - Issue appears isolated to this CompactLogix PLC. We have not noticed any issues with the other PLC, an SLC 5/04 connected via DHRIO in a ControlLogix bridge.

Bad Tag Values Issue Resolution:
Upgrading Allen Bradley drivers (Allen-Bradley Drivers-module.modl) to version 1.6.5.2013122712-rc1 (available in the early access section of Ignition downloads) appears to resolve this issue (thank you Ignition Tech Support). I tested 10 restarts with no tag value issues (previously would happen every 1-5 restarts). Here’s the driver release note:

[quote]Allen-Bradley Drivers - 1.6.5.2013122712-rc1

ControlLogix/CompactLogix Drivers
    Fixed - Fixed bug where subscriptions can be optimized before the ControlLogix or CompactLogix
            product information is read causing incorrect tag values for arrays.[/quote]

Perhaps whether Vision clients were connecting or not was not relevant. Though I never saw tag value issues when all clients were closed prior to restart, I didn’t test that scenario a lot either.

We’re still having issue with Vision client auto logon failing (despite a 90s delay before starting the clients on the gateway). If I close the client that failed to login and then re-open it, auto logon then succeeds. I’ll report back if/when this is resolved.

Vision client auto logon failure resolution:
Increasing the client launcher startup delay from 90s after Ignition service running to 105s after Ignition service running has resolved the failed client auto logons for now. It appears the Gateway must be fully running–possibly for a little time–(not just it’s Windows service running) before a client attempts auto logon, otherwise auto logon fails.

Below is the .cmd (.bat would also work, and is required for older Windows versions) file code that checks to confirm the service is running and then waits a specified time before starting the client launcher. Note initially we would see a slight delay before Ignition service running in the script output (see first three lines of screenshots below), but Windows 7 appears to have reoptimized it’s startup order (or we saved it enough by switching from Aero to Basic theme) and the Ignition service is now consistently running by the time this script checks for it. So we don’t see the first three lines of script output anymore.

[code]:: Check for running state on Ignition service
:: and then wait another 1.25min (necessary to avoid auto logon failure as of 2014.01.02)
:: before starting the Palletizer and Wrapper OIT HMIs

@echo off

:: Check if Ignition service is running. Skip loop if it is; else echo status and begin loop
for /F "tokens=3 delims=: " %%A in (‘sc query “Ignition” ^| findstr “STATE”’) do (
if /I “%%A” EQU “RUNNING” (
goto skipDoUntil
)
)
echo Palletizer and Wrapper OIT HMIs will start automatically…
echo Waiting for Ignition service to reach running state…

:: Wait for Ignition service to start
:doUntil
for /F "tokens=3 delims=: " %%A in (‘sc query “Ignition” ^| findstr “STATE”’) do (
if /I “%%A” EQU “RUNNING” (
goto endDoUntil
)
)
:: Sleep 1s and then repeat loop
timeout /T 1 /nobreak > nul
<nul (set/p progressBlip=.)
goto doUntil
:endDoUntil
echo .
:skipDoUntil

:: Launch Palletizer and Wrapper touch screen OIT HMIs after Ignition service is running
echo Ignition service is running. Waiting for Ignition Gateway ready for login:
echo Starting Palletizer and Wrapper OIT HMIs in 105s (1.75 minutes)…
timeout /T 105 /nobreak
echo(
echo * * * Starting Palletizer and Wrapper OIT HMIs now * * *
echo If you don’t see the Ignition Client Launcher with 10 seconds,
echo use the desktop shortcut ‘Launch Palletizer OIT’ to start it.
echo If this problem persists, or client launcher fails to launch
echo HMIs, call your automation support contact.
C:\data\IgnitionClientLauncher\clientlauncher.exe scope=C project=QClg windowmode=fullscreen screen=0
:: Display last echo for a few moments before closing CMD window
timeout /T 30 /nobreak > nul
[/code]

Here’s the output of the above script while waiting for the Gateway service to start (this and below screenshots were taken with two external clients and one designer attempting to reconnect while Gateway started up):
[attachment=2]OIT Startup - Waiting for Ignition Service to reach RUNNING state.png[/attachment]
I stopped the Gateway using the Gateway Control Utility and then restarted it so we could see Gateway status next to the time delay in the command script–here is the moment the Gateway status switch to running in the Gateway Control Utility. The Gateway is showing running status 32s before we start clientlauncher. When we started clientlauncher 15s earlier this auto logon failed consistently, however that was on boot and the Gateway startup likely took longer due to other things happening at the same time (these screenshots were not taken on startup from reboot as the Gateway Control Utility would not get running soon enough to show status then).
[attachment=1]OIT Startup - Counting down 105s - Ignition Server running at 32s left.png[/attachment]
And here’s the final output as the script launches the native client launcher:
[attachment=0]OIT Startup - Final output - Starting clientlauncher.png[/attachment]