FactroyTalk Gateway OPC Server

Ignition
1

We are using FactoryTalk Gateway version 3.90 as an OPC server with Ignition 7.9 and up until yesterday, everything was working fine. Yesterday morning, we made some fairly extensive changes and downloads and when we went back to configuring in Ignition, all of our tags had bad quality. I checked the OPC server connection and it said “connecting” so I opened FactoryTalk Gateway and it said it was running but after about a minute or so, the status in Gateway would say “stopped”, my OPC connection status in Ignition would say “Stopped” then “Faulted”, FactoryTalk Gateway would say “running” again, and then my Ignition OPC connection would go back to “connecting”. This cycle would repeat endlessly. I tried restarting Linx Enterprise, restarting FactoryTalk Gateway, Disabling and re enabling the OPC server in Ignition’s gateway, and even restarting the server where both gateways are running. Every now and then, seemingly random, the OPC server will connect and everything will be fine for a minute or two then the cycle starts happening again.
I get this fault from Ignition:

com.jniwrapper.win32.com.ComException: COM object method returns error code: 0x800706BA; The RPC server is unavailable.
at com.jniwrapper.win32.com.impl.IUnknownImpl.invokeStandardVirtualMethod(SourceFile:762)
at com.jniwrapper.win32.com.impl.IUnknownImpl.invokeStandardVirtualMethod(SourceFile:741)
at com.inductiveautomation.opccom.interop.opcda.impl.IOPCServerImpl.addGroup(IOPCServerImpl.java:64)
at com.inductiveautomation.opccom.wrapper.subscriptions.COMSubscription$Connector.internalRun(COMSubscription.java:817)
at com.inductiveautomation.opccom.wrapper.OleJob.run(OleJob.java:48)
at com.jniwrapper.win32.MessageLoopThread$ThreadSynchronizedAction.run(MessageLoopThread.java:620)
at com.jniwrapper.win32.MessageLoopThread$LoopThread.run(MessageLoopThread.java:562)

7.9.14 (b2020042813)
Oracle Corporation 1.8.0_181

From Windows Event Viewer on the server, I get the following error every time FactoryTalk Gateway stops and starts again (which is pretty consistently every 60 seconds):

Faulting application name: RSOPCGateway.exe, version: 3.90.0.21, time stamp: 0x5825b9d1
Faulting module name: RnaDaNS.dll, version: 2.90.0.172, time stamp: 0x582556b0
Exception code: 0xc00000fd
Fault offset: 0x000d2627
Faulting process id: 0x1620

I’ve checked to make sure all the RPC services are running and working and I’ve checked to make sure all the other needed services are running. The FactoryTalk Gateway and Ignition Gateway are on the same server so I am not using COM connection, I tried it and it wouldn’t connect at all. As a test, I disabled the faulting OPC server and created a new OPC server under a different name with the exact same settings, but wasn’t referenced anywhere in the project and it made and held the connection almost immediately. I’ve spoken with Rockwell Support and we were able to see all the tags come in with good quality from FactoryTalk Live Data Test Client.

I’m thinking it has to have something to do with RnaDaNS.dll file, but I can’t figure out what. If anybody has any suggestions as to what to do or try, I would be very appreciative.
Thanks in advance!

2

The extensive changes you made, were they on the Ignition side or the FactoryTalk side? What were the extensive changes (tag restructuring, UDT creation, etc…)

3

Sorry about the long delay, I’ve been meaning to update this thread with the solution.

We split a controller into two controllers for PLC memory purposes. When we did that, we took the original controller X and split it into Y and Z. A lot of the tag structures and locations were the same, just the bracketed name for the OPC path was different, so I did a find and replace to change those tags. In doing so, that included the X controller tags, as well as the Y controller tags. Which resulted in a couple thousand tags looking at the same location in the PLC. It must have caused a lot of traffic and confusion on the Gateway side of things not to mention about 5,000 reads were failing. I deleted the X controller folder in Ignition and FT Gateway resolved itself immediately.

Thanks, and sorry again for the delay!

4

Updating this again because last night we had the same issue with the same Windows Logs errors.

We found an obscure known bug in Ignition 8.2 that causes leased tags to scan at the fastest possible rate the gateway will allow and prevent them from being removed from the scan list when the gateway is restarted in redundancy mode. We noticed that the tag count on our FTGateway server had 700,000 tags on scan, up from our nominal 60,000 tags. We placed the Gateway into independent mode instead of redundant and the OPC connection became stable and tag counts were nominal.

5

Time traveling? (8.2 doesn’t exist yet.)

6

Whoops! The Ignition version we are using at the time of this post is 8.0.2; thank you for that catch.

7

That version is ancient and buggy and unsupported. Many, many bugs have been fixed since then, and you really should be on a supported sequence (v8.1.x).

2 Likes