[IGN-7351, IGN-7316] Ignition 8.1.25 and IEC 61850 error

Hello,

I just updated our Ignition gateways and I keep getting error with the IEC driver module.
Im running on Windows 2016 and this is what my backup gateway is showing:

I have tried to remove license / manually uninstall the module on the backup gateway and resync with reactivation but everytime I tried, it failed with the same error message.

I downloadded the file manually and tried to install it on both gateway (master and backup) and I am getting the same error without sync the servers.

I ended up uninstalling the module, because I was getting incompatibility warnings on the gateway status page.

You can just leave it uninstalled for now.

I'll talk with QA on Monday and see if there's something fundamentally broken when using Windows Server 2016 or if it works and it's not working on just on your just system for some reason.

We saw the same thing, I had no use for the module so just uninstalled it as well.

Same OS?

Yes windows Server 2016

I started getting this error after the update (8.1.17 to 8.1.25)

Unexpected error initializing redundancy session. Will try again in 1 minute.

com.inductiveautomation.metro.api.ex.TimeoutExpiredException: TimeoutExpiredException[queueId=service_GatewayRedundancyService, targetIntent=_services.invoke_|2, msgData=com.inductiveautomation.ignition.gateway.redundancy.services.GatewayRedundancyService#registerPeerNode([class com.inductiveautomation.metro.api.ServerId, class com.inductiveautomation.ignition.gateway.cluster.NodeInfo])]

at com.inductiveautomation.metro.impl.queues.StandardQueue.processTimeouts(StandardQueue.java:229)

at com.inductiveautomation.metro.impl.queues.StandardQueue.lambda$maybeScheduleTimeoutChecker$1(StandardQueue.java:211)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

spam every minute only on redundant gateway. We are on version 1607 of Windows with most recent updates.

Everything seems normal on status page...

So in trying to duplicate this issue I have been unsuccessful and Windows Server 2016 seems to be working as expected with my installation of the software. A few questions I have:

  1. Which version of Server 2016 is everyone using? I am currently using Server 2016 Standard Edition with all current patches applied.
  2. It sounds like this is impacting backup gateways. Is is affecting the master Gateway at all?
  3. Are all Gateways running the same OS?
  4. How was Ignition installed? (Installer or ZIP install)
  5. Are you running Ignition or Edge?
  6. Are there any additional programs installed on the Gateway that could be monitoring/possibly blocking a .dll loading? Because we are loading a native library, this could be a possibilty.
  7. Anything else unique about these environments that you think could play into this.

Thanks,
Garth

2016 Standard fully patched.

Non redundant gateway here.

N/A

Installer

Ignition

No but isn't it strange to be putting things in the temp directory and trying to run them from there?

Nothing that comes to mind, this is a bog standard Ignition 8 server with WebDev the primary use.

Ignition is running as a service and, in this case, we are requesting %TEMP% as the location to move the files. Because your Ignition service is running as a User, that is the location it is being stored at. If it were run as a System service, it would likely be stored in c:\Windows\temp. While not wrong, we do have a temp directory within the Ignition install that we should be targeting (and will be making a change for).

I am guessing that AppLocker is preventing DLLs from being loaded from that directory and us making that change will potentially resolve this issue. If we can get this change into a nightly release would it be possible to see if it resolves the issue by installing just the new module?

Garth

It is running as a service, sorry it is actually c:\Windows\Temp
And yes I will test a new version of the module when it is available.

  1. Standard 2016 V1607 OS Build 14393.5648
    2.The Gateway seems fine, at least no error on the module status page
  2. Yes they are running the same OS (2 gateways), they are using a domain account to run as a service.
  3. Installer
  4. Ignition
  5. SentinelOne I'm guessing? (Antivirus?)
  6. I would say IT department :wink: but it was working fine in 8.1.17 without this module.

since it looks like it is related to redundancy I saw something weird in the tag browser of System:

Us changing where the temp directory is located should hopefully resolve the issue you are seeing with the module loading. It would make sense based on everything I am reading about .dll placement and settings that can be applied.

For the other error, it looks like it is occurring when it is attempting to process the initial connection. We don't think that it should have anything to do with the 61850 module, as this would cause the peer incompatible status that you also mentioned seeing, but seeing that a connection was successful seems like there might be disconnects occurring. When you removed the module, did the com.inductiveautomation.metro.api.ex.TimeoutExpiredException error stop occurring?

Garth

The IEC61850 Module doesn't seem to install properly for us on our Windows 10 LTSC gateways:

I've tried on 2 different Windows gateways (both Windows 10 LTSC / 1809) with same result. I don't see a similar error on the one Linux gateway I tested with.

Thanks.

@malone46844, I moved your post into this existing thread, because it's almost certainly the same root cause. If you're able to help Garth work identify what's different between your system and our test systems, it would be greatly appreciated :slight_smile: - even if not, you can be assured we're aware of the issue and working on a fix.

If you are seeing the dependent library error when Ignition attempts to load the 61850 driver, the fix is going to be to download and install the latest Microsoft Visual C++ Redistributable package:

The Triangle Microworks library that we leverage has dependancies on items included in that package. Uninstalling the module is also an option if you don't have a need for it.

We are looking into what can be done to prevent this requirement, but that is the current solution. We appreciate your help in letting us know and providing information to help assist in troubleshooting the issue.

Edit: The following KB article has been posted about this issue:

Garth

2 Likes

Is the IEC 61850 driver compatible with Linux Ignition installations? Mention of Visual C++ libraries would seem to make this MS Windows dependent.

We are receiving this error when attempting to install on OEL8 Linux:

/tmp/iec61850_native2868284322401336071/libcLibrary.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /tmp/iec61850_native2868284322401336071/libTMWBaseCPP.so)

It is compatible with Linux, though we don't test with Oracle distros. You seem to be missing a library, or a minimum version of that library.

Thanks Kevin, awesomely quick response. Will look into this with our Linux admins.