Ignition Backup Gateway, Out Of Date Sync issue

Jobin_Samuel · September 7, 2023, 7:31am

Hi There,

The Ignition version we are using is V 8.1.14

Master Sync status : Good
Backup Sync status : Out Of Date

Force Re-Sync both from Master and Backup gateway, no change
move the config.idb file and restarted the backup gateway, no change

PROD 1 log:

|InternalDb|07Sep2023 12:38:18|Error sending system backup to other gateway.|
| --- | --- | --- |
|com.inductiveautomation.metro.api.ex.RemoteException: Exception occurred on remote machine, message id='4895', error code=GENERAL_PROTOCOL_EXCEPTION [100]
at com.inductiveautomation.metro.impl.protocol.handler.AckErrorHandler.handleMessage(AckErrorHandler.java:76)
at com.inductiveautomation.metro.impl.protocol.websocket.MetroWebSocket.onMessage(MetroWebSocket.java:366)
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:70)
at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.lambda$onBinaryFrame$0(JettyAnnotatedEventDriver.java:113)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)|

PROD 2 log:

|WebSocketConnection|07Sep2023 12:38:18|-> outgoing local='XXXprd01ignt_AAAXXXvlping01-backup' remote='XXXprd01ignt_AAAXXXvlping01-master' method=Metro-RunDownloadThread: Exception occurred in the local system, message id='4895'|
| --- | --- | --- |
|java.io.FileNotFoundException: /tmp/metro/8317a818-ca23-4b51-aa9c-1df38592868e___statetransfer.gwbk (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(Unknown Source)
at java.base/java.io.FileOutputStream.(Unknown Source)
at java.base/java.io.FileOutputStream.(Unknown Source)
at com.inductiveautomation.metro.impl.codecs.FileStreamCodec.decode(FileStreamCodec.java:143)
at com.inductiveautomation.metro.impl.CentralManagerImpl.handleLocal(CentralManagerImpl.java:429)
at com.inductiveautomation.metro.impl.ConnectionWatcher.handle(ConnectionWatcher.java:429)
at com.inductiveautomation.metro.impl.ConnectionWatcher.handle(ConnectionWatcher.java:44)
at com.inductiveautomation.metro.impl.protocol.websocket.WebSocketConnection.forward(WebSocketConnection.java:1357)
at com.inductiveautomation.metro.impl.protocol.websocket.WebSocketConnection$RunDownload.run(WebSocketConnection.java:1926)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)|

|Intent|07Sep2023 12:38:18|Unhandled Intent failure.|
| --- | --- | --- |
|java.io.FileNotFoundException: /tmp/metro/8317a818-ca23-4b51-aa9c-1df38592868e___statetransfer.gwbk (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(Unknown Source)
at java.base/java.io.FileOutputStream.(Unknown Source)
at java.base/java.io.FileOutputStream.(Unknown Source)
at com.inductiveautomation.metro.impl.codecs.FileStreamCodec.decode(FileStreamCodec.java:143)
at com.inductiveautomation.metro.impl.CentralManagerImpl.handleLocal(CentralManagerImpl.java:429)
at com.inductiveautomation.metro.impl.ConnectionWatcher.handle(ConnectionWatcher.java:429)
at com.inductiveautomation.metro.impl.ConnectionWatcher.handle(ConnectionWatcher.java:44)
at com.inductiveautomation.metro.impl.protocol.websocket.WebSocketConnection.forward(WebSocketConnection.java:1357)
at com.inductiveautomation.metro.impl.protocol.websocket.WebSocketConnection$RunDownload.run(WebSocketConnection.java:1926)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)|

PROD 1 status log image.

PROD 2 status log image.

pturmel · September 7, 2023, 11:55am

Permission denied suggests something is wrong with the backup gateway's folder permissions. Consider deactivating it, uninstalling, re-installing (making sure you get just the matching modules), and configuring its backup role again (so it will try to resync).

Or, perhaps, the disk is full.

Jobin_Samuel · September 7, 2023, 12:11pm

Thank you for the comments.

The disk is not full.

When you say de-activate, do you mean to make the master independent and then reinstall the backup?

Thanks in advance.

pturmel · September 7, 2023, 12:13pm

No, I meant move the license back to IA's license servers so you can completely wipe the install folders (otherwise you'd lose the license).

kcollins1 · September 7, 2023, 2:01pm

My spidey sense is firing related to your /tmp folder. What OS are you running? If you run findmnt, are there any special mount configurations for /tmp? I'm remembering some situations where some hardening efforts on /tmp can cause compatibility problems (in which case you can configure a temp folder in Ignition and direct it to use that versus the system-default temp location).

Jobin_Samuel · September 8, 2023, 6:03am

Hi,

The Ignition is installed on a Linux server.

Jobin_Samuel · September 15, 2023, 11:02am

Hi,
Thank you for the reply, but I did not understand the suggested method.

kcollins1 · September 15, 2023, 1:50pm

I probably should have been more specific to ask what flavor of Linux OS that you're running? Ubuntu? RHEL? Ultimately taking a look at the permissions and mount configuration of /tmp (as Phil suggested) will probably be enlightening.

Jobin_Samuel · September 21, 2023, 9:08am

Hi,

The system is Linux 7.

While executing the findmnt, we found below results. Could you please help with next step.

pturmel · September 21, 2023, 12:33pm

Never heard of this. Do you mean RHEL 7 ? (If so, I must say it is really old and troublesome.)

Jobin_Samuel · September 21, 2023, 12:55pm

Hi,

Sorry for the confusion.

I have no knowledge on Linux, from one of the playbook I found something like this, Red Hat Enterprise Linux 7 & OpenJDK8. Not sure whether this makes sense to the team's question.

Jobin_Samuel · September 21, 2023, 12:57pm

Now, I believe the answer is RHEL 7 ( Red Hat Enterprise Linux 7).

kcollins1 · September 21, 2023, 2:08pm

It does look like perhaps you've got some extra hardening on that system. At this point, I'd probably direct Ignition to use a tmp folder under its own installation folder (instead of the global tmp folder at /tmp) for better compatibility. It might be worth touching base with Support for this.

Jobin_Samuel · September 22, 2023, 5:17am

Thank you for the support, I have raised a support ticket and pasted this link to it.

Will update over here once the solution has been found.

Jobin_Samuel · October 19, 2023, 3:57am

Status issue resolved by providing necessary permission to “/tmp/metro” folder in the backup server.

Jobin_Samuel · October 19, 2023, 3:57am

Thank you all for the support.