Gateway faulted status - X11 window server

Ignition
1

Hi every body,

Since yesterday we have some strange issue on our development system : (Cent OS 7 , java 1.8, Ignition 7.9.7). The gateway becomes faulted after some time … and not only once a day. Here is what i have in my wrapper log. Any help please ??

INFO | jvm 2 | 2018/10/10 12:52:17 | I [T.P.Config ] [10:52:17]: Duplicate Detection took: 6 ms
INFO | jvm 2 | 2018/10/10 12:52:17 | I [T.P.Config ] [10:52:17]: Delete duplicates will delete 0 tags.
INFO | jvm 2 | 2018/10/10 12:52:17 | I [S.S.HsqlDataStore ] [10:52:17]: Looking for existing store and forward datacache “Logbook”… store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.HsqlDataStore ] [10:52:18]: … found existing. store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: Checkpoint start store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose start store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose synched store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache open start store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache open end store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose script done store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache commit start store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache commit end store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose end store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: Checkpoint end - txts: 102 store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.HsqlDataStore ] [10:52:18]: store and forward datacache “Logbook” started up successfully. store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.StandardDatasourceSFEngine] [10:52:18]: History sink ‘Logbook’ started successfully. store-forward-name=Logbook
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.HsqlDataStore ] [10:52:18]: Looking for existing store and forward datacache “InterfaceTunnel”… store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.HsqlDataStore ] [10:52:18]: … found existing. store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: Checkpoint start store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose start store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose synched store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache open start store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache open end store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose script done store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache commit start store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: dataFileCache commit end store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: checkpointClose end store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [h.d.H.ENGINE ] [10:52:18]: Checkpoint end - txts: 102 store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.HsqlDataStore ] [10:52:18]: store and forward datacache “InterfaceTunnel” started up successfully. store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [S.S.StandardDatasourceSFEngine] [10:52:18]: History sink ‘InterfaceTunnel’ started successfully. store-forward-name=InterfaceTunnel
INFO | jvm 2 | 2018/10/10 12:52:18 | I [ImageManager ] [10:52:18]: Starting up…
INFO | jvm 2 | 2018/10/10 12:52:18 | I [ProjectManager ] [10:52:18]: ProjectManager starting up…
INFO | jvm 2 | 2018/10/10 12:52:19 | E [SRContext ] [10:52:18]: Error during context startup.
INFO | jvm 2 | 2018/10/10 12:52:19 | java.awt.AWTError: Can’t connect to X11 window server using ‘localhost:10.0’ as the value of the DISPLAY variable.
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:115)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.awt.X11GraphicsEnvironment.(X11GraphicsEnvironment.java:74)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Class.forName0(Native Method)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Class.forName(Class.java:264)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.GraphicsEnvironment.createGE(GraphicsEnvironment.java:103)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:82)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.awt.X11.XToolkit.(XToolkit.java:126)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Class.forName0(Native Method)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Class.forName(Class.java:264)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.Toolkit$2.run(Toolkit.java:860)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.Toolkit$2.run(Toolkit.java:855)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:854)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.Toolkit.getEventQueue(Toolkit.java:1740)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.awt.EventQueue.isDispatchThread(EventQueue.java:1071)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.SwingUtilities.isEventDispatchThread(SwingUtilities.java:1361)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext.reclaim(StyleContext.java:454)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext.addAttribute(StyleContext.java:311)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext$NamedStyle.addAttribute(StyleContext.java:1501)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext$NamedStyle.setName(StyleContext.java:1312)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext$NamedStyle.(StyleContext.java:1259)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext.addStyle(StyleContext.java:107)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext.(StyleContext.java:87)
INFO | jvm 2 | 2018/10/10 12:52:19 | at javax.swing.text.StyleContext.getDefaultStyleContext(StyleContext.java:75)
INFO | jvm 2 | 2018/10/10 12:52:19 | at org.fife.ui.rtextarea.RTextAreaBase.getDefaultFont(RTextAreaBase.java:450)
INFO | jvm 2 | 2018/10/10 12:52:19 | at org.fife.ui.rsyntaxtextarea.Theme.(Theme.java:111)
INFO | jvm 2 | 2018/10/10 12:52:19 | at org.fife.ui.rsyntaxtextarea.Theme.load(Theme.java:341)
INFO | jvm 2 | 2018/10/10 12:52:19 | at org.fife.ui.rsyntaxtextarea.Theme.load(Theme.java:323)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.gui.NamedTheme.(NamedTheme.java:23)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.gui.NamedTheme.(NamedTheme.java:14)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 2 | 2018/10/10 12:52:19 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Class.getEnumConstantsShared(Class.java:3320)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Class.enumConstantDirectory(Class.java:3341)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Enum.valueOf(Enum.java:232)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.handlers.EnumHandler.startElement(EnumHandler.java:39)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.XMLDeserializer$ParseContext.onElementStart(XMLDeserializer.java:864)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.BinaryParser.readElement(BinaryParser.java:151)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.BinaryParser.readElement(BinaryParser.java:157)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.BinaryParser.readElement(BinaryParser.java:157)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.BinaryParser.readElement(BinaryParser.java:157)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.BinaryParser.parse(BinaryParser.java:51)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.XMLDeserializer.deserializeBinary(XMLDeserializer.java:347)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.XMLDeserializer.deserialize(XMLDeserializer.java:257)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.xmlserialization.deserialization.XMLDeserializer.deserialize(XMLDeserializer.java:237)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.project.NamedQueryExecutor.lambda$onProjectUpdated$0(NamedQueryExecutor.java:148)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.ArrayList.forEach(ArrayList.java:1257)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.project.NamedQueryExecutor.onProjectUpdated(NamedQueryExecutor.java:144)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.project.NamedQueryExecutor.(NamedQueryExecutor.java:69)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.project.ProjectSettingsCache.(ProjectSettingsCache.java:94)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.project.ProjectManagerImpl.startup(ProjectManagerImpl.java:1782)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.SRContext.startupInternal(SRContext.java:1335)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:224)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.SRContext.asyncInit(SRContext.java:1036)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.SRContext.access$300(SRContext.java:289)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.gateway.SRContext$2.run(SRContext.java:767)
INFO | jvm 2 | 2018/10/10 12:52:19 | at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO | jvm 2 | 2018/10/10 12:52:19 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 2 | 2018/10/10 12:52:19 | I [SRContext ] [10:52:18]: Ignition["/main", state=STARTING] ContextState = FAULTED

Thank you very much.

2

Does it come back automatically? Or do you need to restart something?

Do you have some stats on CPU/memory/network connectivity for the moments you saw these problems?

3

We need to restart ignition on the server every time it happens:

  • /etc/init.d/ignition stop
  • /etc/init.d/ignition start

Gateway status faulted comes after an undefined time but same logs every time.
It seems like network is perfectly ok. CPU and MEMORY too. I’ll take a screenshot of this at next faulted status.

4

So this part of the log looks like it is from when the gateway started. Do you have the log from when the gateway first went down? Also, what flavor of Java 8 is installed? (Oracle, OpenJDK, etc)

5

Here is the wrapper content with about 2/3 faulted status :

https://we.tl/t-y7jI3klmqR <- wrapper log

openjdk version “1.8.0_171”

6

I think you are going to have to call into Support on this one. The gateway crashed as soon as a client was opened:

INFO | jvm 1 | 2018/10/10 15:08:05 | I [c.i.i.g.s.g.f.Projects$ProjectChangeMonitor] [13:08:05]: Starting up client project monitor
ERROR | wrapper | 2018/10/10 15:10:05 | JVM exited unexpectedly.

It looks like there is a NamedTheme object related to a named query that is causing issues when it is deserialized. Just for kicks, you could also try Oracle Java 8 and see if it shows the same behavior.

7

I found this thread while troubleshooting a new install (AWS Linux 2 AMI, OpenJDK 1.8, Ignition 7.9.12). It looks close enough that the solution I found might be useful to share.

We don’t use the X11 interface at all, but seeing it in the stack trace led me to check whether some of the X11 system packages were required. The gateway that this new one is replacing did in fact have xorg-x11-font-utils and xorg-x11-fonts-Type1 installed.

So I installed xorg-x11-fonts-Type1 and its dependencies, and the problem was resolved on the next gateway startup.

(On RedHat-based systems like this, the relevant command was yum install xorg-x11-fonts-Type1).

1 Like