The following are flooding the gateway console immediately prior to Ignition “locking up.” I am not sure where to even start looking for the cause, any help is greatly appreciated.
INFO | jvm 2 | 2019/01/25 06:05:41 | ERROR [TCPServerChannel UAChannel ] [06:05:41,012]: Uncaught throwable while processing chunk.
INFO | jvm 2 | 2019/01/25 06:05:41 | java.lang.ArrayIndexOutOfBoundsException: 2
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.common.stack.UAChannel.getMessageType(UAChannel.java:758)
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.common.stack.UAChannel.processChunk(UAChannel.java:764)
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.common.stack.UAChannel.access$000(UAChannel.java:62)
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.common.stack.UAChannel$ProcessChunk.run(UAChannel.java:1581)
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.client.stack.SerialExecutionQueue$RunnableExecutor.execute(SerialExecutionQueue.java:100)
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.client.stack.SerialExecutionQueue$RunnableExecutor.execute(SerialExecutionQueue.java:97)
INFO | jvm 2 | 2019/01/25 06:05:41 | at com.inductiveautomation.xopc.client.stack.SerialExecutionQueue$PollAndExecute.run(SerialExecutionQueue.java:75)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.util.concurrent.FutureTask.run(Unknown Source)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 2 | 2019/01/25 06:05:41 | at java.lang.Thread.run(Unknown Source)
the gateway is already running out of memory for another reason and this is a side effect
IT is running a network/security scanner that has found the open OPC UA port and started sending it test data
You might need to call support and work with them to figure out which is going on. They’ll need to look a look at the rest of your logs and your system in general.
The error below shows up periodically… Could it be part of the problem? How do I find the tag involved?
8:19:15 AM TagPropertyRecord Error deserializing value.
java.lang.ClassCastException: Cannot coerce value 'null' into type: class java.lang.Integer
at com.inductiveautomation.ignition.common.TypeUtilities.coerce(TypeUtilities.java:1293)
at com.inductiveautomation.ignition.common.TypeUtilities.datasetFromJSON(TypeUtilities.java:2026)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.config.TagPropertyRecord.getValue(TagPropertyRecord.java:136)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore._loadTags(InternalTagStore.java:1887)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.loadTags(InternalTagStore.java:1709)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore$6.load(InternalTagStore.java:2095)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore$6.load(InternalTagStore.java:2088)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.getDefinition(InternalTagStore.java:2111)
at com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalTagStore.getProperty(InternalTagStore.java:2184)
at com.inductiveautomation.ignition.gateway.sqltags.execution.tags.AbstractGatewayTag.getAttribute(AbstractGatewayTag.java:140)
at com.inductiveautomation.ignition.gateway.sqltags.execution.tags.AbstractExecutableTag.getAttribute(AbstractExecutableTag.java:500)
at com.inductiveautomation.ignition.common.sqltags.TagDefinition.getAttributesFrom(TagDefinition.java:598)
at com.inductiveautomation.ignition.common.sqltags.TagDefinition.<init>(TagDefinition.java:330)
at com.inductiveautomation.ignition.common.sqltags.TagDefinition.<init>(TagDefinition.java:321)
at com.inductiveautomation.ignition.gateway.servlets.gateway.functions.SQLTags.wrap(SQLTags.java:436)
at com.inductiveautomation.ignition.gateway.servlets.gateway.functions.SQLTags.getTag(SQLTags.java:184)
at sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.inductiveautomation.ignition.gateway.servlets.gateway.AbstractGatewayFunction.invoke(AbstractGatewayFunction.java:209)
at com.inductiveautomation.ignition.gateway.servlets.Gateway.doPost(Gateway.java:345)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:85)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:837)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Unknown Source)
We had an issue exactly like the described, it turned out to be scenario 2, we were not notified on time of the pen-testing in place, our IGN logs were flooded with logs like the above and took us a few minutes to realize the problem, we had to restart the gateway v7.8.4.
Would a later version be more resilient to attacks without manual intervention?