Problem with webserver Ignition 8.3

Hello

Honestly, I just downloaded and upgraded Ignition from version 8.1 to 8.3; however, after the update, a problem occurred.

The update is already installed, and everything is operational, but… a problem appeared related to a thread name called webserver-2788 (id=2788). Over time, its resource consumption increases a lot, causing the applications to stop working.

This issue didn’t happen in the previous version. Restarting the service temporarily fixes it, but it’s not tied to any specific schedule or pattern. We’ve tried checking if it’s caused by a particular project or something similar, but there’s no clear indication — it just shows that this thread suddenly starts consuming a lot of resources without specifying which project or element (timer, tag, etc.) it’s related to.
This identifier changes, but it’s always the same issue.

Could you please help me with this?
Thanks in advance.

The first two processes.

Best regards,
Kenneth

Can you post the error message text on here as preformatted text instead of a screenshot?
We cant read anything you posted.

1 Like

Putting on my eagle eyes, I can discern that you've got an RPC call initiating a tag history query. Meaning a Vision client or an Ignition Designer somewhere (probably multiple of them) is issuing tag history queries, and some part of that process (aggregating results, or serializing results, or literally just pushing the bytes down the wire) is consuming a lot of CPU. You've given us barely any information to work with to provide more useful advice.

Think this could be related to the bug I found and discussed with you at ICC? Similar thing happening in it. Shows up as a tag history query sucking up CPU resources, but somewhere in the logs will show an invalid tag path error that's the cause of the problem. It's a long shot being the same issue, but just thought I'd throw it out there.

It's entirely possible; at this point we're reading tea leaves :smiley:

I'm hoping that the underlying architectural changes in 8.3's historian gets us a new swathe of bugs instead of repeating the old ones, but we all know how that goes :laughing:

1 Like
{
      "name": "webserver-1103",
      "id": 1103,
      "state": "RUNNABLE",
      "daemon": false,
      "system": "Platform",
      "scope": "Client",
      "cpuUsage": 12.402765845130226,
      "stacktrace": [
        "app//com.inductiveautomation.ignition.gateway.sqltags.history.query.columns.CalculatingResultNode.finishAggregationWindow(CalculatingResultNode.java:137)",
        "app//com.inductiveautomation.ignition.gateway.sqltags.history.query.columns.CalculatingResultNode.processValue(CalculatingResultNode.java:231)",
        "com.inductiveautomation.sqlhistorian.gateway.query.DatasourceQueryExecutor.loadValue(DatasourceQueryExecutor.java:516)",
        "com.inductiveautomation.sqlhistorian.gateway.query.DatasourceQueryExecutor.loadValue(DatasourceQueryExecutor.java:94)",
        "com.inductiveautomation.historian.gateway.query.execution.AbstractHistoryLoader.processData(AbstractHistoryLoader.java:179)",
        "com.inductiveautomation.historian.gateway.query.writing.HistoryWriter.readData(HistoryWriter.java:314)",
        "com.inductiveautomation.historian.gateway.query.writing.HistoryWriter.execute(HistoryWriter.java:227)",
        "com.inductiveautomation.historian.gateway.HistorianManagerImpl.queryHistory(HistorianManagerImpl.java:1070)",
        "app//com.inductiveautomation.ignition.gateway.sqltags.history.TagHistoryManagerBridge.queryHistory(TagHistoryManagerBridge.java:99)",
        "app//com.inductiveautomation.ignition.gateway.tags.model.ProjectDefaultTagManagerFacade.queryHistory(ProjectDefaultTagManagerFacade.java:475)",
        "app//com.inductiveautomation.ignition.gateway.historian.rpc.TagHistoryBridgeRpcImpl.queryTagHistory(TagHistoryBridgeRpcImpl.java:47)",
        "java.base@17.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
        "java.base@17.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)",
        "java.base@17.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)",
        "java.base@17.0.16/java.lang.reflect.Method.invoke(Unknown Source)",
        "app//com.inductiveautomation.ignition.gateway.rpc.RpcDelegate$DelegateRpcHandler.handle(RpcDelegate.java:271)",
        "app//com.inductiveautomation.ignition.gateway.rpc.RpcRoutes.lambda$handle$1(RpcRoutes.java:199)",
        "app//com.inductiveautomation.ignition.gateway.rpc.RpcRoutes$$Lambda$6572/0x0000000801db3bc8.get(Unknown Source)",
        "app//com.inductiveautomation.ignition.gateway.rpc.RpcRoutes.safelyHandle(RpcRoutes.java:219)",
        "app//com.inductiveautomation.ignition.gateway.rpc.RpcRoutes.handle(RpcRoutes.java:198)",
        "app//com.inductiveautomation.ignition.gateway.rpc.RpcRoutes$$Lambda$1238/0x00000008005584d8.handle(Unknown Source)",
        "app//com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:361)",
        "app//com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:120)",
        "app//com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupCollectionServlet.serviceInternal(RouteGroupCollectionServlet.java:133)",
        "app//com.inductiveautomation.ignition.gateway.dataroutes.AbstractRouteGroupServlet.service(AbstractRouteGroupServlet.java:40)",
        "app//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)",
        "app//org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1379)",
        "app//org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)",
        "app//org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)",
        "app//com.inductiveautomation.catapult.filters.GatewayFilter.doFilter(GatewayFilter.java:116)",
        "app//jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
        "app//org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
        "app//org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)",
        "app//org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)",
        "app//org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:819)",
        "app//org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)",
        "app//org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)",
        "app//org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
        "app//org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
        "app//org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)",
        "app//org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
        "app//org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
        "app//org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
        "app//org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
        "app//org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
        "app//org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
        "app//org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
        "app//org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
        "app//org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
        "app//org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
        "app//org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
        "app//org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
        "app//org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
        "app//org.eclipse.jetty.server.Server.handle(Server.java:182)",
        "app//org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:662)",
        "app//org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:418)",
        "app//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
        "app//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)",
        "app//org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
        "app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)",
        "app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)",
        "app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)",
        "java.base@17.0.16/java.lang.Thread.run(Unknown Source)"
      ]
    },

This mensaje repeat when the cpu is going to 100% , so don't show me nothing and I don't know but in the versión 8;1 this problem don't appeared.
Something knows what message means ? Or how can i look this features such as projects name or something like this ? But for this specific cpu consumption ?
Thanks

Are you getting any error messages in the logs at all prior to the high CPU usage?

This i s the Log when I review in the server because in the Ignition Service don't show me nothing.

I think at this point you need to file a support ticket. They will be able to get a diagnostic dump and gateway backup to try to replicate your issue or can look over your shoulder to see what may be going on.

Support. This forum is not an official support venue.

2 Likes