we upgraded from 7.9 to 8.1 on July 26th, we experienced numerous issues, the latest is the memory leaks: we have 1 master and 1 backup(redundant) gateways, both have 32GB of memory and the master “gobbles” up the memory at least once per day and stops working, we have to restart it and all goes back to normal.
Here is the latest wrapper and a screenshot of the performance page after such an event
https://drive.google.com/drive/folders/145dGjiJcPjhV22Ro1w6WeeEUy8U_JgY0?usp=sharing
Memory leaks have so many possible causes that help from support is almost certainly required. (This forum is not official support…)
1 Like
We have been in contact with support but so far no fix, we call or email on average twice/week, their MO is to send them the wrapper, they could not find a cause, then send them more wrappers, turn on more frequent logs during events …
Perhaps start by fixing the airlink issues that are flooding the wrapper log.
One of the common causes for a memory profile that just shoots straight up from otherwise normal operation like in your screenshot is one or more large or unbounded tag history queries being made.
I do see some evidence of tag history queries timing out in your logs:
2021/09/25 05:57:37 | E [t.h.q.d.DB ] [12:57:37]: default (id=6e7d4104) Error occurred while loading tag information. request-origin=192.168.4.72, session-user=mmendoza, session-project=VVWRA_SCADA, session-id=364F3993, queryid=6e7d4104
2021/09/25 05:57:37 | com.microsoft.sqlserver.jdbc.SQLServerException: Read timed out
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1368)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1355)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1532)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3274)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:4433)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:386)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:185)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:160)
2021/09/25 05:57:37 | at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:281)
2021/09/25 05:57:37 | at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
2021/09/25 05:57:37 | at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.datasource.SRConnectionWrapper$SRPreparedStatement.executeQuery(SRConnectionWrapper.java:1008)
2021/09/25 05:57:37 | at com.inductiveautomation.gateway.tags.history.query.DatasourceHistoryInterface.loadSCExecRecords(DatasourceHistoryInterface.java:430)
2021/09/25 05:57:37 | at com.inductiveautomation.gateway.tags.history.query.DatasourceQueryExecutor.initialize(DatasourceQueryExecutor.java:589)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.sqltags.history.query.AggregateHistoryQueryExecutor.initialize(AggregateHistoryQueryExecutor.java:78)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.sqltags.history.query.HistoryWriter.initLoader(HistoryWriter.java:141)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.sqltags.history.query.HistoryWriter.execute(HistoryWriter.java:211)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.sqltags.history.TagHistoryManagerImpl.queryHistory(TagHistoryManagerImpl.java:537)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.tags.model.ProjectDefaultTagManagerFacade.queryHistory(ProjectDefaultTagManagerFacade.java:436)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.servlets.gateway.functions.QueryTagHistory.invoke(QueryTagHistory.java:96)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.servlets.Gateway.doPost(Gateway.java:405)
2021/09/25 05:57:37 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
2021/09/25 05:57:37 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
2021/09/25 05:57:37 | at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:86)
2021/09/25 05:57:37 | at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1391)
2021/09/25 05:57:37 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
2021/09/25 05:57:37 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
2021/09/25 05:57:37 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
2021/09/25 05:57:37 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
2021/09/25 05:57:37 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.Server.handle(Server.java:500)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
2021/09/25 05:57:37 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
2021/09/25 05:57:37 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
2021/09/25 05:57:37 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
2021/09/25 05:57:37 | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
2021/09/25 05:57:37 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
2021/09/25 05:57:37 | at java.base/java.lang.Thread.run(Unknown Source)