I’m seeing the above whenever I go into my Gateway logs. It says I have over 7000 log items, but none of them show up. There’s no filtering going on, I haven’t even looked at this page until now since installing this Gateway.
Are any errors showing in the browser’s console?
DevTools failed to parse SourceMap: http://localhost:8088/web/status/twotiermenu.js.map
DevTools failed to parse SourceMap: http://localhost:8088/web/status/GatewayBreadcrumb.js.map
DevTools failed to parse SourceMap: http://localhost:8088/web/status/trialbar.js.map
DevTools failed to parse SourceMap: http://localhost:8088/web/status/logviewer.js.map
These are the only items that show up in the console log.
Can you enable devtools and see what’s coming in on the ‘Network’ tab? Also, check the /logs/ directory on the gateway machine - it could be (somewhat ironically) that an error message is occurring in the function that’s meant to retrieve the logs, which is only being logged into those same logs. If you’re not sure what you’re looking at, just post the wrapper.log file here.
wrapper.zip (278.9 KB)
I’ve had to compress the wrapper.log file since it was about 7MB.
In the Network tab in the devtools there don’t seem to be any errors, just a lot of requests for “trial”.
These “trial” requests continue for a very long time.
Yep, error is with the logging system:
INFO | jvm 1 | 2019/06/13 13:54:07 | E [c.i.i.g.l.SQLiteAppenderReader] [18:54:07]: Error executing query against logs database. route-group=status, route-path=/logs
INFO | jvm 1 | 2019/06/13 13:54:07 | java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (database disk image is malformed)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.sqlite.core.DB.newSQLException(DB.java:890)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.sqlite.core.DB.newSQLException(DB.java:901)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.sqlite.core.DB.execute(DB.java:810)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.sqlite.jdbc3.JDBC3PreparedStatement.executeQuery(JDBC3PreparedStatement.java:68)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.logging.SQLiteAppenderReader.executePrepStatement(SQLiteAppenderReader.java:97)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.logging.SQLiteAppenderReader.query(SQLiteAppenderReader.java:170)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.logging.GatewayLoggingManagerImpl.queryLogEvents(GatewayLoggingManagerImpl.java:144)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.web.pages.status.routes.LoggingRoutes.getLogs(LoggingRoutes.java:561)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:247)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:49)
INFO | jvm 1 | 2019/06/13 13:54:07 | at com.inductiveautomation.ignition.gateway.dataroutes.DataServlet.service(DataServlet.java:87)
INFO | jvm 1 | 2019/06/13 13:54:07 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.Server.handle(Server.java:530)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
INFO | jvm 1 | 2019/06/13 13:54:07 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
INFO | jvm 1 | 2019/06/13 13:54:07 | at java.base/java.lang.Thread.run(Unknown Source)
You can safely delete the system_logs.idb file in the /logs/ directory (you’ll obviously lose the currently logged events, but they’re also duplicated into the wrapper.log files). The log file won’t recreate until you restart the gateway, though.
3 Likes
That fixed it! Thank you!
1 Like
Is there any way to do this without resting the service?
Thanks
No.
I'm using Ignition 8.1.22 (b2022110109) on Windows 10 | amd64
Yesterday I noticed I didn'r see any log-entries on the gateway webpage. Finding this topic, I downloaded a backup and restored it, to force gateway restart. This did not resolv my problems.
I deleted the system_logs.idb file and stopped/started the ignition service, but alas to no avail.
Attached the wrapper log of today.:
wrapper.log (903.2 KB)
I don't know about your issue, but to restart the gateway you can use the gwcmd utility in ignition's directory, with the -r or --restart argument.
Yes, that's the best way. It didn't solve my problem though.
Even a ignition update to the same version did not repair it.
Only a complete uninstall en clean install fixed the log-problem.
It looks like the backup that's getting restored has a path defined for the system logs that doesn't exist on this system. I don't remember which file defines the path, but it should be either the ignition.conf file or the logback.xml file. Both are located in the installation directory inside of data.
INFO | jvm 1 | 2023/05/23 10:00:18 | 10:00:18,912 |-WARN in ch.qos.logback.core.db.DataSourceConnectionSource@ae9b50b - Could not discover the dialect to use. java.sql.SQLException: path to 'C:\Program Files\Inductive Automation\Ignition\/var/log/ignition\system_logs.idb': 'C:\Program Files\Inductive Automation\Ignition\var\log' does not exist
INFO | jvm 1 | 2023/05/23 10:00:18 | at java.sql.SQLException: path to 'C:\Program Files\Inductive Automation\Ignition\/var/log/ignition\system_logs.idb': 'C:\Program Files\Inductive Automation\Ignition\var\log' does not exist
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.SQLiteConnection.open(SQLiteConnection.java:235)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.SQLiteConnection.<init>(SQLiteConnection.java:60)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.jdbc3.JDBC3Connection.<init>(JDBC3Connection.java:23)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.jdbc4.JDBC4Connection.<init>(JDBC4Connection.java:19)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.JDBC.createConnection(JDBC.java:104)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.SQLiteDataSource.getConnection(SQLiteDataSource.java:440)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.sqlite.SQLiteDataSource.getConnection(SQLiteDataSource.java:432)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.db.DataSourceConnectionSource.getConnection(DataSourceConnectionSource.java:62)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.db.ConnectionSourceBase.discoverConnectionProperties(ConnectionSourceBase.java:46)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.db.DataSourceConnectionSource.start(DataSourceConnectionSource.java:44)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.logging.SQLiteAppender.configureConnectionSource(SQLiteAppender.java:231)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.logging.SQLiteAppender.start(SQLiteAppender.java:287)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:90)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:309)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:193)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:179)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:165)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:152)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:110)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:53)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.ignition.common.util.LoggerEx$Builder.build(LoggerEx.java:603)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.ignition.gateway.IgnitionGateway.getLogger(IgnitionGateway.java:3663)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.ignition.gateway.IgnitionGateway.<init>(IgnitionGateway.java:355)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.ignition.gateway.IgnitionGateway.main(IgnitionGateway.java:295)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at com.inductiveautomation.catapult.Catapult.main(Catapult.java:8)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at java.base/java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
INFO | jvm 1 | 2023/05/23 10:00:18 | at at java.base/java.lang.Thread.run(Unknown Source)
1 Like
Hello,
I am also facing the same issue, i am not getting logs in my gateway,
wrapper.zip (502.9 KB)
I did deleting the system_logs.idb file and restarted, still not solved. Could you please help me?
1 Like
From inspecting your wrapper logs, it looks like you have some contention with the cleanup actions for the system_logs.idb SQLite database that the web UI uses. Filtering out your log shows these lines:
INFO | jvm 1 | 2023/12/25 15:31:32 | Cleaned up 500 rows in 117801 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:33:28 | Cleaned up 500 rows in 115659 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:35:23 | Cleaned up 500 rows in 114878 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:40:38 | Cleaned up 500 rows in 111008 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:42:26 | Cleaned up 500 rows in 107917 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:44:15 | Cleaned up 500 rows in 108542 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:46:00 | Cleaned up 500 rows in 105137 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:47:44 | Cleaned up 500 rows in 103686 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:49:25 | Cleaned up 500 rows in 101041 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:51:03 | Cleaned up 500 rows in 98219 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:52:37 | Cleaned up 500 rows in 93984 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:54:07 | Cleaned up 500 rows in 90319 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:55:37 | Cleaned up 500 rows in 89875 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:57:09 | Cleaned up 500 rows in 91547 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:58:36 | Cleaned up 500 rows in 87668 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 15:59:58 | Cleaned up 500 rows in 81739 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:01:22 | Cleaned up 500 rows in 84038 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:02:49 | Cleaned up 500 rows in 86610 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:04:10 | Cleaned up 500 rows in 80950 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:05:27 | Cleaned up 500 rows in 76775 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:06:43 | Cleaned up 500 rows in 75968 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:07:55 | Cleaned up 500 rows in 72144 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:09:04 | Cleaned up 500 rows in 69301 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:10:13 | Cleaned up 500 rows in 68878 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:11:20 | Cleaned up 500 rows in 66690 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:12:24 | Cleaned up 500 rows in 64493 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:13:27 | Cleaned up 500 rows in 62795 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:14:28 | Cleaned up 500 rows in 60721 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:15:29 | Cleaned up 500 rows in 60970 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:16:27 | Cleaned up 500 rows in 57542 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:17:25 | Cleaned up 500 rows in 55750 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:18:22 | Cleaned up 500 rows in 53086 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:19:20 | Cleaned up 500 rows in 49535 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:20:18 | Cleaned up 500 rows in 47705 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:21:17 | Cleaned up 500 rows in 45359 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:22:16 | Cleaned up 500 rows in 43996 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:23:14 | Cleaned up 500 rows in 41513 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:24:12 | Cleaned up 500 rows in 38349 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:25:11 | Cleaned up 500 rows in 36720 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:26:10 | Cleaned up 500 rows in 34748 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:27:08 | Cleaned up 500 rows in 32476 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:28:07 | Cleaned up 500 rows in 30759 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:29:05 | Cleaned up 500 rows in 29035 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:30:04 | Cleaned up 500 rows in 27388 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:31:02 | Cleaned up 500 rows in 24716 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:32:00 | Cleaned up 500 rows in 23284 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:33:00 | Cleaned up 500 rows in 22431 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:33:58 | Cleaned up 500 rows in 19506 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:34:55 | Cleaned up 500 rows in 17110 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:35:53 | Cleaned up 500 rows in 14942 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:36:53 | Cleaned up 500 rows in 14048 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:37:50 | Cleaned up 500 rows in 10916 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:38:49 | Cleaned up 500 rows in 8895 ms. Vacuum performed: true*
INFO | jvm 1 | 2023/12/25 16:39:47 | Cleaned up 500 rows in 7076 ms. Vacuum performed: true*
When disk space cleanups are issued, the default is to attempt to clean up 500 rows at a time. This seems to be taking quite a long time to achieve based on some of the timing figures listed (slow disk I/O?). It seems odd that you'd be tripping on the disk space limits so quickly after deleting the logs/system_logs.idb from the Ignition installation path (which you'd have to do with the gateway stopped).
In the near term, you could uncomment this section (by moving the trailing --> line above the <entrylimit>... line) from your data/logback.xml and modify some of the settings:
<!--
Maintenance Settings
entryLimit: The maximum number of entries in the database. However, at any given time, there may be more than this number, due to how cleanup works.
maxEventsPerMaintenance: The number of event that can happen before a maintenance cycle occurs.
minTimeBetweenMaintenance: The minimum time (max frequency) between maintenance events. Takes precedent over max events.
vacuumFrequency: The number of maintenance cycles before a "vacuum" is performed, to recover disk space.
diskspaceCleanupEventCount: The number of oldest events that will be removed when max database size is exceeded.
maxDatabaseSize: Database size in bytes, above which will trigger a maintenance cycle to remove diskspaceCleanupEventCount of oldest events.
On disk, most log events are between 600-800 bytes.
<entryLimit>50000</entryLimit>
<maxEventsPerMaintenance>5000</maxEventsPerMaintenance>
<minTimeBetweenMaintenance>60000</minTimeBetweenMaintenance>
<vacuumFrequency>3</vacuumFrequency>
<diskspaceCleanupEventCount>500</diskspaceCleanupEventCount>
<maxDatabaseSize>104857600</maxDatabaseSize>
-->
Perhaps setting the maxDatabaseSize to higher than 100MB and/or having it cleanup more than the default 500 oldest events via diskspaceCleanupEventCount.
As time permits, consider addressing some of the high volume of errors (it looks like there are a lot of verbose errors being generated by errors in some of your database interactions).
2 Likes