We've deployed an integration with SAP recently via the WebDev module, and are receiving a significant amount of http inbound payloads now. I've got some custom loggers setup to provide some diagnostics information on these payloads, but this has resulted in a significant amount of gateway log entries, to the point where the default configuration is only showing us the last 24hrs or so of gateway logs.
I notice there are some settings in the config file which affect the wrapper logs:
From opening a wrapper.log file in a text editor, it seems like this is recording different things to what is displayed on the gateway log page, is that correct?
And the gateway log messages are instead stored in the system_logs.idb file?
Mine is currently at 40MB, is there a way to allow this to be bigger to retain and view more logs in the gateway?
This got me thinking though, is it possibel for IA to give us an option to store these logs in an external database, just like we do for AuditLog and AlarmEvents??
It would be amazing to have more comprehensive searching and querying on this data which is currently not possible with the limited search capabilities of the gateway status log page filter.
Additional things. The wrapper log also captures gateway print statement output and various logs from DLLs that don't know how to log through java.
I find the wrapper log far preferable to the gateway logs, as the wrapper can be "tailed" from a terminal, displaying in natural order. Multi-line messages aren't garbled, either.
Thanks Phil, I'll have to have a look at the wrapper log again.
So adjusting those config settings should give us more results showing up in the Gateway webpage logs page too then?
Thanks, what version of logback is Ignition currently using as of 8.1.37?
As of logback version 1.2.8 DBAppender no longer ships with logback-classic. However, DBAppender for logback-classic is available under the following Maven coordinates:
ch.qos.logback.db:logback-classic-db:1.2.11.1
Not really sure how to deal with this if it is no longer shipped.
It looks like when we upgraded logback beyond the version it was bundled that we also started explicitly including the logback-classic-db dependency, so you should be good to go regardless.
Getting an error starting up the logback appender.
Its complaining that the JDBC driver doesn't support getGeneratedKeys method OR i've not specified a specific SQL dialect..
I'm trying to use MariaDB which is up and running and also setup in the Ignition gateway as a connection to ensure the credentials etc.. are valid.
org.mariadb.jdbc.Driver
Will try dig a bit further into the two possibile issues above.
I have the following driver .jar files loaded, not sure if the filename matters, and whether logback even references these or if its only for Ignition to use.
I'm not sure how to bring in additional driver .jar for logback if required, as hinted in one of the posts here where "logback-examples-1.0.0.jar" was mentioned.
The user/lib/jdbc folder will only be picked up by dedicated classloaders in Ignition used for database operations. You'll have to copy the mariadb jar to somewhere on the global gateway classpath for logback to pick it up... probablylib/core/gateway is the "best" place, though as you're finding, this approach in general has some problems.
You could also just modify (or duplicate) the system logs config in logback and do the SQLite -> MariaDB step manually using your programming language of choice.
You could also just modify (or duplicate) the system logs config in logback and do the SQLite -> MariaDB step manually using your programming language of choice.
By this do you mean doing something with this "DB" appender?
I'm sure there are ways to do a dump of SQLite data and import into MariaDB / Mysql regularly but it seems a bit messy. Ideally want to get this streaming log messages direct to the external DB if possible.
Will see if I can get it working by playing around with the .jar files a bit more
@richardNZ15 How is this coming? I was trying to follow what you did and realized that your last message was yesterday. I am trying to store gateway logs to MS SQL Server.
This seems to have got it to work. I copied the mysql-connector.jar file that was referenced in my logback .xml configuration, and placed it into the above referenced Ignition folder.
/usr/local/bin/ignition/lib/core/gateway
And lo and behold I get some more promising entries in the Ignition wrapper log about logback initializing:
Could you post a copy of your logback.xml? I'm trying to copy what you did and I keep getting in my log that DBExternal is being skipped. logback.xml (3.1 KB)
Sounds like you need to reference it at the end of the file, at the part i've labelled (2).
Bit hard to upload the file itself, too many hoops to jump through. But there isn't too much to it.
A word of warning. I did this a long time ago, and you need to filter out database error log messages. If you don't, and writing to the database is the cause of the error, it can get out of hand extremely quickly.
I might have interjected with the logging system programmatically, but just something to look out for.
What did you find was an effective filter? Checking for a specific MDC key, or looking at the logger message itself?
How granular did you go with that? Because I imagine it would still be nice to get certain database error log messages recorded, that might relate to other tables.
I can't remember exactly, but I think I programmatically added a DBAppender. I then filtered out a few specific DB loggers, it wasn't all of them. I think the DBAppender I added used system.db calls. Let me see if I can did it up.
