My company uses Grafana Loki for our logs and metrics. I want to use something like Loki Logback Appender to also send my custom module's logs to loki.
Can I do something similar with metrics? Ideally I'd like to:
- Register a custom metric in my module so it's available in the Ignition Metrics Dashboard
- Automatically also send the metric to a preconfigured loki endpoint.
Registering a custom metric is easy; see GatewayContext.getMetricRegistry().
There's no facility I'm aware of to send codahale metrics (the library we use) to logs, because they're different things, but if you particularly want to I'm sure you could write code into your module that introspects the metric registry and, at some rate, sends them on to your loki endpoint. Be aware that a key feature of the metric library we use is that unwatched metrics are ~0 cost; if you suddenly start reading all metrics at some fixed rate you're going to incur some performance penalty.
2 Likes
Oh sorry, that was bad wording on my part. We have two different grafana endpoints, one for metrics and one for logs.
I can use the logback appender to send all the slf4j logs to the logs endpoint, and I want to do something similar with metrics. The endpoint takes a list of Timeseries with labels and datapoints (<timestamp - double>).
So if there an equivalent "appender" where I can apply some transformation to the codahale metrics and forward them to Grafana as they're being collected? I can manually send the metrics in my module for the domain specific stuff, but I also want general Ignition health checks and uptime to go to Grafana.
Nothing built in/"off the shelf", at least to my knowledge. You're going to have to come up with something yourself. Again, though - metrics are late bound; you supply a lambda defining metric collection, so there's no "metric collection" event to watch for - the act of observing is what collects a metric.