Excellent points to make here. You are absolutely right in that what I'm asking for results in a larger demand for resources. My questions are aimed at a use case during evaluation of certain projects, during development and testing. The configurations for which metrics are kept, which are dropped, and what retention sizes should be used, will vary depending on the environment and scale.
Regarding histogram buckets, it appears the internal gateway metrics are using a default exponential bucket, where the underlying data may be misrepresented by these buckets. Operationally, you are 100% correct in that, you can drop many of the histogram bucket labels that aren't entirely insightful for alerting or triage.
I've been attempting to see what it might be like to report metrics from within a certain project, and stumbled upon this type of pattern:
WARNING: I have no idea if this is stable or recommended by IA.
from com.inductiveautomation.ignition.gateway import IgnitionGateway
def observe_histogram_metric(value):
gateway = IgnitionGateway.get()
metrics = gateway.getMetricRegistry()
custom_metric = metrics.histogram("my_custom_histogram")
custom_metric.update(value)
The gateway has a getter for the MetricRegistry, which allows scripts to report their own metrics through calling methods that return or create new metrics.
I'm still exploring exactly how the javaagent <-> dropwizard metrics interaction happens internally, to better understand how histogram buckets are set for a given metric.