TODO: document how to retrieve metrics for M3DB components.
TODO: document how to retrieve logs for M3DB components.
M3DB is integrated with opentracing to provide insight into query performance and errors.
Currently, only Jaeger is supported as a backend.
To enable it, set tracing.backend to "jaeger" (see also our sample local config:
tracing: backend: jaeger # enables jaeger with default configs jaeger: # optional configuration for jaeger -- see # https://github.com/jaegertracing/jaeger-client-go/blob/master/config/config.go#L37 # for options ...
Jaeger can be run locally with docker as described in https://www.jaegertracing.io/docs/1.9/getting-started/.
The default configuration will report traces via udp to localhost:6831; using the all-in-one jaeger container, they will be accessible at
N.B.: for production workloads, you will almost certainly want to use sampler.type=remote with adaptive sampling for Jaeger, as write volumes are likely orders of magnitude higher than read volumes in most timeseries systems.
If you'd like additional backends, we'd love to support them!
File an issue against M3 and we can work with you on how best to add
the backend. The first time's going to be a little rough--opentracing
unfortunately doesn't support Go plugins (yet--see
update model means that adding dependencies directly will update
everything, which isn't ideal for an isolated dependency change.
These problems are all solvable though,
and we'll work with you to make it happen!
Note: all URLs assume a local jaeger setup as described in Jaeger's docs.
Finding slow queries
To find prom queries longer than
minDuration >= <threshold> on
Finding queries with errors
Finding 500 (Internal Server Error) responses