[monitoring] Add OpenTelemetry option #1365
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a set of 3 PRs, proposition to add OpenTelemetry to generate optional traces.
You can check the base PR, the one for handlers and the one for datastores.
The idea is to contribute to #1257 by providing an option to enable traces.
This first PR provide base support, including HTTP & SQL traces.
Traces looks like that (in that PR):

As OpenTelemetry is a framework, this allow processing traces with various compatibles services. Self-hostable examples include jaeger, openobserve, Grafana Tempo, SigNoz. Multiples SaaS solutions exists (some previous tools provide them). As long as there is OpenTelemetry support it can be used, and that also mean it could be integrated into the existing stack of an organization.
As an example, same traces (of last PR) in SigNoz:
Idea behind thoses PR is to use it only for tracing, not metrics nor logging, however it also possible to do so in the future.
Goal is to have it optional behind a flag, and only enabled if needed, mostly for debugging. Configuration is done via OpenTelemetry's own flags.
Documentation is not done yet, but this can be tested with a local jaeger (port 16686) with the following changes: