Skip to main content
Version: Next

0035: Watch events export over OTLP, never block the watch, and never carry a secret

  • Status: Accepted (backfilled 2026-10-06: records a decision already built)
  • Date: 2026-10-06

Context​

A watch's history lived in a file per project. Operators already run an observability backend (Grafana, Datadog, Dynatrace, New Relic, Honeycomb) and want to search and chart Orion there. Three risks come with shipping events off the machine: a plugin per vendor to maintain, an exporter that stalls the watch when the backend is slow, and a credential leaking inside an event.

Decision​

internal/export ships events to one backend, configured machine-wide in ~/.orion/observability.json. Off by default.

  • One standard, not a plugin per vendor. Every listed vendor accepts OTLP logs over HTTP, so a vendor is a preset (which header carries the key), not code. Loki's own push API is kept for a self-hosted Loki without an OTLP endpoint.
  • Never in the way. Events are handed over without blocking, wait in a bounded buffer, and go out in batches from one goroutine. When the buffer is full or the backend refuses them, they are dropped with one warning. If export cannot start, the watch says so and runs without it.
  • No secrets in the config. Each credential is the name of an environment variable, read at start.
  • No secrets in the events. Anything credential-shaped (provider tokens, bearer headers, a password in a URL) is scrubbed before it leaves the machine.
  • Tool-call events are opt-in (IncludeToolEvents): one per tool call is most of the volume a backend bills for.

Consequences​

  • Adding a vendor that speaks OTLP is a preset, not a release of new code.
  • A slow or unreachable backend loses events, never stalls a watch.
  • The scrub is pattern-based; a secret in an unusual shape could pass it, which is why secrets are also kept out of agents' reach by the sandbox.

Alternatives rejected​

  • A plugin per vendor. Five integrations to keep current for one protocol they all already accept.
  • Synchronous export. Makes observability a new way for the watch to fail.
  • Tokens in the config file. A file that is easy to copy, commit or paste.