Self-hosted observability: Vector → OpenObserve over a mesh

What was actually wrong

A multi-site fleet — databases across sites plus application pods on k3s — had no single place to see health and errors. The obvious SaaS options price per host and per ingested GB, meaning visibility gets more expensive exactly as the fleet grows.

What we had to work within

• Cost had to scale sub-linearly with host count — a per-host SaaS model was a non-starter.
• Hosts span sites with no shared private network; telemetry could not cross the public internet in the clear.
• Operable by a very small team — nothing that needs a dedicated observability engineer.
• Keep ingest (and therefore cost and noise) down without losing the signal that matters.

What we weighed

The decision

We ship with Vector — a single lightweight agent — and filter at the edge with VRL so only warn/error/fatal log lines and 60-second status probes ever leave a host. That cuts ingest volume, and therefore cost and noise, at the source instead of paying to store and index everything. It all lands in OpenObserve — one backend for both the database-infra streams and the application logs — reached over a Tailscale mesh, so no agent or backend is exposed to the public internet. WireGuard/headscale were considered and deliberately rejected in favour of a single Tailscale mesh.

• Vector on each host with VRL filtering — ship only warn/error/fatal plus 60-second status probes — to keep volume low.
• OpenObserve as the single store for database-infra streams and application logs.
• All collection over a Tailscale mesh — no public exposure of agents or backend.
• Separate streams for DB infrastructure vs. application logs so the signals stay legible.

The result

The pipeline unifies multi-site database infrastructure and application logs into one self-hosted, mesh-private backend, with edge filtering holding volume (and cost) down. TODO: founder to fill with the real result — actual monthly cost vs. a comparable Datadog/SaaS quote, and ingest volume (GB/day or events/sec). These need real numbers before they are stated publicly.