Three-site HA MariaDB with automated failover

What was actually wrong

A single-primary database was the last single point of failure in an otherwise redundant platform. Losing the primary site meant a manual, error-prone failover, and read traffic had no clean way to scale or stay in-region across geographically separated sites.

What we had to work within

• No hosted/managed database option — the platform runs on our own infrastructure across three independent sites and providers.
• Reads had to scale and stay in-region without the applications rewriting their connection logic — they talk to one endpoint.
• A wrong automatic failover (split-brain, promoting a stale replica) had to be treated as worse than a slow one.
• Cross-site links add latency and the occasional blip; the design has to tolerate that, not assume a clean LAN.

What we weighed

The decision

We run MariaDB 11.4 primary→replica with GTID async replication across three sites — a US primary, an EU replica, and an independent third node that runs only the failover manager (no data), so ties break cleanly instead of risking split-brain. ProxySQL sits in front: the apps use a single endpoint while its read_only monitor polls every 1.5s and weights local reads 1000:1 over remote, keeping roughly 99.9% of reads in-region. For failover we started on Orchestrator, then moved to Signal18 replication-manager (repman) with a single active manager on the witness and deliberately conservative promotion rules — because a bad automatic promotion is worse than a brief manual one. Cloudflare Load Balancing health-checks the application tier and steers traffic US→EU on failure; it sits above the database and never talks to it. Everything rides a Tailscale mesh, so there is no public database exposure and no floating VIP to babysit.

• ProxySQL read/write split (writer/reader hostgroups), read_only monitor at 1.5s, local:remote read weighting 1000:1 (~99.9% of reads served in-region).
• Signal18 replication-manager (repman) with a single active manager on an independent witness node and conservative promotion rules.
• A rehearsed failover drill in both directions — promote the replica, auto-rejoin the ex-primary — measured on throwaway VMs, not trusted from a diagram.
• Cloudflare Load Balancing for app-tier geo steering; a Tailscale mesh so no database endpoint is ever publicly exposed.

The result

We cut over roughly 20 production databases — application deployments plus ~49 scheduled jobs, analytics and WordPress — onto the HA tier. In a failover drill the replica was promoted and made writable in about 16 seconds, with the ex-primary auto-rejoining in about 12 seconds at zero lag, verified in both directions on throwaway VMs before production was touched. ProxySQL adds only ~85µs per query for the routing. TODO: founder to add a production-incident timing and an uptime figure once there is a real event to cite — the ~16s above is drill-verified, not from a production outage.