Observability · Platform engineering · 7 min read ·
SLOs before dashboards
Observability tooling is easy to buy and hard to make useful. Start from what the service promises, measure it from the user's side, and let the alerts, the dashboards and the on-call rota follow from the promise rather than from what the tools export.
Teams that buy an observability platform usually end up with hundreds of dashboards, thousands of time series and a pager that goes off for CPU. None of that tells them whether the service is doing its job. The missing piece is not a tool; it is a sentence that says what the service promises and a number that says how well it is keeping the promise.
Start with the promise
A service-level objective is a target for a service-level indicator over a window: "99.5% of report requests complete within two seconds, measured over 30 days." Writing one forces three decisions that nothing else forces:
- Which indicator? Something the user experiences: request latency at the edge, the fraction of scheduled jobs that ran on time, the age of the freshest data in a dashboard. Not CPU, not queue depth, not pod restarts; those are causes, and causes change.
- What threshold? Measured from what the service does today, then tightened as it improves. A target the service has never met is a wish, not an objective.
- How much failure is acceptable? The remainder is the error budget, and it is the most useful number the team owns: it says whether to ship the risky change this week or to spend the week on reliability.
Three or four objectives per service is plenty. More than that and nobody can say which one matters when they conflict.
Measure from the user's side
The indicator has to be measured where the user is, or as close as possible. Latency measured inside the application excludes the load balancer, the TLS handshake and the queue in front of the pod, all of which the user waits for. Measure at the edge where you can; where you cannot, a synthetic probe that behaves like a user (signs in, opens the page, runs the query) is honest in a way that internal metrics are not.
For a multi-tenant platform, measure per tenant. An aggregate objective at 99.9% can hide one customer at 90%, and that customer is the one who notices.
Alerts follow from the budget, not from thresholds
A static threshold ("alert when latency exceeds two seconds") pages on every blip and sleeps through a slow decline. Alerting on how fast the error budget is being consumed does the opposite: a burn rate that would exhaust the month's budget in an hour pages someone now; a rate that would exhaust it in a week opens a ticket. Two windows (a short one to catch the spike, a long one to confirm it is not noise) remove most false pages without missing real incidents.
Every page must be actionable by the person who receives it. If the runbook step is "wait and see", it is a ticket, not a page. If there is no runbook step, it is a dashboard.
Dashboards, last
With objectives and burn-rate alerts in place, the dashboards design themselves. The top of each service's page shows the objectives and the remaining budget; below that, the handful of causes that explain most budget burn for that service. Everything else is a query someone can run when they need it, not a panel to be scrolled past.
Labels decide whether any of this is possible. A metric without a tenant, environment and version label cannot answer "which tenant, since which deploy". Agree on the label set once, enforce it in the platform's instrumentation library, and the per-tenant and per-release views exist for free.
Logs and traces have a job too
Logs answer "what happened in this request"; traces answer "where did the time go". Neither is the thing to alert on. Both are the thing to reach for once the burn-rate alert fires, which means they need the same labels as the metrics and a correlation id that travels through every hop. A request id that appears in the edge log, the application log and the trace turns an incident from a search into a lookup.
Retention is a cost decision: metrics for a year at low resolution, logs for a month, traces sampled. A platform that keeps everything forever is paying for data nobody queries.
The on-call rota is part of the design
An objective nobody is accountable for is documentation. The team that owns the service owns its objectives, its budget and its pager. If the budget is spent, feature work pauses until it recovers; that rule, written down and honoured once, is what makes the objective real.
A reasonable first month
- Write one objective per user-facing service, measured at the edge.
- Instrument the labels (tenant, environment, version) in the shared library.
- Replace static-threshold pages with two-window burn-rate alerts.
- Build one dashboard per service, objectives first.
- Review the budgets weekly; tighten targets the service keeps easily, relax ones it never meets, and write down why.
The observability and reliability service starts from exactly this list; the platform it produces is what the per-tenant request-rate, latency and queue-depth views on our own stack are built on.
Keep reading
Platform engineering · GitOps · CI/CD · 7 min
Golden paths that engineers actually take
A platform team's output is not the platform; it is the paths other teams take through it. How to build paved roads that are faster than the alternative, keep them honest with the people who use them, and avoid turning them into mandates.
Kubernetes · Security · Platform engineering · 8 min
Isolating tenants on a shared Kubernetes cluster
A namespace is not a security boundary, but a namespace plus five other controls is a usable one. What separates tenants on a shared cluster, what still needs a cluster of its own, and how to decide which customer gets which.
GitOps · Platform engineering · CI/CD · 7 min
GitOps promotion without copying YAML between environments
Argo CD and Flux make deployments declarative; they do not by themselves solve promotion. How to move a change from dev to production with a record, a gate and no hand edits.