Observability

Observability is the ability to understand what a running system is doing from the data it emits. You do not need to reproduce a failure locally to answer why a payment retried, which service dropped a webhook, or why a dead-letter queue grew after a deploy.

Monitoring tells you when something is wrong. Observability helps you explain why it went wrong and what path the work took through the system.

The three pillars

Most production systems combine three signal types.

Logs are the detailed record. Metrics show trends and thresholds. Traces connect the dots when work crosses HTTP calls, queues, workers, and databases.

flowchart TB
Client["Client request"] --> API["API service"]
API -->|"log + trace span"| Broker["Message broker"]
Broker -->|"metric: lag"| Worker["Worker"]
Worker -->|"log + trace span"| DB["Database"]
Worker -->|"metric: failures"| DLQ["Dead-letter queue"]
API -.->|"metrics dashboard"| Dashboard["On-call view"]
Worker -.->|"trace lookup"| Dashboard
DLQ -.->|"alert"| Dashboard

Use all three together. Metrics surface the symptom. Logs explain the local decision. Traces show the cross-service story.

Correlation and context

Async systems fail when each component logs in isolation. A retry looks like a new incident unless every hop shares the same identifiers.

Carry context through the path:

• Request or trace ID on HTTP headers and log fields.
• Message ID and correlation ID on broker payloads and consumer logs.
• Idempotency key when at-least-once delivery can produce duplicates.

function logEvent(level, message, context) {
  console.log(JSON.stringify({
    level,
    message,
    traceId: context.traceId,
    messageId: context.messageId,
    idempotencyKey: context.idempotencyKey,
    service: "billing-worker",
    timestamp: new Date().toISOString(),
  }));
}

type LogContext = {
  traceId: string;
  messageId?: string;
  idempotencyKey?: string;
};

function logEvent(level: string, message: string, context: LogContext) {
  console.log(JSON.stringify({
    level,
    message,
    traceId: context.traceId,
    messageId: context.messageId,
    idempotencyKey: context.idempotencyKey,
    service: "billing-worker",
    timestamp: new Date().toISOString(),
  }));
}

Structured logs make search and alerting possible. Free-text paragraphs are hard to query when an on-call engineer needs every event for one traceId during an incident.

What to instrument first

Start where retries, duplicates, and silent failure already exist.

• HTTP entry points, latency, status codes, saturation, and error rate by route.
• Message brokers, publish rate, consumer lag, retry count, and DLQ depth.
• Webhooks, verification failures, duplicate deliveries, and time to acknowledge versus time to finish processing.
• Background workers, job duration, failure class, and replay outcomes.
• Downstream dependencies, database timeouts, cache misses, and third-party API errors.

For event-driven architectures, the failure path is often more important than the happy path. Instrument publish, consume, ack, nack, retry, and dead-letter transitions explicitly.

Signals that matter in operations

The exact dashboard changes by system, but four questions recur.

• Latency, how long did the work take end to end?
• Traffic, how much work is arriving?
• Errors, what is failing, and with which error type?
• Saturation, which queue, pool, or dependency is running out of headroom?

Pair these with ownership. A chart without a team and runbook still leaves production opaque.

Debugging retries and duplicates

Retries make incidents harder when observability is weak. The same logical action can look like several unrelated errors.

Design for investigation:

• Log the retry attempt number and the reason for retry.
• Record whether a handler short-circuited because of idempotency.
• Keep the original payload reference on DLQ entries, not only the latest error string.
• Compare trace spans before replaying messages from a DLQ.

Without that context, teams either avoid retries and lose reliability, or retry blindly and duplicate side effects.

CI/CD and production change

CI/CD pipelines should end with verification in production, not only a green deploy step. Compare error rate, latency, and saturation against a recent baseline after release. Canary and feature-flag rollouts depend on the same signals.

Treat observability checks as part of the release contract. A deploy that cannot be measured is a deploy that cannot be rolled back confidently.

Trade-offs

Observability has a cost. High-cardinality metrics, verbose logs, and full trace sampling can be expensive. Sampling, retention tiers, and deliberate field design keep signal useful without capturing everything forever.

Under-instrumentation is also costly. Webhook reliability patterns, brokers, and idempotent handlers only stay safe when operators can see duplicates, lag, and poison messages before customers report them.