How to Choose an Observability Tool: Criteria Matrix and Template

Start with the questions, not the feature list

If the goal is to see everything, the cost of an observability tool grows without limit. First decide how quickly you must learn about an outage and whether you must follow a request across services. This site has collected 2 observability-topic posts so far.

Map the service graph before you score tools

Netflix built a platform for analyzing the topology of a large distributed service. As the number of services grows, knowing which service calls which becomes the first question. Put “shows the service topology” in the matrix. The summary gives no load figure, so measure your own cost before you rely on a vendor estimate.

Check the cost of the signal, not only the tool

Pricing usually scales with the volume of data you keep. Score each candidate by what you pay to keep the signals you actually query, and cap retention per signal type. Re-score the matrix when traffic changes, not only when you renew.

Verify alerts against real incidents

After the choice, check alerts against incidents you actually had. Count the incidents that an alert caught, the ones it missed, and the alerts that fired without a real problem. Those three numbers set the starting point for the next review.

Template (copy)

## 1. Questions the tool must answer

- How fast must we know about an outage?
- Must we follow one request across every service it touched?

## 2. Criteria matrix

| Criterion | Weight | Candidate A | Candidate B | How we check |
|---|---|---|---|---|
| Cost per unit of data | | | | |
| Logs, metrics, and traces in one place | | | | |
| Query language the team already knows | | | | |
| Operating load | | | | |

## 3. Checks after rollout

- Did alerts catch real incidents, and how many were false alarms

## 4. Reversal conditions

- Cost ceiling, and the alert gap that triggers a change