Use when alerting is either silent during real incidents or noisy enough that people ignore it.
alert-threshold-calibration.md
You are setting alert thresholds so that an alert firing means something. You are not diagnosing a live incident.
Metrics we want to alert on: {{METRICS_TO_ALERT}}
Historical variation for each, by day of week where available: {{HISTORICAL_VARIATION}}
Who receives the alert and what they can do at that hour: {{ALERT_RECIPIENTS}}
Alerts firing today and how often they are ignored: {{CURRENT_ALERT_NOISE}}
Output:
1. A table: Metric | Comparison basis, as same weekday, trailing 28 days, or versus forecast | Threshold | Minimum duration before it fires | Expected false alarms per month | Recipient from {{ALERT_RECIPIENTS}}.
2. The arithmetic behind each threshold, referring to the spread in {{HISTORICAL_VARIATION}}. Do not give a round number without saying what it corresponds to.
3. Metrics in {{METRICS_TO_ALERT}} that should carry no alert at all, because nobody can act inside the window or because the metric is too noisy at this granularity.
4. Alerts in {{CURRENT_ALERT_NOISE}} to delete, merge with another, or downgrade to a weekly digest, each with the reason.
5. What each alert should say when it fires, and the first check the recipient runs.
Rules:
- Every alert must name an action available to its recipient. An alert with no action is a report.
- Where {{HISTORICAL_VARIATION}} covers less than four weeks, mark the threshold provisional and give a date to revisit it.
- Prefer fewer alerts. State what you deliberately left unmonitored and why.
Replace each placeholder with your own detail. The more specific you are, the less the model invents.
When do I use this rather than tightening the alerts that keep firing?
When alerts are being ignored, which is the point where adjusting one number stops helping. The question here is which metrics deserve an alert at all, given who receives it and what they can do at that hour. Some of the answer is deleting alerts rather than recalibrating them.
What do I need in front of me?
The metrics you want alerts on, historical variation for each broken out by day of week where possible, who receives each alert and what they can act on, and an honest record of what fires today and how often it is ignored. The ignored count is the input people soften and the one that decides the output.
What comes back?
A table per metric with comparison basis, threshold, minimum duration before firing, expected false alarms per month and a named recipient, the arithmetic tying each threshold to the observed spread, metrics that should carry no alert, existing alerts to delete or digest, and the wording each alert should carry. Argue about the false alarm column.
What is the mistake that costs me here?
Setting a threshold with no minimum duration. Duration removes most false alarms without moving the threshold at all, so a tight threshold firing instantly is the noisy configuration you already have. Where variation data covers under four weeks, keep the threshold provisional and diary the date to revisit it.