QuQi
ANALYTICS & REPORTING

Alert threshold calibration

Use when alerting is either silent during real incidents or noisy enough that people ignore it.

alert-threshold-calibration.md
.md herunterladen
You are setting alert thresholds so that an alert firing means something. You are not diagnosing a live incident.

Metrics we want to alert on: {{METRICS_TO_ALERT}}
Historical variation for each, by day of week where available: {{HISTORICAL_VARIATION}}
Who receives the alert and what they can do at that hour: {{ALERT_RECIPIENTS}}
Alerts firing today and how often they are ignored: {{CURRENT_ALERT_NOISE}}

Output:
1. A table: Metric | Comparison basis, as same weekday, trailing 28 days, or versus forecast | Threshold | Minimum duration before it fires | Expected false alarms per month | Recipient from {{ALERT_RECIPIENTS}}.
2. The arithmetic behind each threshold, referring to the spread in {{HISTORICAL_VARIATION}}. Do not give a round number without saying what it corresponds to.
3. Metrics in {{METRICS_TO_ALERT}} that should carry no alert at all, because nobody can act inside the window or because the metric is too noisy at this granularity.
4. Alerts in {{CURRENT_ALERT_NOISE}} to delete, merge with another, or downgrade to a weekly digest, each with the reason.
5. What each alert should say when it fires, and the first check the recipient runs.

Rules:
- Every alert must name an action available to its recipient. An alert with no action is a report.
- Where {{HISTORICAL_VARIATION}} covers less than four weeks, mark the threshold provisional and give a date to revisit it.
- Prefer fewer alerts. State what you deliberately left unmonitored and why.

Vor dem Start ausfüllen

Ersetzen Sie jeden Platzhalter durch Ihre eigenen Angaben. Je genauer Sie sind, desto weniger erfindet das Modell.

  • {{METRICS_TO_ALERT}}
  • {{HISTORICAL_VARIATION}}
  • {{ALERT_RECIPIENTS}}
  • {{CURRENT_ALERT_NOISE}}

So wird das Ergebnis besser

  1. Same weekday comparisons beat trailing averages for anything with a weekend pattern.
  2. A minimum duration kills most false alarms without moving the threshold at all.
  3. Count the ignored alerts honestly; a channel people mute is worse than no alerting.