Alerts setup#

Alerting has three parts, and they are deliberately separate: a rule defines the condition, a contact point is somewhere a notification can go, and a notification policy decides which alerts reach which contact points.

Keeping routing out of the rules is the whole point. Adding a new destination, or changing who is paged overnight, is one policy change rather than an edit to every rule you own.

1. Create contact points#

Alerts → Contact points → Add.

Use Send test before saving. It delivers a real notification, which catches a wrong webhook or a blocked sender immediately rather than during an incident.

2. Create a rule#

The quickest route is from something you are already looking at: open any dashboard panel or graph and choose Create alert. The query comes with it.

Otherwise Alerts → Alert rules → Create rule:

1. Query the thing to watch, as PromQL or by picking a metric and filters.
2. Reduce the result to one number with last, min, max or average. Most series return several values over the evaluation window and the reducer decides which one is compared.
3. Condition: the comparison and threshold, for example above 0.05.
4. For: how long the condition must hold continuously before firing.
5. Severity: info, warning or critical.
6. Category: uptime, metric, trace, log, security, cloud or custom.

Severity and category#

These two fields are what make routing possible, so set them deliberately. Severity should reflect the response expected: critical means someone is woken, warning means someone looks during working hours, info means it is recorded but not pushed.

3. Route with notification policies#

Alerts → Notification policies → Add policy. A policy has matchers, destinations and a mode.

A typical arrangement:

Policies evaluate in priority order, lowest first. Stop prevents further policies from also matching, which is what stops a critical alert both paging the on-call engineer and appearing in the management digest.

A policy with no matchers is the catch-all. Have one, so an alert can never be created that routes nowhere.

Digest mode#

Digest accumulates matching alerts and sends one summary on a schedule, daily or weekly. This is how management gets visibility without a pager. It is the difference between a director's inbox and an engineer's phone, and mixing the two is the fastest way to have alerts ignored.

4. On-call and escalation#

Alerts → On-call:

1. Create a schedule with a rotation and the people in it.
2. Create an escalation policy: notify the on-call engineer, wait a set period, then notify the next person or the whole team if unacknowledged.
3. Attach the escalation policy to your critical notification policy.

Acknowledge from the web interface, by replying to the SMS, or by keypress on the voice call. Acknowledgement stops the escalation.

5. Reduce noise#

Alerts that are ignored are worse than no alerts. Practical measures:

• Set a for-duration on every rule.
• Alert on symptoms users feel, not on every underlying cause. One alert on error rate beats ten on individual components.
• Use maintenance windows for planned work rather than muting after the pager fires.
• Review firing history monthly and delete or retune anything that never led to action.

Testing the whole path#

Create a temporary rule guaranteed to fire, for example vector(1) > 0 with a for-duration of one minute, and confirm it reaches the intended destination. That verifies rule, policy, contact point and escalation together. Delete it afterwards.

Importing existing rules#

Prometheus alerting rule files can be imported under Alerts → Import. Severity, thresholds and for-durations are preserved.