Homelablab Operations

Notification

Pushover

• Old fashion, widely used. Realtime alerts might need proxy (for me).

Serverchan

• via webhook, ref; free-tier is enough; dont need proxy;
• if work, prefer sct_appkey, dont need another app, push to work_channel;
• if home, prefer sc3_appkey, dont need app running, push to status_bar;
• support markdown message;

Group pushing

• tried a few methods to group notification pushing, current method is prometheus rules;

# alertmanager
config:
  receivers:
    - name: "default-receiver"
      pushover_configs:
        - send_resolved: true
          priority: |-
            {{ if eq .Status "firing" }}1{{ else }}0{{ end }}
          title: >-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
            {{ .CommonLabels.alertname }}
          message: |-
            {{- range .Alerts }}
              {{- if ne .Annotations.description "" }}
                {{ .Annotations.description }}
              {{- else if ne .Annotations.summary "" }}
                {{ .Annotations.summary }}
              {{- else if ne .Annotations.message "" }}
                {{ .Annotations.message }}
              {{- else }}
                Alert description not available
              {{- end }}
              {{- if gt (len .Labels.SortedPairs) 0 }}
                <small>
                  {{- range .Labels.SortedPairs }}
                    <b>{{ .Name }}:</b> {{ .Value }}
                  {{- end }}
                </small>
              {{- end }}
            {{- end }}
          sound: gamelan
          ttl: 86400s
          token:
            name: *name
            key: PUSHOVER_TOKEN
          user_key:
            name: *name
            key: PUSHOVER_USER_KEY
      webhook_configs:
        - send_resolved: true
          url: http://apprise.monitoring-system.svc.cluster.local:8001?template=alertmanager

# apprise via `camalot/apprise-webhook`