[Affected in Kublr 1.21.2 and earlier]

Tags: prometheus, alertmanager, elasticsearch


In current Kublr version we have wrong alert rule for detecting potential log collection, based on ELK and prometheus expression: increase(elasticsearch_indices_docs{es_data_node="true"}[15m])<1

In some cases we gets false alarm, for fixing:

1. Go to Alertmanager UI and mute alarm alertname=ElasticsearchNoNewDocuments

2. modify and update Kublr Control Plane cluster specification

spec:
...
  features:
    monitoring:
      values:
        alertmanager:
          alerts:
            - alert: LoggingNoNewDocuments
              annotations:
                description: No new documents in ELK for 15 min!
                summary: '{{$labels.kublr_space}}.{{$labels.kublr_cluster}} - LoggingNoNewDocuments'
              expr: 'sum by (kublr_space, kublr_cluster, cluster) (increase(elasticsearch_indices_docs{es_data_node="true"}[15m])) < 1'
              labels:
                feature: centralizeLogging
                severity: warning