[Supported in Kublr 1.21.0 and later]

Tags: elasticsearch, watchers


By default Elasticsearch is configured with a Basic license and SearchGuard plugin for security and access control. You can enable enterprise features manually.


Before beginning, you need to enter your license key on the cluster management page in Kibana or enable trial license.


Previous Kublr versions

For previous Kublr versions ready logging helm packages for installation:

Kublr v1.17.1

Run Kublr in a Box with override logging version

$ docker run --name kublr -e CLUSTER_CONTROLLER_FEATURE_LOGGING_VERSION=1.17.3-2 -d --restart=unless-stopped -p 9080:9080 kublr/kublr:1.17.1

Upgrade your platform, using Kublr in a Box. Modify logging values section for enable watchers

spec:
...
  features:
    logging:
      elasticsearch:
          cluster:
            xpackIncluded: true
            env:
              XPACK_WATCHER_ENABLED: true
          keystore:
           # see Using Watchers section
            - secretName: elastic-config-secret
            - items:
                - key: slack_url
                  path: xpack.notification.slack.account.kublr.secure_url
              secretName: elastic-config-custom-path
        logstash:
          image:
            name: logstash/logstash
        searchguard:
          elasticsearch:
            image:
              name: kublr/elasticsearch-sg
          enabled: true
          kibana:
            image:
              name: kublr/kibana-sg


Kublr v1.20.2

Modify your cluster specification, change Kublr default logging features chart version to 1.20.3-59 in KCP values:

spec:
  features:
    controlPlane:
      values:
        kublr-api:
          configComponentsVersion:
            logging: 1.20.3-59

Restart Kublr API pod

$ kubectl delete po -n kublr -l app=kcp-kublr-api

Go to Kublr UI and upgrade logging to 1.20.3-59


Using Watchers


https://www.elastic.co/guide/en/elasticsearch/reference/7.12/xpack-alerting.html

https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-es-secure-settings.html



Prepare secrets for use in Elasticsearch keystore:

https://www.elastic.co/guide/en/elasticsearch/reference/7.12/elasticsearch-keystore.html


# Generate system key file using elasticsearch-syskeygen
# https://www.elastic.co/guide/en/elasticsearch/reference/7.12/syskeygen.html

kubectl create secret -n kublr generic elastic-config-secret --from-file=xpack.watcher.encryption_key=./watcher.key

kubectl create secret generic elastic-config-slack -n kublr --from-literal=xpack.notification.slack.account.monitoring.secure_url=https://hooks.slack.com/services/aaa/bbb/aaa

kubectl create secret -n kublr generic elastic-config-custom-path --from-literal=slack_url='https://hooks.slack.com/services/aaa/bbb/ccc' --from-literal=thing_i_don_tcare_about=test

Modify cluster spec:


spec:
  features:
    logging:
      values:
        elasticsearch:
          cluster:
            xpackEnabled: true
            config:
              xpack.watcher.encrypt_sensitive_data: true
            env:
              XPACK_WATCHER_ENABLED: true
          keystore:
            - secretName: elastic-config-secret
            - items:
                - key: slack_url
                  path: xpack.notification.slack.account.kublr.secure_url
              secretName: elastic-config-custom-path
        kibana:
          additionalSettings:
            xpack.encryptedSavedObjects.encryptionKey: Super$ecreKe62312YWordForKibana126%
            xpack.security.encryptionKey: Super$ecreKe62312YWordForKibana126%
            xpack.watcher.enabled: true

Create a simple watcher to testing watcher and slack integration:

https://www.elastic.co/guide/en/elasticsearch/reference/7.13/actions-slack.html

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "kublr*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "now-30m"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 1000
      }
    }
  },
  "actions": {
    "notify-slack": {
      "slack": {
        "account": "monitoring",
        "message": {
          "from": "kublr",
          "to": [
            "#alerts"
          ],
          "text": "Simple Kublr alert",
          "attachments": [
            {
              "color": "warning",
              "title": "Alert Details",
              "text": "{{ctx.payload.hits.total}} events in last 30 mins in kublr logs."
            }
          ]
        }
      }
    }
  }
}