[Supported in Kublr 1.20.1 and later]

Tags: elasticsearch, observability, apm, uptime, heartbeat


Kublr runs a regular open source Elastic stack as a part of platform for log collection and management.


By default Elasticsearch is configured with a Basic license and SearchGuard plugin for security and access control.


You can also enable a number of observability features like a heartbeat, APM, Metrics and alerts.


Application Performance Monitoring

https://www.elastic.co/apm/

https://github.com/elastic/helm-charts/tree/master/apm-server

For enable APM, you need to deploy apm-server-7.10.2 helm chart:

spec:
  packages:
    elk-apm-server:
      chart:
        name: apm-server
        repoUrl: 'https://helm.elastic.co/helm/apm-server'
        version: 7.10.2
      releaseName: elk-apm-server
      namespace: kublr
      helmVersion: v3.4.0
      values:
        apmConfig:
          apm-server.yml: |
            apm-server:
              host: "0.0.0.0:8200"
            queue: {}
            output.elasticsearch:
              username: 'admin'
              password: '${ELASTICSEARCH_PASSWORD}'
              protocol: https
              hosts: ["kublr-logging-elasticsearch-client:9200"]
              ssl.certificate_authorities:
              - /usr/share/apm-server/config/certs/root-ca.pem
              ssl.verification_mode: none
        extraEnvs:
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              key: admin-password
              name: kublr-logging-searchguard
        secretMounts:
        - name: elastic-certificate-pem
          path: /usr/share/apm-server/config/certs
          secretName: kublr-logging-searchguard


Application Performance Monitoring

https://www.elastic.co/guide/en/beats/metricbeat/

https://github.com/elastic/helm-charts/tree/master/metricbeat

https://www.elastic.co/guide/en/beats/metricbeat/7.x/metricbeat-modules.html

For enable Metricbeat, you need to deploy metricbeat-7.10.2 helm chart:

    elk-metricbeat:
      chart:
        name: metricbeat
        repoUrl: 'https://helm.elastic.co/helm/metricbeat'
        version: 7.10.2
      releaseName: elk-metricbeat
      namespace: kublr
      helmVersion: v3.5.2
      values:
        daemonset:
          enabled: false
        deployment:
          extraEnvs:
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: admin-password
                  name: kublr-logging-searchguard
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: aws-secret-access-key
                  name: key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: aws-secret-access-key
                  name: key-secret
          metricbeatConfig:
            metricbeat.yml: |
              logging.json: true
              metricbeat.modules:
              - module: kubernetes
                enabled: false
              - module: aws
                period: 5m
                access_key_id: ${AWS_ACCESS_KEY_ID}
                secret_access_key: ${AWS_SECRET_ACCESS_KEY}
                metricsets:
                - cloudwatch
                metrics:
                - namespace: AWS/EC2
                  resource_type: ec2:instance
                - namespace: AWS/S3
              output.elasticsearch:
                username: 'admin'
                password: '${ELASTICSEARCH_PASSWORD}'
                protocol: https
                hosts: ["kublr-logging-elasticsearch-client:9200"]
                ssl.certificate_authorities:
                - /usr/share/apm-server/config/certs/root-ca.pem
                ssl.verification_mode: none
          secretMounts:
            - name: elastic-certificate-pem
              path: /usr/share/apm-server/config/certs
              secretName: kublr-logging-searchguard
        kube_state_metrics:
          enabled: false

In this example we gets metrics from AWS cloudwatch service.


Uptime monitoring with Elastic Heartbeat service

https://www.elastic.co/beats/heartbeat


Create configuration for your heartbeat service in configmap using spec.packages and RAW helm package:

https://www.elastic.co/guide/en/beats/heartbeat/current/configuration-heartbeat-options.html

spec:
  packages:
    elk-heartbeat:
      releaseName: elk-heartbeat
      namespace: kublr
      chart:
        name: raw
        repoUrl: 'https://charts.helm.sh/incubator/packages'
        version: 0.2.3
      values:
        resources:
          - apiVersion: v1
            kind: ConfigMap
            metadata:
              labels:
                k8s-app: heartbeat
              name: elk-heartbeat
              namespace: kublr
            data:
              heartbeat.yml: |-
                reload.enabled: true 
                reload.period: 60s
                heartbeat.scheduler: 
                   limit: 10 
                heartbeat.monitors:
                - type: http
                   id: service-status
                   name: Service Status
                   hosts: ["http://localhost:80/service/status"]
                   check.response.status: [200]
                   schedule: '@every 5s'

                output.elasticsearch:
                  username: 'admin'
                  password: '${ELASTICSEARCH_PASSWORD}'
                  protocol: https
                  hosts: ["kublr-logging-elasticsearch-client:9200"]
                  ssl.certificate_authorities:
                    - /usr/share/apm-server/config/certs/root-ca.pem                                                                                                                                                                                     
                  ssl.verification_mode: none

Also, you need add deployment manifest to:


spec:
  packages:
    elk-heartbeat:
    ...
      values:
        resources:
        ...
          - apiVersion: apps/v1
            kind: Deployment
            metadata:
              name: elk-heartbeat
              namespace: kublr
              labels:
                k8s-app: heartbeat
            spec:
              selector:
                matchLabels:
                  k8s-app: heartbeat
              template:
                metadata:
                  labels:
                   k8s-app: heartbeat
                spec:
                  volumes:
                  - name: elastic-certificate-pem
                    secret:
                      defaultMode: 420
                      secretName: kublr-logging-searchguard
                  - name: config
                    configMap:
                      name: elk-heartbeat
                  - name: data
                    hostPath:
                      path: /var/lib/heartbeat-data
                      type: DirectoryOrCreate
                  containers:
                  - name: heartbeat
                    image: docker.elastic.co/beats/heartbeat:7.10.2
                    args: [ "-c", "/conf/heartbeat.yml", "-e" ]
                    env:
                    - name: ELASTICSEARCH_PASSWORD
                      valueFrom:
                        secretKeyRef:
                          key: admin-password
                          name: kublr-logging-searchguard
                    - name: NODE_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: spec.nodeName
                    securityContext:
                      runAsUser: 0
                    resources:
                      limits:
                        memory: 200Mi
                      requests:
                        cpu: 100m
                        memory: 200Mi
                    volumeMounts:
                    - name: config
                      mountPath: /conf
                    - name: data
                      mountPath: /usr/share/heartbeat/data
                    - name: elastic-certificate-pem
                      mountPath: /usr/share/apm-server/config/certs