[Supported in Kublr 1.20.1 and later]
Tags: elasticsearch, observability, apm, uptime, heartbeat
Kublr runs a regular open source Elastic stack as a part of platform for log collection and management.
By default Elasticsearch is configured with a Basic license and SearchGuard plugin for security and access control.
You can also enable a number of observability features like a heartbeat, APM, Metrics and alerts.
Application Performance Monitoring
https://github.com/elastic/helm-charts/tree/master/apm-server
For enable APM, you need to deploy apm-server-7.10.2 helm chart:
spec:
packages:
elk-apm-server:
chart:
name: apm-server
repoUrl: 'https://helm.elastic.co/helm/apm-server'
version: 7.10.2
releaseName: elk-apm-server
namespace: kublr
helmVersion: v3.4.0
values:
apmConfig:
apm-server.yml: |
apm-server:
host: "0.0.0.0:8200"
queue: {}
output.elasticsearch:
username: 'admin'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
hosts: ["kublr-logging-elasticsearch-client:9200"]
ssl.certificate_authorities:
- /usr/share/apm-server/config/certs/root-ca.pem
ssl.verification_mode: none
extraEnvs:
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
key: admin-password
name: kublr-logging-searchguard
secretMounts:
- name: elastic-certificate-pem
path: /usr/share/apm-server/config/certs
secretName: kublr-logging-searchguardApplication Performance Monitoring
https://www.elastic.co/guide/en/beats/metricbeat/
https://github.com/elastic/helm-charts/tree/master/metricbeat
https://www.elastic.co/guide/en/beats/metricbeat/7.x/metricbeat-modules.html
For enable Metricbeat, you need to deploy metricbeat-7.10.2 helm chart:
elk-metricbeat:
chart:
name: metricbeat
repoUrl: 'https://helm.elastic.co/helm/metricbeat'
version: 7.10.2
releaseName: elk-metricbeat
namespace: kublr
helmVersion: v3.5.2
values:
daemonset:
enabled: false
deployment:
extraEnvs:
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
key: admin-password
name: kublr-logging-searchguard
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: aws-secret-access-key
name: key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: aws-secret-access-key
name: key-secret
metricbeatConfig:
metricbeat.yml: |
logging.json: true
metricbeat.modules:
- module: kubernetes
enabled: false
- module: aws
period: 5m
access_key_id: ${AWS_ACCESS_KEY_ID}
secret_access_key: ${AWS_SECRET_ACCESS_KEY}
metricsets:
- cloudwatch
metrics:
- namespace: AWS/EC2
resource_type: ec2:instance
- namespace: AWS/S3
output.elasticsearch:
username: 'admin'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
hosts: ["kublr-logging-elasticsearch-client:9200"]
ssl.certificate_authorities:
- /usr/share/apm-server/config/certs/root-ca.pem
ssl.verification_mode: none
secretMounts:
- name: elastic-certificate-pem
path: /usr/share/apm-server/config/certs
secretName: kublr-logging-searchguard
kube_state_metrics:
enabled: falseIn this example we gets metrics from AWS cloudwatch service.
APM Module for Kibana
For enable Kibana APM agent, modify cluster specification:
spec:
features:
logging:
values:
env:
ELASTIC_APM_ACTIVE: 'true'
ELASTIC_APM_SERVER_URL: 'http://elk-apm-server-apm-server.kublr:8200'Uptime monitoring with Elastic Heartbeat service
https://www.elastic.co/beats/heartbeat
Create configuration for your heartbeat service in configmap using spec.packages and RAW helm package:
https://www.elastic.co/guide/en/beats/heartbeat/current/configuration-heartbeat-options.html
spec:
packages:
elk-heartbeat:
releaseName: elk-heartbeat
namespace: kublr
chart:
name: raw
repoUrl: 'https://charts.helm.sh/incubator/packages'
version: 0.2.3
values:
resources:
- apiVersion: v1
kind: ConfigMap
metadata:
labels:
k8s-app: heartbeat
name: elk-heartbeat
namespace: kublr
data:
heartbeat.yml: |-
reload.enabled: true
reload.period: 60s
heartbeat.scheduler:
limit: 10
heartbeat.monitors:
- type: http
id: service-status
name: Service Status
hosts: ["http://localhost:80/service/status"]
check.response.status: [200]
schedule: '@every 5s'
output.elasticsearch:
username: 'admin'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
hosts: ["kublr-logging-elasticsearch-client:9200"]
ssl.certificate_authorities:
- /usr/share/apm-server/config/certs/root-ca.pem
ssl.verification_mode: noneAlso, you need add deployment manifest to:
spec:
packages:
elk-heartbeat:
...
values:
resources:
...
- apiVersion: apps/v1
kind: Deployment
metadata:
name: elk-heartbeat
namespace: kublr
labels:
k8s-app: heartbeat
spec:
selector:
matchLabels:
k8s-app: heartbeat
template:
metadata:
labels:
k8s-app: heartbeat
spec:
volumes:
- name: elastic-certificate-pem
secret:
defaultMode: 420
secretName: kublr-logging-searchguard
- name: config
configMap:
name: elk-heartbeat
- name: data
hostPath:
path: /var/lib/heartbeat-data
type: DirectoryOrCreate
containers:
- name: heartbeat
image: docker.elastic.co/beats/heartbeat:7.10.2
args: [ "-c", "/conf/heartbeat.yml", "-e" ]
env:
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
key: admin-password
name: kublr-logging-searchguard
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: config
mountPath: /conf
- name: data
mountPath: /usr/share/heartbeat/data
- name: elastic-certificate-pem
mountPath: /usr/share/apm-server/config/certs