This article describes a solution to enable external monitoring of Kublr monitoring components.

External services may be used such as:


Prometheus and Alertmanager health check endpoints /-/healthy can be used for external monitoring.


These endpoints are behind Kublr authentication, so if necessary, they can be exposed as unauthenticated endpoints.


This can be done by creating corresponding ingress rules (replace the hostnames, or adjust the ingress rules as necessary):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-watchdog
  namespace: kublr
spec:
  rules:
    # change to the correct prometheus hostname
    - host: prometheus.example.com
      http:
        paths:
          - backend:
              serviceName: kublr-monitoring-prometheus
              servicePort: 9090
            path: /-/healthy
            pathType: ImplementationSpecific
  tls:
    - hosts:
        # change to the correct prometheus hostname
        - prometheus.example.com
      secretName: kublr-monitoring-prometheus-tls
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alertmanager-watchdog
  namespace: kublr
spec:
  rules:
    # change to the correct alertmanager hostname
    - host: alerts.example.com
      http:
        paths:
          - backend:
              serviceName: kublr-monitoring-alertmanager
              servicePort: 9093
            path: /-/healthy
            pathType: ImplementationSpecific
  tls:
    - hosts:
        # change to the correct alertmanager hostname
        - alerts.example.com
      secretName: kublr-monitoring-alertmanager-tls


It is possible to include the ingress rules in the cluster specification using "raw" helm chart: 

spec:
  ...
  packages:
    monitoring-watchdog:
      releaseName: monitoring-watchdog
      namespace: kublr
      helmVersion: v3.4.0
      chart:
        name: raw
        repoUrl: 'https://charts.helm.sh/incubator/packages'
        version: 0.2.3
      values:
        resources:
          - apiVersion: extensions/v1beta1
            kind: Ingress
            metadata:
              name: prometheus-watchdog
              namespace: kublr
            spec:
              rules:
                - host: prometheus.example.com
                  http:
                    paths:
                      - backend:
                          serviceName: kublr-monitoring-prometheus
                          servicePort: 9090
                        path: /-/healthy
                        pathType: ImplementationSpecific
              tls:
                - hosts:
                    - prometheus.example.com
                  secretName: kublr-monitoring-prometheus-tls
          - apiVersion: extensions/v1beta1
            kind: Ingress
            metadata:
              name: alertmanager-watchdog
              namespace: kublr
            spec:
              rules:
                - host: alerts.example.com
                  http:
                    paths:
                      - backend:
                          serviceName: kublr-monitoring-alertmanager
                          servicePort: 9093
                        path: /-/healthy
                        pathType: ImplementationSpecific
              tls:
                - hosts:
                    - alerts.example.com
                  secretName: kublr-monitoring-alertmanager-tls