Tags: kcp, monitoring


TABLE OF CONTENTS


Change/increase Prometheus scrape timeout


spec:
  features:
    monitoring:
      values:
        prometheus:
          config:
            scrapeTimeout: 60s


Customize Alert rules


spec:
  features:
    monitoring:
      values:
        alertmanager:
          alerts:
          - alert: PrometheusTargetDown
             expr: "100 * (up{job=~"cluster-.*"} == 0) / (up{job=~"cluster-.*"}) > 0"
             for: 10m
             labels:
               kublr_space: demo
               kublr_cluster: kcp-demo
               severity: warning
             annotations:
               summary: 'Prometheus targets down!'
               description: '{{ "{{" }} printf "%0.00f" $value {{ "}}" }}% of the Prometheus job are down!'


Add custom Prometheus recording rules via config map


You can use a custom config map for storing additional rules files


spec:
  features:
    monitoring:
      values:
        prometheus:
          config:
            extraRulesConfigmaps:
              - name: prometheus-rules-configmap
                fileMask: '*.yaml'


Customize OIDC Roles name for access


spec:
  features:
    monitoring:
      values:
        grafana:
          authentication:
            oidc:
              role: "admin"
        prometheus:
          authentication:
            oidc:
              role: "admin"
        alertmanager:
          authentication:
            oidc:
              role: "admin"


Add custom scrape jobs

Note that the scrape rules need to be added in the specification of the cluster in which the metrics need to be scraped, not in the Kublr Control Plane cluster specification.

If the scrape rules are added in the KCP cluster spec, it will only affect the KCP cluster, not the managed clusters.


spec:
  features:
    monitoring:
      values:
        prometheus:
          config:
            scrapeJobs: |
              - job_name: istiod
                kubernetes_sd_configs:
                - namespaces:
                    names:
                      - istio-system
                  role: endpoints
                relabel_configs:
                - action: keep
                  regex: istiod;http-monitoring
                  source_labels:
                    - __meta_kubernetes_service_name
                    - __meta_kubernetes_endpoint_port_name