TABLE OF CONTENTS


Recreate Keycloak Data PVC


Make sure that you have admin access to the KCP Kubrnetes cluster with cluster admin permissions.


NB! DANGEROUS OPERATION! This operation may lead to a data loss, make sure that you back up keycloak data.


TODO: not tested on keycloak and postgres HA configurations


1. Scale keycloak and keycloak database down to 0

kubectl -n kublr scale statefulset kcp-keycloak --replicas=0
kubectl -n kublr scale statefulset kcp-keycloak-postgres --replicas=0


2. (if necessary) Delete keycloak data PVC.

If deleted, the PVC will be recreated and re-allocated on the next step:

kubectl -n kublr delete pvc data-kcp-keycloak-postgres-0

# etc if HA mode is used for the kecloak database
kubectl -n kublr delete pvc data-kcp-keycloak-postgres-1
kubectl -n kublr delete pvc data-kcp-keycloak-postgres-2


3. (if necessary) Mark keycloak to reinitialize Kublr objects (see also the next section for more information):

kubectl patch secret -n kublr kcp-keycloak-lock \
  --type json -p='[{"op": "remove", "path": "/data/realm-init.lock"}]'


4. Scale keycloak postgres and keycloak back up (adjust the number of replicas as required) and verify that the PVC was recreated

kubectl -n kublr scale statefulset kcp-keycloak-postgres --replicas=1
kubectl -n kublr scale statefulset kcp-keycloak --replicas=1


Recreating Kublr Objects in Keycloak


Kublr requires a number of objects - realm, groups, users, roles, role mappings, etc - to exist in Keycloak.

The objects are created automatically and can be recreated using the following procedure.

If the procedure is used when the objects already exist, it will try to recover them as much as possible without modifications to those objects that already exist.


1. Mark keycloak to reinitialize Kublr objects (see also the next section for more information):

kubectl patch secret -n kublr kcp-keycloak-lock \
  --type json -p='[{"op": "remove", "path": "/data/realm-init.lock"}]'


2. Restart keycloak pods:

kubectl -n kublr delete pod -l=app.kubernetes.io/name=app-keycloak

# OR with immediate restart

kubectl -n kublr delete pod -l=app.kubernetes.io/name=app-keycloak \
  --force --grace-period=0


Refresh Kublr Componets JWKS keys


When Kublr keycloak objects are recreated, Kublr components have to recreate their corresponding client credentials.

Each client will also re-create a corresponding OIDC client in Keycloak kublr-ui realm.


1. Clear the client JWKS init markers:

# for all components

kubectl patch -n kublr $(kubectl -n kublr get secret -o name | grep -- '-jwks') \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'

# OR for specific components

kubectl patch -n kublr secret/kcp-kublr-api-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kcp-kublr-ui-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kcp-backup-controller-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kcp-cluster-controller-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kcp-feature-controller-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kcp-terraform-controller-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kublr-logging-controller-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kublr-logging-kibana-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kublr-monitoring-controller-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kublr-monitoring-prometheus-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kublr-monitoring-alertmanager-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'
kubectl patch -n kublr secret/kublr-monitoring-grafana-jwks \
  --type json -p='[{"op": "remove", "path": "/data/client-init.lock"}]'


2. Restart clients:

# All Kublr pods (not recommended for large installations)

kubectl -n kublr delete $(kubectl -n kublr get pods -o name)

# OR specific components
# The list of components as well as labels used may change with Kublr versions.

kubectl -n kublr delete pods -l=app=kcp-kublr-api
kubectl -n kublr delete pods -l=app=kcp-kublr-ui
kubectl -n kublr delete pods -l=app=kcp-backup-controller
kubectl -n kublr delete pods -l=app=kcp-cluster-controller
kubectl -n kublr delete pods -l=app=kcp-feature-controller
kubectl -n kublr delete pods -l=app=kcp-terraform-controller

kubectl -n kublr delete pods -l=app=kublr-logging-controller
kubectl -n kublr delete pods -l=k8s-app=kublr-logging-kibana

kubectl -n kublr delete pods -l=app=kublr-monitoring-monitoring-controller
kubectl -n kublr delete pods -l=app=kublr-monitoring-alertmanager
kubectl -n kublr delete pods -l=app=kublr-monitoring-grafana
kubectl -n kublr delete pods -l=app=kublr-monitoring-prometheus