Tags: security


TABLE OF CONTENTS


Overview


Kublr-provisioned Kubernetes clusters maintain their own secret store that is used to exchange keys, certificates and other credentials between master and worker nodes and the clients. In particular, Kublr Control Plane accesses a cluster's secret store to get the cluster Kubeconfig file to communicate with the cluster API.


Clusters provisioned in the clouds use cloud-specific secret store type: AWS - S3 buckets, Azure - storage account blob store containers, GCP - Google cloud storage buckets.

On-prem clusters (vSphere, vCD, BYOI) on the other hand use a custom secret storage implemented by Kublr agents running on the master nodes.


Kublr-managed Kubernetes clusters will automatically generate and rotate the secret packages when certificates are expired or are missing from the package. This property is used to implement on-demand rotation of the secrets.


This article describes the secrets rotation procedure using admin kubeconfig file rotation as an example.


Prerequisites

  • Bash
  • AWS / Azure / GCP CLI tools
  • openssl
  • curl
  • jq


1. Prepare Parameters and/or Shell Functions

AWS

The only parameter needed to rotate AWS Kublr Kubernetes cluster secrets is the name of S3 bucket for the cluster.

This name may be found in the cluster specification spec.secretStore.awsS3.s3BucketName section.

S3_BUCKET=...
Azure

To work with Azure cluster you need to know its secret store Azure Storage Account and Storage Container names. These parameters may be found in the cluster specification spec.secretStore.azureAS.storageAccountName and spec.secretStore.azureAS.storageContainerName sections correspondingly.

STORAGE_ACCOUNT=...
STORAGE_CONTAINER=...
GCP

To work with a GCP cluster you need to know the Google project ID and its secret store storage bucket. These parameters may be found in the cluster specification spec.locations.[0].gcp.projectId and spec.secretStore.googleGCS.bucketName sections correspondingly.

PROJECT_ID=...
STORAGE_BUCKET=...
Bare metal / on-prem (vSphere, vCD, BYOI)

On-prem Kublr Kubernetes clusters use a secret store implemented by the Kublr Agent running on masters. This secret store API uses the authinication scheme similar to that of AWS S3. The scheme requires generating request cryptographic signature, so the preparatation steps are a bit more complicated than those for other clouds.


Parameters ncessary to access kublr secret store may be found in the cluster specification spec.master.locations.[0].baremetal.hosts[0].address, spec.secretStore.kublrAgent.port, spec.secretStore.kublrAgent.accessKeys[role="master"].accessKeyId, and spec.secretStore.kublrAgent.accessKeys[role="master"].secretAccessKey sections correspondingly.


MASTER_ADDRESS_HOST=...
MASTER_ADDRESS_PORT=11251
MASTER_ADDRESS="${MASTER_ADDRESS_HOST}:${MASTER_ADDRESS_PORT:-11251}"
ACCESS_KEY_ID=...
SECRET_ACCESS_KEY=...


In addition to specifying the secret store connection parameters, a shell function implementing the request signing algorithm needs to be registered:


# Usage: kublrsign <unix-date> <http-method> <http-path> <content-sha256>
#   <unix-date> - unix date; if empty, current date is used
#   <http-method> - http method; if empty, "GET" is used
#   <http-path> - http path; if empty, "/secret-store" is used
#   <content-sha256> - requests content hash; if empty, default value of sha256("") is used
#
# Example: kublrsign "$(date -u '+%s')" "GET" "/secret-store" ""
#
function kublrsign() {
  local CONTENT_HASH="${1:-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855}"

  local DATE="${2:-"$(date -u '+%s')"}"
  local DATE1123="$(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')"

  local SCOPE="$(date -u  -d "@${DATE}" '+%Y%m%d')/cluster/secret_store/kublr_request"

  local CANONICAL_REQ="${3:-GET}
${4:-"/secret-store"}

date:${DATE1123}
x-kublr-content-sha256:${CONTENT_HASH}

date;x-kublr-content-sha256
${CONTENT_HASH}"

  local CANONICAL_REQ_HASH="$(echo -n "${CANONICAL_REQ}" |
    openssl dgst -sha256 -binary | xxd -p -c 64)"

  local STRING_TO_SIGN="KUBLR-HMAC-SHA256
$(date -u  -d "@${DATE}" '+%Y%m%dT%H%M%SZ')
${SCOPE}
${CANONICAL_REQ_HASH}"

  local SIGNATURE="$(echo -n "${STRING_TO_SIGN}" |
    openssl sha256 -hex -mac HMAC -macopt "key:${SECRET_ACCESS_KEY}" -binary |
    xxd -p -c 64)"

  local AUTH="KUBLR-HMAC-SHA256 Credential=${ACCESS_KEY_ID}/${SCOPE},SignedHeaders=date;x-kublr-content-sha256,Signature=${SIGNATURE}"
  echo -n "${AUTH}"
}

2. Check packages in the secret store

AWS
aws s3api list-objects-v2 \
  --bucket "${S3_BUCKET}" \
  --prefix 'data/master/master-' \
  --output json |
  jq -r '[.Contents[].Key|select(endswith(".tgz"))]|sort|reverse'
Azure
az storage blob list \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --prefix data/master/master- \
  --output json |
  jq '[.[].name|select(endswith(".tgz"))]|sort|reverse'
GCP
gsutil ls -p "${PROJECT_ID}" \
  "gs://${STORAGE_BUCKET}/data/master/master-*.tgz" |
  jq -R --arg pref "gs://${STORAGE_BUCKET}/data/master/" \
  '[.|ltrimstr($pref)]' | jq -s 'add|sort|reverse'
Bare metal / on-prem (vSphere, vCD, BYOI)
DATE="$(date -u '+%s')"
AUTH="$(kublrsign "${DATE}" GET /secret-store )"
curl -k -XGET \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "authorization: ${AUTH}" \
  https://${MASTER_ADDRESS}/secret-store |
  jq -r '[.contents[].name|select(endswith(".tgz") and startswith("data/master/master-"))]|
    sort|reverse'

3. Download current package

AWS
PACKAGE_CUR="$(aws s3api list-objects-v2 \
  --bucket "${S3_BUCKET}" \
  --prefix 'data/master/master-' \
  --output json |
  jq -r '[.Contents[].Key|select(endswith(".tgz"))]|max_by(.)|
    ltrimstr("data/master/")|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

aws s3api get-object \
  --bucket "${S3_BUCKET}" \
  --key "data/master/${PACKAGE_CUR}.tgz" "${PACKAGE_CUR}.tgz"
Azure
PACKAGE_CUR="$(az storage blob list \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --prefix 'data/master/master-' \
  --output json |
  jq -r '[.[].name|select(endswith(".tgz"))]|max_by(.)|
    ltrimstr("data/master/")|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

az storage blob download \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --name "data/master/${PACKAGE_CUR}.tgz" \
  --file "${PACKAGE_CUR}.tgz"
GCP
PACKAGE_CUR="$(gsutil ls -p "${PROJECT_ID}" \
  "gs://${STORAGE_BUCKET}/data/master/master-*.tgz" |
  jq -Rr --arg pref "gs://${STORAGE_BUCKET}/data/master/" \
  '[.|ltrimstr($pref)]' | jq -sr 'add|max_by(.)|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

gsutil cp \
  "gs://${STORAGE_BUCKET}/data/master/${PACKAGE_CUR}.tgz" \
  "${PACKAGE_CUR}.tgz"
Bare metal / on-prem (vSphere, vCD, BYOI)
DATE="$(date -u '+%s')"
AUTH="$(kublrsign "${DATE}" GET /secret-store )"
PACKAGE_CUR="$(curl -k -XGET \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "authorization: ${AUTH}" \
  https://${MASTER_ADDRESS}/secret-store |
  jq -r '[.contents[].name|select(endswith(".tgz") and startswith("data/master/master-"))]|
    max_by(.)|ltrimstr("data/master/")|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

DATE="$(date -u '+%s')"
AUTH="$(kublrsign "${DATE}" GET "/secret-store/data/master/${PACKAGE_CUR}.tgz" )"
curl -k -XGET \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "authorization: ${AUTH}" \
  "https://${MASTER_ADDRESS}/secret-store/data/master/${PACKAGE_CUR}.tgz" > \
  "${PACKAGE_CUR}.tgz"

4. Generate new package


# Generate the new package name

PACKAGE_NEW="${PACKAGE_CUR}.tgz.$(date -u '+%F-%H-%M-%S')"
echo "PACKAGE_NEW=${PACKAGE_NEW}"

# Generate the new package by excluding
# the secret files to be rotated

gzip -d -c "${PACKAGE_CUR}.tgz" |
  tar --delete \
    './config' './basic_auth.csv' \
    './known_tokens.csv' './kubecfg.crt' './kubecfg.key' \
  | gzip - > "${PACKAGE_NEW}.tgz"

5. Upload the new package

AWS
aws s3api put-object \
  --bucket "${S3_BUCKET}" \
  --key "data/master/${PACKAGE_NEW}.tgz" \
  --body "${PACKAGE_NEW}.tgz"
Azure
az storage blob upload \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --name "data/master/${PACKAGE_NEW}.tgz" \
  --file "${PACKAGE_NEW}.tgz"
GCP
gsutil cp \
  "${PACKAGE_NEW}.tgz" \
  "gs://${STORAGE_BUCKET}/data/master/${PACKAGE_NEW}.tgz"
Bare metal / on-prem (vSphere, vCD, BYOI)
DATE="$(date -u '+%s')"
SHA256="$(openssl dgst -sha256 -binary < "${PACKAGE_NEW}.tgz" | xxd -p -c 64)"
AUTH="$(kublrsign "${DATE}" PUT "/secret-store/data/master/${PACKAGE_NEW}.tgz" "${SHA256}")"
curl -k -XPUT \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: ${SHA256}" \
  -H "authorization: ${AUTH}" \
  --data-binary "@${PACKAGE_NEW}.tgz" \
  "https://${MASTER_ADDRESS}/secret-store/data/master/${PACKAGE_NEW}.tgz"