Tags: security

TABLE OF CONTENTS

Overview

Kublr-provisioned Kubernetes clusters maintain their own secret store that is used to exchange keys, certificates and other credentials between master and worker nodes and the clients. In particular, Kublr Control Plane accesses a cluster's secret store to get the cluster Kubeconfig file to communicate with the cluster API.

Clusters provisioned in the clouds use cloud-specific secret store type: AWS - S3 buckets, Azure - storage account blob store containers, GCP - Google cloud storage buckets.

On-prem clusters (vSphere, vCD, BYOI) on the other hand use a custom secret storage implemented by Kublr agents running on the master nodes.

Kublr-managed Kubernetes clusters will automatically generate and rotate the secret packages when certificates are expired or are missing from the package. This property is used to implement on-demand rotation of the secrets.

This article describes the secrets rotation procedure using admin kubeconfig file rotation as an example. To fulfil this particular senario you can follow steps below or download and execute bash script in the end of the article.

Prerequisites

Bash
AWS / Azure / GCP CLI tools
openssl
curl
jq

1. Prepare Parameters and/or Shell Functions

AWS

The only parameter needed to rotate AWS Kublr Kubernetes cluster secrets is the name of S3 bucket for the cluster.

This name may be found in the cluster specification spec.secretStore.awsS3.s3BucketName section.

S3_BUCKET=...

Azure

To work with Azure cluster you need to know its secret store Azure Storage Account and Storage Container names. These parameters may be found in the cluster specification spec.secretStore.azureAS.storageAccountName and spec.secretStore.azureAS.storageContainerName sections correspondingly.

STORAGE_ACCOUNT=...
STORAGE_CONTAINER=...

GCP

To work with a GCP cluster you need to know the Google project ID and its secret store storage bucket. These parameters may be found in the cluster specification spec.locations.[0].gcp.projectId and spec.secretStore.googleGCS.bucketName sections correspondingly.

PROJECT_ID=...
STORAGE_BUCKET=...

Bare metal / on-prem (vSphere, vCD, BYOI)

On-prem Kublr Kubernetes clusters use a secret store implemented by the Kublr Agent running on masters. This secret store API uses the authinication scheme similar to that of AWS S3. The scheme requires generating request cryptographic signature, so the preparatation steps are a bit more complicated than those for other clouds.

Parameters ncessary to access kublr secret store may be found in the cluster specification spec.master.locations.[0].baremetal.hosts[0].address, spec.secretStore.kublrAgent.port, spec.secretStore.kublrAgent.accessKeys[role="master"].accessKeyId, and spec.secretStore.kublrAgent.accessKeys[role="master"].secretAccessKey sections correspondingly.

MASTER_ADDRESS_HOST=...
MASTER_ADDRESS_PORT=11251
MASTER_ADDRESS="${MASTER_ADDRESS_HOST}:${MASTER_ADDRESS_PORT:-11251}"
ACCESS_KEY_ID=...
SECRET_ACCESS_KEY=...

In addition to specifying the secret store connection parameters, a shell function implementing the request signing algorithm needs to be registered:

# Usage: kublrsign <unix-date> <http-method> <http-path> <content-sha256>
#   <unix-date> - unix date; if empty, current date is used
#   <http-method> - http method; if empty, "GET" is used
#   <http-path> - http path; if empty, "/secret-store" is used
#   <content-sha256> - requests content hash; if empty, default value of sha256("") is used
#
# Example: kublrsign "$(date -u '+%s')" "GET" "/secret-store" ""
#
function kublrsign() {
  local CONTENT_HASH="${1:-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855}"

  local DATE="${2:-"$(date -u '+%s')"}"
  local DATE1123="$(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')"

  local SCOPE="$(date -u  -d "@${DATE}" '+%Y%m%d')/cluster/secret_store/kublr_request"

  local CANONICAL_REQ="${3:-GET}
${4:-"/secret-store"}

date:${DATE1123}
x-kublr-content-sha256:${CONTENT_HASH}

date;x-kublr-content-sha256
${CONTENT_HASH}"

  local CANONICAL_REQ_HASH="$(echo -n "${CANONICAL_REQ}" |
    openssl dgst -sha256 -binary | xxd -p -c 64)"

  local STRING_TO_SIGN="KUBLR-HMAC-SHA256
$(date -u  -d "@${DATE}" '+%Y%m%dT%H%M%SZ')
${SCOPE}
${CANONICAL_REQ_HASH}"

  local SIGNATURE="$(echo -n "${STRING_TO_SIGN}" |
    openssl sha256 -hex -mac HMAC -macopt "key:${SECRET_ACCESS_KEY}" -binary |
    xxd -p -c 64)"

  local AUTH="KUBLR-HMAC-SHA256 Credential=${ACCESS_KEY_ID}/${SCOPE},SignedHeaders=date;x-kublr-content-sha256,Signature=${SIGNATURE}"
  echo -n "${AUTH}"
}

2. Check packages in the secret store

AWS

aws s3api list-objects-v2 \
  --bucket "${S3_BUCKET}" \
  --prefix 'data/master/master-' \
  --output json |
  jq -r '[.Contents[].Key|select(endswith(".tgz"))]|sort|reverse'

Azure

az storage blob list \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --prefix data/master/master- \
  --output json |
  jq '[.[].name|select(endswith(".tgz"))]|sort|reverse'

GCP

gsutil ls -p "${PROJECT_ID}" \
  "gs://${STORAGE_BUCKET}/data/master/master-*.tgz" |
  jq -R --arg pref "gs://${STORAGE_BUCKET}/data/master/" \
  '[.|ltrimstr($pref)]' | jq -s 'add|sort|reverse'

Bare metal / on-prem (vSphere, vCD, BYOI)

DATE="$(date -u '+%s')"
AUTH="$(kublrsign "${DATE}" GET /secret-store )"
curl -k -XGET \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "authorization: ${AUTH}" \
  https://${MASTER_ADDRESS}/secret-store |
  jq -r '[.contents[].name|select(endswith(".tgz") and startswith("data/master/master-"))]|
    sort|reverse'

3. Download current package

AWS

PACKAGE_CUR="$(aws s3api list-objects-v2 \
  --bucket "${S3_BUCKET}" \
  --prefix 'data/master/master-' \
  --output json |
  jq -r '[.Contents[].Key|select(endswith(".tgz"))]|max_by(.)|
    ltrimstr("data/master/")|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

aws s3api get-object \
  --bucket "${S3_BUCKET}" \
  --key "data/master/${PACKAGE_CUR}.tgz" "${PACKAGE_CUR}.tgz"

Azure

PACKAGE_CUR="$(az storage blob list \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --prefix 'data/master/master-' \
  --output json |
  jq -r '[.[].name|select(endswith(".tgz"))]|max_by(.)|
    ltrimstr("data/master/")|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

az storage blob download \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --name "data/master/${PACKAGE_CUR}.tgz" \
  --file "${PACKAGE_CUR}.tgz"

GCP

PACKAGE_CUR="$(gsutil ls -p "${PROJECT_ID}" \
  "gs://${STORAGE_BUCKET}/data/master/master-*.tgz" |
  jq -Rr --arg pref "gs://${STORAGE_BUCKET}/data/master/" \
  '[.|ltrimstr($pref)]' | jq -sr 'add|max_by(.)|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

gsutil cp \
  "gs://${STORAGE_BUCKET}/data/master/${PACKAGE_CUR}.tgz" \
  "${PACKAGE_CUR}.tgz"

Bare metal / on-prem (vSphere, vCD, BYOI)

DATE="$(date -u '+%s')"
AUTH="$(kublrsign "${DATE}" GET /secret-store )"
PACKAGE_CUR="$(curl -k -XGET \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "authorization: ${AUTH}" \
  https://${MASTER_ADDRESS}/secret-store |
  jq -r '[.contents[].name|select(endswith(".tgz") and startswith("data/master/master-"))]|
    max_by(.)|ltrimstr("data/master/")|rtrimstr(".tgz")')"
echo "PACKAGE_CUR=${PACKAGE_CUR}"

DATE="$(date -u '+%s')"
AUTH="$(kublrsign "${DATE}" GET "/secret-store/data/master/${PACKAGE_CUR}.tgz" )"
curl -k -XGET \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  -H "authorization: ${AUTH}" \
  "https://${MASTER_ADDRESS}/secret-store/data/master/${PACKAGE_CUR}.tgz" > \
  "${PACKAGE_CUR}.tgz"

4. Generate new package

# Generate the new package name

PACKAGE_NEW="${PACKAGE_CUR}.tgz.$(date -u '+%F-%H-%M-%S')"
echo "PACKAGE_NEW=${PACKAGE_NEW}"

# Generate the new package by excluding
# the secret files to be rotated

gzip -d -c "${PACKAGE_CUR}.tgz" |
  tar --delete \
    './config' './basic_auth.csv' \
    './known_tokens.csv' './kubecfg.crt' './kubecfg.key' \
  | gzip - > "${PACKAGE_NEW}.tgz"

5. Upload the new package

AWS

aws s3api put-object \
  --bucket "${S3_BUCKET}" \
  --key "data/master/${PACKAGE_NEW}.tgz" \
  --body "${PACKAGE_NEW}.tgz"

Azure

az storage blob upload \
  --account-name "${STORAGE_ACCOUNT}" \
  --container-name "${STORAGE_CONTAINER}" \
  --name "data/master/${PACKAGE_NEW}.tgz" \
  --file "${PACKAGE_NEW}.tgz"

GCP

gsutil cp \
  "${PACKAGE_NEW}.tgz" \
  "gs://${STORAGE_BUCKET}/data/master/${PACKAGE_NEW}.tgz"

Bare metal / on-prem (vSphere, vCD, BYOI)

DATE="$(date -u '+%s')"
SHA256="$(openssl dgst -sha256 -binary < "${PACKAGE_NEW}.tgz" | xxd -p -c 64)"
AUTH="$(kublrsign "${DATE}" PUT "/secret-store/data/master/${PACKAGE_NEW}.tgz" "${SHA256}")"
curl -k -XPUT \
  -H "date: $(date -u  -d "@${DATE}" '+%a, %d %b %Y %T %Z')" \
  -H "x-kublr-content-sha256: ${SHA256}" \
  -H "authorization: ${AUTH}" \
  --data-binary "@${PACKAGE_NEW}.tgz" \
  "https://${MASTER_ADDRESS}/secret-store/data/master/${PACKAGE_NEW}.tgz"

Script to rotate admin kubeconfig file

AWS

awsRotateSecrets -p YOUR_AWS_CLI_PROFILE_NAME -b YOUR_AWS_S3_BUCKET_NAME

Azure

azureRotateSecrets -a YOUR_STORAGE_ACCOUNT_ID -c YOUR_STORAGE_CONTAINER_ID

GCP

gcpRotateSecrets -p YOUR_GCP_PROJECT_ID -b YOUR_STORAGE_BUCKET

Kublr

How can we help you today?

Rotate Secrets in Kublr Kubernetes Cluster Secrets Package Print

Overview

Prerequisites

1. Prepare Parameters and/or Shell Functions

2. Check packages in the secret store

3. Download current package

4. Generate new package

5. Upload the new package

Script to rotate admin kubeconfig file

How can we help you today?

Rotate Secrets in Kublr Kubernetes Cluster Secrets Package Print

Overview

Prerequisites

1. Prepare Parameters and/or Shell Functions

2. Check packages in the secret store

3. Download current package

4. Generate new package

5. Upload the new package

Script to rotate admin kubeconfig file

Related Articles