Tags: tiller


TABLE OF CONTENTS


Issue Description


1. When creating a new or updating an exiting cluster, the cluster create or update process is stuck on the "Helm/Tiller" condition not OK with all other higher priority conditions being OK.


2. "Helm/Tiller" condition is not OK for an existing cluster with all other conditions being OK.


Background


Early versions of Kublr (1.17 and below) used Helm 2 to install in the cluster additional packages such as ingress controller and centralized monitoring and logging addons. This required that Tiller (Helm 2 server-side in-cluster component) was installed in each cluster.


Tiller unavailability could cause cluster malfunctioning and as a result Kublr includes Tiller deployment in each cluster by default and checks that a Tiller pod is available and healthy in a cluster before completing cluster operations such as cluster creation or update.


Starting with Kublr 1.18 Helm packages are deployed by Helm 3. It simplifies operations and increases security as Helm 3 does not need server-side (Tiller) component to deploy packages.

At the same time Kublr kept the requirement to run Tiller in the clusters for backward compatibility and to ensure that automatic cluster upgrade and migration works.


Kublr configured Tiller using an image from Google image repository: gcr.io/kubernetes-helm/tiller:v2.14.3

Unfortunately this image was discontinued in Google repository in August 2021.

This created issues for some versions of Kublr due to the fact that Kublr cluster create and update operations cannot complete until Tiller is successfully started, and Tiller cannot start because the image is not available in the image repository.


Solutions


Option 1. Use Kublr Control Plane and Agent versions that do not require Tiller


The described issue is fixed in the following Kublr versions:

  • Kublr Control Plane - 1.21.2 and above
  • Kublr Operator - 1.21.2 and above
  • Kublr Agent 1.17 - 1.17.17-25 and above
  • Kublr Agent 1.18 - 1.18.20-26 and above
  • Kublr Agent 1.19 - 1.19.13-48 and above
  • Kublr Agent 1.20 - 1.20.9-18 and above
  • Kublr Agent 1.21 - 1.21.3-10 and above
  • Kublr Agent 1.22 and above - all versions


Option 2. Override Tiller Image in the cluster specification


If newer versions of Kublr Control Plane and Kublr Agent cannot be used, a valid Tiller image must be specified in a cluster spec.


For existing cluster


To do that for an existing cluster first check current Tiller image using the following command:


echo "$(kubectl get deployments \
  -n kube-system tiller-deploy \
  -o jsonpath='{.spec.template.spec.containers[0].image}')"


If the image used is gcr.io/kubernetes-helm/tiller:v2.14.3, then update it to use either the Kublr public image registry (recommended), or Docker Hub public registry, or your custom private registry if available:


kubectl patch deployment \
  -n kube-system tiller-deploy \
  --type='json' \
  -p='[{"op": "replace",
        "path": "/spec/template/spec/containers/0/image",
        "value":"cr.kublr.com/kubernetes-helm/tiller:v2.14.3"}]'


(Optional) Include the following snippet into the cluster specification:


spec:
  master:
    kublrAgentConfig:
      kublr:
        docker_image:
          helm_tiller: cr.kublr.com/kubernetes-helm/tiller:v2.14.3


You can use other tiller images and/or versions as necessary.


For new cluster


For a new cluster it is enough to just include the snippet above in the cluster specification, there is no need to run the kubectl patch command.


Option 3. Use Air-Gapped Kublr Deployment


Air-Gapped Kublr deployment is described in Kublr Air-Gapped deployment documentation.

When deployed in Air-Gapped mode, Kublr uses a local private image repository to get required docker images including Tiller docker image.