[Supported in Kublr 1.20.0 and later]

[Important changes between Kublr 1.21 and 1.22]


TABLE OF CONTENTS


Overview


Kublr allows full customization of VM (virtual machines) and VMSS (virtual machine scale sets), including customizing VM OS images and plans.


One of the frequently asked questions is how to run a Kublr Kubernetes cluster on virtual machines that require terms acceptance, including for example Ubuntu FIPS-enabled image.


This article shows how VM and VMSS resource definitions created by Kublr can be customized in the Kublr cluster specification using Ubuntu FIPS-enabled image as an example.


Specifying VM Image


VM and VMSS images are defined in Kublr for each node group separately in the node group's locations[0].azure.osDisk property. It can be defined for both/either master and worker groups.


An image can be defined via one and ony one (for each node groups) of the following combinations:

  • image publisher, offer, sku, and optional version with Kublr properties imagePublisher, imageOffer, imageSku, and imageVersion
  • [starting with Kublr 1.20.1] image resource id with Kublr property imageResourceId
  • [deprecated, use imageResourceId instead] image resource id of an image in a resource group in the same subscription as the cluster with Kublr properties imageResourceGroup and imageId


Note, that there is an important change in the cluster specification structure between Kublr 1.21 and Kublr 1.22.

  • In Kublr 1.21 the property imageVersion was used to provide the image SKU, and the actual Azure image version had to be changed via ARM template extras section.
  • In Kublr 1.22 this inconsistency was corrected, and Azure image SKU is specified via imageSku Kublr property, and imageVersion Kublr property is used to provide the Azure image version.
  • Migration between the old and the new convention for existing clusters is done automatically on the first update of the cluster in Kublr Control Plane 1.22.


When image version is not specified, latest is used by Azure.


Example specification in Kublr 1.22+ via publisher/offer/sku/version


spec:
  ...
  master:
    locations:
      - locationRef: default
        azure:
          osDisk:
            type: image
            imagePublisher: canonical
            imageOffer: 0001-com-ubuntu-server-focal
            imageSku: 20_04-lts
            # imageVersion property is optional;
            # if not specified, 'latest' is used
            imageVersion: 20.04.202107200
  nodes:
    - ...
      locations:
        - locationRef: default
          azure:
            osDisk:
              type: image
              imagePublisher: canonical
              imageOffer: 0001-com-ubuntu-server-focal
              imageSku: 20_04-lts
              # imageVersion property is optional;
              # if not specified, 'latest' is used
              imageVersion: 20.04.202107200


Example specification in Kublr pre-1.22 via publisher/offer/sku/version


spec:
  ...
  master:
    locations:
      - locationRef: default
        azure:
          osDisk:
            type: image
            imagePublisher: canonical
            imageOffer: 0001-com-ubuntu-server-focal
            imageVersion: 20_04-lts
          armTemplateExtras:
            scaleSet:
              properties:
                virtualMachineProfile:
                  storageProfile:
                    imageReference:
                      version: 20.04.202107200
            virtualMachine:
              properties:
                storageProfile:
                  imageReference:
                    version: 20.04.202107200
  ...
  nodes:
    - ...
      locations:
        - locationRef: default
          azure:
            osDisk:
              type: image
              imagePublisher: canonical
              imageOffer: 0001-com-ubuntu-server-focal
              imageVersion: 20_04-lts
            armTemplateExtras:
              scaleSet:
                properties:
                  virtualMachineProfile:
                    storageProfile:
                      imageReference:
                        version: 20.04.202107200
              virtualMachine:
                properties:
                  storageProfile:
                    imageReference:
                      version: 20.04.202107200

Note that both virtualMachine and scaleSet section are customized and included in the node group definition. It is safe to do so although unnecessary if you know specific groupType for this node group.


Using Images with Custom Plan (e.g. FIPS)


First of all make sure that the image terms are accepted under the subscription you are using to deploy cluster:


# list all available images
az vm image list --all

# list all images available from a specific publisher under a specific offer
az vm image list --offer 0001-com-ubuntu-pro-bionic-fips --publisher canonical --all

# show a specific image properties
az vm image terms show --urn Canonical:0001-com-ubuntu-pro-bionic-fips:pro-fips-18_04:18.04.202010201
az vm image terms show --offer 0001-com-ubuntu-pro-bionic-fips --publisher canonical --plan pro-fips-18_04

# accept term for a specific image
az vm image terms accept --offer 0001-com-ubuntu-pro-bionic-fips --publisher canonical --plan pro-fips-18_04

Check Microsoft Azure documentation for more information about the command line tools and APIs used to work with images:


Customize Kublr cluster specification to use a specific image under a specific plan:


spec:
  ...
  master:
    locations:
      - locationRef: default
        azure:
          # FIPS enabled ubuntu image
          osDisk:
            imageOffer: 0001-com-ubuntu-pro-bionic-fips
            imagePublisher: canonical
            imageSku: pro-fips-18_04
            type: image
            diskSizeGb: 40

          armTemplateExtras:

            # everything specified in this section will be directly merged into
            # the AvailabilitySet resource definition(s) created by Kublr for this
            # node group
            availabilitySet:

            # everything specified in this section will be directly merged into
            # the VM resource definition(s) created by Kublr for this node group
            virtualMachine:
              plan:
                name: pro-fips-18_04
                product: 0001-com-ubuntu-pro-bionic-fips
                publisher: canonical

            # everything specified in this section will be directly merged into
            # the VMSS resource definition(s) created by Kublr for this node group
            scaleSet:
              plan:
                name: pro-fips-18_04
                product: 0001-com-ubuntu-pro-bionic-fips
                publisher: canonical

In the example above only master node group is customized. Worker node groups can be customized the same way.


Note that both virtualMachine and scaleSet section are customized and included in the node group definition. It is safe to do so although unnecessary if you know specific groupType for this node group.


Node groups with groupType = 'AvailabilitySet' (default for stateful groups and the master) will only use overrides in the virtualMachine and availabilitySet sections and ignore the scaleSet section.


Node groups with groupType = 'VirtualMachineScaleSet' (default for stateless groups and the workers) will only use overrides in the scaleSet section and ignore the virtualMachine section.


Overrides for groups with groupType = 'AvailabilitySetLegacy' (inherited from clusters created by Kublr 1.19 and before) are not supported.


References


Refer to Azure documentation for more information about VM and VMSS customizable properties: