TABLE OF CONTENTS


Tags: aws, persistence, storage, efs, security, encryption, kms


Kublr Cluster Configuration


The following cluster specification excerpt deploys a cluster on AWS and additionally creates an EFS and integrates it with the cluster using NFS-client-provisioner.


Important notes:

  • The following spec is not a full cluster specification;
    it only includes excerpts, that have to be added to a full cluster specification in order to set up EFS integration.
  • The configuration depends on the cluster AZs used and the subnet configuration in the cluster.
    EFS mount targets must be created in each AZ in one of the cluster subnets available in that AZ.
    As a result the list of Mount Targets specified in the cluster spec should be updated according to the cluster node groups configuration changes.
  • Note that .lcl TLD is used for cluster-local hosted zone.
    DO NOT use .local as it may be used by AWS. 


spec:
  locations:
    - aws:

        # EFS configuration depends on the AZs used, so take note
        # of the availability zones section
        availabilityZones:
          - us-east-1a
          - us-east-1c

        resourcesCloudFormationExtras:

          # EFS is configured in this resource
          CustomEFS:
            Type: AWS::EFS::FileSystem
            Properties: {}

          # One MountTarget resource must be created for each AZ used by the cluster
          # Kublr uses AZ enumeration convention so that AZ "a" takes number 0,
          # AZ "b" - 1, AZ "c" - 2 etc

          # MountTarget for AZ "a"
          CustomEFSMT0:
            Type: AWS::EFS::MountTarget
            Properties:
              FileSystemId: { Ref: CustomEFS }
              SecurityGroups: [ { "Fn::GetAtt": [ NewVpc, DefaultSecurityGroup ] } ]
              # for a cluster with private nodes only "SubnetNodePrivate..."
              # may have to be used
              SubnetId: { Ref: SubnetNodePublic0 }

          # MountTarget for AZ "c"
          CustomEFSMT2:
            Type: AWS::EFS::MountTarget
            Properties:
              FileSystemId: { Ref: CustomEFS }
              SecurityGroups: [ { "Fn::GetAtt": [ NewVpc, DefaultSecurityGroup ] } ]
              SubnetId: { Ref: SubnetNodePublic2 }

          # private hosted zone is used so that EFS mount targets have
          # a stable DNS name inside the cluster
          CustomPrivateHostedZone:
            Type: AWS::Route53::HostedZone
            Properties:
              Name: mycluster.kublr.lcl
              VPCs: 
                - VPCId: { Ref: NewVpc }
                  VPCRegion: { Ref: 'AWS::Region' }
          CustomPrivateHostedZoneRecordSetEFS:
            Type: AWS::Route53::RecordSet
            Properties:
              HostedZoneId: { Ref: CustomPrivateHostedZone }
              Name: efs.mycluster.kublr.lcl
              ResourceRecords:
                - { "Fn::Sub": [ '${EFS}.efs.${AWS::Region}.amazonaws.com', { EFS: { Ref: CustomEFS } } ] }
              TTL: "300"
              Type: CNAME

  # packages section allows to specify additional helm packages to be
  # deployed to the cluster
  packages:

    # in this case we deploy NFS provisioner to integrate Kubernetes with EFS
    nfs-subdir-external-provisioner:
      helmVersion: '3.4.0'
      namespace: kube-system
      releaseName: nfs-subdir-external-provisioner
      chart:
        name: nfs-subdir-external-provisioner
        url: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/releases/download/nfs-subdir-external-provisioner-3.0.0/nfs-subdir-external-provisioner-3.0.0.tgz
      values:
        nfs:
          server: efs.mycluster.kublr.lcl
          path: /

        # uncomment if you want the EFS storage class to be default

        #storageClass:
        #  defaultClass: true

  # uncomment if you want EFS storage class to be default, and Kublr-specified
  # storage class (EBS) not to be default

  #features:
  #  system:
  #    values:
  #      storageClass:
  #        defaultClass: false


EFS KMS Encryption


EFS file system can be encrypted using AWS KMS keys.


Modify the above cluster specification as follows to create encrypted EFS:


spec:
  locations:
    - aws:
        resourcesCloudFormationExtras:
          CustomEFS:
            Type: AWS::EFS::FileSystem
            Properties:
              Encrypted: true
              # if KmsKeyId is omitted, the default account key will be used by AWS
              KmsKeyId: 'alias/kms-key-id'



AWS IAM Configuration


For Kublr to be able to provision and manage clusters with additional AWS resources, the Kublr IAM user should have corresponding permissions.

In case of EFS the following permissions need to be added to Kublr IAM account in addition to standard policies described in Kublr documentation:


{
  "Version": "2012-10-17",
  "Statement": [
    {
     "Sid" : "Stmt1PermissionForAllEFSActions",
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:*",
        "route53:*"
      ],
      "Resource": "*"
    }
  ]
}