Tags: aws, security, network


By default Kublr automatically creates all AWS resources required to run a secure Kubernetes cluster in a single AWS CloudFormation stack.


This includes a set of standard subnets for master and worker nodes of the cluster as well as for NAT gateways and public ELBs when necessary.


At the same time, sometimes it is necessary to use custom network topology to implement complex or extended use-cases.


Kublr supports it via specification of existing subnet.


It is also possible to create custom subnets as a part of the cluster specification as described in the article.


spec:
  nodes:
    - locations:
        - aws:
            # each AWS instance group (workers or master) in Kublr runs
            # in a specified set of AWS availability zones
            availabilityZones:
              - us-east-1c
              - us-east-1f

            # subnetIds property allows to specify subnets to be used
            # by the nodes of this instance group in each availability zone.
            # subnetIds array must have the same length as the availabilityZones
            # set for this instance group.
            # An empty string value means that the worker nodes in
            # the corresponding AZ will run in a default Kublr-generate subnet;
            # while non-empty string specifies the ID of an existing subnet that
            # needs to be used by worker nodes in this AZ.
            subnetIds:
              - ''
              - subnet-123456789

The example above shows how subnets created outside Kublr can be used.


This approach has certain limitations, in particular the subnets must already exist before the cluster is created or updated.


To simplify this, custom subnet definition(s) can be included in the cluster specification and referred to from the subnetIds properties as follows:


spec:
  locations:
    - name: aws1
      aws:
        # custom subnet definition in CloudFormation format is
        # included in the location specification.
        resourcesCloudFormationExtras:

          SubnetCustom:
            Type: 'AWS::EC2::Subnet'
            Properties:
              # for custom subnets the user is responsible for the correct
              # definition: correct AZ, non-conflicting CIDR, public/private
              # properties, routing etc
              AvailabilityZone: 'us-east-1f'
              CidrBlock: '172.16.96.0/20'
              MapPublicIpOnLaunch: true
              Tags:
                - Key: Name
                  Value: { 'Fn::Sub': [ '${AWS::StackName}-subnet-custom', {} ] }
                - Key: KubernetesCluster
                  Value: { Ref: KubernetesCluster }
                - Key: { 'Fn::Sub': [ 'kubernetes.io/cluster/${KubernetesCluster}', {} ] }
                  Value: 'owned'
              VpcId: { Ref: NewVpc }

          # in this example the subnet is associated with the default route
          # created by Kublr.
          # At the same time it is also possible to specify a custom route
          # table as well.
          RtAssocSubnetCustom:
            Type: 'AWS::EC2::SubnetRouteTableAssociation'
            Properties:
              RouteTableId: { Ref: RouteTablePublic }
              SubnetId: { Ref: SubnetCustom }

  nodes:
    - locations:
        - aws:
            availabilityZones:
              - us-east-1c
              - us-east-1f

            # here instead of a fixed string with subnet ID a CloudFormation
            # Ref function is used
            subnetIds:
              - ''
              - { Ref: SubnetCustom }