Tags: aws, security, network


TABLE OF CONTENTS


Overriding Default Kublr-generated Security Groups


By default Kublr automatically creates all AWS resources required to run a secure Kubernetes cluster in a single AWS CloudFormation stack.


This includes a set of standard security groups for master and worker nodes of the cluster.


At the same time, sometimes it is necessary to use more restrictive or permissive security groups to implement complex or extended use-cases.


Kublr supports it via specification of existing security groups.


It is also possible to create custom security groups as a part of the cluster specification as described in the article.


The following cluster specification snippets show cluster specification fields related to security groups management in Kublr:


spec:
  locations:
    - aws:
        # use this to tell Kublr that a default security group
        # for master should NOT be created and added to master instances
        skipSecurityGroupMaster: true

        # use this to tell Kublr that a default security group
        # for workers should NOT be created and added to worker instances
        skipSecurityGroupNode: true

        # use this to tell Kublr that a default security group
        # for existing VPC should NOT be created and added to cluster instances
        skipSecurityGroupDefault: true

        # use this property to specify addtional security groups to be added
        # to all instances
        existingSecurityGroupIds:
          - sg-123456789
          - sg-123456790

  master:
    locations:
      - aws:
          # add these security groups to the master nodes
          existingSecurityGroupIds:
            - sg-123456791

  nodes:
    - locations:
        - aws:
            # add these security groups to worker nodes in this instance group
            existingSecurityGroupIds:
              - sg-123456792


Including Custom Security Group Definitions in Kublr Cluster Specification


[Since Kublr 1.19.0] extra resources

[Since Kublr 1.19.3] Cloudformation Ref in the existingSecurityGroupIds property


The previous example shows how security groups created outside Kublr can be used.


This approach has certain limitations, in particular the security groups must already exist before the cluster is created or updated.


To simplify this, custom security group definition(s) can be included in the cluster specification and referred to from the existingSecurityGroupIds properties as follows:


spec:
  locations:
    - aws:
        # note that instead of a fixed string id we are using
        # a CloudFormation Ref function to reference the custom
        # security group included in resourcesCloudFormationExtras property below
        existingSecurityGroupIds:
          - { Ref: SgCustom }

        # additional resources to be included in the cluster's CloudFormation template
        resourcesCloudFormationExtras:
          SgCustom:
            Type: 'AWS::EC2::SecurityGroup'
            Properties:
              VpcId: { Ref: NewVpc }
              GroupDescription: { Ref: 'AWS::StackName' }
              SecurityGroupEgress:
                - CidrIp: 0.0.0.0/0
                  FromPort: 0
                  IpProtocol: -1
                  ToPort: 65535
              SecurityGroupIngress:
                - CidrIp: 0.0.0.0/0
                  FromPort: 22
                  IpProtocol: tcp
                  ToPort: 22
                - CidrIp: 0.0.0.0/0
                  FromPort: 443
                  IpProtocol: tcp
                  ToPort: 443
                - CidrIp: 0.0.0.0/0
                  FromPort: 30000
                  IpProtocol: tcp
                  ToPort: 32767
                - CidrIp: 0.0.0.0/0
                  FromPort: 30000
                  IpProtocol: udp
                  ToPort: 32767
              Tags:
                - Key: Name
                  Value: { 'Fn::Sub': [ '${AWS::StackName}-sg-custom', {} ] }
                - Key: KubernetesCluster
                  Value: { Ref: KubernetesCluster }

Reference:

Default Kublr-generated Master Security Group


By default the following rules are defined for the master security group:

  • All outgoing traffic is allowed
  • All incoming traffic to ports 22, Kubernetes API Port (usually 443 or 6443), and 30000-32767
  • All incoming traffic to all ports from other locations of the same cluster.
    This is rarely used, only in the clusters that have more than one location configured.


  SgMaster:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: !Ref 'AWS::StackName'

      # All outgoing traffic

      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0
          FromPort: 0
          IpProtocol: -1
          ToPort: 65535

      # All incoming traffic to ports 22, <k8s-api>, 30000-32767
      # All incoming traffic to all ports from other locations of the same cluster

      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          FromPort: 22
          IpProtocol: tcp
          ToPort: 22
        - CidrIp: 0.0.0.0/0
          FromPort: {{@root.cluster.spec.spec.network.apiServerSecurePort}}
          IpProtocol: tcp
          ToPort: {{@root.cluster.spec.spec.network.apiServerSecurePort}}
        - CidrIp: 0.0.0.0/0
          FromPort: 30000
          IpProtocol: tcp
          ToPort: 32767
        - CidrIp: 0.0.0.0/0
          FromPort: 30000
          IpProtocol: udp
          ToPort: 32767
        {{#each cluster.aws.accounts}}
        {{#each regions}}
        {{#each stacks}}
        - CidrIp: {{vpcCidrBlock}}
          FromPort: 0
          IpProtocol: -1
          ToPort: 65535
        {{/each}}
        {{/each}}
        {{/each}}
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}-sg-master'
        - Key: KubernetesCluster
          Value: !Ref KubernetesCluster
        - Key: k8s.io/role/master
          Value: 1
      VpcId: {{#if @root.stack.vpcId}}{{@root.stack.vpcId}}{{else}}!Ref NewVpc{{/if}}


Default Kublr-generated Worker Security Group


By default the following rules are defined for the worker security group:

  • All outgoing traffic is allowed
  • All incoming traffic to ports 22 and 30000-32767
  • All incoming traffic to all ports from other locations of the same cluster.
    This is rarely used, only in the clusters that have more than one location configured.


  SgNode:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: !Ref 'AWS::StackName'

      # All outgoing traffic

      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0
          FromPort: 0
          IpProtocol: -1
          ToPort: 65535

      # All incoming traffic to ports 22, 30000-32767
      # All incoming traffic to all ports from other locations of the same cluster

      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          FromPort: 22
          IpProtocol: tcp
          ToPort: 22
        - CidrIp: 0.0.0.0/0
          FromPort: 30000
          IpProtocol: tcp
          ToPort: 32767
        - CidrIp: 0.0.0.0/0
          FromPort: 30000
          IpProtocol: udp
          ToPort: 32767
        {{#each cluster.aws.accounts}}
        {{#each regions}}
        {{#each stacks}}
        - CidrIp: {{vpcCidrBlock}}
          FromPort: 0
          IpProtocol: -1
          ToPort: 65535
        {{/each}}
        {{/each}}
        {{/each}}
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}-sg-node'
        - Key: KubernetesCluster
          Value: !Ref KubernetesCluster
        - Key: k8s.io/role/node
          Value: 1
      VpcId: {{#if @root.stack.vpcId}}{{@root.stack.vpcId}}{{else}}!Ref NewVpc{{/if}}


Standard Kublr-generated "Default" Security Group (only for existing VPC)


If the cluster is deployed into an existing VPC, Kublr will create one more Security Group that will be assigned to all master and worker nodes and will allow all traffic on all ports within the same group.