Install k8s cluster in AWS with kOps

Installing our k8s cluster with kOps involves initiating its state store. Let's go ahead and do that, and then we will continue with configuring it before pushing big red button that says "Deploy".

We start off simple:

$ export KOPS_STATE_STORE=s3://bucket1
$ export KOPS_NAME=docks-www-tutorials.sherzod.com
$ kops create cluster --name $KOPS_NAME --zones=us-east-1a --cloud aws

I0211 16:07:09.493025  955069 new_cluster.go:1072]  Cloud Provider ID = aws
I0211 16:07:09.744421  955069 subnets.go:182] Assigned CIDR 172.20.32.0/19 to subnet us-east-1a
Previewing changes that will be made:
...
...
... < list of resources to be created >
...
Must specify --yes to apply changes

Cluster configuration has been created.

Suggestions:
 * list clusters with: kops get cluster
 * edit this cluster with: kops edit cluster docks-www-tutorials.sherzod.com
 * edit your node instance group: kops edit ig --name=docks-www-tutorials.sherzod.com nodes-us-east-1a
 * edit your master instance group: kops edit ig --name=docks-www-tutorials.sherzod.com master-us-east-1a

Finally configure your cluster with: kops update cluster --name docks-www-tutorials.sherzod.com --yes --admin

This simply initializes our cluster state, and doesnt create anything. Rather than providing everything in CLI, we are going to download the config and edit the values directly. That way we can check that config into a VCS (eg Git) and use it as source of truth. Plus, that will fit nicely into our eventual CI/CD pipeline for this cluster.

To get configuration to an entire cluster, run this command:

$ kops get $KOPS_NAME -o yaml > cluster-manifest.yaml

Feel free to commit this file into a Git repo for tracking the changes. The file used in this post can be found in this repo.

If we look at the resulting config file, there should be three sections:

  • kind: Cluster
  • kind: InstanceGroup, name: master-us-east-1a
  • kind: InstanceGroup, name: nodes-us-east-1a

What's nice about kOps that it picks sensible defaults like [kind="InstanceGroup"].spec.image, node instance sizes, kubernetes version (which may not be current, but not old either like EKS'). Of course, for production those values won't do

Let's start configuring the cluster and get going. I went ahead and created the resulting cluster with desired configuration, and exported its manifest again. We can diff the latter file to the original cluster config, and see what changes need to be made.

diff --git a/cluster-manifest.yaml b/cluster-manifest.yaml
index d970964..7c9a0c4 100644
--- a/cluster-manifest.yaml
+++ b/cluster-manifest.yaml
@@ -6,10 +6,27 @@ metadata:
 spec:
   api:
     dns: {}
+  authentication:
+    aws: {}
   authorization:
     rbac: {}
+  certManager:
+    enabled: true
   channel: stable
   cloudProvider: aws
+  clusterAutoscaler:
+    enabled: true
+    expander: least-waste
+    balanceSimilarNodeGroups: false
+    awsUseStaticInstanceList: false
+    scaleDownUtilizationThreshold: "0.5"
+    skipNodesWithLocalStorage: true
+    skipNodesWithSystemPods: true
+    newPodScaleUpDelay: 0s
+    scaleDownDelayAfterAdd: 10m0s
+    image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.0
+    cpuRequest: "100m"
+    memoryRequest: "300Mi"
   configBase: s3://com.sherzod.docks-www-tutorials/docks-www-tutorials.sherzod.com
   etcdClusters:
   - cpuRequest: 200m
@@ -29,6 +46,16 @@ spec:
   iam:
     allowContainerRegistry: true
     legacy: false
+    serviceAccountExternalPermissions:
+      # this bit creates IAM roles
+      # create the service accounts separately
+      # kubectl create sa ec2-readonly
+      - name: aws-reader
+        namespace: default
+        aws:
+          policyARNs:
+            - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
+            - arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess
   kubelet:
     anonymousAuth: false
   kubernetesApiAccess:
@@ -36,66 +63,95 @@ spec:
   - ::/0
   kubernetesVersion: 1.22.6
   masterPublicName: api.docks-www-tutorials.sherzod.com
-  networkCIDR: 172.20.0.0/16
+  metricsServer:
+    enabled: true
+  networkCIDR: 10.2.0.0/22
+  networkID: vpc-04fb9d63a5a16d599
   networking:
-    kubenet: {}
+    calico: {}
   nonMasqueradeCIDR: 100.64.0.0/10
+  serviceAccountIssuerDiscovery:
+    discoveryStore: s3://com.sherzod.docks-www-tutorials-pub
+    enableAWSOIDCProvider: true
   sshAccess:
-  - 0.0.0.0/0
-  - ::/0
+  - 1.2.3.4/32 # use your own IP address
+  sshKeyName: "kops-ssh-key"
   subnets:
-  - cidr: 172.20.32.0/19
-    name: us-east-1a
+  - cidr: 10.2.0.0/26
+    id: subnet-0fa7ebf92dae29925
+    name: us-east-1a-publ
+    type: Public
+    zone: us-east-1a
+  - cidr: 10.2.0.64/26
+    id: subnet-01ea1e4c285875c89
+    name: us-east-1b-publ
     type: Public
+    zone: us-east-1b
+  - cidr: 10.2.2.0/25
+    id: subnet-086c890182a284389
+    name: us-east-1a-priv
+    type: Private
     zone: us-east-1a
+  - cidr: 10.2.2.128/25
+    id: subnet-0f616cfdedd17b617
+    name: us-east-1b-priv
+    type: Private
+    zone: us-east-1b
   topology:
     dns:
       type: Public
     masters: public
-    nodes: public
+    nodes: private

 ---

 apiVersion: kops.k8s.io/v1alpha2
 kind: InstanceGroup
 metadata:
-  creationTimestamp: "2022-02-11T21:07:10Z"
+  creationTimestamp: null
   labels:
     kops.k8s.io/cluster: docks-www-tutorials.sherzod.com
   name: master-us-east-1a
 spec:
+  cloudLabels:
+    application: kops
   image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220131
   instanceMetadata:
     httpPutResponseHopLimit: 3
     httpTokens: required
-  machineType: t3.medium
+  machineType: t3a.medium
   maxSize: 1
   minSize: 1
   nodeLabels:
     kops.k8s.io/instancegroup: master-us-east-1a
   role: Master
+  rootVolumeType: gp3
   subnets:
-  - us-east-1a
+  - us-east-1a-publ

 ---

 apiVersion: kops.k8s.io/v1alpha2
 kind: InstanceGroup
 metadata:
-  creationTimestamp: "2022-02-11T21:07:10Z"
+  creationTimestamp: null
   labels:
     kops.k8s.io/cluster: docks-www-tutorials.sherzod.com
   name: nodes-us-east-1a
 spec:
+  cloudLabels:
+    application: kops
   image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220131
   instanceMetadata:
     httpPutResponseHopLimit: 1
     httpTokens: required
-  machineType: t3.medium
+  machineType: t3a.medium
   maxSize: 1
   minSize: 1
   nodeLabels:
     kops.k8s.io/instancegroup: nodes-us-east-1a
   role: Node
+  rootVolumeType: gp3
+  rootVolumeSize: 64
   subnets:
-  - us-east-1a
+  - us-east-1a-priv

We are going to talk about spec.authentication, spec.certManager, spec.clusterAutoscaler, spec.iam.serviceAccountExternalPermissions, spec.metricsServer, and serviceAccountIssuerDiscovery separately.

So what are we starting with?

  • for the network piece, use custom networkCIDR and supply networkID to instruct kOps to use an existing VPC
  • swap out kubenet with calico for enhanced configuration (like IPPools and network policies)
  • restrict the SSH access to non 0.0.0.0/0 one (usually an office IP) that's reflected on security group created by kOps, and use the ssh key we created before
  • use the existing subnets, and define the four we created for this configuration
  • mark the nodes under private topology (i.e. skip public EIPs)
  • modify the instance groups to use the updated subnets, billing tag(s)

Once we are happy with our configuration, we can replace the initial one with command:

$ kops replace -f cluster-manifest.yaml
$ kops update cluster --name $KOPS_NAME

...

$ kops update cluster --name $KOPS_NAME --yes

After these commands, the cluster should be up and running within 10 minutes. The output from kops update cluster will mention exporting kubeconfig via kops export kubeconfig --admin that sets the current kubectl context. Use --admin=48h for setting kubeconfig session duration for 48 hours (default is 18 hourse; see ops export kubeconfig --help). In our future posts we are going to authenticate into our cluster using aws-iam-authenticator which is recommended way to control access, especially in organizations using federated authentication and SSO.

Once you see master and node(s) come online, you have a running kubernetes cluster :)

$ kubectl get nodes
NAME                        STATUS   ROLES                  AGE   VERSION
ip-10-2-0-54.ec2.internal   Ready    control-plane,master   2d    v1.22.6
ip-10-2-2-42.ec2.internal   Ready    node                   2d    v1.22.6

In our next post we are going to stand up nodes backed by spot instance(s), and use EC2 resource at "clearance" prices :) Let's go.