Delete cluster

During testing, or maybe when a project has come to an end - there might be a necessity to delete a cluster created with Okctl.

Delete a cluster

# Usage
okctl --cluster-declaration <path to cluster declaration> delete cluster

# Example:
okctl --cluster-declaration cluster.yaml delete cluster

Delete a cluster manually

The delete operation may have failed for several reasons. In the following sections, we list common causes of problems and possible remedies.

Delete all Kubernetes resources

Using kubectl

The following applies if you can access your cluster with kubectl.

Delete everything in all namespaces except kube-system. In kube-system, some core components, such as AWS load balancer controller are running. These controllers create AWS resources on your behalf. By removing all resources from all namespaces, we allow these services to clean up themselves.

MacOS

for each in $(kubectl get ns -o jsonpath="{.items[*].metadata.name}" | grep -v kube-system);
do
  kubectl delete ns $each
done

Linux

for each in $(kubectl get ns -o jsonpath="{.items[*].metadata.name}" | tr ' ' \\n | grep -v kube-system);
do
  kubectl delete ns $each
done

Manually

If you're not able to access the cluster with kubectl, you can delete the following through the AWS console instead:

EC2 -> Load Balancers -> Delete any load balancers with tags matching the name of your cluster (try the filter tag: elbv2.k8s.aws/cluster : mycluster-myenv)
EC2 -> Auto Scaling Groups -> Delete any auto-scaling groups with a name matching your cluster

Remove Fargate Profile

Log in to the AWS console for your account, then go to Elastic Kubernetes Service > Clusters. Select your cluster, then in the Configuration find the Compute sub tab. Find the Fargate Profiles items and delete the fp-default profile.

Remove generated route53 records

Log in to the AWS console for your account, then go to route 53 and select the relevant hosted zone. The only two records that should remain so that Cloud Formation is able to delete it is the SOA record, and the NS record.

Delete objects in S3 bucket

Log in to the AWS console for your account, then go to Cloud Formation -> Stacks. Find the stack with the name okctl-s3bucket-<cluster name>-<database name>, where <cluster name> and <database name> can be found in your cluster declaration file. Click on this stack, open the "Resources" tab, and click on the link in the "Physical ID" column.

You will now see a ZIP file in an S3 bucket. Delete this file.

Delete the cloud formation stacks in reverse order

Log in to the AWS console for your account, then go to Cloud Formation -> Stacks. If your AWS account contains many stacks, filter the list by the cluster name and environment. All stacks that contain Okctl or eksctl should be deleted. We invoke the eksctl CLI for creating a number of resources for us, which is why these also need to be removed.

Start deleting the stacks in order from newest to oldest, if a stack ends up in a state where the delete operation fails. Take a closer look at the events generated by the delete operation on the stack. Frequently you will find a resource that has failed to delete for some reason. Follow this resource to its corresponding page. Maybe a subnet has failed to delete because there are active network interfaces on it. By following the resource and trying to delete it manually, you will be given additional information. Perhaps you need to disconnect the interfaces from an EC2 machine first, etc. Once you have resolved the problem, go back and try to delete the cloud formation stack again.

Remove secrets

Log in to the AWS console for your account, then go to Systems Manager -> Parameter store. Here you might find leftover secrets.

Remove deploy key(s) in the infrastructure-as-code (IAC) repository

Clean up the IAC repository

Delete the directory infrastructure/<cluster name>.

If the cluster you are deleting is the last one in this IAC repository, you can delete the infrastructure directory.

List all AWS resources created by Okctl

It's possible to list all AWS resources created by Okctl, by running the command below. This can be a useful command to run to find any missing resources.

aws resourcegroupstaggingapi get-resources \
--tag-filters Key=alpha.okctl.io/okctl-version,Values=dev \
--tags-per-page 100

Conclusion

This article demonstrates two ways of deleting a cluster, either using okctl delete cluster or doing it manually.