Okctl relies on services in AWS and GitHub to provide its functionality. In the following sections, we describe some core services we use from the cloud provider.

Cloud components

Cloud providers offer a vast array of functionality for:

  • Networking
  • Computation
  • DNS
  • Certificates
  • Databases
  • Block storage
  • Artificial intelligence

In Okctl we use a subset of this functionality to provide a platform for running production workloads:

This isn't an exhaustive list of integrations but gives some idea of the types of services we integrate with.

Amazon Web Services (AWS)

With Okctl we use AWS as our cloud operator; there is no particular reason for preferring AWS to other cloud vendors, such as Microsoft Azure or Google Cloud. In Oslo kommune, we can use any of these, but there are a number of teams that have greater experience with AWS.

Virtual Private Cloud (VPC)

Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways. You can use both IPv4 and IPv6 in your VPC for secure and easy access to resources and applications.

Elastic Kubernetes Service (EKS)

Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed Kubernetes (k8s) service. Being a fully managed service, AWS ensures that the control plane is secure, reliable and scalable. This allows us to focus more on application security.

K8s is an open-source system for automating deployment, scaling, and management of containerized applications. It provides a powerful platform to build applications on top of.

AWS Route53

AWS Route53 (Route53) is a highly available and scalable Domain Name System (DNS) web service. It is designed to give developers and businesses an extremely reliable and cost-effective way to route end users to Internet applications by translating names like www.example.com into the numeric IP addresses like 192.0.2.1 that computers use to connect to each other.

dig test.oslo.systems NS +short

ns-327.awsdns-40.com.
ns-612.awsdns-12.net.
ns-1706.awsdns-21.co.uk.
ns-1322.awsdns-37.org.

AWS Certificate manager (ACM)

AWS Certificate Manager (ACM) lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with AWS services, and your internal connected resources. SSL/TLS certificates secure network communication and establish the identity of websites over the Internet as well as resources on private networks.

AWS Systems Manager (SSM) Parameter Store

AWS Systems Manager (SSM) gives you visibility and control of your infrastructure on AWS. Systems Manager provides a unified user interface so you can view operational data from multiple AWS services and allows you to automate operational tasks across your AWS resources.

The parameter store provides centralized storage and management of secrets and configuration data such as passwords, database strings, and license codes. We can encrypt values, or store as plain text, and secure access at every level.

Github components

Github is at its core a Version Control System (VCS). Since its inception Github has grown to provide a wide array of functionality to interact with a product's source code. In Oslo kommune, we have decided to use Github as our primary source control system. It is therefore natural to make use of the functionality Github has on offer for implementing Continuous Integration (CI), Continuous Deployment (CD), Authentication and Authorisation, within a product.

Github Actions (GHA)

Github Actions make it easy to automate, customize, and execute software development workflows from your repository. It is possible to discover, create, and share actions to perform any job one would like, including CI/CD, and combine actions in a completely customized workflow.

Github Organisations and Teams

Github Organisations are shared accounts where businesses and open-source projects can collaborate across many projects at once. Owners and administrators can manage member access to the organization's data and projects with sophisticated security and administrative features.

Github Teams allows one to organize members into teams that reflect the company or group's structure with cascading access permissions and mentions.

Kubernetes components

Kubernetes is highly configurable and extensible. We make use of this fact with Okctl and bind our Kubernetes cluster setup tighter together with AWS. We do this by deploying a variety of applications into the cluster to manage various aspects of an application setup for us.

Kubernetes External Secrets

Kubernetes External Secrets allows you to use external secret management systems, like AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.

We have installed external secrets and configured it to use SSM Parameter store as a backend. This means that we can store secrets in SSM and eventually have them made available as a Kubernetes secret resource that we can reference in our deployment manifests.

AWS Load Balancer Controller

AWS Load Balancer Controller satisfies the Kubernetes ingress resources by provisioning Application Load Balancers (ALB). An ALB  functions at the application layer, the seventh layer of the Open Systems Interconnection (OSI) model. After the load balancer receives a request, it evaluates the listener rules in priority order determining which rule to apply, and then selects a target from the target group for the rule action. We use ALBs, among other things, to route traffic from the internet into a pod (container).

We have configured AWS Load Balancer Controller to work with a cluster, which means that a user doesn't have to manage the life-cycle of an ALB outside their cluster. Instead, one can attach annotations to the ingress object and have the controller create these for one.

Example

The following ingress resource will result in the creation of a public ALB. In this example, we only use a subset of the available annotations, but they demonstrate how you can:

  1. Configure HTTP to HTTPS redirect
    alb.ingress.kubernetes.io/actions.ssl-redirect: \
        '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
  1. Define the ports to listen on
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
  1. The TLS certificate to use
    alb.ingress.kubernetes.io/certificate-arn: arn:::certificate/...
  1. Define a custom health check endpoint on the service
    alb.ingress.kubernetes.io/healthcheck-path: /health

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-backend-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: arn:::certificate/...
  labels:
    app: test-backend
spec:
  rules:
    - host: test-backend.oslo.systems
      http:
        paths:
          # A path like this is required for the SSL redirect to function
          # This rule must also be the first in the list.
          - path: /*
            backend:
              serviceName: ssl-redirect
              servicePort: use-annotation
          - path: /*
            backend:
              serviceName: test-backend-service
              servicePort: 80

ExternalDNS

ExternalDNS synchronizes exposed Kubernetes Services and Ingresses with DNS providers.

We have configured ExternalDNS to work with Route53, which looks at the host part of an ingress definition and creates a DNS entry for that host. This functionality works in tandem with the AWS Load Balancer Controller so traffic is routed to the correct deployment.

Example

To understand ExternalDNS, focus on the following line:

/spec/rules/0/host: test-backend.oslo.systems

ExternalDNS will simply look at the defined host and create a Route53 DNS entry that it associates with the ALB created in the AWS Load Balancer Controller example.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-backend-ingress
  labels:
    app: test-backend
spec:
  rules:
    - host: test-backend.oslo.systems
      http:
        paths:
          - path: /*
            backend:
              serviceName: test-backend-service
              servicePort: 80

Cluster Autoscaler

Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster. On AWS it does this by adjusting the size of the Autoscaling Group of the cluster's nodegroup. We have configured the cluster autoscaler to automatically adjust the default nodegroup that we add to a cluster. Starting at 1 node it can scale up and down to a maximum of 10 nodes.

Amazon Elastic Block Store (EBS) CSI driver

Amazon Elastic Block Store (EBS) CSI driver provides a CSI interface used by Container Orchestrators to manage the lifecycle of Amazon EBS volumes. This makes it possible to create PersistentVolumeClaims in the Kubernetes cluster.

Kubernetes Prometheus

Kubernetes Prometheus is a stack meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. For a better idea of what possibilities this stack offers take a look at some examples.

Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Tempo

Tempo is an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki. Tempo can be used with any of the open-source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.

ArgoCD

NB: While this describes what Argo CD is and how it can be used, the intent is to facilitate our users in the creation of these resources.

What value does ArgoCD provide

With Okctl we use ArgoCD for managing application deployment. ArgoCD is based on a gitops model:

GitOps is a way to do Kubernetes cluster management and application delivery.  It works by using Git as a single source of truth for declarative infrastructure and applications. With GitOps, the use of software agents can alert on any divergence between Git with what's running in a cluster, and if there's a difference, Kubernetes reconcilers automatically update or rollback the cluster depending on the case. With Git at the center of your delivery pipelines, developers use familiar tools to make pull requests to accelerate and simplify both application deployments and operations tasks to Kubernetes.

The primary advantages of using a gitops model are:

  1. The entire system described declaratively.
  2. The canonical desired system state versioned in Git.
  3. Approved changes that can be automatically applied to the system.  
  4. Software agents to ensure correctness and alert on divergence.

Setup of Argo CD

When you create a full cluster we configure Argo CD for you, this configuration consists of a couple of operations:

  1. Add a Github deploy key to your Github repository with read-only access. This means that Argo CD can poll for changes to your git repository and see if it needs to deploy a new version or not.
  2. Configure an ingress resource and make the Argo CD UI available on the following URL: https://argocd.{product/team}.oslo.systems.
  3. Setup a Github Oauth app to manage access to the Argo CD UI. Only the configured Github organization and team that are set during the configuration are given access to this UI.