TheCB4

Full Stack Solution, Swift Vapor + MongoDB behind Traefik Proxy with Let's Encrypt TLS in a Kuerbernetes Cluster with Ceph as Block Storage using Ansible + Terraform + Helm

Introduction

I want to easily deploy a web app that can scale. Doing this can take a considerable amount of time to do and historically was done with a lot of custom scripts. Over the years tools have been developed to speed up this process and make it more repeatable. I am going to walk through how I decided to put these tools together to get deployment down to roughly 15 minutes. This was my first time learning some of these tools so looking forward to the feedback on building out this stack.


What are the tools:

  • Ansible is a platform that is focused on automation of just about any task. With a huge community, this tool allows you to connect these tasks in complex ways to deliver any type of solution. These tasks can be run on your local computer or a remote server.
    • Terraform is relatively new and is meant to be “Infrastructure as Code”. Instead of defining tasks, you define resources in HCL (HashiCorp Configuration Language) through providers. These resources represent your infrastructure stack.
    • Kubernetes is a cloud based infrastructure that is designed to be able to scale up and down quickly through predefined states. The “cluster” is always working to maintain that state and will create/destroy underlying resources to do so. Within the cluster there are pods that have connected software elements running on a container based process. Multiple containers are connected together in a pod.
    • Helm is a tool to package up Kubernetes resources and deploy them easily into the cluster. These resources could be a database or an application.
    • Docker is the container solution that is pretty familiar to most at this point. A container usually contains one process. This is the database process or the application process.
    • MongoDB is the NoSQL database solution that is highly scalable.
    • Vapor is the Server Side Swift platform
    • Traefik is a modern proxy server that allows for scalability and load balancing. The interface is fairly intuitive and integrates well with Cert Manager.
    • Let's Encrypt is a non-profit TLS certificate authority that provides https certificates at no cost.
    • Digital Ocean, who just went public, is a cloud solution provider including managing Domains.
    • Namecheap. This is where I host my Top Level Domain (TLD)

How do they work.

Much like a cake, the order of the ingredients matter. By using each tool for what it does best, I am able to deploy the layers in a repeatable fashion.

  • Ansible operates as a general purpose task manager to run terraform.
  • Terraform plans and applies resource state. This includes creating the Kuerbenetes cluster on Digital Ocean. I’ve created several resource groups I’ve called “Formations” to build up the infrastructure resources.
  • Each of the ‘Formations uses Helm to deploy a set of Kubernetes resources that have been packaged up into “Charts”.
  • Within the Charts there are Docker images, many of these images are built by either the community of bony the software provider themselves. This is the case with the Mongo database that we will deploy. We will use Docker to package up our own image of the Vapor application for deployment. We will use a sharded mongo database a shared storage infrastructure (Ceph). Traefik will be used with Let’s Encrypt to create a TLS proxy for the application and the Traefik Proxy. This will also be deployed as a Helm Chart.
  • What will you learn. In this post you will learn how to package up your vapor application and deploy it behind a proxy server. This will be done using a combination of Ansible, Terraform, Helm, and Docker. The software running will be Mongo DB, Vapor for Backend with Vapor Leaf as the front end and Traefik as the proxy server. We will be deploying on a Digital Ocean Kubernetes Cluster. We will use Let’s Encrypt for TLS management.

Prerequisites

  • A DigitalOcean account. If you do not have one, sign up for a new account.
  • A DigitalOcean Personal Access Token, which you can create via the DigitalOcean control panel. Instructions to do that can be found in this link: How to Generate a Personal Access Token.
  • A Top Level Domain that is already connected to Digital Ocean. I get mine from Namecheap. I manage the domain through Digital Ocean to make doing things like this easier. Managing the top level domain is outside the scope of this article.
  • A Docker Hub account. They are free for what we're going to do.
  • Some familiarity with Vapor, MongoDB, and Kubernetes.

The Work


The Vapor Application

  • download the completed Ray Wenderlich app
  • Modify the mongo+app.swift file so that you build the connection string from environment variables.

public var mongDBConnectionString: String {
  
  let user = Environment.get("DB_USER") ?? "generic"
  let password = Environment.get("DB_USER_PASSWORD") ?? "generic"
  let server = Environment.get("DB_SERVER") ?? "generic"
  let database = Environment.get("DATABASE") ?? "generic"

  let string = "mongodb://\(user):\(password)@\(server):27017/\(database)"
  print(string)
  return string
}

  • Modify the configuration swift file to connect to Mongo using this connection string ` try app.initializeMongoDB(connectionString: app.mongDBConnectionString) `

Docker-Compose and MongoDB

  • Add MongoDB to the docker-compose file
  • Modify the vapor application to accept the environment variables you created in the prior step.
  • Modify the vapor application 'image' to be where you would like the image to reside. To keep it easy we will use Docker Hub.

services:
  app:
    image: yourdomain/vapor-blog:1.0
    environment:
      <<: *shared_environment
      DB_USER: "root"
      DB_USER_PASSWORD: "yourmongopassword"
      DB_SERVER: "mongo-service"
      DATABASE: "socialbird"
    ports:
      - '8080:8080'
    # user: '0' # uncomment to run as root for testing purposes even though Dockerfile defines 'vapor' user.
    command: ["serve", "--env", "production", "--hostname", "0.0.0.0", "--port", "8080"]

  # Add 
  mongo_service:
    image: mongo:latest
    container_name: "mongo-service"
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: yourmongopassword
    ports:
      - '27017:27017'

Run Docker locally to confirm everything works...


Concerns about this in production

  • Access The first issue here is that we can’t get to our ‘localhost’ externally. There are tools to handle this like ngrok. That isn’t necessarily sustainable and risks your local computer going down.
  • Scale The second issue is that if you need more CPU, more RAM, more Disk, you have to go buy a new computer and connect it to the first one and somehow get those pieces working together.
  • Security The third issue is that having your solution hosted on your local machine opens you up to quite a bit of security issues. Also, given the move of the web to https, there is more difficult.

Why use an orchestration

TL;DR:


Kubernetes on Digital Ocean solves some of these issues. For access and security Kubernetes solves these issues by creating a closed network for the various services. You have to expose the services explicitly on an IP address. For scale, this is where kubernetes really shines.The infrastructure is setup in a way to automatically scale up based on your limits. All of that is great, now you just need a place to host. That is where Digital Ocean comes in. Digital Ocean is a cloud platform that allows you to host simple servers, kubernetes clusters, or even one-click applications. For this article, we will take advantage of the kubernetes cluster capabilities. The drawback to Kubernetes is that it’s complicated AF to deploy. Many of the elements require long missives of YAML that have to be deployed together. The good part is that many of those elements are similar elements and they themselves can be packaged up. This is where Helm comes in. Using something called a Chart, kubernetes resources are packaged and deployed together. So we can go to the Digital Ocean website, start a new kubernetes cluster, and deploy resources via Helm. The issue with this process is that there are still manual steps. I still have to interact with a web page to set up the server. I also still have to interact with a web page to add the IP address of the cluster to a sub domain record. The alternative used to be writing a bunch of shell scripts. That is where Terraform comes in. I can convert the Helm deployments to HCL. I have packaged each one of these sets of HCL steps into modules that represent each layer of the solution that I’ve called Formations. The helpful part of Terraform is that it includes configuration for the cluster itself and for the digital ocean domain management. This allows me to write out HCL for defining the cluster the same way I define the services that will go on the cluster.


Server folder structure


├── bin
│   └── manage.sh
├── charts+values
│   ├── ceph-storage
│   ├── cert-manager
│   ├── echo-service
│   ├── mongo-database
│   ├── proxy-dashboard
│   ├── traefik-proxy
│   └── vapor-application
├── formations
│   ├── application
│   ├── cluster
│   ├── database
│   ├── echo-server
│   ├── env.tfvars
│   ├── main.tf
│   ├── proxy
│   ├── storage
│   └── variables.tf
├── images
│   ├── blog
│   └── deployment
└── orchestration
    ├── ansible.cfg
    ├── group_vars
    ├── inventory
    ├── roles
    └── strategy.yaml
  • images: Docker images that have to be built
  • charts: Kurbernetes Resources
  • formations: Terraform HCL
  • orchestration: Ansible playbooks

Orchestration

As mentioned ansible is a general purpose orchestration tool. We will use this to drive Terraform.


├── ansible.cfg
├── group_vars
│   └── server
│       ├── vars.yaml
│       └── vault
├── inventory
├── roles
│   └── Orc
│       ├── tasks
│       │   └── main.yaml
│       └── templates
│           └── tfvars.j2
└── strategy.yaml

ansible.cfg is where we will maintain default elements of the ansible configuration


[defaults]
# Debug is on
default_debug = True

#Inventory file
inventory = ./inventory

#We are only using localhost so we don't need to do host key checking
host_key_checking = False

# Provide standard output as YAML so it's human readable
# https://www.shellhacks.com/ansible-human-readable-output-format/
stdout_callback = yaml

# Profile the tasks so we know how long Terraform takes
# https://dbaclass.com/article/print-execution-time-tasks-ansible/
callback_whitelist = profile_tasks

# The Ansible Vault password file. Used to manage for the Digital Ocean token mentione earlier in the article.
vault_password_file = ./.vault_pass

inventory is where we will maintain the hosts that ansible will connect to in order to perform the orchestration. As mentioned, we are only connecting to the local machine.


[server]
localhost

group_vars/vars.yaml has the configuration we will run for a production and testing environment.


---
terraform:
  production:
    cluster_name: "yourdomain-k8s-production"
    cluster_region: "nyc1"
    node_size: "s-4vcpu-8gb"
    node_count: 3
  testing:
    cluster_name: "yourdomain-k8s-testing"
    cluster_region: "nyc1"
    node_size: "s-2vcpu-4gb"
    node_count: 3

.vault_pass is where we will store the vault password in plain text. It will not be part of the repo. For the purposes of this article we will keep it very simple with 'myvaultpassword' as the vault password. By running the following command ansible-vault create group_vars/database/vault

group_vars/vault is where we will store the Digital Ocean access token that was created. The details of the process are here.


ansible-vault create group_vars/database/vault

This is where you will place your Digital Ocean token


---
digital_ocean_token: supersecretpassword

In the roles/Orc/tasks/main.yaml you will reference the role that is defined to run Terraform


- name: "Swift Vapor + MongoDB behind Traefik with Let's Encrypt TLS in a Kuerbernetes Cluster with Ceph as Block Storage using Ansible + Terraform + Helm"
  hosts: localhost
  connection: local
  roles:
    - Orc

The role, which I've titled 'Orc', will house the actions we will perform. We have Terraform Deploy and Terraform Destroy.


- name: Substitute Terraform Variables
  template:
    src: templates/tfvars.j2
    dest: "{{ playbook_dir }}/terraform/env.tfvars"
  when: (operation == "deploy")

- name: "Terraform Deploy"
  community.general.terraform:
    project_path: '{{ playbook_dir }}/../formations'
    state: "present"
    force_init: true
    init_reconfigure: true
    variables:
      do_token: '{{ hostvars[inventory_hostname].digital_ocean_token }}'
    variables_files:
      - "{{ playbook_dir }}/../formations/env.tfvars"
  when: (operation == "deploy")

What are we doing here?

First we are doing some variable substitution. We have the ability to setup a production or testing environment. By passing the variable env = 'production or 'testing' we can have a different name for the cluster, different sizes, regions, etc.


cluster_name = "{{ terraform['%s' | format(env)].cluster_name }}"
cluster_region = "{{ terraform['%s' | format(env)].cluster_region }}"
worker_size = "{{ terraform['%s' | format(env)].node_size }}"
worker_count = {{ terraform['%s' | format(env)].node_count }}

Second, we ask Ansible to run Terraform init / plan / apply in one step. This is accomplished with the 'present' state. We are passing the digitaloceantoken that is stored in Ansible Vault. We only run this action when operation is 'deploy'. This eliminates the need to have a bash shell script to run each of the three steps separately. This also allows for the passing of variable files as well.

This leads us to our formations. These are Terraform modules.

There formations are tied together using a top-level terraform file (tf). The first part of the file lists the 'required providers' that will be used. Providers provide the nomenclature for declaring state of the solution.


terraform {
  required_version = ">= 0.14"
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = ">= 2.4.0"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = ">= 2.0.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.0.1"
    }
  }
}

There are three required providers. The first is the Digital Ocean provider for creating the cluster. The next is the Kubernetes provider to access Kubernetes resources. Last is the helm provider which deploys Helm Charts.

The first providers drives the first formation. The Digital Ocean Kubernetes Cluster.


provider "digitalocean" {
  token = var.do_token
}

locals {
  # cluster_name = "yourdomain-k8s-${random_id.cluster_name.hex}"
  cluster_name = var.cluster_name
}

module "cluster" {
  source             = "./cluster/digital-ocean"
  do_token           = var.do_token
  cluster_name       = local.cluster_name
  cluster_region     = var.cluster_region

  worker_size        = var.worker_size
  worker_count       = var.worker_count
}

Notice that this is where we use the Digital Ocean Access Token through Ansible.


Where ./cluster/digital-ocean is:


terraform {
  required_version = ">= 0.14"
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = ">= 2.4.0"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = ">= 2.0.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.0.1"
    }
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.7.0"
    }
  }
}

data "digitalocean_kubernetes_versions" "current" {
  version_prefix = var.cluster_version
}

resource "digitalocean_kubernetes_cluster" "primary" {
  name    = var.cluster_name
  region  = var.cluster_region
  version = data.digitalocean_kubernetes_versions.current.latest_version

  node_pool {
    name       = "default"
    size       = var.worker_size
    node_count = var.worker_count
  }
}

We are defining a Kubernetes cluster based on the latest version of Kubernetes. This can be modified, but makes sense to use the latest. Then we simply define the resource, based on the provider. This includes the name, region, version, and node pool definition. Terraform talks to Digital Ocean and spins everything up.


From there we have a series of steps that we go through for each module. 1. We configure the Kubernetes and Helm providers with the configuration data from the cluster resource we created. 2. We create the distributed storage capability on top of Digital Ocean Volumes so we can run MongoDB in a sharded manner. 3. Traefik as the proxy with Cert Manager for TLS. 4. The proxy dashboard so we can see our backend Vapor application. 5. MongoDB to run using rook-ceph storage class. 6. Vapor application


provider "kubernetes" {
  host             = module.cluster.info.endpoint
  token            = module.cluster.info.kube_config[0].token
  cluster_ca_certificate = base64decode(
    module.cluster.info.kube_config[0].cluster_ca_certificate
  )
}

provider "helm" {
  kubernetes {
    host             = module.cluster.info.endpoint
    token            = module.cluster.info.kube_config[0].token
    cluster_ca_certificate = base64decode(
      module.cluster.info.kube_config[0].cluster_ca_certificate
    )
  }
}

resource "kubernetes_namespace" "solution" {
  metadata {
    name = "solution"
  }
}

module "proxy" {
  source   = "./proxy"
  namespace = kubernetes_namespace.solution.metadata.0.name
  depends_on = [module.cluster]
}

module "storage" {
  source   = "./storage"
  depends_on = [module.cluster]
}

# It takes a while for ceph to get working
resource "time_sleep" "wait_for_ceph" {
  depends_on = [module.storage]
  create_duration = "300s"
}

module "database" {
  source   = "./database"
  namespace = kubernetes_namespace.solution.metadata.0.name
  depends_on = [time_sleep.wait_for_ceph]
}

module "application" {
  source   = "./application"
  namespace = kubernetes_namespace.solution.metadata.0.name
  depends_on = [module.database]
}

This reduces security risks for the cluster because I never have to print out my cluster configuration or save it to file. It only exists as part of the deployment.

The Cluster

The cluster is pretty straight forward.


# required resources not listed, but same as main.tf

data "digitalocean_kubernetes_versions" "current" {
  version_prefix = var.cluster_version
}

resource "digitalocean_kubernetes_cluster" "primary" {
  name    = var.cluster_name
  region  = var.cluster_region
  version = data.digitalocean_kubernetes_versions.current.latest_version

  node_pool {
    name       = "default"
    size       = var.worker_size
    node_count = var.worker_count
  }
}

Only two things being done here. 1. Determining the latest version of Kubernetes on Digital Ocean 2. Creating the Kubernetes cluster based on information passed in via the variable substitution in Ansible

There's also some output that we will use in subsequent modules.


output "info" {
  value = digitalocean_kubernetes_cluster.primary
}

This is the cluster info resource. We will pull the necessary credentials to execute the other work.

The Proxy

I am using Traefik as the Proxy / Loadbalancer for this solution. Traefik plays well with Let's Encrypt Certificate Manager for TLS. A few things we will do here.

  1. Install Traefik into the cluster
  2. Install Let's Encrypt Certificate Manager Custom Resource Definitions (CRDs). Let's encrypt stores the certificate as a secret in Kubernetes that is used with the IngressRoute definition for each backend.
  3. Access the traefik service to get the external IP of the loadbalancer
  4. Create a subdomain for the traefik dashboard
  5. Deploy the traefik dashboard backend with TLS

resource "helm_release" "traefik" {
  name       = "traefik"
  repository = "https://helm.traefik.io/traefik"
  chart      = "traefik"
  namespace = "solution"

  values = [
    file("${path.module}/../../charts+values/traefik-proxy/values.yaml")
  ]

  atomic = true
  timeout = 600
  cleanup_on_fail = true 
}

resource "helm_release" "cert_manager" {
  name       = "cert-manager"
  repository = "https://charts.jetstack.io"
  chart      = "cert-manager"
  namespace = "solution"

  values = [
    file("${path.module}/../../charts+values/cert-manager/values.yaml")
  ]

  atomic = true
  timeout = 600
  cleanup_on_fail = true

  depends_on = [helm_release.traefik]
}


# Use this to get the load balancer external IP for DNS configuration
data "kubernetes_service" "traefik" {
  metadata {
    name      = "traefik"
    namespace = "solution"
  }

  depends_on = [helm_release.cert_manager]
}

resource "digitalocean_record" "subdomain" {
  domain = "yourdomain.net"
  type = "A"
  name = "traefik"
  # https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/ingress
  value =  data.kubernetes_service.traefik.status.0.load_balancer.0.ingress.0.ip

  depends_on = [
    data.kubernetes_service.traefik
  ]
}

resource "helm_release" "proxy_dashboard" {
  name       = "proxy-dashboard"
  chart      = "${path.module}/../../charts+values/proxy-dashboard"
  namespace = var.namespace

  depends_on = [
    digitalocean_record.subdomain
  ]

}

With the TLS resources as


# cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: traefik-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: cavelle@yourdomain.net
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: traefik.yourdomain.net-tls
    # Add a single challenge solver, HTTP01
    solvers:
      - http01:
          ingress:
            class: traefik-cert-manager
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: traefik-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: cavelle@yourdomain.net
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: traefik.yourdomain.net-tls
    # Add a single challenge solver, HTTP01
    solvers:
      - http01:
          ingress:
            class: traefik-cert-manager
---
# traefik-dashboard-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: traefik-yourdomain-cert
spec:
  commonName: traefik.yourdomain.net
  secretName: traefik.yourdomain.net-tls
  dnsNames:
    - traefik.yourdomain.net
  issuerRef:
    name: traefik-staging
    kind: ClusterIssuer

And the dashboard resources themselves as


---
apiVersion: v1
kind: Secret
metadata:
  name: traefik-dashboard-auth
data:
  # https://traefik.io/blog/install-and-configure-traefik-with-helm/
  users: |2
    somelong64bitencodedpassword
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-dashboard-basicauth
spec:
  basicAuth:
    secret: traefik-dashboard-auth
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`traefik.yourdomain.net`)
      kind: Rule
      Middlewares:
        - name: traefik-dashboard-basicauth
      services:
        - name: api@internal
          kind: TraefikService
  tls:
    secretName: traefik.yourdomain.net-tls

Notice that I've added some middlware here. This is to provide authentication so that everyone and their mother doesn't have access to the backend services.

The Storage

Ceph Was chosen as a way to abstract the data volumes away so that it behaves as 'one' across pods. This was necessary to run a Sharded Mongo Database. This allows for horizontal scaling of the database to help with High Availability. The underlying storage class for Ceph is is the Digital Ocean storage class.


# required resources not listed, but same as main.tf

# Create namespace
resource "kubernetes_namespace" "rook-ceph" {
  metadata {
    name = "rook-ceph"
  }
}

resource "helm_release" "rook_ceph_operator" {
  name       = "rook-ceph"
  repository = "https://charts.rook.io/release"
  chart      = "rook-ceph"
  namespace = "rook-ceph"
  atomic = true
  timeout = 600
  cleanup_on_fail = true
}

resource "helm_release" "ceph-storage" {
  name       = "ceph-storage"
  chart      = "${path.module}/../../charts+values/ceph-storage"
  namespace = "rook-ceph"
  atomic = true
  timeout = 600
  cleanup_on_fail = true
  depends_on = [
    helm_release.rook_ceph_operator
  ]
}

I won't go into the details of Ceph, but I used the Quick Start with Ceph and Block Storage and packaged it up into a chart that is used.

The Database

So the database is a Sharded Mongo Database. Mongo has a lot of great capability with respect to scalability, performance, flexibility.


resource "helm_release" "mongodb_sharded" {
  name       = "mongo-k8s"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "mongodb-sharded"
  namespace = var.namespace

  cleanup_on_fail = true

  values = [
    file("${path.module}/../../charts+values/mongo-database/values.yaml")
  ]

}

With the values file as:


global:
  storageClass: rook-ceph-block
mongodbRootPassword: yourmongopassword
shards: 2
common:
  mongodbSystemLogVerbosity: 3
  serviceAccount:
    create: true

Notice that here we're using the rook-ceph-block storage. That storage is on top of digital ocean block storage. Rook Ceph requests a certain amount of block storage from Digital Ocean then passes it out as other resources make persistent volume claims based on the rook-ceph storage class. Note the password. This is the password that we will pass to our application.


# Excerpt from Ceph CRDs Chart
volumeClaimTemplates:
  - metadata:
      name: data
      # if you are looking at giving your OSD a different CRUSH device class than the one detected by Ceph
      # annotations:
      #   crushDeviceClass: hybrid
    spec:
      resources:
        requests:
          storage: 10Gi
      # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
      storageClassName: do-block-storage
      volumeMode: Block
      accessModes:
        - ReadWriteOnce

The Application


resource "helm_release" "vapor_application" {
  name       = "vapor-application"
  chart      = "${path.module}/../../charts+values/vapor-application"
  namespace = var.namespace
}

# Use this to get the load balancer external IP for DNS configuration
data "kubernetes_service" "traefik" {
  metadata {
    name      = "traefik"
    namespace = var.namespace
  }
  depends_on = [helm_release.vapor_application]
}

resource "digitalocean_record" "subdomain" {
  domain = "yourdomain.net"
  type = "A"
  name = "vapor-blog"
  # https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/ingress
  value =  data.kubernetes_service.traefik.status.0.load_balancer.0.ingress.0.ip

  depends_on = [
    data.kubernetes_service.traefik
  ]
}

We deploy the Vapor Application the same way as the proxy. 1. Deploy the application via a local Helm Chart. 2. Get the service associated with the application 3. Create a subdomain record on digital ocean

The application is based on the image that we built earlier in the article. The service or IngressRoute is similar to traefik IngressRoute but without the auth middleware.


---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: vapor-application-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`vapor-blog.yourdomain.net`)
      kind: Rule
      services:
        - name: {{ .Values.service.name }}
          port: 8080
  tls:
    secretName: vapor-blog.yourdomain.net-tls

This IngressRoute is just like the Traefik IngressRoute but does not have the middleware for authentication. This is immediately accessible via the web.

To make this easier, I created a Docker image that has Ansible, Terraform, kubctl, doctl, to run this in CI. So when I commit, it creates a new cluster.


FROM ubuntu:20.04

RUN DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true \
  && apt-get -q update \
  && apt-get -q dist-upgrade -y \
  && apt-get install -y \
  sudo openssh-server wget curl unzip \
  apt-transport-https ca-certificates \
  software-properties-common snapd \
  && rm -rf /var/lib/apt/lists/*

RUN useradd -m -p --disabled-password --user-group --create-home --comment "" -s /bin/bash ansible

RUN usermod -aG sudo ansible

RUN su ansible

RUN sudo apt-get -q update && \
    sudo apt-add-repository --yes --update ppa:ansible/ansible && \
    sudo apt-get install -y ansible

RUN ansible-galaxy collection install community.general

RUN /bin/echo -e "[local]\nlocalhost ansible_connection=local" > /etc/ansible/hosts ;\
  ssh-keygen -q -t ed25519 -N '' -f /root/.ssh/id_ed25519 ;\
  mkdir -p ~/.ssh && echo "Host *" > ~/.ssh/config && echo " StrictHostKeyChecking no" >> ~/.ssh/config

RUN wget https://github.com/digitalocean/doctl/releases/download/v1.58.0/doctl-1.58.0-linux-amd64.tar.gz && \
    tar xf doctl-1.58.0-linux-amd64.tar.gz && \
    sudo mv doctl /usr/local/bin

RUN sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
  echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update && \
  sudo apt-get install -y kubectl

RUN wget https://get.helm.sh/helm-v3.5.3-linux-amd64.tar.gz && \
  tar xf helm-v3.5.3-linux-amd64.tar.gz && \
  ls -alF && \
  sudo mv linux-amd64/helm /usr/local/bin

RUN sudo wget https://releases.hashicorp.com/terraform/0.14.7/terraform_0.14.7_linux_amd64.zip && \
  sudo unzip terraform_0.14.7_linux_amd64.zip && \
  sudo mv terraform /usr/local/bin/

I use Gitlab for most of my dev work with a mirror to github. the gitlab CI file is pretty straight forward:


stages:
  - servers

build:
  image: docker.io/yourdomain/orc-deployment:latest
  stage: servers
  script:
    - cd Servers
    - echo "$MY_VAULT_PASSWORD" > orchestration/.vault_pass
    - bin/manage.sh deploy testing
  only:
    - main

Notice I am passing in my vault password from an environment variable. This is a protected environment variable in Gitlab so it's never shown. I also keep it out of the git repository by adding it to the .gitignore file.

the manage.sh file is pretty straight forward.


#!/usr/bin/env sh

export ANSIBLE_CONFIG=./orchestration/ansible.cfg
ansible-playbook -v orchestration/strategy.yaml -e operation=$1 -e env=$2

With all of these components, I can run the manage script with the operation and the env


> bin/manage.sh deploy testing

Once this is all deployed, I can go back and use Helm to update any aspect of the solution. The full project can be found on github

Conclusion

So three major elements to what I've done here. - Deployment Tools... Ansible, Terraform, Helm - Infrastructure Stack... Kubernetes Cluster on Digital Ocean with Ceph Storage - Software Stack... Vapor Application behind Traefik Proxy with Let's Encrypt TLS and MongoDB.

The goal was to use each element for what they are best suited. What do you think? Too much abstraction? Too many pieces? Or do they all play together nicely to create a repeatable deployment?

Tagged with: