Hello, and welcome to my first blog post!

Today we’re going to look into what is required to deploy a hexo site to a kubernetes cluster.

In this post we take a closer look at Dockerizing applications, Kubernetes deployment files and Gitlab CI Jobs to do these things for us.

Requirements

We’re going to use the following tools:

  1. jojoxd/deploy-tools
  2. A set-up Kubernetes cluster managed by GitLab
  3. GitLab CI Runner

Before getting started

A note on my Docker Repository;

My Docker repository is located at docker.jojoxd.nl:5005, which is a GitLab default instance. you should change this to your own server wherever you find it in this post.

Getting started

First, we’re going to create a hexo project

1
2
3
4
5
6
7
8
9
hexo init 'jojoxd-nl'

cd jojoxd-nl

git init
git remote add origin <git-url>

git commit
git push

This should push the site to your gitlab instance. (it should automatically create the repository.)

Generating Site Content

Firstly, we have to create a gitlab CI job to generate the content using hexo.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#
# .gitlab-ci.yml
#

stages:
  - compile

generate-content:
  stage: compile
  image: node:12

  script:
    # We're using npx to automatically download hexo.
    - yarn install
    - npx hexo generate

  artifacts:
    paths:
      - public/

When you git push this, it should generate an artifact with your site content.

Setting up Kubernetes deployment files

The Kubernetes files are templates, just like in Helm, but we don’t have any fancy if statements and such, it is only variable substitution.

deploy-tools uses envsubst internally to “template” the files.

Service

The service file groups multiple pods from a deployment together under a single Service.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#
# k8s/service.yaml
#
apiVersion: v1
kind: Service

metadata:
  name: "${CI_ENVIRONMENT_SLUG}-svc"
  namespace: "${KUBE_NAMESPACE}"

  labels:
    app: "${LABEL_APP}"

  annotations:
    app.jojoxd.nl/ref: "${LABEL_REF}"

spec:
  selector:
    app: "${LABEL_APP}"

  ports:
    - name: "${CI_ENVIRONMENT_SLUG}"
      port: 80

Notice that we use CI_ENVIRONMENT_SLUG as a prefix in metadata.name, this allows GitLab to monitor the resource and ingress usage for the site.

Note: you can use your own domain for the ref annotation, e.g. ref.my-domain.org: ${LABEL_REF}

Deployment

The Deployment actually deploys the Docker image to your Kubernetes cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#
# k8s/deployment.yaml
#
apiVersion: apps/v1
kind: Deployment

metadata:
  name: "${CI_ENVIRONMENT_SLUG}"
  namespace: "${KUBE_NAMESPACE}"

  labels:
    app: "${LABEL_APP}"

  annotations:
    app.jojoxd.nl/ref: "${LABEL_REF}"

spec:
  replicas: 1

  selector:
    matchLabels:
      app: "${LABEL_APP}"

  template:
    metadata:
      labels:
        app: "${LABEL_APP}"

    spec:
      containers:
        - name: hexo
          image: "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}"

          ports:
            - containerPort: 80
              name: "${CI_ENVIRONMENT_SLUG}"

Securing the image repository

Note: This will only work with Gitlab Docker Registry

Create a Deploy Token for your project, you can find it in Project Settings > Repository > Deploy Tokens

It MUST be named gitlab-deploy-token, or it won’t be available in your CI environment
You MUST do this step if your project is set to private or protected.

Patch the following files:

1
2
3
4
5
6
7
8
9
#
# k8s/deployment.yaml
#

spec:
  template:
    spec:
      imagePullSecrets:
        - name: "${DEPLOY_TOKEN_SECRET}"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/usr/bin/env bash
#
# deploy.sh (Created later in this tutorial)
#

{snip}

# Add these to the env:: statements
env::exists CI_REGISTRY
env::exists CI_DEPLOY_USER
env::exists CI_DEPLOY_PASSWORD
env::exists GITLAB_USER_EMAIL
env::exists DEPLOY_TOKEN_SECRET "gitlab-deploy-token"

{snip}

# Setup Service
kubectl::apply_file k8s/service.yaml "" false

# Add this:
kubectl create secret docker-registry "${DEPLOY_TOKEN_SECRET}" \
  -n "${KUBE_NAMESPACE}"
  --docker-server="${CI_REGISTRY}" \
  --docker-username="${CI_DEPLOY_USER}" \
  --docker-password="${CI_DEPLOY_PASSWORD}" \
  --docker-email="${GITLAB_USER_EMAIL}"

# Setup Deployment
kubectl::apply_file k8s/deployment.yaml "jojoxd-nl" false

{snip}

This will generate a Secret containing the deploy-key you created in GitLab.

Ingress

The ingress connects the service to a domain

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#
# k8s/ingress.yaml
#
apiVersion: networking.k8s.io/v1beta1
kind: Ingress

metadata:
  name: "${CI_ENVIRONMENT_SLUG}"
  namespace: "${KUBE_NAMESPACE}"

  labels:
    app: "${LABEL_APP}"

  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"

    nginx.ingress.kubernetes.io/rewrite-target: "/"

    app.jojoxd.nl/ref: "${LABEL_REF}"

spec:
  tls:
    - hosts:
      - "${APP_HTTP_HOSTNAME}"
      secretName: "${CI_ENVIRONMENT_SLUG}-tls"

  rules:
    - host: "${APP_HTTP_HOSTNAME}"
      http:
        paths:
          - path: "${APP_HTTP_PATH}"
            backend:
              serviceName: "${CI_ENVIRONMENT_SLUG}-svc"
              servicePort: 80

Deployment

Containerizing your site

To run your site in Kubernetes, you need to create a Docker image.

This is very easy to do; we only need to have a webserver and the generated files for it.

1
2
3
4
5
6
7
8
9
10
#
# Dockerfile
#
FROM nginx:1.16.0-alpine

EXPOSE 80/tcp 80/udp

COPY ./public /usr/share/nginx/html

CMD ["nginx", "-g", "daemon off;"]

Building Container using GitLab CI

We’re going to use GitLab CI to create and push our app container to the container repository.

The complete .gitlab-ci.yml can be found at the end of this tutorial

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#
# .gitlab-ci.yml
#

stages:
  - compile
  - release

{snip}

release-image:
  stage: release
  image: docker:latest
  services:
    - 'docker:dind'

  script:
    - docker login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" "${CI_REGISTRY}"
    # Note the dot at the end, it is important:
    - docker build --pull -t "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}" -f Dockerfile .
    - docker login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" "${CI_REGISTRY}"
    - docker push "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}"

  dependencies:
    - generate-content

{snip}

We add a dependency on generate-content, so the public artifact will be re-downloaded from GitLab.
Otherwise, the job could be run on a different runner, where the folder is not available, and the build will therefore fail.

You should git push this so you can check if your container is being built correctly

Testing the container

Before continuing, we’re going to test the container using Docker.

1
docker run --rm -p 8080:80 -t docker.jojoxd.nl:5005/jojoxd-k8s/jojoxd-nl:master

You should now be able to see the site using localhost:8080

Setting up deploy-tools

For the deployment, we’re going to use deploy-tools, my own feature-poor bash-implemented version of helm.

1
2
3
4
# In the jojoxd-nl folder

mkdir k8s
git submodule add https://gitlab.jojoxd.nl/jojoxd-k8s/deploy-tools.git k8s/deploy-tools

Creating deploy script

The deploy-script should be in the root of your repository. (I named it deploy.sh)

Firstly, we should load the deploy-tools.sh file

1
source $(which deploy-tools.sh || echo "k8s/deploy-tools/deploy-tools.sh")

We specify it with a which first, as we’re not going to setup GitLab to pull submodules, and instead use the Docker image provided by deploy-tools to gain access to the tools and kubectl.

Next up, we ensure all our environment variables are set.

We use the CI_* variables to be able to track our deployments using an annotation later.

1
2
3
4
5
6
7
8
9
10
11
12
13
env::exists KUBE_NAMESPACE

env::exists CI_COMMIT_REF_SLUG
env::exists CI_ENVIRONMENT_SLUG
env::exists CI_COMMIT_SHORT_SHA

env::exists CI_REGISTRY_IMAGE

env::exists LABEL_APP "jojoxd-nl"
env::exists LABEL_REF "${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}"

gitlab_ci::http_hostname APP_HTTP_HOSTNAME
gitlab_ci::http_path APP_HTTP_PATH

Next up, we’re going to call kubectl to push all our configuration to the cluster.

1
2
3
4
5
6
7
8
9
10
11
12
#!/usr/bin/env bash

kubectl create namespace "${KUBE_NAMESPACE}"

# Setup Service
kubectl::apply_file k8s/service.yaml "" false

# Setup Deployment
kubectl::apply_file k8s/deployment.yaml "${CI_ENVIRONMENT_SLUG}" false

# Setup Ingress
kubectl::apply_file k8s/ingress.yaml "" false

If you want to use imagePullSecrets, add that now

Locally testing deployment using minikube

It’s always a good idea to test your deployment before pushing it to your production server; let’s use minikube to deploy it to a freshly created cluster.

Note: You should push your changes (including your .gitlab-ci.yml with the release step) to the server, So you can use the generated Docker image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
minikube start

minikube addon enable ingress

minikube dashboard

# in a different terminal (Preferably BASH)
# Setup required environment variables for local development (I usually use a bash script that exports them)
export KUBE_NAMESPACE="jojoxd-nl"
export CI_COMMIT_REF_SLUG="master"
export CI_ENVIRONMENT_SLUG="develop"
export CI_COMMIT_SHORT_SHA="aabbccdd"
export CI_REGISTRY_IMAGE="docker.jojoxd.nl:5005/jojoxd-k8s/jojoxd-nl:master"
export CI_ENVIRONMENT_URL="jojoxd.local"

# Start the deployment by using the deploy script
./deploy.sh

If you get an error that the script can’t source deploy-tools,
you might need to run git submodule update --init before retrying

If everything is right, you should see that the namespace, deployment, service and ingress are created in the dashboard.

To access your deployment, look at the ingress IP, and add the following to your /etc/hosts file:
<ingress-ip> jojoxd.local

You should now be able to connect to jojoxd.local in your browser.

Note: You might need to disable HSTS in the browser to be able to view the site.

Setting up Deployment using Gitlab CI

For GitLab to deploy your site, it should only need to call deploy.sh.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#
# .gitlab-ci.yml
#

stages:
  - compile
  - release
  - deploy

deploy:
  stage: deploy
  when: manual
  only:
    - master

  image: docker.jojoxd.nl:5005/jojoxd-k8s/deploy-tools:latest

  environment:
    name: production
    url: https://jojoxd.nl

  script:
    - ./deploy.sh

Commit this, let the compile & release run, and click the play button on this new stage.

It should now push your site to your cluster.