• servers
  • knowlege-backup
  • meta
  • cd
  • forgejo
  • git

How This Blog is Deployed

I mentioned last month I had changed the underlying tech driving this blog to be primary written in golang. I may release the structure of that application at some point, but for now I thought I would document how I deploy this service to my VPS.

My VPS is hosted on hetzner, I try to host most of my external-facing things there to avoid my in home NAS from being too much of an attack surface. My previous deployment process was faily simple but not pipeline-driven.

Previous Deploys

  1. Write new content or add a new feature
  2. run make build
  3. run make sync

This did a docker build which exported the contents of the static site generator’s output to a local folder, then make sync ran rsync to push the changes to a folder on my vps. My site at that time was a nginx service sitting behind my nginx proxy.

I wanted to keep a similiar process with the rewrite, but I wanted it to be backed by a pipeline, so I could write anywhere with access to my git server, and deploy with a push. I also wanted this to be much more secure and locked down than my previous action, which involved my primary user copying files to the service. This user had full sudo access to the machine, which is not ideal security practice.

After some research and experimentation, here is where things stand.

Server User Setup

  • I have a user deploy on my VPS. My VPS is named ext(yes this is a departure from my naming system). This user has a home directory in my service folder.
sudo useradd -M -d /var/docker-service/blog/ -s /usr/sbin/nologin deploy
sudo chown -R deploy:deploy /var/docker-service/blog/
  • Then I created a wrapper script named /usr/local/bin/blog-compose for running docker-compose commands in that directory. `
#!/bin/bash
set -e
cd /var/docker-service/blog
exec docker compose -f /var/docker-service/blog/docker-
compose.yml "$@"
  • Make this script executable
sudo chmod +x /usr/local/bin/blog-compose
  • Now you need to allow that script to be run with sudo as your user. This allows this user to perform docker commands without needing to be a member of the docker group
sudo visudo -f /etc/sudoers.d/blog
### contents
deploy ALL=(root) NOPASSWD: /usr/local/bin/blog-compose
deploy ALL=(root) NOPASSWD: /usr/local/bin/blog-compose *
  • Finally, you need to generate an ssh key and add it to /var/docker-service/blog/.ssh/authorized_key file. I added a passphrase for additional security. There are 1 million articles on this on the web, so I won’t repeat those details.

Docker Compose

I run all my services as docker containers. This is partly for encapsulation and partly because it makes adding and removing services simple. I front the containers with nginx-proxy and it’s acme-companion, which makes SSL super low maintenance. Because of this, the site is run as a Docker container. Here is the corresponding compose file.

Notes: I use scratch as my base image for running the site, this makes for a smaller image and reduces potential attack surfaces. Because of this I am mounting the root SSH keys and timezone data into the container. Without these, requests to my mealie instance fail as it cannot validate the ssh key and using a timezone to gatekeep post publishing fails as it cannot find timezone data.

services:
  connermccall.com:
# I run a docker registry internally for personal projects
    image: registry.connermccall.me/personal/connermccall.com:latest
    environment:
      - "URL=connermccall.com"
# Recipe retrieval requires a key
      - "MEALIE_API_KEY=${MEALIE_API_KEY}"
# there are the important things, this tell the proxy and companion what domains we are serving this site on. 
      - "LETSENCRYPT_EMAIL=conner@connermccall.com"
      - "LETSENCRYPT_HOST=connermccall.com,www.connermccall.com"
      - "VIRTUAL_HOST=connermccall.com,www.connermccall.com"
      - "VIRTUAL_PORT=8080"
    networks:
      - network-nginx-proxy
    restart: "unless-stopped"
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro"
      - "/usr/share/zoneinfo:/usr/share/zoneinfo:ro"
    logging:
      driver: syslog
      options:
        tag: "docker/connermccall-com"
# I mark this as external as it is created with tofu elsewhere. The container has to be on this network for the proxy and companion to recognize it
networks:
  network-nginx-proxy:
    external: true

CI Setup

I use Forgejo as my forge. It has been rock solid for the past several years and I have a runner configured on my local NAS. This gives me more compute than my VPS provides along with plenty of disk space and access to local resources.

I will walk through the pertinent parts of my script, the full version is at the end of this post.

The first five steps are checking out the container, setting up build details, and actually building and pushing the container. Once that is done we need to update the running container on my VPS, which we do through SSH.

I chose to protect my ssh key with a passphrase. This passphrase is stored in 1Password. They provide a github action for fetching secrets. This takes a Service Token and then a map of keys to values. In this case we map the password field from the secret named ssh_passphrase in the forgejo vault to the output SSH_KEY_PASSPHRASE.

- name: Fetch Passphrase
  id: fetch_passphrase
  uses: https://github.com/1password/load-secrets-action@v4
  env:
    OP_SERVICE_ACCOUNT_TOKEN: ${{ env.OP_SERVICE_ACCOUNT_TOKEN }}
    SSH_KEY_PASSPHRASE: op://forgejo/ssh_passphrase/password

Once we have the passphrase we need to update the image and recreate the container on my vps. I used an ssh action to accomplish this. We need to pass the ssh key, the passphrase, the hostname, user, and ideally the host fingerprint to the command. Finally the command must be run with sudo so that the user has access to the necessary permissions for interacting with docker.

- name: Update compose
  uses: https://github.com/appleboy/ssh-action@v1.2.5
  with:
     username: ${{ vars.USER }}
     host: ${{ secrets.HOST }}
     key: ${{ secrets.SSH_KEY }}
     passphrase: ${{ steps.fetch_passphrase.outputs.SSH_KEY_PASSPHRASE }}
     fingerprint: ${{ secrets.HOST_FINGERPRINT}}
     script: |
        sudo /usr/local/bin/blog-compose up -d --pull always --force-recreate

Gotchas

  • Ensure you set the permissions correctly on the .ssh directory and authorized key file.
ext:~$ chmod 700 .ssh/
ext:~$ chmod 600 .ssh/authorized_keys
  • I technically named my user something different which included a dot in their username. This tripped up sudo when I added the file user visudoers.d with the actual username. sudo ignores files with dots in their name. The easy fix was renaming the file with a - instead of a dot. But to confirm that you successfully granted the user permission you can run.
sudo -l -U deploy
  • You may need to try a couple different host keys to figure out which one will work. I initially used the ssh_host_ecdsa_key fingerprint and the pipeline failed. I settled on the ssh_host_ed25519_key.
ext:~$ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:XXXX....XX root@ext (ED25519)

You need the 2nd string, including the SHA256 part. SHA256:XXXX....XX

Full Pipeline

name: Build and Deploy New Version

env:
  IMAGE_NAME: connermccall.com
  REPO: registry.connermccall.me/personal


on:
  push:
    branches:
      - 'main'

jobs:
  docker-build:
    name: Build and push docker image
    runs-on: docker
    container:
      image: catthehacker/ubuntu:act-latest
    steps: 
      - name: Checkout
        uses: actions/checkout@v4
      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v6
        with:
          images: ${{ env.REPO }}/${{ env.IMAGE_NAME }}
          flavor: latest=auto
          tags: |
            type=ref,event=branch
            type=sha
            type=raw,value=latest,enable={{is_default_branch}}
      - name: Resolve version info
        id: version
        run: |
          echo "GIT_HASH=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          push: true
          context: .
          cache-from: type=registry,ref=${{ env.REPO }}/${{ env.IMAGE_NAME }}:latest
          cache-to: type=inline
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          build-args: |
            ASSET_HASH=${{ env.GIT_HASH }}
      - name: Fetch Passphrase
        id: fetch_passphrase
        uses: https://github.com/1password/load-secrets-action@v4
        env:
          OP_SERVICE_ACCOUNT_TOKEN: ${{ env.OP_SERVICE_ACCOUNT_TOKEN }}
          SSH_KEY_PASSPHRASE: op://forgejo/ssh_passphrase/password
      - name: Print masked secret
        run: 'echo "Secret: ${{ steps.fetch_passphrase.outputs.SSH_KEY_PASSPHRASE }}"'
      - name: Update compose
        uses: https://github.com/appleboy/ssh-action@v1.2.5
        with:
          username: ${{ vars.USER }}
          host: ${{ secrets.HOST }}
          key: ${{ secrets.SSH_KEY }}
          passphrase: ${{ steps.fetch_passphrase.outputs.SSH_KEY_PASSPHRASE }}
          fingerprint: ${{ secrets.HOST_FINGERPRINT}}
          script: |
           sudo /usr/local/bin/blog-compose up -d --pull always --force-recreate
Feedback:
💙 Likes 📅 Now ⚙️ Defaults 🖥️ Self Hosted