NixOS: Secrets in the Git repo with sops-nix

My server pulls its configuration from Git, but most services need a password or a token, and those don't belong in a repo in plain text. With sops-nix they sit encrypted right next to the configuration, and only the server and I can read them.

In part 2 I set up the server so that it rebuilds itself from Git every hour. Whoever can read the configuration repo can therefore see everything in it. My repo is private, but a plain-text password would still be a bad idea. Everything in the configuration ends up in the Nix store when the system is built, and the store is readable by all users of the server by default.

sops is a tool that encrypts only the values of a YAML file. The names stay readable, and in the Git diff I can see which secret I touched.

The state after this part is in the nixos-series repository on the branch part-3-secrets-with-sops-nix, and the comparison of the two branches shows what is added compared to part 2. The encrypted secrets.yaml is missing there; you create it with your own keys.

Who can read what?

sops encrypts with age. I use an age key for this that I use for nothing else. For the server I use the SSH host key it has had since the installation: ssh-to-age converts it into an age key, and NixOS derives the private key from it by itself at boot. I don't have to create anything on the server, and there is no second key I would have to back up.

The secrets are encrypted for two recipients: for me, so that I can edit them, and for the server, so that it can decrypt them at boot.

Step 1: Tools and keys

I install sops and age with Homebrew on the Mac and with the package manager of your choice elsewhere:

brew install sops age

ssh-to-age converts SSH keys into age keys. If you have Nix on your machine (part 1 requires it), there is nothing to install: nix run nixpkgs#ssh-to-age -- replaces every call of ssh-to-age below. Without Nix, the releases page has a ready-made binary.

I create my age key where sops looks for it on its own. On macOS that is ~/Library/Application Support/sops/age/keys.txt, on Linux ~/.config/sops/age/keys.txt:

d="$HOME/Library/Application Support/sops/age"
mkdir -p "$d"
age-keygen -o "$d/keys.txt"

age-keygen prints the public key (age1...), which I need for the configuration in a moment. It is also in the file as a comment. The file is readable only by me but not encrypted, and if you lose it, you can no longer get at your secrets. So a copy belongs in your password manager. With the wrong path, sops later reports no identity matched any of the recipients.

I don't actually keep the key on disk myself. It lives in my password manager, and with the environment variable SOPS_AGE_KEY_CMD I tell sops which command fetches it from there. That works with any password manager that has a command line. Here I stick to the simple variant.

For the server, I read the public host key over SSH and convert it:

ssh root@<ipv4> cat /etc/ssh/ssh_host_ed25519_key.pub \
  | ssh-to-age

This output also starts with age1.

Step 2: Define the recipients

In the root of the repo there is a .sops.yaml. It tells sops for whom to encrypt which file:

keys:
  - &ich age1...
  - &hetzner age1...
creation_rules:
  - path_regex: secrets/secrets\.yaml$
    key_groups:
      - age:
          - *ich
          - *hetzner

The two keys are the public keys from step 1. The pattern secrets/secrets\.yaml$ applies to the file I create next.

Step 3: The first secret

sops secrets/secrets.yaml opens the file in the editor. If it doesn't exist yet, sops creates it encrypted when you save. As an example I use the credentials for Miniflux, the feed reader I run on my server. It only stands in here: any service that reads its credentials from a file works the same way. Miniflux expects a file with KEY=VALUE lines, so I store that as a multi-line value:

miniflux-admin: |
  ADMIN_USERNAME=daniel
  ADMIN_PASSWORD=a-long-password

After saving, the file in the repo looks roughly like this (shortened):

miniflux-admin: ENC[AES256_GCM,data:3fG...,type:str]
sops:
  age:
    - recipient: age1...
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        ...

The name miniflux-admin is in plain text, the value isn't. To read or change it, I open the file again with sops secrets/secrets.yaml, and a single secret can be printed with, for example, sops -d --extract '["miniflux-admin"]' secrets/secrets.yaml.

Step 4: Connect NixOS

In flake.nix, sops-nix is added as an input and its module goes into the list:

inputs = {
  # ...
  sops-nix = {
    url = "github:Mic92/sops-nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };
};

outputs = { self, nixpkgs, disko, sops-nix, ... }: {
  nixosConfigurations.hetzner = nixpkgs.lib.nixosSystem {
    modules = [
      disko.nixosModules.disko
      sops-nix.nixosModules.sops
      # ...
    ];
  };
};

In configuration.nix there are three things: where the encrypted file lives, where the server gets its key from, and which secrets it provides from it. Miniflux gets the path to the secret as adminCredentialsFile:

{ config, ... }:
{
  sops.defaultSopsFile = ../../secrets/secrets.yaml;
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];

  sops.secrets.miniflux-admin.restartUnits = [
    "miniflux.service"
  ];

  services.miniflux = {
    enable = true;
    adminCredentialsFile =
      config.sops.secrets.miniflux-admin.path;
  };
}

config.sops.secrets.miniflux-admin.path is /run/secrets/miniflux-admin. The file only comes into being when the system is activated on the server. Only the encrypted value is in the repo and in the Nix store. restartUnits makes Miniflux restart when I change the password. Without that line, the service doesn't notice the new value.

Then commit and push as usual, and an hour later at the latest it is running. If you don't want to wait, trigger the rebuild as in part 2. A look at the server shows whether it worked:

ssh root@<ipv4> "systemctl status miniflux"
ssh root@<ipv4> "ls -l /run/secrets"

When the server is reinstalled

The catch with this setup: the server decrypts with its host key. If I reinstall it, it creates a new one, and the old contents of secrets.yaml are no longer readable for it. That's why I stored the host key once in the repo, encrypted, for me only:

ssh root@<ipv4> cat /etc/ssh/ssh_host_ed25519_key \
  | sops encrypt --filename-override secrets/hostkey.enc \
      --input-type binary --output-type binary /dev/stdin \
  > secrets/hostkey.enc

For that, .sops.yaml needs a second rule with secrets/hostkey\.enc$ that contains only *ich. On a reinstall I put the key in a directory that nixos-anywhere copies to the new server with --extra-files. I start the rescue system for that as in part 1:

umask 077
mkdir -p /tmp/extra/etc/ssh
sops decrypt --input-type binary --output-type binary \
  secrets/hostkey.enc \
  > /tmp/extra/etc/ssh/ssh_host_ed25519_key
ssh-keygen -y -f /tmp/extra/etc/ssh/ssh_host_ed25519_key \
  > /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub
nix run github:nix-community/nixos-anywhere -- \
  --build-on-remote \
  --extra-files /tmp/extra \
  --flake .#hetzner \
  root@<ipv4>
rm -rf /tmp/extra

That way the server has the same age key as before from the start and can decrypt its secrets right away.

If you set up a server from scratch, you can skip the detour of reading the key out. You create the host key straight in the --extra-files directory, convert its public part with ssh-to-age and put it in .sops.yaml. You pass the key along at installation, and the server can decrypt its secrets from the very first boot:

umask 077
mkdir -p /tmp/extra/etc/ssh
ssh-keygen -q -t ed25519 -N '' \
  -f /tmp/extra/etc/ssh/ssh_host_ed25519_key
ssh-to-age < /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub

I encrypt the private key for secrets/hostkey.enc as above, except that sops reads the local file this time instead of the output of ssh. I delete the directory only after the installation.

In the next part, the server finally serves a website.