NixOS: git push instead of nixos-rebuild

When I want to change something on my server, I edit a file, commit and push. An hour later at the latest, the new configuration is running, and I never log in to the server.

In the first part I installed NixOS on an empty Hetzner VM with nixos-anywhere and still pushed every change to the server with nixos-rebuild from my own machine. That works from any machine that runs Nix. I started out on Linux with NixOS. By now I also want to update my infrastructure from the Mac, without having to install Nix there.

So I turn the direction around: the server pulls its configuration from a Git repository by itself, in my case on Codeberg.

The idea isn't new. NixOS ships system.autoUpgrade, a timer that rebuilds a flake regularly, but it has no option to verify signatures. There are also ready-made tools that let a server pull its own configuration, for example comin or auto-upgrade-with-pinch. comin in particular looks promising and can also verify SSH-signed commits. But I didn't know about it when I set up my server, so I built the mechanism myself. What mattered to me was that only signed commits get built, as an extra safety net. Whether I'll switch to comin later, I still have to look into.

I got to this setup in three steps, and the server works after each one. At the end of the article you'll find the complete configuration. The state after this part is in the nixos-series repository on the part-2-git-push-instead-of-nixos-rebuild branch, and the comparison of the two branches shows what is new compared to part 1.

A new file

I continue from where part 1 ended. Everything for the auto-rebuild goes into a separate file so configuration.nix stays readable:

infra/
├── flake.nix
└── hosts/hetzner/
    ├── auto-rebuild.nix   (new)
    ├── configuration.nix
    ├── disko.nix
    └── hardware.nix

For NixOS to load the file, it goes into the list of modules in flake.nix:

modules = [
  disko.nixosModules.disko
  ./hosts/hetzner/hardware.nix
  ./hosts/hetzner/configuration.nix
  ./hosts/hetzner/auto-rebuild.nix
];

auto-rebuild.nix is built like hardware.nix. The first line brings pkgs into scope so I can use packages like git, and everything from the next steps goes between the curly braces:

{ pkgs, ... }:
{
  # services go here
}

Step 1: The server builds from Git

First the server needs a copy of the repo, and it goes into /etc/nixos, because that is where NixOS looks for its configuration by default. My repo is private, so the server gets a deploy key: a separate SSH key that may only read this one repo. These two commands create the key on the server:

ssh root@<ipv4> \
  "ssh-keygen -t ed25519 -f /root/.ssh/infra-pull -N ''"
ssh root@<ipv4> "cat /root/.ssh/infra-pull.pub"

I add the output of the second command on Codeberg, in the repo settings under Deploy Keys, without write access. If your repo is public you don't need the key and can clone over https.

Nobody on the server can answer the question whether to trust Codeberg's host key. Without an entry, the clone fails with Host key verification failed. So I put Codeberg's host key into auto-rebuild.nix. ssh-keyscan -t ed25519 codeberg.org prints the value. Compare it with the fingerprint Codeberg publishes:

programs.ssh.knownHosts."codeberg.org".publicKey =
  "ssh-ed25519 AAAA...";

Work the server should do on its own, I describe in NixOS as a systemd service. The first one clones the repo at boot, but only if /etc/nixos isn't a Git repo yet. That way a new server sets itself up as well.

systemd.services.clone-infra-repo = {
  wantedBy = [ "multi-user.target" ];
  after = [ "network-online.target" ];
  wants = [ "network-online.target" ];
  path = [ pkgs.git pkgs.openssh ];
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
  };
  script = ''
    if [ ! -d /etc/nixos/.git ]; then
      export GIT_SSH_COMMAND="ssh -i /root/.ssh/infra-pull"
      rm -rf /etc/nixos
      git clone git@codeberg.org:<user>/infra.git /etc/nixos
      git -C /etc/nixos config \
        core.sshCommand "$GIT_SSH_COMMAND"
    fi
  '';
};

wantedBy starts the service at boot, after and wants make it wait for the network, and path lists the programs the script may use. oneshot means the script runs once and is done, unlike a web server that keeps running.

The second service does the actual rebuild and needs the clone, so it points to the first one with requires and after. It fetches the latest state and builds with nixos-rebuild switch, the same command as in part 1, just on the server itself. If nothing has changed, it stops right away. --ff-only only accepts commits that build on the current state, and a rewritten history aborts. A systemd timer, something like a cron job, starts the whole thing: five minutes after boot and every hour after that.

systemd.services.nixos-auto-rebuild = {
  after = [ "network-online.target" "clone-infra-repo.service" ];
  wants = [ "network-online.target" ];
  requires = [ "clone-infra-repo.service" ];
  restartIfChanged = false;
  path = [ pkgs.git pkgs.openssh ];
  serviceConfig.Type = "oneshot";
  script = ''
    set -euo pipefail
    cd /etc/nixos
    git fetch origin
    remote=$(git rev-parse origin/main)
    if [ "$(git rev-parse HEAD)" = "$remote" ]; then
      exit 0
    fi
    git merge --ff-only "$remote"
    /run/current-system/sw/bin/nixos-rebuild switch \
      --flake /etc/nixos#hetzner
  '';
};

systemd.timers.nixos-auto-rebuild = {
  wantedBy = [ "timers.target" ];
  timerConfig = {
    OnBootSec = "5min";
    OnUnitActiveSec = "1h";
  };
};

One line isn't obvious: restartIfChanged = false. On a switch, NixOS restarts every service whose definition has changed. If I change something in this service, NixOS would restart it in the middle of its own switch. The module behind system.autoUpgrade sets the same line. That is exactly what happened to me: the rebuild ended itself while switching, and the log only said the service had been stopped. It took me a while to understand that it had shot itself.

I commit and push this state and put it on the server one last time with nixos-rebuild from the Mac. From then on, git push is enough.

Step 2: Signed commits only

So far the server builds everything that lands in main. Whoever can push to my repo can therefore rebuild my server. That's why it should only build commits I have signed.

Since version 2.34, Git can sign with SSH keys instead of GPG, and in my opinion that makes it a lot easier: an SSH key is quickly created. On the Mac I set three options in the root of my repository (infra/). Without --global they only apply to this repo (I use them globally by now, but for this series I don't want to change your whole Git configuration):

git config gpg.format ssh
git config user.signingkey ~/.ssh/id_ed25519.pub
git config commit.gpgsign true

From now on Git signs every commit automatically. In return, the server has to know which key is valid. For that, auto-rebuild.nix gets a let block at the top, in which I define a variable. It stands for a small file with my public key: first a label, in my case my email address, then the content of ~/.ssh/id_ed25519.pub (cat ~/.ssh/id_ed25519.pub shows it). I write the line myself, and Nix creates the file when it builds, so I don't have to put it anywhere.

{ pkgs, ... }:
let
  allowedSigners = pkgs.writeText "allowed-signers" ''
    me@example.org ssh-ed25519 AAAA...
  '';
in
{
  # services as before
}

In the rebuild service's script, a loop goes right before git merge. It checks every commit that is new. Nix replaces ${allowedSigners} with the path of the generated file.

for c in $(git rev-list HEAD.."$remote"); do
  git -c gpg.format=ssh \
    -c gpg.ssh.allowedSignersFile=${allowedSigners} \
    verify-commit "$c"
done

If a commit isn't signed, the script aborts and the server stays on the old state. I pay the price with pull requests: on Codeberg I can't just click Merge, because Codeberg doesn't sign the merge commit with my key. So I merge locally and then push.

Step 3: When a build fails

The script still has a bug. It merges first and builds afterwards. If the build fails, the checkout is already up to date, and on the next run there seems to be nothing to do. The server only tries again with the next commit.

The fix: the script no longer compares origin/main with the checkout but with the system that is running right now. For that, the system has to know which commit it was built from. That is the only change outside auto-rebuild.nix: in flake.nix, self is added to the arguments of outputs, plus one line in the list of modules.

outputs = { self, nixpkgs, disko, ... }: {
  nixosConfigurations.hetzner = nixpkgs.lib.nixosSystem {
    modules = [
      { system.configurationRevision = self.rev or "dirty"; }
      # ...
    ];
  };
};

On the server, nixos-version --configuration-revision prints that commit, and the script now compares against it. If a build fails, the server tries again an hour later.

This also means the server builds when the checkout and origin/main match, for example after a fresh clone. In that case git rev-list returns no commit. So the loop starts with "$remote" and checks origin/main itself as well.

Last but not least, one more case: a build that hangs. With a oneshot service, systemd waits indefinitely, and as long as it runs, the timer doesn't start a new one. TimeoutStartSec = "1h" aborts it after an hour.

The finished file

This is what hosts/hetzner/auto-rebuild.nix looks like in the end:

{ pkgs, ... }:
let
  allowedSigners = pkgs.writeText "allowed-signers" ''
    me@example.org ssh-ed25519 AAAA...
  '';
in
{
  programs.ssh.knownHosts."codeberg.org".publicKey =
    "ssh-ed25519 AAAA...";

  systemd.services.clone-infra-repo = {
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
    path = [ pkgs.git pkgs.openssh ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
    };
    script = ''
      if [ ! -d /etc/nixos/.git ]; then
        export GIT_SSH_COMMAND="ssh -i /root/.ssh/infra-pull"
        rm -rf /etc/nixos
        git clone git@codeberg.org:<user>/infra.git /etc/nixos
        git -C /etc/nixos config \
          core.sshCommand "$GIT_SSH_COMMAND"
      fi
    '';
  };

  systemd.services.nixos-auto-rebuild = {
    after = [ "network-online.target" "clone-infra-repo.service" ];
    wants = [ "network-online.target" ];
    requires = [ "clone-infra-repo.service" ];
    restartIfChanged = false;
    path = [ pkgs.git pkgs.openssh ];
    serviceConfig = {
      Type = "oneshot";
      TimeoutStartSec = "1h";
    };
    script = ''
      set -euo pipefail
      cd /etc/nixos
      sw=/run/current-system/sw/bin
      git fetch origin
      remote=$(git rev-parse origin/main)
      running=$($sw/nixos-version --configuration-revision || true)
      if [ "$running" = "$remote" ]; then
        exit 0
      fi
      for c in "$remote" $(git rev-list HEAD.."$remote"); do
        git -c gpg.format=ssh \
          -c gpg.ssh.allowedSignersFile=${allowedSigners} \
          verify-commit "$c"
      done
      git merge --ff-only "$remote"
      $sw/nixos-rebuild switch --flake /etc/nixos#hetzner
    '';
  };

  systemd.timers.nixos-auto-rebuild = {
    wantedBy = [ "timers.target" ];
    timerConfig = {
      OnBootSec = "5min";
      OnUnitActiveSec = "1h";
    };
  };
}

If you don't want to wait an hour, start the rebuild from your machine and look at the log:

ssh root@<ipv4> "systemctl start nixos-auto-rebuild"
ssh root@<ipv4> "journalctl -u nixos-auto-rebuild -n 30"

One catch remains: the server now gets everything from Git, and some things don't belong there in plain text, such as passwords and API tokens. How I put them into the repo encrypted, so that only I and the server can read them, I'll show in the next part with sops-nix.