TIL: NixOS deletes store paths that only config files mention

Last night my server suddenly stopped updating itself. Every hour a notification came in with the error No such file or directory. The path behind it starts with /nix/store/, and git fetch failed on it every time.

The culprit was a path in /etc/nixos/.git/config. On the first clone, my service ran git config core.sshCommand with the value of ${pkgs.openssh}/bin/ssh, which expands to something like /nix/store/98xy…-openssh-10.5p1/bin/ssh.

Nix puts every package in its own directory under /nix/store, with a hash in the name. An update overwrites nothing; the old version stays until garbage collection (nix-gc) cleans up. It deletes whatever nothing points to anymore: no system state Nix keeps around, no other package. To find out, Nix only looks at what it manages itself. My .git/config isn't part of that, and a cron job, a script or a dotfile would have been the same. After the update to openssh 10.6p1, the old version was surplus as far as Nix was concerned, and garbage collection ran at midnight.

The fix: I now set GIT_SSH_COMMAND as environment on the service. The variable takes precedence over core.sshCommand, and since it lives in the system configuration, Nix keeps the package around:

systemd.services.nixos-auto-rebuild = {
  environment.GIT_SSH_COMMAND =
    "${pkgs.openssh}/bin/ssh -i /root/.ssh/infra-pull";
};

A Nix pro probably wouldn't have made this mistake. Afterwards I went through my Nix configuration again, carefully, looking for similar spots. I did get it right in the series, by the way: there it's just ssh, resolved through path.