nix-builder-autoscaler

nix-builder-autoscaler provides elastic Nix remote builder capacity for Guardian Project jobs using buildbot-nix.

The reason this was created is because we don't have the budget for dedicated always-on build hardware.

So the idea is that Buildbot waits for builder capacity before it starts a nix-build job, and idle builders should disappear some time after jobs finish.

The autoscaler launches EC2 nix builder instances, waits until they are reachable through Tailscale and HAProxy, hands Buildbot a reservation for a ready slot, and later drains and terminates unused capacity. It uses EC2 Spot by default and can use on-demand instances for nested virtualization workloads when configured.

The Buildbot instance has a single master/single worker config like upstream buildbot-nix expects. HAPoroxy is used to present a logical single nix-builder host to buildbot-nix, this was inspired by Garnix's yensid

Pieces

The project has two main runtime pieces:

1. agent/: the autoscaler daemon and autoscalerctl CLI. The daemon owns the slot database, reservation API, scheduler, EC2 runtime, HAProxy binding, health checks, and metrics.

2. buildbot-ext/: the Buildbot integration. The extension patches Buildbot */nix-build builders with a capacity gate step at the beginning and a reservation release step at the end. It also lets the Buildbot host send Nix distributed builds through the HAProxy-backed builder cluster.

The nix/modules/ directory contains NixOS modules that package and wire these pieces into hosts:

• services.nix-builder-autoscaler runs the daemon and can generate the HAProxy slot configuration.
• services.buildbot-nix.nix-build-autoscaler installs the Buildbot extension and configures Nix remote builder access.

How it works

Buildbot (via the extension) creates a reservation before a Nix build starts. The autoscaler assigns that reservation to a ready slot if one exists. If no ready slot has capacity, the scheduler launches an EC2 instance into an empty slot, subject to the configured minimum, maximum, warm pool, and timeout settings.

The reconciler moves each slot through the runtime states:

1. launching: EC2 accepted the instance launch.
2. booting: the instance is running.
3. binding: the daemon found the instance's Tailscale IP and enabled its HAProxy backend slot.
4. ready: HAProxy health checks pass and Buildbot can use the slot.
5. draining or terminating: the slot is being removed after release, idle timeout, interruption, or failure.

Buildbot waits until the reservation becomes ready, then runs the build through the configured Nix remote builder alias. When the build finishes, the release step releases the reservation. Idle slots drain and terminate after the configured cooldowns.

Development

Common checks:

nix flake check
nix build .#nix-builder-autoscaler
nix build .#buildbot-autoscale-ext
nix fmt

Useful local CLI commands against a running daemon:

autoscalerctl status
autoscalerctl slots
autoscalerctl reservations
autoscalerctl drain <slot-id>
autoscalerctl reconcile-now

The daemon listens on /run/nix-builder-autoscaler/daemon.sock by default. NixOS deployments should configure the service modules rather than hand-writing daemon config files.