Rolling my own CI/CD

Anyway, I write software. And my ever-growing collection of personal projects kept dragging along with it exactly those kinds of problems that are mentioned in the introduction. First of all, I kept finding out that they either had broken builds (typically on systems-that-are-not-mine), or more trivially, that some dependency decided to produce new warnings. Second of all, if someone else wanted to use what I had made, he’d have a hard time, seeing as essentially nothing has found its way to a distribution yet. He’d need to follow my terse instructions, and build it from source code—​which is only easy on BTW-I-use-Arch, because I maintain scripts for development builds of my software in the Arch User Repository. It doesn’t take much effort to publish installable packages for non-rolling distros, such as Ubuntu, but it’s a ton of mind-numbing work to do it by hand with each project release, much less commit.

In short, a CI/CD solution had the potential to be of great help, mostly to my mental well-being. Though I had a few conditions: I wasn’t about to pay for this, since I didn’t actually need it, and it couldn’t stop providing service out of the random, because that would bring me back to the beginning.

My experiences with public providers had generally been bad. In the past, I got acquainted with Travis CI, Open Build Service, Launchpad, and wanted to try out Wercker. Of these, the one with an ever-outdated instance of Ubuntu went all commercial, the packager-specific one was super convoluted, the pile of technical debt was a lot of pain to work with (then someone succeeded in having my account terminated), and the remaining one just disappeared.

On the other hand, the rather simple CI/CD that people had put together from shell scripts at my former job was more or less exactly what it needed to be. It was flexible, and the occasional problems had easy solutions.

I knew that the essence of this kind of automation is trivial: it runs shell scripts when triggered by a git push. The only hard parts are when and how to run them. And I believed I could do away with a lot of accidental complexity that comes with popular self-hosted solutions, so I set out to build something neat.

In the rest of this article, I’ll guide you through setting up the CI system I’ve put together. On the highest level, it looks like this:

Everything runs on the same physical machine (yet SSH makes it possible to also delegate tasks remotely to cover annoyances like macOS). When someone pushes to a repository, the event is recorded in a queue. The CI then launches any appropriate virtual machines from a snapshot, sends them work, and waits for the results. Those are attached to Gitea commits over its REST API. Build failures are forwarded to the administrator over e-mail or IRC. Eventually, build artifacts can be stored for download.

Gitea just happens to be what I use to manage my repositories. Gogs or Forgejo integration would look nearly identical. And GitLab or GitHub users should delete that crap.

While it might make some sense to use containers for all CI targets that are Linux distributions rather than full-blown virtual machines for everything, it would also add a lot of headaches concerning management.

The host machine is assumed to be Arch Linux, with a running instance of Gitea. You should have a few tens of gigabytes of spare disk space, and several gigabytes of free memory. The more RAM, the better, since it allows for running our VMs without writing to disk.

Luckily, there seems to be a thing called ‘cloud-init’ that unifies virtual machine setup across various operating systems, including BSD derivatives. You simply download a ‘cloud-enabled’ image which has that package pre-installed, and pass the VM a settings file to boot with. Let’s install a utility to enable passing the file as a virtual drive:

Next, while it might seem sufficient to use passwords to log in to the machines, it also costs very little to create an SSH keypair. Additionally, it will be convenient to place proxy settings for the guests in a special file. Let’s put all these files in a subdirectory:

Some cloud images can be found linked in OpenStack documentation. For demonstration purposes, we will use Debian, which behaves very well, and doesn’t need special casing:

To configure the Debian image, add the following snippet to your acid.yaml:

acid trivially launches the given script or binary, and expects it to eventually make the target machine available over SSH at the specified address, keeping it that way for as long as it doesn’t exit.

Once connected and made sure of cloud-init's completion, we can make excellent use of shell options to get us the behaviour we want: -e to exit on the first error, and -x to print commands as they’re being executed. The following system upgrade is something that could be done within cloud-init, but it would also make progress invisible, which isn’t particularly desirable for actions that can take long to finish. Finally, we set up the stage for project scripts. Private repositories might need adjustments for authentication there.

Due to the repetitive nature of launching QEMU, let’s split out the common part first:

I’ll trust you, dear reader, to figure most of it out yourself. The gist of it is that it creates a throw-away overlay disk image in your /tmp, which on most Linux distributions is backed by system memory, generates a cloud-init configuration drive, and launches an instance of QEMU that forwards the host machine’s port 8022 to the guest’s SSH port.

Cloud images are distributed tiny, and blow up on boot to fill up the whole device, so we give them a bit more space to breathe. In practice, none of my builds require more than 6 gibibytes, and that includes a GTK+ project with two Win32 cross-builds. Unfortunately, it is not possible to keep the overlay image entirely in QEMU’s process memory—​it must be in some way backed by the filesystem. Meaning, you can choose between wearing down your SSD, if you have one, or risking that you cause a dangerous kind of memory pressure when you put it in a tmpfs.

2 gibibytes of RAM were never a problem either. Here we can make the process sacrifice itself to the OOM killer, at least.

That was the complicated part. The Debian-specific script is then just this:

We use an extra file descriptor to pass the cloud-init script fragment, so that acid gets access to QEMU’s standard input stream.

Other Linux distributions and BSDs may require considerably more effort to get the MITM proxying to work, which will be your homework. Configuring certificates and global environment variables is apparently a hard problem.

Congratulations, we’re almost done! You can now put stuff like the following contrived example in your acid.yaml file, and have it run automatically once you push to the repository:

Just remember to make the daemon reload its configuration when you happen to change it:

The easiest way to inspect failures is to put a very long sleep at the end of build scripts, then connect to the machine (assuming they’re all configured similarly, otherwise some yq magic would be in order here, as illustrated later on):

And you can certainly launch the runners independently, with the help of another simple script:

Truth be told, I was procrastinating over this endeavour for many years, almost scrapping the idea in favour of simple Nix-based build checking instead. But I’m happy about what I’ve ended up with. While a few areas remain that deserve more love, I’ve achieved my main goals. And I still think setting up something like buildbot the way I want would take me a similar amount of time.

I have deployed acid on most of my projects, so feel free to go have a look.

I’ve skipped over how acid is actually put together. There is surprisingly little to it, because my language of choice—​Go—​comes with batteries included, and a ton of things can be achieved through shelling out.