Reflections on Hosting the Hoster

"This will be the last time I let myself build on Kubernetes," I negotiated with myself. It was 2025, I had written a new OCI registry from scratch (artisanally, as we did in those days), I had the domain, and I was eager to get my software out. Kubernetes had been my go-to solution for delivering software for ten years. Kubernetes was familiar and safe.

But the question that presented itself insisted on an answer: why was I building a registry for bootable operating system images if I didn't believe the best way to deliver my registry was as a bootable operating system image?

And so I resolved that Kheeper must be delivered as a bootc OCI image. With that constraint in place, a solution emerged that was far from elegant. I provisioned a CentOS VM on GCP, built and scp'd my OCI archive, and ran bootc install to-existing-root. For the next few months each release was one podman build, one scp, and one bootc upgrade. Manual and slow, but I was indeed dogfooding.

Hosting Like It's 2014

By the time winter athletes were departing Milan Olympic village I was reconsidering a second haven I had become a bit too comfortable with over the years: the cloud. Once you're shipping your server images as OCI artifacts the convenience advantage of the cloud starts to disappear. Looking around, I found an older 16-core server with 32GB memory on Hivelocity for $51 a month.

But at this point I hadn't yet built the ipxe integration for Kheeper. Still the price of bare metal was enticing. It felt like a regression, but I installed AlmaLinux and configured the server over SSH. While I wasn't dogfooding bootc images in this period, I was learning to swim in the systemd pond. Goodbye GCP.

During my years on Kubernetes I had forgotten to care about how my software interfaces with hardware. In fact, I was so uncomfortable with the host system that I initially opted to run Postgres in a container with a volume mount. Perhaps I feared the dnf install postgresql-server was going to pollute my pristine host?

Thesis, Antithesis, Synthesis

Fast-forward to spring. I'd finished the integrations to make running bootc images on GCP and AWS easy. This time when I launched an instance it wasn't generic CentOS; it was fedora-bootc with features to autoregister with Kheeper and automatically boot releases when pushed to the registry. And so the Kheeper registry was hosted on the Kheeper registry.

The Epiphany

Only it was hosted on a registry running on AlmaLinux configured manually over SSH. It was not yet Kheeper hosting Kheeper. The bottom turtle was not a bootable operating system image. Could that even work?

Yes. It can and it does. The realization came that bootc doesn't need access to a registry during upgrade. It completes the pull before it ever begins a reboot. Once it has the image it doesn't matter if the registry that hosts the image is available. It doesn't matter if the registry that hosts the image is the same system that is being rebooted.

Present & Future

As I write this, Kheeper EU is hosted on Kheeper US, and Kheeper US is strictly self-hosted on Kheeper US. Both regions are running on OVH Cloud dedicated servers at significant cost savings over the cloud. I can coordinate two minutes of downtime for rebooting into upgrade images when the servers have no traffic.

The logical next step is HA with a load balancer and two servers for each region. Drain a server, reboot it, then proceed with the next. And everything mutually hosted with US hosting EU and EU hosting US.

But someday - if the scale warrants - I may just decide to go back to GKE, my familiar old friend.