Kheeper

Tips and Tricks

We run bootc in production across all of our infrastructure. These tips come from problems we've hit and solved along the way.

Ship containers with your image using bound images

If your host runs containers via Quadlet, the default behavior is for Podman to pull them on first boot. That means your services wait on a network pull before they can start — and if the pull fails, they don't start at all.

Logically bound images solve this. When bootc pulls a host update, it also pulls any bound images into read-only storage, so they're available immediately at boot with no separate podman pull.

Three pieces wire up a bound image:

1. The .image file — declares the OCI reference. Place it in /usr/share/containers/systemd/:

# /usr/share/containers/systemd/my-app.image
[Image]
Image=docker.io/org/my-app:v1.2.3

2. A symlink in bound-images.d — tells bootc to pull this image when deploying the OS. Without the symlink, bootc ignores the .image file:

RUN install -d -m 0755 /usr/lib/bootc/bound-images.d \
    && ln -s ../../../share/containers/systemd/my-app.image \
           /usr/lib/bootc/bound-images.d/my-app.image

3. The .container quadlet — the systemd unit that runs the container. Add GlobalArgs=--storage-opt additionalimagestore=/usr/lib/bootc/storage so Podman can find the image in bootc's read-only store:

# /usr/share/containers/systemd/my-app.container
[Container]
Image=docker.io/org/my-app:v1.2.3
GlobalArgs=--storage-opt additionalimagestore=/usr/lib/bootc/storage

The Image= in both files must match exactly.

Containerfile pattern

COPY my-app.image /usr/share/containers/systemd/my-app.image

RUN install -d -m 0755 /usr/lib/bootc/bound-images.d \
    && ln -s ../../../share/containers/systemd/my-app.image \
           /usr/lib/bootc/bound-images.d/my-app.image

COPY my-app.container /usr/share/containers/systemd/my-app.container

Tips

  • Pin your image tags in the .image file — never use :latest. When you update the version, update both the .image and .container files.
  • Use bound images for anything integral to the host — services that must be present at first boot with no network dependency. Use normal Quadlet auto-pull for containers that can tolerate a pull delay or that change independently of the host image.

Check that NetworkManager isn't timing out on unused NICs

This is especially common on bare-metal hosts with multiple NICs.

After a reboot, services that depend on network-online.target can take 45–60+ seconds to start. If your services are slow to come up after a reboot, NetworkManager might be the culprit.

What happens

NetworkManager-wait-online.service blocks network-online.target until every managed interface has finished activation. On bare-metal hosts with multiple NICs, a secondary NIC may have link but no DHCP server. NetworkManager auto-activates a profile on it, the DHCP transaction times out after 45 seconds, and everything downstream waits.

How to check

# Look for interfaces stuck in "connecting" or "disconnected"
nmcli device status

# See how long wait-online took
systemctl status NetworkManager-wait-online.service --no-pager

# Check for DHCP timeouts
journalctl -u NetworkManager.service | grep -iE 'dhcp|fail|timeout'

Signs of the problem:

  • An interface in connecting state that isn't your primary uplink.
  • dhcp4 (<iface>): activation: beginning transaction (timeout in 45 seconds) on a NIC with no DHCP server.

How to fix

Tell NetworkManager to ignore the unused NIC:

# Replace IFACE with the interface name (e.g. eno2, enp2s0f1np1)
printf '[keyfile]\nunmanaged-devices=interface-name:IFACE\n' \
  | sudo tee /etc/NetworkManager/conf.d/99-unmanaged-IFACE.conf
sudo systemctl restart NetworkManager

Verify it worked:

nmcli device status          # IFACE should show "unmanaged"
sudo systemctl restart NetworkManager-wait-online  # should finish in <5s

This is a per-host fix because NIC names differ across hardware. Don't bake it into the image unless all your hosts share the same NIC layout.

Next: kheeper auth