Kheeper

Backups

Every Kheeper host comes with an S3-shaped object store and a localhost proxy that lets any S3 client (wal-g, restic, rclone, the AWS SDK, etc.) write to it without managing credentials. This guide explains how the pieces fit together and walks through a complete wal-g setup for PostgreSQL.

Concepts

Object store

Each host has a dedicated namespace inside an org-scoped bucket on us.kheeper.com. Paths are S3 path-style:

us.kheeper.com/<org>/<host>/<your data>

A host can only read and write under its own <org>/<host>/... prefix. The org owner can read across all hosts in the org for restore and migration purposes.

Authentication is the same ed25519 host-JWT every other kheeper command uses — there is no sigv4, no per-host static credentials, and nothing to rotate on your side.

Storage and bandwidth are billed at the same rates as the registry — see pricing.

Object store proxy

The Kheeper object store is not S3 compatible. We provide a proxy that you can point your S3 clients at.

kheeper object-proxy

By default this will listen on 127.0.0.1:5000. The credentials and signature supplied by the S3 client will be dropped and the proxy will use the host's authentication key.

Quick start: env-file for an S3 client

kheeper hosts me --env prints an env-file with everything an S3 client needs to talk to the local proxy:

$ kheeper hosts me --env
AWS_ACCESS_KEY_ID=dummy
AWS_SECRET_ACCESS_KEY=dummy
AWS_REGION=us-east-1
AWS_DEFAULT_REGION=us-east-1
AWS_ENDPOINT=http://127.0.0.1:5000
WALG_S3_PREFIX=s3://myorg/web-01
AWS_S3_FORCE_PATH_STYLE=true
WALG_S3_FORCE_PATH_STYLE=true

wal-g + PostgreSQL

The canonical setup is published at github.com/kheepercom/public/tree/main/postgres. The snippets below are excerpts — read the repo for the full, working image.

1. Run the proxy as a system service

kheeper-object-proxy.service keeps the proxy running on loopback. It runs as root so it can read the host's ed25519 private key from /etc/kheeper/default/:

[Unit]
Description=Localhost S3 proxy: wal-g -> kheeper objects backend
After=network-online.target
Wants=network-online.target
ConditionPathExists=/etc/kheeper/default/identity.json

[Service]
Environment=KHEEPER_CONFIG=/etc/kheeper
ExecStart=/usr/local/bin/kheeper object-proxy --listen 127.0.0.1:5000
Restart=on-failure
RestartSec=3
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadOnlyPaths=/etc/kheeper

[Install]
WantedBy=multi-user.target

2. Render the env-file at boot

kheeper-walg-env.service runs once at boot and writes /etc/kheeper/walg.env. Any unit that wants wal-g env vars takes it as an EnvironmentFile=:

[Unit]
Description=Render /etc/kheeper/walg.env from this host's identity
After=network-online.target
Wants=network-online.target
Before=postgresql.service walg-base-backup.service walg-retention.service
ConditionPathExists=/etc/kheeper/default/identity.json

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=KHEEPER_CONFIG=/etc/kheeper
ExecStart=/bin/sh -c '/usr/local/bin/kheeper hosts me --env > /etc/kheeper/walg.env.tmp && install -m 0644 /etc/kheeper/walg.env.tmp /etc/kheeper/walg.env && rm /etc/kheeper/walg.env.tmp'

[Install]
WantedBy=multi-user.target

3. Wire wal-g into PostgreSQL

Append the archive settings to postgresql.conf:

archive_mode = on
archive_command = 'wal-g wal-push %p'
archive_timeout = 60
restore_command = 'wal-g wal-fetch %f %p'

And give postgresql.service the same env-file via a drop-in:

[Unit]
Wants=kheeper-object-proxy.service kheeper-walg-env.service
After=kheeper-object-proxy.service kheeper-walg-env.service

[Service]
EnvironmentFile=-/etc/kheeper/walg.env

With this in place, every WAL segment Postgres archives is pushed via the proxy to your host's prefix in the object store.

4. Schedule base backups

walg-base-backup.timer fires daily and runs:

[Service]
Type=oneshot
User=postgres
Group=postgres
EnvironmentFile=/etc/kheeper/walg.env
Environment=PGHOST=/var/run/postgresql
ExecStart=/usr/local/bin/wal-g backup-push /var/lib/pgsql/data

5. Prune old backups

walg-retention.timer keeps the most recent 7 full backups and discards the rest:

[Service]
Type=oneshot
User=postgres
Group=postgres
EnvironmentFile=/etc/kheeper/walg.env
ExecStart=/usr/local/bin/wal-g delete retain FULL 7 --confirm

Restoring

Restoring to the same host is just wal-g backup-fetch against the same WALG_S3_PREFIX — credentials and prefix are already in walg.env.

To restore onto a different host (say, web-01 died and you're rebuilding it as web-02), the new host can't read web-01's prefix by default. Run the restore from a workstation where you have org-owner credentials:

WALG_S3_PREFIX=s3://myorg/web-01 \
AWS_ENDPOINT=http://127.0.0.1:5000 \
AWS_ACCESS_KEY_ID=dummy AWS_SECRET_ACCESS_KEY=dummy \
AWS_S3_FORCE_PATH_STYLE=true WALG_S3_FORCE_PATH_STYLE=true \
wal-g backup-fetch /var/lib/pgsql/data LATEST

Then rsync the data directory across to the new host before starting Postgres.

Using other S3 clients

Any S3-compatible tool works the same way. Examples assuming walg.env is sourced:

# rclone
rclone --s3-endpoint=$AWS_ENDPOINT --s3-force-path-style ls myorg-host:myorg/web-01/

# AWS CLI
aws --endpoint-url=$AWS_ENDPOINT s3 ls s3://myorg/web-01/

# restic
RESTIC_REPOSITORY=s3:$AWS_ENDPOINT/myorg/web-01/restic restic init

The proxy sees standard S3 requests, so the only configuration most tools need is the endpoint URL, path-style addressing, and any dummy credentials they insist on.

Next: kheeper auth