Docs Blog Apps Pricing

Getting Started

1. Install the CLI 2. Login 3. Build and Push 4. Boot on GCP 5. Boot on Bare Metal

Guides

Authentication Pushing and Pulling Images Configuring Images Working with Hosts Working with GCP Working with Bare Metal latitude.sh OVHcloud Working with AWS Backups Tips and Tricks

CLI Reference

kheeper auth kheeper authorizations kheeper bots kheeper clouds kheeper contexts kheeper hosts kheeper images kheeper orgs kheeper push kheeper releases kheeper status

Backups

Every Kheeper host comes with an S3-shaped object store and a localhost proxy that lets any S3 client (wal-g, restic, rclone, the AWS SDK, etc.) write to it without managing credentials. This guide explains how the pieces fit together and walks through a complete wal-g setup for PostgreSQL.

Concepts

Object store

Each host has a dedicated namespace inside an org-scoped bucket on us.kheeper.com. Paths are S3 path-style:

us.kheeper.com/<org>/<host>/<your data>

A host can only read and write under its own <org>/<host>/... prefix. The org owner can read across all hosts in the org for restore and migration purposes.

Authentication is the same ed25519 host-JWT every other kheeper command uses — there is no sigv4, no per-host static credentials, and nothing to rotate on your side.

Storage and bandwidth are billed at the same rates as the registry — see pricing.

Object store proxy

The Kheeper object store is not S3 compatible. We provide a proxy that you can point your S3 clients at.

kheeper object-proxy

By default this will listen on 127.0.0.1:5000. The credentials and signature supplied by the S3 client will be dropped and the proxy will use the host's authentication key.

Quick start: env-file for an S3 client

kheeper hosts me --env prints an env-file with everything an S3 client needs to talk to the local proxy:

$ kheeper hosts me --env
AWS_ACCESS_KEY_ID=dummy
AWS_SECRET_ACCESS_KEY=dummy
AWS_REGION=us-east-1
AWS_DEFAULT_REGION=us-east-1
AWS_ENDPOINT=http://127.0.0.1:5000
WALG_S3_PREFIX=s3://myorg/web-01
AWS_S3_FORCE_PATH_STYLE=true
WALG_S3_FORCE_PATH_STYLE=true

wal-g + PostgreSQL

The canonical setup is published at github.com/kheepercom/public/tree/main/postgres. The snippets below are excerpts — read the repo for the full, working image.

1. Run the proxy as a system service

kheeper-object-proxy.service keeps the proxy running on loopback. It runs as root so it can read the host's ed25519 private key from /etc/kheeper/default/:

[Unit]
Description=Localhost S3 proxy: wal-g -> kheeper objects backend
After=network-online.target
Wants=network-online.target
ConditionPathExists=/etc/kheeper/default/identity.json

[Service]
Environment=KHEEPER_CONFIG=/etc/kheeper
ExecStart=/usr/local/bin/kheeper object-proxy --listen 127.0.0.1:5000
Restart=on-failure
RestartSec=3
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadOnlyPaths=/etc/kheeper

[Install]
WantedBy=multi-user.target

2. Render the env-file at boot

kheeper-walg-env.service runs once at boot and writes /etc/kheeper/walg.env. Any unit that wants wal-g env vars takes it as an EnvironmentFile=:

[Unit]
Description=Render /etc/kheeper/walg.env from this host's identity
After=network-online.target
Wants=network-online.target
Before=postgresql.service walg-base-backup.service walg-retention.service
ConditionPathExists=/etc/kheeper/default/identity.json

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=KHEEPER_CONFIG=/etc/kheeper
ExecStart=/bin/sh -c '/usr/local/bin/kheeper hosts me --env > /etc/kheeper/walg.env.tmp && install -m 0644 /etc/kheeper/walg.env.tmp /etc/kheeper/walg.env && rm /etc/kheeper/walg.env.tmp'

[Install]
WantedBy=multi-user.target

3. Wire wal-g into PostgreSQL

Append the archive settings to postgresql.conf:

archive_mode = on
archive_command = 'wal-g wal-push %p'
archive_timeout = 60
restore_command = 'wal-g wal-fetch %f %p'

And give postgresql.service the same env-file via a drop-in:

[Unit]
Wants=kheeper-object-proxy.service kheeper-walg-env.service
After=kheeper-object-proxy.service kheeper-walg-env.service

[Service]
EnvironmentFile=-/etc/kheeper/walg.env

With this in place, every WAL segment Postgres archives is pushed via the proxy to your host's prefix in the object store.

4. Schedule base backups

walg-base-backup.timer fires daily and runs:

[Service]
Type=oneshot
User=postgres
Group=postgres
EnvironmentFile=/etc/kheeper/walg.env
Environment=PGHOST=/var/run/postgresql
ExecStart=/usr/local/bin/wal-g backup-push /var/lib/pgsql/data

5. Prune old backups

walg-retention.timer keeps the most recent 7 full backups and discards the rest:

[Service]
Type=oneshot
User=postgres
Group=postgres
EnvironmentFile=/etc/kheeper/walg.env
ExecStart=/usr/local/bin/wal-g delete retain FULL 7 --confirm

Restoring

Restoring to the same host is just wal-g backup-fetch against the same WALG_S3_PREFIX — credentials and prefix are already in walg.env.

To restore onto a different host (say, web-01 died and you're rebuilding it as web-02), the new host can't read web-01's prefix by default. Run the restore from a workstation where you have org-owner credentials:

WALG_S3_PREFIX=s3://myorg/web-01 \
AWS_ENDPOINT=http://127.0.0.1:5000 \
AWS_ACCESS_KEY_ID=dummy AWS_SECRET_ACCESS_KEY=dummy \
AWS_S3_FORCE_PATH_STYLE=true WALG_S3_FORCE_PATH_STYLE=true \
wal-g backup-fetch /var/lib/pgsql/data LATEST

Then rsync the data directory across to the new host before starting Postgres.

Using other S3 clients

Any S3-compatible tool works the same way. Examples assuming walg.env is sourced:

# rclone
rclone --s3-endpoint=$AWS_ENDPOINT --s3-force-path-style ls myorg-host:myorg/web-01/

# AWS CLI
aws --endpoint-url=$AWS_ENDPOINT s3 ls s3://myorg/web-01/

# restic
RESTIC_REPOSITORY=s3:$AWS_ENDPOINT/myorg/web-01/restic restic init

The proxy sees standard S3 requests, so the only configuration most tools need is the endpoint URL, path-style addressing, and any dummy credentials they insist on.