zrepl 0.7.0 on AWS User Guide
Overview
This guide covers launching and using the zrepl 0.7.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg backup and recovery listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.
zrepl is an open source replication daemon for ZFS. It takes snapshots on a schedule, prunes them on a retention grid, and replicates them incrementally from one machine to another with zfs send and zfs receive, resuming an interrupted transfer where it stopped.
The cloudimg image is a ready ZFS backup target for your other ZFS machines:
- A ZFS pool on its own volume. First boot creates the pool
tank(lz4 compression) on a separate 20 GiB gp3 EBS volume that you can grow on its own, withtank/datafor your own files andtank/sinkfor the datasets your other machines replicate here. zrepl snapshotstank/dataevery 15 minutes and keeps them on a grid (every snapshot for an hour, hourly for a day, daily for two weeks). - A sink that only accepts your machines. zrepl listens on TCP 8888 with mutual TLS. First boot generates a certificate authority for this instance, the sink's own certificate and a first client certificate (
client1); a machine can only connect with a certificate from this CA whose name is on the sink's list. There is no password: the client keys are the credentials, andzrepl-add-clientandzrepl-revoke-clientmanage them. - Received data cannot take over the instance. Whatever properties a client's stream carries, a dataset received into
tank/sinkis never mounted and hasexec,setuidanddevicesoff, so a client cannot mount a dataset over/etcor plant a setuid program. The client's own values are kept, and come back when you restore. - Locked down. The Prometheus metrics endpoint listens on
127.0.0.1only, zrepl and the first boot service cannot reach the instance metadata service, and the AMI requires IMDSv2. Nothing listens on the network except SSH and the sink. - Built from source. zrepl 0.7.0 is compiled from the upstream release (pinned by SHA-256) with the pinned Go 1.27.3 toolchain, with its dependencies raised to versions that fix the published advisories, on fully patched Ubuntu 24.04 LTS with unattended security upgrades switched on. OpenZFS comes from Ubuntu's own packages.
What is on the image: zrepl 0.7.0 (/opt/zrepl, licence and build provenance in /usr/share/doc/zrepl), OpenZFS (zfsutils-linux and the kernel's ZFS module), Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.
Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large
Prerequisites
- An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
- A security group that allows TCP 22 (SSH) from your own networks and TCP 8888 (the sink) from the machines that will replicate to it only. Keep both closed to the internet.
- For replication, one or more source machines with OpenZFS and zrepl installed. zrepl is a single binary for Linux and FreeBSD; the zrepl installation guide covers the packages.
Step 1: Launch the instance
In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick zrepl 0.7.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair, and a security group as above. The AMI attaches the 20 GiB data volume for the pool automatically; increase its size in the launch wizard's storage section to fit what you will replicate. The pool is created on that volume on first boot.
With the AWS CLI (replace the values in angle brackets):
aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
--instance-type m5.large --key-name <your-key-pair> \
--security-group-ids <your-security-group> --subnet-id <your-subnet> \
--metadata-options HttpTokens=required,HttpEndpoint=enabled \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=zrepl-sink}]'
Step 2: Connect to your instance
| Version | SSH user |
|---|---|
| zrepl 0.7.0 on Ubuntu 24.04 | ubuntu |
ssh -i <your-key.pem> ubuntu@<public-ip>
Step 3: Check the service and the pool
First boot sets the ZFS host id, creates the pool, generates the certificates and then starts zrepl:
systemctl is-active zrepl
sudo test -f /var/lib/cloudimg/zrepl-firstboot.done && echo FIRSTBOOT_DONE
active
FIRSTBOOT_DONE
Only SSH and the sink listen on the network; the metrics endpoint (9811) is on loopback:
sudo ss -lntp | awk 'NR>1 {print $4, $6}' | sed -E 's/pid=[0-9]+,fd=[0-9]+/.../g' | sort
*:8888 users:(("zrepl",...))
0.0.0.0:22 users:(("sshd",...),("systemd",...))
127.0.0.1:9811 users:(("zrepl",...))
127.0.0.53%lo:53 users:(("systemd-resolve",...))
127.0.0.54:53 users:(("systemd-resolve",...))
[::]:22 users:(("sshd",...),("systemd",...))
The pool and its datasets. tank/sink has no mount point by design (Step 9):
sudo zpool list -o name,size,health tank
sudo zfs list -o name,used,avail,mountpoint -r tank
NAME SIZE HEALTH
tank 19.5G ONLINE
NAME USED AVAIL MOUNTPOINT
tank 792K 18.9G /tank
tank/data 96K 18.9G /tank/data
tank/sink 192K 18.9G none
tank/sink/client1 96K 18.9G none
zrepl runs two jobs, sink and snap_data:
sudo zrepl status --mode raw | jq -r '.Jobs | keys[]' | grep -v '^_'
sink
snap_data
sudo zrepl status opens zrepl's interactive status view (press q to leave it).
Step 4: Read the connection details
First boot writes the sink's address, the certificate fingerprints and where the first client bundle is to /root/zrepl-credentials.txt (root only). The file holds no secret itself; the private keys it points to do:
sudo cat /root/zrepl-credentials.txt
# zrepl 0.7.0 - generated on first boot by zrepl-firstboot.service
# Every key and certificate below is unique to this instance; none was baked into the image. The private keys are the
# credentials of this sink: anyone holding a client key listed in client_cns can replicate to it. Keep them secret.
#
ZREPL_SINK_ADDRESS=172.31.84.59:8888
ZREPL_SINK_PUBLIC_ADDRESS=54.211.111.161:8888
ZREPL_SERVER_CN=zrepl-sink
ZREPL_POOL=tank
ZREPL_DATA_DISK=/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0c57ec3d5c2e8ec9b
ZREPL_CA_CERT=/etc/zrepl/tls/ca.crt
ZREPL_CA_CERT_HASH=<ZREPL_CA_CERT_HASH>
ZREPL_SERVER_CERT_HASH=<ZREPL_SERVER_CERT_HASH>
ZREPL_CLIENT1_BUNDLE=/etc/zrepl/clients/client1
ZREPL_CLIENT1_CERT_HASH=<ZREPL_CLIENT1_CERT_HASH>
#
# Status: sudo zrepl status Self test: sudo zrepl-selftest Config: /etc/zrepl/zrepl.yml
# A new client machine: sudo zrepl-add-client <name> (bundle and an example push job in /etc/zrepl/clients/<name>/)
# Remove one: sudo zrepl-revoke-client <name>
# User guide: https://www.cloudimg.co.uk/guides/zrepl-0-7-0-on-ubuntu-24-04-aws/
Your instance shows its own addresses and fingerprints. What each value is for:
| Value | Used for |
|---|---|
ZREPL_SINK_ADDRESS |
The address your source machines connect to from inside the VPC (private IP and port 8888) |
ZREPL_SINK_PUBLIC_ADDRESS |
The public address, for sources outside the VPC (open 8888 to those sources only) |
ZREPL_SERVER_CN |
The name clients check the sink's certificate against (server_cn in their push job) |
ZREPL_CA_CERT_HASH, ZREPL_SERVER_CERT_HASH |
SHA-256 fingerprints of this instance's CA and sink certificates |
ZREPL_CLIENT1_BUNDLE |
The first client bundle: client1.crt, client1.key, ca.crt and an example push job |
Step 5: Run the self test
zrepl-selftest checks the whole path without touching your data. It creates a small throwaway pool in a private temporary directory, starts a second zrepl daemon that replicates one snapshot over TLS to this sink as client1, checks that the snapshot arrived intact, and then removes everything it made:
sudo zrepl-selftest 2>&1 | tail -2
replicated cloudimgselftest/src@cloudimg_selftest_20261108053359 over TLS as client1 to tank/sink/client1/cloudimgselftest/src (guid 5801883061040581056 matches)
ZREPL_SELFTEST_OK
Step 6: Local snapshots of tank/data
Anything you keep under /tank/data is snapshotted every 15 minutes:
sudo zfs list -H -t snapshot -o name -r tank/data | head -3
tank/data@zrepl_20261108_053140_000
Every snapshot is browsable, read only, under .zfs/snapshot. This writes a file, takes a snapshot by hand, deletes the file and copies it back from the snapshot (zrepl leaves snapshots you take yourself alone):
echo "notes from cloudimg" | sudo tee /tank/data/notes.txt >/dev/null
sudo zfs snapshot tank/data@before-change
sudo rm /tank/data/notes.txt
sudo cat /tank/data/.zfs/snapshot/before-change/notes.txt
sudo cp /tank/data/.zfs/snapshot/before-change/notes.txt /tank/data/
ls /tank/data
sudo zfs destroy tank/data@before-change
notes from cloudimg
notes.txt
To change the schedule or the retention grid, edit the snap_data job in /etc/zrepl/zrepl.yml and restart zrepl (sudo zrepl configcheck && sudo systemctl restart zrepl).
Step 7: Replicate a source machine to the sink
Each source machine gets its own client bundle. The first one, client1, already exists. Copy it to the source machine, from your workstation through the instance (the key is root only on the instance, so it is staged in a private directory first and removed after the copy):
ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo tar -C /etc/zrepl/clients/client1 -cf - client1.crt client1.key ca.crt push-job-example.yml' > client1-bundle.tar
scp client1-bundle.tar root@<source-host>:/root/
rm client1-bundle.tar
On the source machine, as root, put the files in /etc/zrepl/ (zrepl installed, as in Prerequisites):
# on the source machine, as root (not on the sink at <public-ip>)
mkdir -p /etc/zrepl && tar -C /etc/zrepl -xf /root/client1-bundle.tar && rm /root/client1-bundle.tar
chmod 600 /etc/zrepl/client1.key
push-job-example.yml is a ready push job for that machine. Add it under jobs: in the source machine's /etc/zrepl/zrepl.yml, set address to the sink's ZREPL_SINK_ADDRESS (a DNS name you point at the instance works too) and the filesystems filter to the datasets to protect:
jobs:
- name: push_to_cloudimg_sink
type: push
connect:
type: tls
address: "<sink-address>:8888"
ca: "/etc/zrepl/ca.crt"
cert: "/etc/zrepl/client1.crt"
key: "/etc/zrepl/client1.key"
server_cn: "zrepl-sink"
filesystems:
"<your-pool>/<dataset><": true
send:
encrypted: false
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
pruning:
keep_sender:
- type: not_replicated
- type: last_n
count: 10
keep_receiver:
- type: grid
grid: 1x1h(keep=all) | 24x1h | 30x1d
regex: "^zrepl_"
On the source machine, check the configuration, restart zrepl and start the first replication at once instead of waiting for the next interval:
# on the source machine, as root (not on the sink at <public-ip>)
zrepl configcheck && systemctl restart zrepl
zrepl signal wakeup push_to_cloudimg_sink
The datasets arrive on the sink under tank/sink/<client name>/<the source's dataset path>. On the sink:
sudo zfs list -r -o name,used,mountpoint tank/sink/client1
If a source dataset is encrypted with ZFS native encryption, set send: encrypted: true in its push job: the sink then stores the ciphertext and never sees the key.
Step 8: Add and remove client machines
zrepl-add-client <name> mints a new client certificate for another source machine, puts its bundle and an example push job in /etc/zrepl/clients/<name>/, adds the name to the sink's list and restarts zrepl. A name is never issued twice:
sudo zrepl-add-client office-nas
sudo ls /etc/zrepl/clients/office-nas
client office-nas: bundle in /etc/zrepl/clients/office-nas/ (certificate sha256 212c856ca2953889cec38f5438f192178705274da98469a7eaf7226ac394f433)
ca.crt
office-nas.crt
office-nas.key
push-job-example.yml
zrepl-revoke-client <name> takes the name off the list (its certificate is refused from the next connection), shreds the bundle's private key and restarts zrepl. The datasets the client already replicated stay under tank/sink/<name> until you destroy them. The last client cannot be revoked; stop the sink instead (sudo systemctl disable --now zrepl):
sudo zrepl-revoke-client office-nas
sudo ls /etc/zrepl/clients/
client office-nas: removed from client_cns; its certificate is refused
client1
issued.list
Step 9: Restore from the sink
Received datasets are never mounted on the sink, and the properties the client sent are kept as "received" values beside the sink's own overrides. These are the sink's settings:
sudo zfs get -o name,property,value,source canmount,mountpoint,exec,setuid,devices tank/sink
sudo sed -n '/^ recv:/,/^ placeholder:/p' /etc/zrepl/zrepl.yml | grep -v placeholder
NAME PROPERTY VALUE SOURCE
tank/sink canmount off local
tank/sink mountpoint none local
tank/sink exec off local
tank/sink setuid off local
tank/sink devices off local
recv:
properties:
override:
"canmount": "off"
"mountpoint": "none"
"exec": "off"
"setuid": "off"
"devices": "off"
"org.openzfs.systemd:ignore": "on"
Read files on the sink. Clone a snapshot to a mount point you choose, read only. Replace the dataset and snapshot with yours (sudo zfs list -t snapshot -r tank/sink lists them):
# on the sink (<public-ip>), with your own dataset and snapshot
sudo zfs clone -o mountpoint=/mnt/restore -o readonly=on -o setuid=off -o exec=off tank/sink/client1/<your-pool>/<dataset>@<snapshot> tank/restore
ls /mnt/restore
sudo zfs destroy tank/restore
Send a dataset back. On the source machine (or a new one), receive a snapshot from the sink over SSH. zfs send -b sends the client's original properties, so the dataset comes back as it was:
ssh -i <your-key.pem> ubuntu@<public-ip> sudo zfs send -b tank/sink/client1/<your-pool>/<dataset>@<snapshot> | zfs receive <your-pool>/<dataset>-restored
Step 10: Check the security posture
The sink proves its identity with its certificate and refuses a client without one. A client certificate signed by another CA, or signed by this CA but with a name that is not on the list, is refused too:
sudo sh -c 'echo | timeout 10 openssl s_client -connect 127.0.0.1:8888 -CAfile /etc/zrepl/tls/ca.crt -verify_hostname zrepl-sink -verify_return_error 2>&1; true' | grep -oE '^Verification: OK|alert certificate required'
Verification: OK
alert certificate required
With the client1 certificate the sink answers with zrepl's protocol handshake:
sudo sh -c 'B=/etc/zrepl/clients/client1; sleep 2 | timeout 10 openssl s_client -connect 127.0.0.1:8888 -CAfile $B/ca.crt -cert $B/client1.crt -key $B/client1.key -verify_hostname zrepl-sink -verify_return_error -quiet 2>/dev/null' | head -c 50; echo
0000000056 ZREPL_ZFS_REPLICATION PROTOVERSION=0007
Client certificates are for client authentication only and are issued by this instance's own CA:
sudo openssl x509 -noout -subject -issuer -ext extendedKeyUsage -in /etc/zrepl/clients/client1/client1.crt
subject=CN = client1
issuer=CN = zrepl CA 608f5028
X509v3 Extended Key Usage:
TLS Web Client Authentication
The metrics endpoint answers on loopback only, and zrepl and its first boot service cannot reach the instance metadata service:
curl -s -o /dev/null -w 'metrics on 127.0.0.1:9811 -> HTTP %{http_code}\n' http://127.0.0.1:9811/metrics
P=$(hostname -I | awk '{print $1}'); curl -s -m 5 -o /dev/null -w 'metrics on the private IP -> %{http_code}\n' http://$P:9811/metrics || true
systemctl show zrepl zrepl-firstboot -p Id -p IPAddressDeny
metrics on 127.0.0.1:9811 -> HTTP 200
metrics on the private IP -> 000
IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
Id=zrepl.service
IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
Id=zrepl-firstboot.service
zrepl runs as root because zfs receive needs it; the sink's received-property overrides (Step 9) are what keep a client's data from acting on the instance.
Step 11: Grow the pool
Increase the data volume's size in the EC2 console (Volumes, Modify volume) or with aws ec2 modify-volume, wait until its state is optimizing or completed, then let the pool use the new space:
D=$(sudo sed -n 's/^ZREPL_DATA_DISK=//p' /root/zrepl-credentials.txt)
sudo zpool online -e tank "$D"
sudo zpool list -o name,size tank
The pool also has autoexpand=on, so a reboot picks up a larger volume on its own. To keep a second copy of the sink, take EBS snapshots of the data volume, or add a push job on the sink that replicates tank/sink to another zrepl sink.
Step 12: Server components
| Component | Path |
|---|---|
| zrepl | /opt/zrepl/bin/zrepl (read only, owner root; zrepl on the PATH) |
| Licence and build provenance | /usr/share/doc/zrepl/ |
| Configuration | /etc/zrepl/zrepl.yml (mode 0600) |
| CA and sink certificate and key | /etc/zrepl/tls/ (mode 0700) |
| Client bundles | /etc/zrepl/clients/<name>/, issued names in /etc/zrepl/clients/issued.list |
| Pool | tank on the data volume, tank/data at /tank/data, tank/sink (not mounted) |
| Service units | zrepl.service, zrepl-firstboot.service in /etc/systemd/system/ |
| Helpers | /usr/local/sbin/zrepl-add-client, zrepl-revoke-client, zrepl-selftest |
| Connection details | /root/zrepl-credentials.txt (mode 0600) |
zrepl version 2>&1 | head -1
grep -E '^(source|toolchain|changes)' /usr/share/doc/zrepl/BUILD-PROVENANCE.txt | cut -c1-120
zfs version
client: zrepl version=v0.7.0 go=go1.27.3 GOOS=linux GOARCH=amd64 Compiler=gc
source: https://github.com/zrepl/zrepl/archive/refs/tags/v0.7.0.tar.gz (tag commit 2c12675ee18329059e7f69a11acd90424f
toolchain: go1.27.3 linux/amd64 (https://go.dev/dl/go1.27.3.linux-amd64.tar.gz, sha256 65efbc80fdc1fa2cd2631cad5b481777a
changes: dependency raise only (go.mod, no source patch): google.golang.org/grpc@v1.83.2 golang.org/x/net@v0.62.0 gola
zfs-2.2.2-0ubuntu9.5
zfs-kmod-2.4.1-1ubuntu5.1
Logs: sudo journalctl -u zrepl (zrepl logs to syslog).
Step 13: Security recommendations
- Allow port 8888 in the security group from your source machines only, and port 22 from your own networks only.
- Treat every client key as a credential: one bundle per machine, delete the bundle from the source machine's staging copies after setup, and revoke a machine's name when it is retired.
- Use
send: encrypted: truefor datasets that should stay encrypted at rest on the sink with a key it never holds. - Keep a second copy of the sink: EBS snapshots of the data volume, or a second sink.
- Run
sudo zrepl-selftestafter configuration changes, and watchsudo zrepl statusfor failed replications. - Keep the operating system patched; unattended security upgrades are on, and
sudo apt-get update && sudo apt-get upgradeapplies everything else.
Support and licensing
zrepl is open source under the MIT licence and OpenZFS under the CDDL; there is no per user or per CPU licence fee. The zrepl licence is in /usr/share/doc/zrepl/. cloudimg is not affiliated with or endorsed by the zrepl project or OpenZFS. The AMI price covers cloudimg's hardening, first boot setup and support.
- Email: support@cloudimg.co.uk
- Website: https://www.cloudimg.co.uk
- Support: 24/7, with a 24 hour response SLA