S3Proxy on AWS User Guide

Overview

This image runs S3Proxy, an open source Java application that implements the Amazon S3 API and proxies requests to a storage backend, from the pinned upstream release s3proxy version 3.3.0 (github.com/gaul/s3proxy, Apache License 2.0). This appliance configures the local filesystem backend (the Apache jclouds filesystem provider), so a plain instance and its dedicated EBS data volume become an Amazon S3-compatible object storage endpoint. Point any S3 SDK, the AWS CLI, s3cmd, rclone, restic or a backup tool at it and read and write buckets and objects exactly as you would against Amazon S3, entirely within your own account.

The appliance is built defensively so it is never an open, anonymous endpoint:

• The S3 gateway binds to loopback 127.0.0.1:8080 only and is never directly internet-exposed.
• An nginx front end terminates TLS on port 443 with a per-instance self-signed certificate and reverse-proxies SigV4-signed requests to the gateway. The client's signed Host header is preserved so signature validation succeeds through the proxy.
• Port 80 issues a 301 redirect to HTTPS and serves an unauthenticated /healthz endpoint for load balancer probes.

S3Proxy can run with authorization disabled, but this appliance sets s3proxy.authorization=aws-v2-or-v4, so every request must be AWS Signature Version 4 (or Version 2) signed and there is no anonymous mode. On the first boot of every deployed instance, a one-shot service generates a unique access key and secret key, writes them into the gateway configuration and into /root/s3proxy-info.txt with file mode 0600, and the gateway service is configured to refuse to start until a key pair is present. No credentials are baked into the image.

Bucket and object data lives on a dedicated 30 GiB EBS data volume mounted at /var/lib/s3proxy, captured into the AMI so every instance is re-provisioned with the same layout. The gateway jar and JRE stay on the OS disk.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network
• The AWS CLI version 2 installed locally if you plan to deploy from the command line or connect an S3 client remotely

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for S3Proxy. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger as a balanced default. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and create or select a security group that allows inbound port 22 from your management network. To let S3 clients reach the endpoint from other machines, additionally open inbound TCP 443 restricted to your application or client subnets, and optionally 80 for the health probe. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation, which generates the per-instance key pair and TLS certificate, takes only a few seconds after the instance state becomes Running and the status checks pass.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg S3Proxy Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens port 22 from your management network.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":30,"VolumeType":"gp3"}}' \
  --metadata-options 'HttpTokens=required,HttpEndpoint=enabled' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=s3proxy}]'

The AMI also carries a second 30 GiB volume definition for /var/lib/s3proxy; it is attached automatically from the AMI's block device mapping.

Step 3: Connect to the Instance over SSH

Connect to the instance with SSH as the default login user for the operating system variant you launched. The login user differs by variant:

ssh -i /path/to/your-key.pem ubuntu@<instance-public-ip>

Step 4: Confirm S3Proxy Is Running

S3Proxy runs under systemd as s3proxy.service, with nginx fronting it. Confirm both are active:

systemctl is-active s3proxy nginx

Expected output:

active
active

The gateway is bound to loopback only, while nginx owns the public TLS port. You can see the split with ss:

sudo ss -tlnp | grep -E ':8080|:443'

You will see S3Proxy listening on 127.0.0.1:8080 (shown as [::ffff:127.0.0.1]:8080) and nginx listening on 0.0.0.0:443. The unauthenticated health endpoint answers over TLS:

curl -ks -o /dev/null -w '%{http_code}\n' https://127.0.0.1/healthz

It returns 200.

Step 5: Read the Per-Instance Access Key and Secret Key

The S3 credentials are generated uniquely on first boot and written to a root-only file. Read them with sudo:

sudo cat /root/s3proxy-info.txt

The file contains S3PROXY_ACCESS_KEY, S3PROXY_SECRET_KEY and S3PROXY_ENDPOINT_URL. Store the secret key somewhere safe; it is the only credential that authenticates requests to your endpoint.

Step 6: Confirm SigV4 Is Enforced

There is no anonymous access. An unsigned request is rejected with 403:

curl -ks -o /dev/null -w '%{http_code}\n' https://127.0.0.1/

It returns 403, confirming that every request must be SigV4-signed with your per-instance key pair.

Step 7: Run an S3 Object Round Trip

From the instance, export your per-instance key pair and run a full create-bucket, upload, list and read-back cycle against the endpoint. The certificate is self-signed per instance, so pass --no-verify-ssl (or trust the certificate). S3Proxy uses path-style addressing because the endpoint is an IP address.

export AWS_ACCESS_KEY_ID=$(sudo grep '^S3PROXY_ACCESS_KEY=' /root/s3proxy-info.txt | cut -d= -f2-)
export AWS_SECRET_ACCESS_KEY=$(sudo grep '^S3PROXY_SECRET_KEY=' /root/s3proxy-info.txt | cut -d= -f2-)
export AWS_DEFAULT_REGION=us-east-1
aws --no-verify-ssl --endpoint-url https://127.0.0.1/ s3 mb s3://my-first-bucket
echo "hello from s3proxy" > /tmp/hello.txt
aws --no-verify-ssl --endpoint-url https://127.0.0.1/ s3 cp /tmp/hello.txt s3://my-first-bucket/
aws --no-verify-ssl --endpoint-url https://127.0.0.1/ s3 ls s3://my-first-bucket/
aws --no-verify-ssl --endpoint-url https://127.0.0.1/ s3 cp s3://my-first-bucket/hello.txt -
aws --no-verify-ssl --endpoint-url https://127.0.0.1/ s3 rb s3://my-first-bucket --force

The commands above read your per-instance key pair straight from /root/s3proxy-info.txt, so you can copy and paste them on the instance. The object round trips through the gateway and is stored on the data volume under /var/lib/s3proxy/data.

Step 8: Connect a Remote S3 Client

From any machine that can reach the instance on port 443, point the AWS CLI (or boto3, s3cmd, rclone or restic) at https://<instance-public-ip>/ with the same per-instance key pair. Because the certificate is self-signed, either trust /etc/nginx/tls/s3proxy.crt on the client or pass the tool's insecure flag while testing:

export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-key>
export AWS_DEFAULT_REGION=us-east-1
aws --no-verify-ssl --endpoint-url https://<instance-public-ip>/ s3 ls

In boto3, pass endpoint_url='https://<instance-public-ip>/', aws_access_key_id, aws_secret_access_key, and config=boto3.session.Config(s3={'addressing_style': 'path'}). For production, front the appliance with your own domain and a CA-signed certificate.

Step 9: Network Security Model

Every request on port 443 is SigV4-authenticated, but network isolation is still your first line of defence. The shipped default security group opens only port 22, so port 443 (and 80) are not reachable from off the instance until you open them deliberately. When you do:

• Open 443 only to the application or client subnets that need to read and write objects. Restrict it to known CIDR ranges rather than 0.0.0.0/0.
• Optionally open 80 only where a load balancer needs the unauthenticated /healthz probe; all other port 80 traffic is redirected to HTTPS.
• Keep port 22 restricted to your management network.

Because the secret key is transmitted only inside a SigV4 signature and TLS protects the transport, keys never cross the wire in plaintext. Rotate the key pair at any time by editing s3proxy.identity and s3proxy.credential in /etc/s3proxy/s3proxy.conf and running sudo systemctl restart s3proxy.

Step 10: Maintenance

Keep the operating system patched with the standard package manager, for example sudo apt-get update && sudo apt-get -y upgrade on Ubuntu, and reboot when a new kernel is installed. Bucket and object data persists on the dedicated EBS volume at /var/lib/s3proxy across reboots; take regular EBS snapshots of that volume for backups. Review the nginx access and error logs under /var/log/nginx/ for client activity. The S3Proxy service logs are available with sudo journalctl -u s3proxy.

Screenshots

S3Proxy running under systemd, with the gateway bound to loopback, nginx terminating TLS on 443, and the health endpoint returning 200:

A full S3 object round trip through aws-cli against the S3Proxy endpoint over TLS, returning the exact object content:

SigV4 enforcement: an unsigned request is refused with 403, a wrong secret key is denied, and the per-instance key pair is accepted:

Support

cloudimg provides 24/7 technical support for this product by email at support@cloudimg.co.uk and by live chat. Our engineers can help with deployment, pointing the AWS CLI, boto3, s3cmd, rclone and restic at the S3Proxy endpoint, security group and network isolation guidance for the S3 API port, TLS termination with your own domain and certificate, and filesystem-backend and data-volume sizing. For critical issues we target a one hour average response, and general inquiries are handled during the same business day.