Toxiproxy Resilience Testing Proxy on AWS User Guide

Overview

This guide covers the deployment and use of Toxiproxy on AWS using cloudimg AWS Marketplace images. Toxiproxy is a TCP proxy for deterministic resilience and chaos testing. It sits between your application and its upstream dependencies (databases, caches, queues, APIs) and injects controllable failures called toxics — latency, bandwidth limits, timeouts, connection resets and data slicing — so you can reproduce a slow database or a flaky network exactly, on demand, instead of waiting for it to happen in production.

The image installs the official Toxiproxy release binaries (v2.12.0, pinned and SHA256-verified at build time) and runs toxiproxy-server under systemd, so a working proxy is available within minutes of launch with no manual compilation.

A self-contained demo on first boot. The image ships a bundled loopback HTTP upstream, a demo proxy named demo that forwards a public port to it, and a sample latency toxic already applied — so listing the proxies and curling through the demo port shows a real, working, failure-injecting proxy the moment the instance boots.

Secure by design — the control API is never exposed unauthenticated. The Toxiproxy control API is unauthenticated and can create proxies to arbitrary destinations, so this image binds it to the loopback interface only (127.0.0.1:8474). For remote and programmatic access it is fronted by an nginx TLS reverse proxy on port 8443, gated by an HTTP Basic credential generated uniquely on each instance at first boot and never baked into the image. Only SSH (22), the demo proxy's data port (8080) and the authenticated control-API endpoint (8443) are opened; the raw control-API port 8474 is never exposed to the network.

What is included:

• Toxiproxy 2.12.0 (official release binaries, pinned and SHA256-verified at build), run under systemd as the unprivileged toxiproxy user (toxiproxy.service)

• A self-contained demo: a bundled loopback upstream (toxiproxy-demo-upstream.service) and a demo proxy on port 8080 with a sample 500 ms latency toxic, live on first boot

• The toxiproxy-cli command line client for driving the control API locally

• An nginx TLS front door on port 8443 for the control API, gated by a per-instance HTTP Basic credential generated on first boot and never baked into the image

• The raw control API bound to loopback only, proven refused on the instance's non-loopback address

• Unattended security upgrades left enabled so the operating system keeps receiving patches

Connecting to your instance

Connect over SSH as the default login user for your operating system variant, using the EC2 key pair you selected at launch.

Once connected, read the per-instance connection facts and control-API credential from the root-only credentials file:

sudo cat /root/toxiproxy-credentials.txt

This file (mode 0600, root only) holds this instance's public host, the authenticated control-API URL, the per-instance HTTP Basic username and password, and the demo proxy URL. The credential is unique to this instance — it is minted on first boot and is different on every instance.

Confirming the services are running

Toxiproxy runs under systemd alongside the bundled demo upstream and the nginx control-API front door. Check them and the listening ports:

systemctl is-active toxiproxy-demo-upstream.service toxiproxy.service nginx.service

active active active

sudo ss -tlnp | grep -E ':8474|:8080|:8443|:8899'

LISTEN 0  511        0.0.0.0:8443   0.0.0.0:*
LISTEN 0  5        127.0.0.1:8899   0.0.0.0:*
LISTEN 0  4096     127.0.0.1:8474   0.0.0.0:*
LISTEN 0  4096             *:8080         *:*

Note the binds: the raw control API (8474) and the demo upstream (8899) listen on 127.0.0.1 only, while the demo proxy data port (8080) and the nginx TLS front door (8443) are network-reachable.

The bundled demo proxy

Drive the control API locally with toxiproxy-cli. List the proxies to see the bundled demo:

export TOXIPROXY_URL=http://127.0.0.1:8474
toxiproxy-cli list

demo    [::]:8080   127.0.0.1:8899  enabled 1

Inspect it to see the sample latency toxic that ships applied:

toxiproxy-cli inspect demo

cloudimg_latency    type=latency    stream=downstream   toxicity=1.00   attributes=[    jitter=0    latency=500 ]

Seeing a toxic shape traffic

The demo proxy forwards http://<instance-ip>:8080/ to the bundled loopback upstream. With the sample 500 ms latency toxic applied, a request through the proxy takes about half a second; remove the toxic and the same request returns almost instantly:

# through the proxy, with the sample 500ms toxic
curl -s -o /dev/null -w 'http_code=%{http_code}  time_total=%{time_total}s\n' http://127.0.0.1:8080/

http_code=200  time_total=0.502454s

# remove the toxic and try again
toxiproxy-cli toxic remove -n cloudimg_latency demo
curl -s -o /dev/null -w 'http_code=%{http_code}  time_total=%{time_total}s\n' http://127.0.0.1:8080/

http_code=200  time_total=0.001751s

The round-trip drops from ~500 ms to ~2 ms — the toxic is demonstrably shaping the traffic. Re-add it (or add your own) at any time:

toxiproxy-cli toxic add -t latency -a latency=500 -n cloudimg_latency -d demo

Security: loopback-only control API and the authenticated front door

The Toxiproxy control API is unauthenticated by design. This image never exposes it unauthenticated. The raw control API answers only on loopback, and is refused on the instance's own non-loopback address:

curl -s -o /dev/null -w '  127.0.0.1:8474 -> %{http_code}\n' http://127.0.0.1:8474/version
PRIV=$(hostname -I | awk '{print $1}')
curl -s -m 3 -o /dev/null -w "  $PRIV:8474 -> %{http_code}\n" http://$PRIV:8474/version || echo "  $PRIV:8474 -> connection refused"

  127.0.0.1:8474 -> 200
  172.31.83.36:8474 -> connection refused

For remote access, the nginx TLS front door on port 8443 requires the per-instance HTTP Basic credential — a request with no credential or a wrong credential is rejected with 401, and only the per-instance credential is accepted:

U=$(sudo grep ^CONTROL_API_USER= /root/toxiproxy-credentials.txt | cut -d= -f2-)
P=$(sudo grep ^CONTROL_API_PASSWORD= /root/toxiproxy-credentials.txt | cut -d= -f2-)
curl -sk -o /dev/null -w '  no credential           -> %{http_code}\n' https://127.0.0.1:8443/version
curl -sk -u "$U:wrong" -o /dev/null -w '  wrong credential        -> %{http_code}\n' https://127.0.0.1:8443/version
curl -sk -u "$U:$P" -o /dev/null -w '  per-instance credential -> %{http_code}\n' https://127.0.0.1:8443/version

  no credential           -> 401
  wrong credential        -> 401
  per-instance credential -> 200

To call the control API remotely from your workstation (the TLS certificate is a per-instance self-signed cert, so pass -k or trust it):

curl -k -u toxictl:<your-password> https://<public-ip>:8443/proxies

Using Toxiproxy against your own dependencies

Point a new proxy at your own upstream dependency, then inject toxics to reproduce failure modes. For example, put a proxy in front of a database on db.internal:5432 and add latency:

export TOXIPROXY_URL=http://127.0.0.1:8474
toxiproxy-cli create -l 0.0.0.0:15432 -u db.internal:5432 mydb
toxiproxy-cli toxic add -t latency -a latency=2000 -n slow_db -d mydb

Now point your application at the proxy (15432) instead of the database directly, and it experiences a consistent 2-second delay. Other toxic types include bandwidth, timeout, slow_close, reset_peer and slicer. Remove a toxic with toxiproxy-cli toxic remove -n <name> <proxy> and delete a proxy with toxiproxy-cli delete <proxy>.

Open the matching port in your instance's security group to expose a proxy's data plane to your test clients. Keep the control API loopback-only or behind the authenticated front door — never expose port 8474 directly.

Support

cloudimg provides 24/7 technical support for this product by email and live chat.

• Contact: support@cloudimg.co.uk
• Deployment and initial configuration
• Driving the Toxiproxy control API and the toxiproxy-cli command
• Designing toxics to reproduce your production failure modes
• Wiring the authenticated control-API front door to your own domain and TLS certificate

cloudimg is not affiliated with or endorsed by Shopify or the Toxiproxy project. Toxiproxy is distributed under the MIT license. All product and company names are trademarks or registered trademarks of their respective holders.