LiteLLM on AWS User Guide

Overview

This image runs LiteLLM 1.88, the open source gateway that puts a single OpenAI-compatible API in front of more than 100 large language model providers - OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Google and self hosted models - on Ubuntu 24.04 LTS. LiteLLM is installed into a dedicated Python virtual environment under /opt/litellm on Python 3.12 and run by an unprivileged litellm system account under a systemd service that starts the proxy on boot and restarts it on failure.

The proxy listens on the loopback address 127.0.0.1:4000 and is never exposed directly. nginx is installed as a reverse proxy on port 80 that forwards to it. The unauthenticated liveliness and readiness probes are open; the OpenAI-compatible API is gated by a master key.

The proxy is secured by a master key (an sk- key). On the first boot of every deployed instance a one-shot service generates a fresh master key, unique to that instance, and writes it to /root/litellm-credentials.txt (mode 0600, readable only by root). Two instances launched from the same AMI never share a key.

LiteLLM ships no model weights and calls out to the provider endpoints you configure, so the image is CPU only. The default security group opens port 22 (SSH) and port 80 (HTTP) only.

Prerequisites

• An AWS account subscribed to this product in AWS Marketplace.
• An EC2 key pair in your target region for SSH access.
• A security group allowing inbound TCP 22 (SSH) from your IP and TCP 80 (HTTP) from your users.
• Recommended instance type: m5.large (the proxy is lightweight; size up for high request volumes).
• API keys for the LLM provider(s) you intend to route to.

Connecting to your instance

Step 1 - Launch from the AWS Marketplace console

1. Open the product page in AWS Marketplace and choose Continue to Subscribe, then Continue to Configuration.
2. Select the LiteLLM 1.88 on Ubuntu 24.04 delivery option and your region, then Continue to Launch.
3. Choose your instance type, VPC/subnet, key pair and the security group described above, and launch.

Step 2 - Launch from the AWS CLI

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type m5.large \
  --key-name your-key \
  --security-group-ids sg-xxxxxxxx \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=litellm}]'

Step 3 - Connect to your instance

ssh -i your-key.pem ubuntu@<instance-public-ip>

Step 4 - Confirm the services are running

systemctl is-active litellm.service nginx.service
ss -tln | grep -E ':80 |:4000 '
curl -s http://127.0.0.1/health/liveliness

Expected output:

active
active
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*
LISTEN 0      2048       127.0.0.1:4000      0.0.0.0:*
LISTEN 0      511             [::]:80           [::]:*
"I'm alive!"

Step 5 - Retrieve your master key

sudo cat /root/litellm-credentials.txt

# LiteLLM - generated on first boot by litellm-firstboot.service
LITELLM_URL=http://<instance-public-ip>/
LITELLM_MASTER_KEY=sk-<your-unique-key>

The master key authenticates every call to the OpenAI-compatible API and mints scoped virtual keys.

Step 6 - Add a provider

Edit /etc/litellm/config.yaml and add your providers under model_list, then restart the proxy:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0

sudo systemctl restart litellm.service

Put provider secrets in /etc/litellm/litellm.env (for example OPENAI_API_KEY=...) so os.environ/... references resolve.

Step 7 - Call the OpenAI-compatible API

Every request carries the master key as a Bearer token. List the configured models:

KEY=$(sudo grep '^LITELLM_MASTER_KEY=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s http://127.0.0.1/models -H "Authorization: Bearer $KEY"

{"data":[],"object":"list"}

The list is empty until you add providers in Step 6. Point any OpenAI SDK at http://<instance-public-ip>/ with the master key as the API key, and call chat completions at /chat/completions. The interactive API documentation is served at http://<instance-public-ip>/.

Step 8 - Virtual keys, budgets and the admin UI

The master key mints scoped virtual keys for teams and applications, each with its own budget and rate limits. Virtual keys, spend persistence and the admin UI at /ui require a database: set DATABASE_URL in /etc/litellm/litellm.env to a PostgreSQL instance and restart the service. See the LiteLLM documentation for the key-management API.

Step 9 - Confirm the runtime

/opt/litellm/venv/bin/pip show litellm | grep ^Version

Version: 1.88.0

Enabling HTTPS

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

certbot edits the nginx site at /etc/nginx/sites-available/cloudimg-litellm to add the TLS listener and arranges automatic renewal.

Backup and maintenance

• Your configuration lives in /etc/litellm/config.yaml (providers) and /etc/litellm/litellm.env (the master key and provider secrets). Back these up.
• Restart with sudo systemctl restart litellm.service; logs are in the journal: sudo journalctl -u litellm.service.
• The proxy is stateless by default. If you enable DATABASE_URL, back up that database to preserve virtual keys and spend history.

Support

cloudimg provides 24/7 technical support for this image by email and chat, covering LiteLLM deployment, configuring providers and model routing, load balancing and fallbacks, virtual keys and budgets, TLS termination and scaling. Contact details are on the AWS Marketplace listing.