LiteLLM 1.88.0 on AWS User Guide

Overview

This guide covers launching and verifying LiteLLM 1.88.0 on Ubuntu 24.04 from cloudimg on Amazon EC2. It is one version of the cloudimg Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace; you pick it as the software version when you subscribe and launch. The other LLM servers, gateways and chat interfaces in the listing are described on the listing's guide hub.

LiteLLM is an open-source LLM gateway. It puts one OpenAI-compatible API (/v1/chat/completions, /v1/embeddings, /v1/models and the rest) in front of more than 100 model providers: OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Google Vertex AI, Mistral, Ollama and many more. Your applications call one endpoint with one key, and you change or load-balance the providers behind it in a configuration file without touching client code.

The image ships no model weights and no GPU libraries: LiteLLM forwards requests to the providers you configure, so a small CPU instance is enough.

What is on the image:

• LiteLLM 1.88.0 (the MIT-licensed proxy core) in a Python 3.12 virtual environment at /opt/litellm/venv, run by litellm.service as the unprivileged litellm user
• LiteLLM bound to loopback only (127.0.0.1:4000), with nginx as the only network-facing service
• HTTPS on port 443 with a self-signed certificate created on your instance's first boot; port 80 redirects to HTTPS
• A per-instance master key, salt key and UI password, generated on first boot into root-only files
• An empty model list in /etc/litellm/config.yaml, ready for your providers
• litellm-set-hostname, a helper that serves LiteLLM under your own DNS name and re-issues the certificate for it

Default user: ubuntu · Architecture: x86_64 · Root volume: 8 GiB gp3

Security posture

• No default key. The master key (LITELLM_MASTER_KEY), the salt key that encrypts stored provider credentials (LITELLM_SALT_KEY) and the UI password are generated on each instance's first boot. None of them exists in the image, so no two instances share one.
• Keys never in a world-readable file or on a command line. They live in /etc/litellm/litellm-secrets.env (mode 0600, root) and are loaded by systemd. config.yaml refers to the master key by name (os.environ/LITELLM_MASTER_KEY) and holds no secret.
• Fails closed. litellm.service refuses to start unless a well-formed per-instance master key exists, and nginx will not start before first boot has created the keys. If first boot cannot prove that an anonymous call and a wrong key are refused and the master key is accepted, it stops nginx rather than leave the gateway open.
• Every API and management route needs a key. Anonymous calls get 401; a key LiteLLM does not know is refused by LiteLLM itself before any provider is contacted.
• Only nginx faces the network. It answers only for the configured hostname (any other Host header gets its connection closed), uses TLS 1.2 or 1.3, refuses cross-origin POST requests and WebSocket upgrades (403), never passes LiteLLM's permissive CORS headers on, and replaces any client-supplied X-Forwarded-For with the real peer address.
• The hostname comes from configuration, never from a request. First boot reads it from EC2 instance metadata (IMDSv2) or from /etc/cloudimg/litellm-hostname.
• No instance metadata access for the gateway. LiteLLM fetches URLs on your behalf (provider api_base values, image URLs in requests, pass-through and MCP targets), so litellm.service is blocked from the instance metadata service (169.254.169.254 and fd00:ec2::254). A request can never read the instance's role credentials through the gateway. Amazon Time Sync (169.254.169.123) is unaffected.
• IMDSv2 required. The AMI is registered with ImdsSupport v2.0, so instances launched from it refuse token-less (IMDSv1) metadata requests.
• MIT core only. The litellm[proxy] package depends on litellm-enterprise, which carries BerriAI's proprietary Enterprise licence. cloudimg removes it, and the build fails if any part of it remains. The open-source gateway features in this guide do not need it.
• Sandboxed service and per-instance identity. litellm.service runs with NoNewPrivileges, ProtectSystem=full, ProtectHome, PrivateTmp and PrivateDevices. SSH host keys and the machine ID are generated fresh on every instance.

Prerequisites

1. An AWS account and permission to launch EC2 instances
2. An EC2 key pair in the Region you launch in
3. API keys for the model providers you want LiteLLM to call (OpenAI, Anthropic, Bedrock and so on), or a self-hosted model server such as Ollama

Recommended instance type: m5.large. LiteLLM is a gateway and holds no model, so its 2 vCPUs and 8 GiB of memory are ample; the model work happens at your providers.

Connecting to your instance

Step 1: Launch the instance

Option A: AWS Marketplace console

1. Open the cloudimg Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace and choose View purchase options, then Subscribe. One subscription covers every version in the listing.
2. Once the subscription is active, choose Launch your software.
3. Choose One-click launch from AWS Marketplace (or Launch from EC2 Console for the full launch wizard). In Version, choose LiteLLM 1.88.0 on Ubuntu 24.04, then choose your Region.
4. Pick an instance type (m5.large recommended; the software charge is $0.04 per vCPU per hour) and your key pair.
5. Security group: the listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks your applications call from.
6. Choose your VPC and subnet, leave the metadata defaults (the image already requires IMDSv2), then choose Launch.

Option B: AWS CLI

Run this on your own workstation. It creates a security group that allows SSH from your management address and HTTPS from your application network, then launches the instance. Replace the AMI ID with the one shown for your Region on the Marketplace launch page, and the VPC, subnet, key pair and CIDR ranges with your own:

SG=$(aws ec2 create-security-group --group-name litellm-gateway \
  --description "LiteLLM gateway" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=litellm-gateway}]'

You do not need --metadata-options: the AMI already makes IMDSv2 required.

Step 2: Connect and find your keys

Connect as ubuntu with the key pair you chose:

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes within a minute of launch. It sets the hostname LiteLLM is served as (the instance's public DNS name when it has one), creates the TLS certificate for it, generates the keys, starts LiteLLM and nginx, and writes /root/litellm-credentials.txt. These are the entries in that file (the values are not shown here):

sudo grep -E '^LITELLM_(HOST|URL|UI_USERNAME)=' /root/litellm-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/litellm-credentials.txt

LITELLM_HOST=ec2-3-84-115-65.compute-1.amazonaws.com
LITELLM_URL=https://ec2-3-84-115-65.compute-1.amazonaws.com/
LITELLM_UI_USERNAME=admin
LITELLM_HOST=
LITELLM_URL=
LITELLM_MASTER_KEY=
LITELLM_UI_USERNAME=
LITELLM_UI_PASSWORD=

To read your master key, run sudo cat /root/litellm-credentials.txt in your own session. LITELLM_MASTER_KEY (an sk- string) is the admin key for the whole gateway: keep it secret and store it in your secrets manager.

Step 3: Confirm the gateway is running

The first-boot unit, LiteLLM and nginx are active, the installed version is 1.88.0 under the MIT licence, and only sshd and nginx listen beyond loopback. LiteLLM itself is on 127.0.0.1:4000.

systemctl is-active litellm-firstboot.service litellm.service nginx.service
/opt/litellm/venv/bin/pip show litellm | grep -E '^(Name|Version|License)'
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u

active
active
active
Name: litellm
Version: 1.88.0
License-Expression: MIT
0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:4000 users:(("litellm"))
127.0.0.53%lo:53 users:(("systemd-resolve"))
127.0.0.54:53 users:(("systemd-resolve"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

Step 4: Call the API with the master key

LiteLLM answers on https://<LITELLM_HOST>/. An anonymous call is refused by LiteLLM's own auth layer, and the master key lists the configured models (none yet):

H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
KEY=$(sudo grep '^LITELLM_MASTER_KEY=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/health/liveliness"; echo
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/v1/models"; echo
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/v1/models" -H "Authorization: Bearer $KEY"; echo

"I'm alive!"
{"error":{"message":"Authentication Error, No api key passed in.","type":"auth_error","param":"None","code":"401"}}
{"data":[],"object":"list"}

From your own machine, copy the instance's certificate (/etc/ssl/litellm/litellm.crt) and pass it to your client, or add it to your trust store. Any OpenAI-compatible SDK works: set its base URL to https://<LITELLM_HOST>/v1 and its API key to the master key (or, once you add a database, a virtual key).

Step 5: Add your model providers

Models are defined in /etc/litellm/config.yaml and provider keys belong in /etc/litellm/litellm-secrets.env (root only), never in the YAML file. The shipped configuration is:

sudo grep -vE '^\s*#' /etc/litellm/config.yaml | sed '/^$/d'

model_list: []
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings:
  drop_params: true

For example, to offer OpenAI's gpt-4o and Anthropic's Claude Sonnet through the gateway, edit the file with sudo nano /etc/litellm/config.yaml and replace model_list: [] with:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

Then add the keys to the secrets file and restart LiteLLM:

sudo nano /etc/litellm/litellm-secrets.env      # add one line each for OPENAI_API_KEY and ANTHROPIC_API_KEY
sudo systemctl restart litellm.service

A chat completion then goes through the gateway like any OpenAI call:

curl --cacert litellm.crt https://<LITELLM_HOST>/v1/chat/completions \
  -H "Authorization: Bearer <master key>" -H 'Content-Type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello"}]}'

Self-hosted servers work the same way: an Ollama instance in your VPC is model: ollama/llama3.1 with api_base: http://<ollama-private-ip>:11434. The provider list and the parameters for each provider are in the LiteLLM documentation.

Step 6: Virtual keys, budgets and the admin UI

Virtual keys (scoped keys per application, with their own budgets, rate limits and model lists), spend tracking and the admin UI at https://<LITELLM_HOST>/ui/ store their state in a PostgreSQL database. This image does not configure one, so out of the box the master key is the only key and the UI login reports that no database is connected. Asking for a virtual key shows the same:

H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
KEY=$(sudo grep '^LITELLM_MASTER_KEY=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/litellm/litellm.crt "https://$H/key/generate" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"models":[]}'; echo

{"error":{"message":"{'error': 'DB not connected. See https://docs.litellm.ai/docs/proxy/virtual_keys'}","type":"internal_server_error","param":"None","code":"500"}}

To turn these features on, point LiteLLM at a PostgreSQL database you control, such as an Amazon RDS for PostgreSQL instance in the same VPC (keep it private to the VPC). Add the connection string to the secrets file and follow the LiteLLM virtual keys documentation for the database client setup:

sudo nano /etc/litellm/litellm-secrets.env
# add: DATABASE_URL=postgresql://litellm:<db-password>@<rds-endpoint>:5432/litellm
sudo systemctl restart litellm.service

The UI then signs in as admin with LITELLM_UI_PASSWORD from your credentials file. Keep LITELLM_SALT_KEY unchanged once the database holds provider credentials: LiteLLM encrypts them with it.

Step 7: Serve LiteLLM under your own DNS name

Port 80 redirects to HTTPS on the configured hostname, and the certificate covers that name, the instance's public and private addresses, localhost and 127.0.0.1:

H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://$H/v1/models"
openssl x509 -in /etc/ssl/litellm/litellm.crt -noout -subject -enddate -ext subjectAltName

301 https://ec2-3-84-115-65.compute-1.amazonaws.com/v1/models
subject=CN = ec2-3-84-115-65.compute-1.amazonaws.com, O = cloudimg
notAfter=Jan 21 11:37:14 2029 GMT
X509v3 Subject Alternative Name: 
    DNS:ec2-3-84-115-65.compute-1.amazonaws.com, IP Address:3.84.115.65, IP Address:172.31.85.155, DNS:ip-172-31-85-155.ec2.internal, IP Address:127.0.0.1, DNS:localhost

To use your own name, point a DNS record at the instance (an Elastic IP keeps the address stable across stops), then run the helper. It records the name in /etc/cloudimg/litellm-hostname, issues a new self-signed certificate for it, updates the credentials file and reloads nginx:

sudo litellm-set-hostname <your-dns-name>

For a publicly trusted certificate, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate, or replace /etc/ssl/litellm/litellm.crt and litellm.key with your own and run sudo systemctl reload nginx.

Step 8: Check the security controls

LiteLLM's service carries the metadata block, and nginx refuses cross-origin writes and requests for any hostname it was not configured with:

systemctl show -p IPAddressDeny litellm.service
H=$(sudo grep '^LITELLM_HOST=' /root/litellm-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'cross-origin POST: %{http_code}\n' --cacert /etc/ssl/litellm/litellm.crt \
  -X POST -H 'Origin: https://evil.example' "https://$H/key/generate"
code=$(curl -s -o /dev/null -w '%{http_code}' -H 'Host: evil.example' http://127.0.0.1/ || true); echo "unknown Host: ${code} (connection closed by nginx)"

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
cross-origin POST: 403
unknown Host: 000 (connection closed by nginx)

Maintenance

• Logs: sudo journalctl -u litellm.service -f for the gateway, /var/log/nginx/ for HTTPS access.
• Configuration changes: edit /etc/litellm/config.yaml or the secrets file, then sudo systemctl restart litellm.service.
• Operating system updates: Ubuntu's unattended security upgrades stay on. Apply the rest with sudo apt update && sudo apt upgrade in a maintenance window.
• Upgrading LiteLLM: upgrade inside the virtual environment, remove litellm-enterprise again if you do not hold a BerriAI Enterprise licence, and restart:

sudo /opt/litellm/venv/bin/pip install --upgrade 'litellm[proxy]'
sudo /opt/litellm/venv/bin/pip uninstall -y litellm-enterprise
sudo systemctl restart litellm.service

• Backups: /etc/litellm/ holds the whole configuration (config.yaml and the root-only secrets file). Keep a copy somewhere safe; an AMI or EBS snapshot of the instance also captures it.

Support

cloudimg provides 24/7 support for this image by email and live chat: launching and connecting, provider configuration, TLS and hostnames, and upgrades.

Trademarks

LiteLLM is developed by BerriAI, Inc. Ubuntu is a registered trademark of Canonical Ltd. All other product and company names are trademarks or registered trademarks of their respective holders. cloudimg is not affiliated with or endorsed by them.