Infinity 0.0.77 on AWS User Guide

Overview

This guide covers launching and verifying Infinity 0.0.77 on Ubuntu 24.04 from cloudimg on Amazon EC2. It is one version of the cloudimg Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace; you pick it as the software version when you subscribe and launch. The other LLM servers, gateways and chat interfaces in the listing are described on the listing's guide hub.

Infinity is an open-source (MIT) REST server for text embeddings, reranking and classification. It serves Hugging Face models such as BGE, GTE, E5 and mixedbread with dynamic batching, and exposes an OpenAI-compatible embeddings API (/embeddings and /models), so code written for the OpenAI SDK, LangChain or LlamaIndex can point at it with a base URL and an API key. Use it to turn documents and queries into vectors for semantic search and retrieval-augmented generation (RAG), with the data never leaving your VPC.

This image runs Infinity on the CPU with the CPU build of PyTorch, so it needs no GPU and runs on general purpose instance types. A small open-weights embedding model, BAAI/bge-small-en-v1.5 (384 dimensions, MIT licence), is downloaded at build time, so the API answers as soon as the instance is up.

What is on the image:

• Infinity 0.0.77 (infinity-emb[server,torch]==0.0.77) in a Python 3.12 virtual environment at /opt/infinity/venv, run by systemd as the unprivileged infinity user
• PyTorch 2.14.1, CPU build (no CUDA libraries), and the BAAI/bge-small-en-v1.5 model cached under /var/lib/infinity/hf
• Infinity bound to loopback only (127.0.0.1:7997), with nginx as the only network-facing service
• HTTPS on port 443 with a self-signed certificate created on your instance's first boot; port 80 redirects to HTTPS
• A per-instance API key (sent as Authorization: Bearer, the way OpenAI SDK clients authenticate) and a per-instance login (HTTP Basic, user admin), both generated on first boot into a root-only file
• infinity-set-hostname, a helper that serves Infinity under your own DNS name and re-issues the certificate for it

Default user: ubuntu · Architecture: x86_64 · Root volume: 20 GiB gp3

Security posture

• No default credential. The API key and the login password are generated on each instance's first boot and stored only in nginx (the password as a bcrypt hash) and in a root-only file. Neither is in the image, so no two instances share one.
• Every API route needs a credential. Infinity has no authentication of its own, so nginx enforces it in front of everything: /embeddings, /rerank, /classify, /models, /metrics and the /docs page. A request needs either the API key or the login. Only GET /health (which returns a timestamp) and nginx's own /healthz are open, for load balancer health checks.
• Fails closed. nginx will not start until first boot has created the certificate and the credentials, and first boot stops nginx again if its own check of the credential gate fails.
• Only nginx faces the network. It answers only for the configured hostname (any other Host header gets its connection closed), uses TLS 1.2 or 1.3, refuses cross-origin POST requests and WebSocket upgrades (403), strips CORS headers, and replaces any client-supplied X-Forwarded-For, X-Real-IP or Forwarded header with the real peer address.
• The hostname comes from configuration, never from a request. First boot reads it from EC2 instance metadata (IMDSv2), or from /etc/cloudimg/infinity-hostname if you set one.
• No instance metadata access for the server. The infinity and nginx services are denied the instance metadata service (169.254.169.254 and fd00:ec2::254) by systemd IPAddressDeny, so no request can make them read the instance's role credentials. Infinity also never fetches image or audio URLs on this image: the bundled model is text-only, so such requests are refused before any download.
• IMDSv2 required. The AMI is registered with ImdsSupport v2.0, so instances launched from it refuse token-less (IMDSv1) metadata requests.
• Telemetry off and no self-update. Infinity's anonymous usage statistics and Hugging Face Hub telemetry are disabled, and the instance never upgrades Infinity on its own: it runs exactly the release in the version title.
• Per-instance identity. SSH host keys and the machine ID are generated fresh on every instance.

Prerequisites

1. An AWS account and permission to launch EC2 instances
2. An EC2 key pair in the Region you launch in

Recommended instance type: m5.large. Infinity runs the bundled model on the CPU; 2 vCPUs and 8 GiB of memory embed short texts in milliseconds. For heavy batch work or larger models, choose more vCPUs (for example c5.2xlarge or m5.2xlarge); the software charge is per vCPU.

Connecting to your instance

Version	Operating system	SSH user
Infinity 0.0.77 on Ubuntu 24.04	Ubuntu 24.04 LTS	ubuntu

Step 1: Launch the instance

Option A: AWS Marketplace console

1. Open the cloudimg Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace and choose View purchase options, then Subscribe. One subscription covers every version in the listing.
2. Once the subscription is active, choose Launch your software.
3. Choose One-click launch from AWS Marketplace (or Launch from EC2 Console for the full launch wizard). In Version, choose Infinity 0.0.77 on Ubuntu 24.04, then choose your Region.
4. Pick an instance type (m5.large recommended; the software charge is $0.04 per vCPU per hour) and your key pair.
5. Security group: the listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks your applications call from.
6. Choose your VPC and subnet, leave the metadata defaults (the image already requires IMDSv2), then choose Launch.

Option B: AWS CLI

Run this on your own workstation. It creates a security group that allows SSH from your management address and HTTPS from your application network, then launches the instance. Replace the AMI ID with the one shown for your Region on the Marketplace launch page, and the VPC, subnet, key pair and CIDR ranges with your own:

SG=$(aws ec2 create-security-group --group-name infinity \
  --description "Infinity embeddings API" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=infinity}]'

You do not need --metadata-options: the AMI already makes IMDSv2 required. An embeddings API usually serves applications inside your VPC; keep 443 closed to the internet unless clients outside need it.

Step 2: Connect and find your credentials

Connect as ubuntu with the key pair you chose:

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes within a minute of launch. It sets the hostname Infinity is served as (the instance's public DNS name when it has one, else its IP address), creates the TLS certificate for it, generates the API key and the login password, starts Infinity on loopback and loads the model, then starts nginx and checks the credential gate. It writes /root/infinity-credentials.txt (mode 0600, root). These are the entries in that file (the secrets are not shown here):

sudo grep -E '^INFINITY_(HOST|URL|MODEL|USERNAME)=' /root/infinity-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/infinity-credentials.txt
sudo stat -c '%a %U:%G' /root/infinity-credentials.txt

INFINITY_HOST=ec2-54-210-8-252.compute-1.amazonaws.com
INFINITY_URL=https://ec2-54-210-8-252.compute-1.amazonaws.com/
INFINITY_MODEL=BAAI/bge-small-en-v1.5
INFINITY_USERNAME=admin
INFINITY_HOST=
INFINITY_URL=
INFINITY_MODEL=
INFINITY_API_KEY=
INFINITY_USERNAME=
INFINITY_PASSWORD=
600 root:root

Value	What it is for
INFINITY_URL	The base URL of this instance's API
INFINITY_MODEL	The model id to put in each request
INFINITY_API_KEY	API key, sent as Authorization: Bearer <key> (OpenAI SDK api_key)
INFINITY_USERNAME / INFINITY_PASSWORD	HTTP Basic login, for your browser on the /docs page

To read the secrets, run sudo cat /root/infinity-credentials.txt in your own session and store them in your secrets manager.

Step 3: Confirm Infinity is running

The first-boot unit, Infinity and nginx are active. Only sshd and nginx listen beyond loopback; Infinity itself is on 127.0.0.1:7997, so nginx is the only way in. The open health endpoint returns a timestamp:

systemctl is-active infinity-firstboot.service infinity.service nginx.service
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u
H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" "https://$H/health"; echo

active
active
active
0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:7997 users:(("infinity_emb"))
127.0.0.53%lo:53 users:(("systemd-resolve"))
127.0.0.54:53 users:(("systemd-resolve"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))
{"unix":1792456780.069389}

/health and nginx's /healthz reveal nothing sensitive, so they are safe to use as load balancer health checks.

Step 4: Prove the credential gate and the web defaults

Requests without a credential are refused, and so are an API key and a password that are wrong by one character. This instance's own API key and password are accepted. A cross-origin POST is refused even with the right key, an unknown Host header gets no answer, and plain HTTP redirects to the configured HTTPS name:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
C=(-s -o /dev/null --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1")
curl "${C[@]}" -w 'no credential -> %{http_code}\n' "https://$H/models"
curl "${C[@]}" -w 'wrong API key -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %sX"\n' "$K") "https://$H/models"
curl "${C[@]}" -w 'wrong password -> %{http_code}\n' -K <(printf 'user = "admin:%sX"\n' "$P") "https://$H/models"
curl "${C[@]}" -w 'this instance API key -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") "https://$H/models"
curl "${C[@]}" -w 'this instance password -> %{http_code}\n' -K <(printf 'user = "admin:%s"\n' "$P") "https://$H/models"
curl "${C[@]}" -w 'cross-origin POST -> %{http_code}\n' -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -X POST -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{}' "https://$H/embeddings"
curl -sk -o /dev/null -w 'unknown Host -> %{http_code}\n' -H 'Host: evil.example' https://127.0.0.1/health || true
curl -s -o /dev/null -w 'http -> %{http_code} %{redirect_url}\n' -H "Host: $H" http://127.0.0.1/models

no credential -> 401
wrong API key -> 401
wrong password -> 401
this instance API key -> 200
this instance password -> 200
cross-origin POST -> 403
unknown Host -> 000
http -> 301 https://ec2-54-210-8-252.compute-1.amazonaws.com/models

The secrets are passed to curl through a config on a file descriptor (-K), so they never appear on a command line, in the process list or in your shell history.

Step 5: Create an embedding

Send text to /embeddings with the API key. The response is in the OpenAI format: one vector per input, plus token usage:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
M=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" \
  -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -H 'Content-Type: application/json' "https://$H/embeddings" \
  -d "{\"model\":\"$M\",\"input\":[\"cloudimg makes self-hosted AI easy\"]}" \
  | jq '{object, model, vectors: (.data | length), dimensions: (.data[0].embedding | length), first3: .data[0].embedding[0:3], usage}'

{
  "object": "list",
  "model": "BAAI/bge-small-en-v1.5",
  "vectors": 1,
  "dimensions": 384,
  "first3": [
    -0.08709850162267685,
    -0.047451201826334,
    0.014566515572369099
  ],
  "usage": {
    "prompt_tokens": 34,
    "total_tokens": 34
  }
}

Step 6: Rank documents by meaning (semantic search)

Infinity embeds a whole batch in one call. Embed a query together with candidate documents and compare the vectors with cosine similarity: the document that answers the query scores highest, although it shares few words with it:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
M=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" \
  -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") \
  -H 'Content-Type: application/json' "https://$H/embeddings" \
  -d "{\"model\":\"$M\",\"input\":[\"What is the capital of France?\",\"Paris is the capital of France.\",\"Berlin is a city in Germany.\"]}" \
  | python3 -c '
import json, math, sys
v = [d["embedding"] for d in json.load(sys.stdin)["data"]]
cos = lambda a, b: sum(x * y for x, y in zip(a, b)) / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
print(f"{len(v)} vectors of {len(v[0])} dimensions")
print(f"query vs Paris sentence:  {cos(v[0], v[1]):.3f}")
print(f"query vs Berlin sentence: {cos(v[0], v[2]):.3f}")'

3 vectors of 384 dimensions
query vs Paris sentence:  0.812
query vs Berlin sentence: 0.578

In a RAG application you store the document vectors in a vector database (pgvector, Qdrant, Weaviate, Chroma and so on) and embed each incoming query the same way.

Step 7: List the served model

/models lists what Infinity is serving, the same call the OpenAI SDK makes for client.models.list():

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
K=$(sudo grep '^INFINITY_API_KEY=' /root/infinity-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/infinity/infinity.crt --resolve "$H:443:127.0.0.1" \
  -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") "https://$H/models" \
  | jq '{object, data: [.data[] | {id, backend, capabilities}]}'

{
  "object": "list",
  "data": [
    {
      "id": "BAAI/bge-small-en-v1.5",
      "backend": "torch",
      "capabilities": [
        "embed"
      ]
    }
  ]
}

The bundled model can embed. /rerank and /classify need a reranker or classifier model (Step 10).

Step 8: Call Infinity from your application

Infinity serves the OpenAI paths at the root, with no /v1 prefix. Point the OpenAI Python SDK at the instance and pass the API key. The client needs to trust the self-signed certificate: copy /etc/ssl/infinity/infinity.crt from the instance to your client (or install your own certificate, Step 9):

pip install openai httpx

import httpx
from openai import OpenAI

client = OpenAI(
    base_url="https://<your-infinity-host>",          # INFINITY_URL without the trailing slash
    api_key="<your INFINITY_API_KEY>",
    http_client=httpx.Client(verify="infinity.crt"),  # the instance's certificate
)
resp = client.embeddings.create(model="BAAI/bge-small-en-v1.5",
                                input=["first document", "second document"])
print(len(resp.data), len(resp.data[0].embedding))  # 2 384

LangChain's OpenAIEmbeddings and LlamaIndex's OpenAIEmbedding take the same base_url (or api_base) and api_key. Tools that only speak HTTP Basic can use admin and INFINITY_PASSWORD instead.

The interactive API page

Infinity's built-in OpenAPI page is at https://<your-infinity-host>/docs. Your browser asks for the login: use admin and INFINITY_PASSWORD. The page lists every route:

Open POST /embeddings, choose Try it out, enter a request body and choose Execute to see the vectors Infinity returns:

GET /models shows the served model with its batch statistics:

The page loads its scripts and styles (Swagger UI) from the jsDelivr CDN in your browser, so it needs internet access on the browser side; the API itself does not.

Step 9: Serve Infinity under your own DNS name

Point a DNS record at the instance (an Elastic IP keeps the address stable), then tell Infinity the name. The helper re-issues the self-signed certificate for it, updates nginx's accepted names, records the name in /etc/cloudimg/infinity-hostname (kept across reboots) and updates INFINITY_URL:

sudo infinity-set-hostname embeddings.example.com

infinity-set-hostname: Infinity is served at https://embeddings.example.com/

To use a certificate from your own CA or from Let's Encrypt instead, replace /etc/ssl/infinity/infinity.crt and /etc/ssl/infinity/infinity.key (keep the key mode 0600, owner root) and run sudo systemctl reload nginx.

Step 10: Serve a different model

The served model is the MODEL line in /etc/infinity/infinity.env. To serve a reranker, a classifier or a larger embedding model, change it to the model's Hugging Face id and restart Infinity. The instance downloads the model on that restart, so it needs outbound HTTPS to huggingface.co for that one start:

sudo sed -i 's|^MODEL=.*|MODEL=BAAI/bge-reranker-base|' /etc/infinity/infinity.env
sudo systemctl restart infinity.service
sudo journalctl -u infinity.service -n 20 --no-pager

/rerank then answers with relevance scores. Serve only models from publishers you trust: loading a model runs its configuration and weights through transformers and PyTorch. Larger models need more memory; check the model card and size the instance to match. The first-boot gate uses the configured model, so set the model you want before you create an image of your own instance.

Step 11: Check the security controls

The certificate was made for this instance on first boot, and both services are denied the instance metadata service. The last line runs a metadata request from inside Infinity's own service control group, exactly as a request handled by Infinity would:

H=$(sudo grep '^INFINITY_HOST=' /root/infinity-credentials.txt | cut -d= -f2-)
openssl s_client -connect 127.0.0.1:443 -servername "$H" </dev/null 2>/dev/null | openssl x509 -noout -subject -enddate
systemctl show -p IPAddressDeny --value infinity.service nginx.service
CG=$(systemctl show -p ControlGroup --value infinity.service)
sudo bash -c "echo \$\$ > /sys/fs/cgroup$CG/cgroup.procs; exec curl -s -m 5 -o /dev/null -w 'metadata from inside infinity.service -> %{http_code}\n' -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' http://169.254.169.254/latest/api/token" || true

subject=CN = ec2-54-210-8-252.compute-1.amazonaws.com, O = cloudimg
notAfter=Jan 22 00:37:14 2029 GMT
168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128

168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128
metadata from inside infinity.service -> 000

The installed release is the one in the version title, PyTorch is the CPU build with no CUDA runtime and no NVIDIA packages, and telemetry is off:

/opt/infinity/venv/bin/pip show infinity-emb | grep -E '^(Name|Version|License)'
/opt/infinity/venv/bin/python -c 'import torch; print("torch", torch.__version__, "cuda:", torch.version.cuda)'
/opt/infinity/venv/bin/pip list --format=freeze | grep -ciE '^(nvidia|triton)' || true
sudo grep -E '^(MODEL|INFINITY_ANONYMOUS_USAGE_STATS|DO_NOT_TRACK|HF_HUB_DISABLE_TELEMETRY)=' /etc/infinity/infinity.env

Name: infinity_emb
Version: 0.0.77
License: MIT
torch 2.14.1+cpu cuda: None
0
MODEL=BAAI/bge-small-en-v1.5
INFINITY_ANONYMOUS_USAGE_STATS=0
DO_NOT_TRACK=1
HF_HUB_DISABLE_TELEMETRY=1

Step 12: Rotate the API key or the login password

To issue a new API key, write it into nginx's key map and reload nginx; the old key stops working at once. The key is generated on the instance and handed to sudo tee on standard input, so it never appears on a sudo command line (which sudo writes to the system logs). The map file keeps its mode 0600, owner root:

NEWKEY="inf-$(openssl rand -hex 24)"
printf '"Bearer %s" 1;\n' "$NEWKEY" | sudo tee /etc/nginx/infinity-apikey.map >/dev/null
sudo systemctl reload nginx
printf 'New API key: %s\n' "$NEWKEY"

To change the login password, htpasswd reads it from standard input, so it never appears on a command line, in your shell history or in the system logs. Enter the new password when prompted, then reload nginx:

sudo htpasswd -i -B /etc/nginx/.infinity.htpasswd admin
sudo systemctl reload nginx

Then put the new values in /root/infinity-credentials.txt (or delete that file) so it does not hold stale secrets.

Maintenance

• Logs: sudo journalctl -u infinity.service for Infinity, /var/log/nginx/ for HTTPS access, sudo journalctl -u infinity-firstboot.service for first boot.
• Restart: sudo systemctl restart infinity.service (reloads the model; /health answers again once it is loaded), or sudo systemctl reload nginx.
• Operating system updates: Ubuntu's unattended security upgrades stay on, so nginx, OpenSSL and the rest of the OS keep receiving patches. Apply the rest with sudo apt update && sudo apt upgrade in a maintenance window.
• Upgrading Infinity: the release is pinned and the instance never upgrades it on its own. To move to a newer release deliberately, install it into the virtual environment with the CPU build of PyTorch already present (sudo /opt/infinity/venv/bin/pip install 'infinity-emb[server,torch]==<version>'), then restart the service. Keep the CPU build of PyTorch: if pip wants to change torch, install the matching CPU wheel from https://download.pytorch.org/whl/cpu first, otherwise pip pulls the much larger CUDA build from PyPI. Test a new release on a copy of the instance first.
• Throughput: Infinity batches concurrent requests automatically. Send several texts per request and run requests in parallel rather than one text at a time.

Troubleshooting

401 Unauthorized from every request. Send the key as Authorization: Bearer <INFINITY_API_KEY> exactly (OpenAI SDK api_key), or the login admin / INFINITY_PASSWORD, over https://.

curl: (60) SSL certificate problem. The certificate is self-signed. Pass --cacert with a copy of /etc/ssl/infinity/infinity.crt, or install your own certificate (Step 9). Use the configured hostname (INFINITY_HOST), which the certificate covers.

nginx will not start. It waits for first boot to finish and for the certificate and credentials to exist, by design. Check first boot with sudo journalctl -u infinity-firstboot.service --no-pager | tail -20.

400 "does not support" from /rerank or /classify. The served model cannot do that task. Serve a reranker or classifier model (Step 10).

Infinity does not start after a model change. Read sudo journalctl -u infinity.service -n 50 --no-pager: the usual causes are a mistyped model id, no outbound HTTPS to huggingface.co for the download, or too little memory for the model.

Support

cloudimg provides 24/7 support for this image by email and live chat: launching and connecting, API clients and SDKs, TLS and hostnames, model changes and upgrades.

Infinity is developed by Michael Feil and contributors and is released under the MIT licence. BGE models are published by the Beijing Academy of Artificial Intelligence (BAAI). PyTorch is a trademark of The Linux Foundation. Ubuntu is a registered trademark of Canonical Ltd. OpenAI is a trademark of OpenAI, Inc. All other product and company names are trademarks or registered trademarks of their respective holders. cloudimg is not affiliated with or endorsed by them.