Text Embeddings Inference on AWS User Guide

Overview

This image runs Text Embeddings Inference (TEI) 1.9, Hugging Face's high-throughput, low-latency server for text embedding and reranking models (huggingface/text-embeddings-inference, Apache-2.0), on Ubuntu 24.04 LTS. TEI serves an OpenAI-compatible /v1/embeddings endpoint (plus the native /embed and /rerank), so existing OpenAI SDK code works unchanged. This is a CPU build: it runs the official Hugging Face CPU container on ordinary general-purpose instances with no GPU, no NVIDIA driver and no CUDA toolkit to manage.

The server runs as the official TEI container under Docker, bound to the loopback address 127.0.0.1:8000; nginx fronts it on port 80 with HTTP Basic Authentication. The public /health endpoint stays open; /embed, /v1/embeddings, /rerank and /info require the password. The default security group opens port 22 (SSH) and port 80 (HTTP) only, so 8000 is not reachable externally.

On the first boot of every deployed instance a one-shot service generates a fresh password, unique to that instance, and writes it to /root/tei-credentials.txt (mode 0600, root only). The embedding model lives under /var/lib/tei/data (mounted into the container at /data) on a dedicated, independently resizable EBS data volume. A compact open-weights model (BAAI/bge-small-en-v1.5, MIT, 384-dim) is pre-downloaded at build time and served by default, so the API returns vectors immediately with no download; you serve a different model by editing MODEL in /etc/tei/tei.env.

Prerequisites

• An AWS account subscribed to this product in AWS Marketplace.
• An EC2 key pair in your target region for SSH access.
• A security group allowing inbound TCP 22 (SSH) from your IP and TCP 80 (HTTP) from your users.
• A general-purpose instance. Recommended: m5.large — embedding models are light and run comfortably on 2 vCPU. Use m5.xlarge/m5.2xlarge (or the compute-optimised c5/c6i family) for higher throughput or larger models.

Connecting to your instance

Step 1 - Launch from the AWS Marketplace console

1. Open the product page in AWS Marketplace and choose Continue to Subscribe, then Continue to Configuration.
2. Select the Text Embeddings Inference 1.9 on Ubuntu 24.04 delivery option and your region, then Continue to Launch.
3. Choose a general-purpose instance type (m5.large or larger), your VPC/subnet, key pair and the security group described above, and launch.

Step 2 - Launch from the AWS CLI

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type m5.large \
  --key-name your-key \
  --security-group-ids sg-xxxxxxxx \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=tei}]'

Step 3 - Connect to your instance

ssh -i your-key.pem ubuntu@<instance-public-ip>

Step 4 - Confirm the services

systemctl is-active docker.service tei.service nginx.service
curl -s http://127.0.0.1/health -w ' HTTP %{http_code}\n'

Expected output:

active
active
active
 HTTP 200

Step 5 - Retrieve your password

sudo cat /root/tei-credentials.txt

# Text Embeddings Inference - generated on first boot by tei-firstboot.service
TEI_URL=http://<instance-public-ip>/
TEI_MODEL=BAAI/bge-small-en-v1.5
TEI_USERNAME=admin
TEI_PASSWORD=<your-unique-password>

Step 6 - Inspect the served model

The /health endpoint is open; everything else requires the password (user admin).

PASS=$(sudo grep '^TEI_PASSWORD=' /root/tei-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PASS http://127.0.0.1/info | jq -c '{model_id, max_input_length}'

{"model_id":"BAAI/bge-small-en-v1.5","max_input_length":512}

Step 7 - Generate embeddings (OpenAI-compatible)

PASS=$(sudo grep '^TEI_PASSWORD=' /root/tei-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PASS http://127.0.0.1/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"bge-small-en-v1.5","input":["the quick brown fox","retrieval augmented generation"]}' \
  | jq -c '{model, count:(.data|length), dim:(.data[0].embedding|length)}'

{"model":"BAAI/bge-small-en-v1.5","count":2,"dim":384}

The native TEI endpoint /embed returns the raw vector array directly:

PASS=$(sudo grep '^TEI_PASSWORD=' /root/tei-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PASS http://127.0.0.1/embed -H 'Content-Type: application/json' \
  -d '{"inputs":"hello world"}' | jq -c '{dim:(.[0]|length)}'

{"dim":384}

From your own machine, target the instance's public IP instead of 127.0.0.1. The /embed (native) and /v1/embeddings (OpenAI-compatible) endpoints both require HTTP Basic Auth.

Step 8 - Use the OpenAI SDK

from openai import OpenAI
import base64
auth = base64.b64encode(b"admin:<your-unique-password>").decode()
client = OpenAI(base_url="http://<instance-public-ip>/v1", api_key="unused",
                default_headers={"Authorization": "Basic " + auth})
r = client.embeddings.create(model="bge-small-en-v1.5", input=["hello world"])
print(len(r.data[0].embedding))

Feed the vectors into a vector database such as Weaviate or Chroma to build a retrieval-augmented-generation pipeline.

Serving a different model

TEI serves the model named in its environment file. Edit and restart:

sudo sed -i 's#^MODEL=.*#MODEL=BAAI/bge-large-en-v1.5#' /etc/tei/tei.env
sudo systemctl restart tei.service

The model is downloaded into /var/lib/tei/data on start. TEI also serves reranking models (query the /rerank endpoint) and sequence-classification models. Larger models benefit from a bigger instance (m5.xlarge or a c6i compute-optimised type).

Enabling HTTPS

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

certbot edits the nginx site at /etc/nginx/sites-available/cloudimg-tei to add the TLS listener and arranges automatic renewal.

Backup and maintenance

• Model weights live under /var/lib/tei/data on a dedicated EBS volume. Snapshot that volume to back up downloaded models, or re-download by model name.
• The password is in the nginx htpasswd file /etc/nginx/.tei.htpasswd; rotate it with sudo htpasswd /etc/nginx/.tei.htpasswd admin.
• Restart with sudo systemctl restart tei.service; logs: sudo journalctl -u tei.service and sudo docker logs tei.

Support

cloudimg provides 24/7 technical support for this image by email and chat, covering TEI deployment, model selection, instance sizing, batching and throughput tuning, the OpenAI-compatible API, TLS termination and scaling. Contact details are on the AWS Marketplace listing.