LocalAI on AWS User Guide

Overview

This image runs LocalAI, the free, open source, self hosted alternative to the OpenAI API. LocalAI exposes the same OpenAI compatible REST API for chat completions, completions, embeddings and more, but runs entirely on your own infrastructure with no external API calls and no GPU required.

The LocalAI binary is installed under /opt/localai and runs as a dedicated unprivileged localai system account under a systemd service that starts it on boot and restarts it on failure. The models directory and the CPU inference backend live at /var/lib/localai, which is a dedicated, independently resizable EBS data volume. A small CPU friendly instruct model named smollm2-135m-instruct is pre pulled, so the chat API and the web interface work the moment the instance boots.

The LocalAI web UI ships with no built in authentication, so the inference server binds to the loopback interface only and is never exposed directly. An nginx reverse proxy publishes the web UI and the OpenAI compatible API on port 80. The web UI is protected by HTTP Basic authentication, and the API is gated by a LocalAI API key sent as a bearer token. Both the admin password and the API key are generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share credentials. They are written to /root/localai-aws-credentials.txt with mode 0600 so that only the root user can read them.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the web UI and the API
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for LocalAI. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.large or larger; LocalAI runs CPU inference, which benefits from memory and vCPUs. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the web UI and the API. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a few seconds after the instance state becomes Running and the status checks pass.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg LocalAI Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=localai}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

Step 4: Retrieve the Admin Password and API Key

The web UI admin password and the LocalAI API key are unique to your instance and were generated on first boot. Read them as root:

sudo cat /root/localai-aws-credentials.txt

The file lists the web UI URL, the admin user (admin) and the generated password, and the LocalAI API key, along with example API calls. Keep these credentials somewhere safe.

Step 5: Sign In to the Web UI

The LocalAI web UI is served on port 80 by nginx behind HTTP Basic authentication. In a browser, go to:

http://<instance-public-ip>/

You are prompted for credentials. Sign in as admin with the password from the credentials file. The pre pulled model is already loaded and selected, so you can start chatting immediately; nginx supplies the API key to the UI for you.

The Home dashboard shows live system status, the loaded model and quick links to install more models or open the chat assistant.

The Install Models page is the model gallery, where you can browse and install hundreds of additional open source models onto the instance with one click.

Step 6: Confirm LocalAI Is Running

Over SSH, confirm the inference server and the nginx proxy are active and that the ports are listening:

sudo systemctl is-active localai nginx
sudo ss -tlnp | grep -E ':(80|8080) '

You should see both services reported as active, the inference server listening on 127.0.0.1:8080 (loopback only), and nginx listening on port 80. Confirm the server reports ready through the public liveness probe:

curl -s -o /dev/null -w 'readyz: %{http_code}\n' http://127.0.0.1/readyz

Step 7: List the Available Models

The pre pulled model appears on the OpenAI compatible models endpoint. The API is gated by the LocalAI API key, so read the key from the credentials file and pass it as a bearer token. On the instance:

KEY=$(sudo grep '^localai.api.key=' /root/localai-aws-credentials.txt | cut -d= -f2-)
curl -s -H "Authorization: Bearer ${KEY}" http://127.0.0.1/v1/models

You should see smollm2-135m-instruct in the returned list.

Step 8: Call the Chat Completions API

LocalAI answers the OpenAI compatible chat completions endpoint. Send a prompt to the pre pulled model on the instance:

KEY=$(sudo grep '^localai.api.key=' /root/localai-aws-credentials.txt | cut -d= -f2-)
curl -s -H "Authorization: Bearer ${KEY}" -H 'Content-Type: application/json' \
  -X POST http://127.0.0.1/v1/chat/completions \
  -d '{"model":"smollm2-135m-instruct","messages":[{"role":"user","content":"Say hello in one short sentence."}],"max_tokens":32}'

The response is the standard OpenAI chat completion JSON, with the model's reply in choices[0].message.content.

From outside the instance, send the same request to http://<instance-public-ip>/v1/chat/completions with the same bearer token. Point any OpenAI client library at http://<instance-public-ip>/v1 and set its API key to your LocalAI API key, and existing applications work unchanged against the local model.

Step 9: Verify the API Key Gates the API

The OpenAI compatible API is protected by your per instance API key. A request with a wrong key is rejected. On the instance:

curl -s -o /dev/null -w 'wrong key: %{http_code}\n' \
  -H 'Authorization: Bearer sk-localai-wrongkey000000' http://127.0.0.1/v1/models

This returns 401, confirming the key is enforced. Keep the API key secret and rotate it by editing LOCALAI_API_KEY in /etc/localai/localai.env (and the matching nginx files under /etc/nginx/conf.d/) and restarting the services.

Step 10: Install More Models from the Gallery

Open the Install Models page in the web UI to browse the gallery, then click Install on any model to download and configure it automatically. You can also install a model over the API. The model and its configuration are written to the models directory on the data volume.

Step 11: The Data Volume

The models directory and the CPU inference backend live on a dedicated EBS volume mounted at /var/lib/localai. This keeps your models off the operating system disk and lets you resize or snapshot them independently. Confirm the mount with:

df -h /var/lib/localai

To grow the models store, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device. Larger models need more disk and memory, so size the instance and the volume to the models you intend to run.

Step 12: Enable HTTPS

The web UI and API are served over plain HTTP on port 80 by nginx. For production use, place them behind TLS. Obtain a certificate for your domain (for example with a managed certificate on an Application Load Balancer in front of the instance, or with Certbot installed on the instance), then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:8080 exactly as the bundled site does for port 80, keeping the HTTP Basic authentication and the API key in place. Restrict the security group so ports 80 and 443 are reachable only from the networks that use the service.

Step 13: Backup and Maintenance

Back up your models and configuration by snapshotting the /var/lib/localai EBS volume. Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade and reboot when a new kernel is installed; LocalAI and nginx start automatically on boot.

Support

This image is published and supported by cloudimg. Support covers deployment, model installation and configuration, the OpenAI compatible API, prompt templates, embeddings, the model gallery, TLS and performance tuning. Contact cloudimg through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.