Mage AI on AWS User Guide

Overview

This image runs Mage AI, the open source data pipeline tool for transforming and integrating data. Mage gives data engineers and analysts a browser based, notebook style editor for building, running and monitoring batch and streaming pipelines from modular blocks written in Python, SQL and R, with a built in scheduler, a run history and a data preview at every step.

Mage AI is installed from PyPI into a dedicated Python virtual environment at /opt/mage/venv and runs as a dedicated unprivileged mage system account under a systemd service that starts it on boot and restarts it on failure. The Mage project directory and the SQLite metadata and user database live under /var/lib/mage, which is a dedicated, independently resizable EBS data volume.

Mage's own user authentication is enabled, so the application binds to the loopback interface only and is never exposed directly. An nginx reverse proxy publishes the pipeline editor and the REST API on port 80. The owner password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/mage-ai-aws-credentials.txt with mode 0600 so that only the root user can read it.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the pipeline editor and the REST API
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Mage AI. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.medium or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the pipeline editor and the REST API. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation generates the owner password a short while after the instance state becomes Running and the status checks pass. The Python application takes a minute or two to finish its first start, so allow a little time before the editor answers.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Mage AI Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.medium \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=mage-ai}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

Step 4: Retrieve the Owner Password

The Mage owner password is unique to your instance and was generated on first boot. Read it as root:

sudo cat /root/mage-ai-aws-credentials.txt

The file lists the editor URL, the owner email (admin@cloudimg.local), the owner username (admin) and the generated password, along with a REST API example. Keep this password somewhere safe.

Step 5: Sign In to the Pipeline Editor

The Mage pipeline editor is served on port 80 by nginx. In a browser, go to:

http://<instance-public-ip>/

You are presented with a sign in page. Sign in with the email admin@cloudimg.local and the password from the credentials file. After signing in you land on the pipelines dashboard, which lists the pipelines in the project with their type and status.

Open a pipeline to enter the notebook style editor. The editor shows the block tree on the left and each block's code and configuration in the centre. Blocks are small pieces of Python, SQL or R wired together into a directed graph; you can run a block, preview its output, and add data loader, transformer and data exporter blocks from the palette.

The pipeline runs view lists every executed run with its trigger, status and timing, drawn from the built in scheduler and run history. Open a run to inspect its block level logs.

Step 6: Confirm Mage AI Is Running

Over SSH, confirm the application and the nginx proxy are active and that the ports are listening:

sudo systemctl is-active mage nginx

sudo ss -tlnp | grep -E ':(80|6789) '

You should see both services reported as active, the application listening on 127.0.0.1:6789 (loopback only), and nginx listening on port 80. You can also check the application's health endpoint over loopback:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6789/api/status

This returns 200 when Mage is ready.

Step 7: Use the REST API

The Mage REST API is served on the same port 80 behind the same login. First create a session to obtain a token. From a remote client, POST your owner email and password to /api/sessions (the public API key shown is the standard Mage client key and is not a secret). The shape of the request is:

curl -s -X POST http://<instance-public-ip>/api/sessions \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --data '{"session":{"email":"admin@cloudimg.local","password":"<password>"}}'

The response contains a session.token, which is a JSON Web Token. Mage's bearer authentication expects the OAuth access token that is carried inside that JWT, not the JWT string itself, so decode the token claim from the JWT payload and use that as the bearer. The following one liner signs in, extracts the access token and lists the pipelines, all over loopback on the instance:

ACCESS_TOKEN=$(curl -s -X POST http://127.0.0.1/api/sessions \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --data "{\"session\":{\"email\":\"admin@cloudimg.local\",\"password\":\"$(sudo grep '^mage.owner.pass=' /root/mage-ai-aws-credentials.txt | cut -d= -f2-)\"}}" \
  | python3 -c 'import sys,json,base64; jwt=json.load(sys.stdin)["session"]["token"]; p=jwt.split(".")[1]; p+="="*(-len(p)%4); print(json.loads(base64.urlsafe_b64decode(p))["token"])')
curl -s http://127.0.0.1/api/pipelines \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'

From a remote client, substitute http://<instance-public-ip> for http://127.0.0.1 and pass the owner password in place of the sudo grep lookup.

Step 8: Build and Schedule Pipelines

In the editor, create a new pipeline and add blocks from the palette. A typical batch pipeline has a data loader block that reads from a database, warehouse, file or API, one or more transformer blocks written in Python or SQL, and a data exporter block that writes the result to your destination. Run each block to preview its output as you build.

When the pipeline is ready, create a trigger to schedule it. Mage supports schedule triggers on a cron interval, event triggers and API triggers. The scheduler runs active triggers automatically and records every run in the run history with its logs and status.

Database and warehouse connections are configured in the project's io_config.yaml, which lives in the project directory on the data volume. Secrets such as passwords are encrypted with a per instance key held in the .mage_data directory, which is generated on first boot and never shipped in the image.

Step 9: The Data Volume

The Mage project directory and the SQLite metadata and user database live on a dedicated EBS volume mounted at /var/lib/mage. This keeps your pipelines and owner account off the operating system disk and lets you resize or snapshot them independently. Confirm the mount with:

df -h /var/lib/mage

To grow the project store, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device. The Mage project is at /var/lib/mage/cloudimg and the metadata database is the SQLite file /var/lib/mage/mage-metadata.db.

Step 10: Enable HTTPS

The pipeline editor and REST API are served over plain HTTP on port 80 by nginx. For production use, place them behind TLS. Obtain a certificate for your domain (for example with a managed certificate on an Application Load Balancer in front of the instance, or with Certbot installed on the instance), then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:6789 exactly as the bundled site does for port 80, keeping the WebSocket upgrade headers in place so the live editor and terminal work. Restrict the security group so ports 80 and 443 are reachable only from the networks that build and run pipelines.

Step 11: Backup and Maintenance

Back up your work by snapshotting the /var/lib/mage EBS volume, which captures the project directory, the pipelines and the metadata database in one consistent image. Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade and reboot when a new kernel is installed; Mage and nginx start automatically on boot. To manage users, sign in as the owner and open the workspace users settings, where you can add users and assign roles.

Support

This image is published and supported by cloudimg. Support covers deployment, pipeline design, block development, database and warehouse integration, scheduling and triggers, user management, TLS and runtime tuning. Contact cloudimg through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.