Mage AI, the open source data pipeline tool for transforming and integrating data, preinstalled as a system service with the browser based pipeline editor published on port 80 behind Mage's own user authentication via an nginx proxy. A unique owner password is generated on first boot. Backed by 24/7 cloudimg support.
Overview
Mage AI is an open source data pipeline tool for transforming and integrating data. It gives data engineers and analysts a browser based notebook style editor for building, running and monitoring batch and streaming pipelines from modular blocks written in Python, SQL and R, with a built in scheduler, a run history and a data preview at every step. This image delivers Mage AI fully installed and configured as a system service, so a production grade pipeline editor is running within minutes of launch.
Application Stack
Mage AI installed from PyPI into a dedicated Python virtual environment and run by a dedicated unprivileged service account. The Mage project directory and the SQLite metadata and user database stored on a dedicated data disk so your pipelines and owner account are independently resizable and survive instance replacement. A systemd service that starts Mage on boot and restarts it on failure. An nginx reverse proxy that publishes the pipeline editor on port 80 with WebSocket support for the live editor and terminal.
Pipelines As Code
Build pipelines from modular blocks, each a small piece of Python, SQL or R, wired together into a directed graph. Load data from databases, warehouses, files and APIs, transform it with pandas or SQL, preview the output of every block as you write it, then export to your destination. Pipelines are stored as code and configuration so they can be exported, version controlled and promoted between environments. The built in scheduler runs pipelines on a cron or event trigger and the run history shows every run with its logs and status.
Secure First Boot
On the first boot of your instance a one shot service generates a fresh owner password, unique to that instance, configures it as the Mage owner account and writes the password to a root only file. Mage binds to loopback only and the editor is never exposed without authentication. No shared or default credentials ship in the image.
Ready To Use
The pipeline editor is served on port 80 through nginx. Sign in with the generated owner credentials to build pipelines, run blocks, schedule triggers, browse run history and manage users. The Mage REST API is available behind the same login for automation.
cloudimg Support
24/7 technical support by email and chat. Help with deployment, pipeline design, block development, database and warehouse integration, scheduling and triggers, user management, TLS and runtime tuning.
Use Cases
Batch and streaming ETL and ELT pipelines. Data integration between databases, warehouses and APIs. Scheduled transformation jobs feeding dashboards and reports. Notebook style data engineering for analysts. A self hosted alternative to hosted pipeline orchestration services.
All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.