dbt Core on AWS User Guide

Overview

This image runs dbt Core, the open source data transformation framework that lets analytics engineers build, test and document data models in plain SQL, applying software engineering best practices to the transform step of an ELT pipeline. The image is delivered as a ready-to-use, self-contained analytics engineering workstation: dbt Core is installed with the DuckDB adapter and a complete working sample project, so the moment an engineer connects over SSH they can run a full transform, test and document workflow with no external data warehouse and no setup.

This is a headless command-line tool. There is no web interface and no credentials to manage. dbt Core is installed in a dedicated Python virtual environment at /opt/dbt/venv, and a wrapper at /usr/local/bin/dbt puts dbt on every login user's PATH so no manual virtual-environment activation is needed. The DuckDB adapter ships preconfigured to materialise models into a local DuckDB database file, so the entire toolchain works with no Snowflake, BigQuery or Redshift account. The instance security group opens port 22 only.

The local DuckDB warehouse the adapter builds into lives on a dedicated, independently resizable EBS data volume mounted at /var/lib/dbt, at /var/lib/dbt/warehouse.duckdb. The built warehouse sits on durable storage rather than the operating system disk and can be resized or snapshotted independently.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for dbt Core. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.small or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a few seconds after the instance state becomes Running and the status checks pass; the dbt toolchain and the sample project are ready by the time you can connect.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg dbt Core Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens port 22 from your management network.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.small \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=dbt-core}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

A welcome banner prints the most useful commands, and sudo cat /root/dbt-core-info.txt shows the full getting-started notes: what is installed, the sample project path, the local DuckDB warehouse path, and how to connect your own data platform.

Step 4: Verify dbt and the DuckDB Adapter

dbt Core is installed in a virtual environment at /opt/dbt/venv, and the dbt wrapper on the system path runs it. Confirm the installed version, which is pinned to dbt Core 1.11.11 with the dbt-duckdb 1.10.1 adapter on this image:

dbt --version

You should see installed: 1.11.11 for Core and duckdb: 1.10.1 in the Plugins section. The bundled, self-contained sample project ships read only at /opt/dbt/jaffle_shop. List its files to see the structure: raw CSV seeds, staging models, a tested mart, and the profiles.yml that points the DuckDB adapter at the data volume:

find /opt/dbt/jaffle_shop -type f | sort

Step 5: Build the Sample Project

The sample project materialises into a local DuckDB warehouse on the data volume, so it needs no external data platform and no credentials. DBT_PROFILES_DIR is exported to the project for every login shell, so dbt finds the profile from anywhere. Work in your own writable copy so the shipped project stays pristine:

cp -r /opt/dbt/jaffle_shop ~/jaffle_shop
cd ~/jaffle_shop && pwd

Resolve any packages the project declares. The bundled project needs none, so this is a quick no-op that confirms the command works:

cd ~/jaffle_shop && dbt deps

Load the raw CSV seeds into the local DuckDB warehouse:

cd ~/jaffle_shop && dbt seed

Build the staging views and the mart tables in dependency order:

cd ~/jaffle_shop && dbt run

Run the schema and data tests on the models:

cd ~/jaffle_shop && dbt test

You can run all three steps in dependency order with a single command. dbt build seeds, runs and tests the whole project, and is the command you will most often use:

cd ~/jaffle_shop && dbt build

You should see every seed loaded, every model created and every test pass, ending with a Done. PASS=21 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=21 summary and Completed successfully.

Step 6: Generate Documentation and Query the Warehouse

dbt compiles your project into a browsable documentation site with model descriptions, column-level detail and a lineage graph. Generate the documentation catalog:

cd ~/jaffle_shop && dbt docs generate

To browse the documentation, run dbt docs serve and forward the port, or copy the generated target/ directory to a static host. The models dbt built live in the local DuckDB warehouse on the data volume. Query it with the bundled DuckDB Python to list the tables and views dbt materialised and read from a mart:

/opt/dbt/venv/bin/python -c "import duckdb; con = duckdb.connect('/var/lib/dbt/warehouse.duckdb', read_only=True); print(con.execute('select table_type, table_name from information_schema.tables order by 1, 2').fetchall()); print(con.execute('select customer_id, first_name, number_of_orders, customer_lifetime_value from customers order by customer_id limit 5').fetchall())"

You will see the raw_* seed tables, the stg_* staging views and the customers and orders mart tables, then the first rows of the customers mart with their order counts and lifetime value, confirming the toolchain works end to end.

Step 7: The Data Volume

The local DuckDB warehouse lives on a dedicated EBS volume mounted at /var/lib/dbt, at /var/lib/dbt/warehouse.duckdb. This keeps the built warehouse off the operating system disk and lets you resize or snapshot it independently. Confirm the mount with:

df -h /var/lib/dbt

The sample project's profile at /opt/dbt/jaffle_shop/profiles.yml points the DuckDB path at this volume, so every dbt run and dbt build materialises into durable storage. To grow the volume, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device.

Step 8: Connect Your Own Data Platform

To use dbt against your own warehouse, install the matching adapter into the environment and add a profile. Install an adapter such as Snowflake, BigQuery or Redshift into the dbt virtual environment:

sudo /opt/dbt/venv/bin/pip install dbt-snowflake

Then create a ~/.dbt/profiles.yml for your project that points at your warehouse with its connection details, and run dbt debug to confirm the connection resolves before running your models. The bundled DuckDB profile remains available for local development and prototyping, so you can build and test models locally before promoting them to your cloud warehouse.

Support

This image is published and supported by cloudimg. Support covers project structure, model materializations, tests and snapshots, packages and macros, incremental models, documentation, connecting to your data platform, and CI integration. Contact cloudimg through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.