Vector Observability Pipeline on AWS User Guide

Overview

This image runs Vector, the high-performance, end-to-end open source observability data pipeline by Datadog. Vector collects, transforms and routes logs, metrics and traces from many sources, reshapes them through a configurable pipeline written in a purpose-built transform language (VRL), then routes the results to dozens of destinations. The image is delivered as a ready-to-use telemetry pipeline appliance: vector.service is enabled and starts on boot running a self-contained demo pipeline, so the moment an engineer connects over SSH the service is already processing events.

This is a headless observability-pipeline service. There is no web interface and no credentials to manage. Vector is a single self-contained binary driven entirely by a plain-text configuration file at /etc/vector/vector.yaml. Its optional management and GraphQL API is enabled but bound to loopback only (127.0.0.1:8686), so vector top and the health endpoint answer on the instance without ever being exposed to the network. The demo pipeline uses a self-contained event generator with no network listener, so the image opens no data port at all; the instance security group opens port 22 only, and you open whatever ingest or forward ports your own pipeline needs.

Vector is developed by Datadog, Inc. This image is not affiliated with or endorsed by Datadog.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Vector. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a few seconds after the instance state becomes Running and the status checks pass; vector.service is already running its demo pipeline by the time you can connect.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Vector Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens port 22 from your management network.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=vector}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

At login the welcome banner summarises the service and where to find things. The full first-boot details are in /root/vector-credentials.txt (a details file, not a secret — Vector has no login):

sudo cat /root/vector-credentials.txt

Step 4: Verify the Service

Confirm the installed version and that the daemon is active and running the demo pipeline.

vector --version
systemctl status vector.service

vector 0.57.0 (x86_64-unknown-linux-gnu 8832452 2026-07-14 20:58:30.491174540)

● vector.service - Vector
     Loaded: loaded (/usr/lib/systemd/system/vector.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/vector.service.d
             └─10-cloudimg.conf
     Active: active (running) since Tue 2026-07-21 06:09:08 UTC; 54s ago
       Docs: https://vector.dev
    Process: 27456 ExecStartPre=/usr/bin/vector validate /etc/vector/vector.yaml (code=exited, status=0/SUCCESS)
   Main PID: 27460 (vector)

The service is pinned to /etc/vector/vector.yaml by a systemd drop-in, and vector validate runs as an ExecStartPre so the daemon never starts on an invalid config.

Step 5: Watch the Demo Pipeline Move Events

The shipped pipeline is a complete, self-contained round-trip: a demo_logs source generates Apache-style log events, a remap transform parses each event with VRL and stamps a cloudimg-demo marker (proving the transform ran), and the events fan out to a console sink (visible in the journal) and a file sink on disk. Watch the console sink:

sudo journalctl -u vector -n 1 -o cat | jq .

{
  "env": "cloudimg-demo",
  "host": "175.116.65.213",
  "message": "PATCH /user/booperbot124 HTTP/2.0",
  "method": "PATCH",
  "path": "/user/booperbot124",
  "processed_by": "vector",
  "protocol": "HTTP/2.0",
  "size": 4520,
  "status": 307,
  "timestamp": "2026-07-21T06:10:02Z",
  "user": "cache_cowboy"
}

The same parsed events accumulate on disk in the file sink:

sudo tail -n 1 /var/lib/vector/demo-out/apache-*.log

The env: cloudimg-demo and method, status, host fields confirm the source, transform and sink are all working together end to end.

Step 6: The Optional GraphQL API (Loopback Only)

Vector's optional management API and GraphQL playground are enabled but bound to loopback only, so local tooling can read pipeline health and live throughput while nothing is reachable off the instance. Check the health endpoint:

curl -s -o /dev/null -w 'GET /health -> HTTP %{http_code}\n' http://127.0.0.1:8686/health

GET /health -> HTTP 200

For a live, interactive view of per-component throughput, run vector top (a full-screen terminal dashboard that reads the same loopback API). Press q to quit.

The listening sockets confirm the secure-by-default posture: the API is on loopback, and the only public port is SSH.

ss -tlnH | awk '{print $4}' | sort -u

0.0.0.0:22            # SSH (the only public port)
127.0.0.1:8686        # Vector GraphQL API (loopback only)
127.0.0.53%lo:53      # systemd-resolved stub (loopback)

Step 7: Edit the Pipeline

Vector reads its whole pipeline from one file, /etc/vector/vector.yaml. Replace the demo source and sinks with your own:

• Sources collect: file (tail logs), journald, syslog, socket, http_server, prometheus_scrape, kafka, aws_s3, and many more.
• Transforms process: remap (VRL — parse, enrich, redact, reshape), filter, sample, route, aggregate, dedupe.
• Sinks ship: elasticsearch, loki, clickhouse, kafka, aws_s3, aws_cloudwatch_logs, http, file, console, and dozens more.

Always validate a change before it goes live, then reload the service:

sudo vector validate /etc/vector/vector.yaml
sudo systemctl reload vector

If your new sink needs a credential (for example an API token for an HTTP or object-storage sink), supply it via an environment file at deploy time rather than baking it into the image. Note that Vector 0.57 disables ${VAR} environment interpolation by default as a security-hardening measure, so any such use is explicit and opt-in — see the Vector configuration reference. Remember to open whatever ingest or forward ports your real pipeline needs in the instance security group.

Step 8: Operating Notes

• Config lives at /etc/vector/vector.yaml; a pristine copy of the shipped demo is kept at /usr/local/share/vector/vector.yaml.sample.
• Restart vs reload: sudo systemctl reload vector applies config changes without a full restart; sudo systemctl restart vector fully restarts the daemon.
• Logs: the service log is sudo journalctl -u vector; the console sink writes structured JSON there.
• No baked secret: the shipped config carries no secrets, and Vector has no login. /root/vector-credentials.txt holds per-instance details only.
• Security group: only SSH 22 is open by default. The management API stays on loopback; expose it, or any ingest ports, only on your own terms.

Support

This image is provided by cloudimg with 24/7 technical support by email and chat. We can help with pipeline design, source and sink configuration, VRL transform authoring, parsing and enrichment, buffering and backpressure, routing to your observability backend and upgrade planning. Contact support@cloudimg.co.uk.

Vector is developed by Datadog, Inc. This image is not affiliated with or endorsed by Datadog. All product and company names are trademarks or registered trademarks of their respective holders.