Developer Tools AWS

Concourse CI 8.3.3 on AWS User Guide

| Product: Concourse CI 8.3.3 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Concourse CI 8.3.3 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg Hardened Apache Subversion (SVN) & CI/CD Tools listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Concourse is an open source continuous integration and delivery system. Everything is a pipeline written in YAML: resources (a Git repository, an image, an S3 bucket, a timer) feed jobs, and every task in a job runs in a fresh container. You drive it from the web interface and from fly, its command line client.

The cloudimg image adds:

  • A complete single-node install. The Concourse web node (API, web interface and the TSA worker gateway), one containerd worker and PostgreSQL 16 run on the instance as systemd services. The database is reached over its local socket only and has no network listener and no password.
  • Nothing shared between instances. The image ships no password, key or certificate. First boot creates the administrator password, the session signing key, the TSA host key, the worker key, the database encryption key and the HTTPS certificate on your instance. Concourse, the worker and nginx do not start until first boot has finished.
  • HTTPS from the first boot, and sign-in first. nginx serves Concourse on port 443 with a certificate generated on your instance; port 80 redirects to HTTPS. Anyone who is not signed in gets the sign-in page; the API answers them with 401 except for /api/v1/info and the fly download.
  • Task containers fenced in. Pipeline tasks run unprivileged, in a user namespace, and cannot reach the instance metadata service (so they cannot pick up the instance's IAM role), the instance itself over IPv4 or IPv6, or Concourse's own internal ports. They can reach the internet, which most pipelines need.
  • The web node talks to loopback only. The kernel drops every packet between concourse-web and anything but 127.0.0.1/::1, so its internal ports cannot be reached from your VPC or from a task, and a pipeline cannot use it to reach other machines in your network.
  • Worker APIs closed to local accounts. The worker's garden and baggageclaim APIs (and the TSA's tunnels to them) have no authentication of their own, so a kernel rule lets only root and the Concourse web node open them: no other account on the instance can use them to start a container.
  • A rate-limited sign-in. 30 sign-in requests a minute from one address in every spelling of the endpoint, and cross-origin state-changing requests are refused.
  • Metadata service blocked for every Concourse service, PostgreSQL, nginx and the helper commands; the AMI requires IMDSv2.
  • fly preinstalled, and logged in without typing a password. sudo concourse-fly-login saves a fly target for you.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on. Concourse and fly are the official 8.3.3 release, checked against upstream's published SHA-256 sums.

Trademark notice. Concourse is an open source project released under Apache-2.0. "Concourse" is used here only to describe the software this image runs. cloudimg is not affiliated with, endorsed by, or sponsored by the Concourse project.

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (tasks pull their container images, and most pipelines fetch code).
  • A security group that allows TCP 443 (the web interface and fly) and TCP 80 (redirects to HTTPS) from the networks that should reach them, and TCP 22 (SSH) from your own networks only. Concourse needs no other inbound port: the web node, the TSA, the worker and PostgreSQL listen on loopback only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Concourse CI 8.3.3 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose t3.large or larger (builds run on the instance itself; m5.xlarge suits a busy team), your key pair and a security group as above. The root volume is 20 GiB; give it more if your pipelines build large images or caches.

Step 2: Connect to your instance

Version SSH user
Concourse CI 8.3.3 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot takes about a minute. Then the web node, the worker, PostgreSQL, nginx and the first boot unit are active:

systemctl is-active concourse-web concourse-worker postgresql@16-main nginx concourse-ci-firstboot

active
active
active
active
active

nginx is the only service listening on the network (ports 80 and 443, plus SSH on 22):

sudo ss -tlnp | awk 'NR==1 || /nginx|sshd|concourse/' | awk '{print $1, $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g'

State Local Peer
LISTEN 127.0.0.1:8888 users:(("concourse"))
LISTEN 0.0.0.0:22 users:(("sshd"),("systemd"))
LISTEN 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
LISTEN 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
LISTEN 127.0.0.1:2222 users:(("concourse"))
LISTEN 127.0.0.1:8079 users:(("concourse"))
LISTEN 127.0.0.1:8080 users:(("concourse"))
LISTEN 127.0.0.1:7777 users:(("concourse"))
LISTEN 127.0.0.1:7776 users:(("concourse"))
LISTEN 127.0.0.1:7787 users:(("concourse"))
LISTEN 127.0.0.1:7788 users:(("concourse"))
LISTEN [::]:22 users:(("sshd"),("systemd"))
LISTEN [::]:80 users:(("nginx"),("nginx"),("nginx"))
LISTEN *:33073 users:(("concourse"))
LISTEN [::]:443 users:(("nginx"),("nginx"),("nginx"))
LISTEN *:37725 users:(("concourse"))

The web node listens on 127.0.0.1:8080 (API and web interface) and 127.0.0.1:2222 (the TSA, where the worker registers); the worker's garden and baggageclaim APIs are on 7777 and 7788. The two *:<random> ports are the TSA's tunnels to the worker: Concourse always opens them on all addresses and has no setting to change that, so this image fences them in the kernel instead (the web node exchanges packets with loopback only; Step 11 shows the rule). The port numbers change every time the worker reconnects. Garden, baggageclaim and the tunnels have no authentication of their own, so a second kernel rule lets only root and the web node open them (also in Step 11).

The API's info endpoint is the one call that answers without a session; the pipeline list does not, and port 80 redirects to HTTPS:

curl -s --cacert /etc/nginx/ssl/concourse.crt https://127.0.0.1/api/v1/info; echo
curl -s -o /dev/null -w 'anonymous pipeline list: %{http_code}\n' --cacert /etc/nginx/ssl/concourse.crt https://127.0.0.1/api/v1/teams/main/pipelines
curl -s -o /dev/null -w 'port 80: %{http_code}\n' http://127.0.0.1/

{"version":"8.3.3","worker_version":"3.0","feature_flags":{"build_rerun":false,"cache_streamed_volumes":false,"global_resources":false,"resource_causality":false},"external_url":"https://<public-ip>","cluster_name":"cloudimg"}
anonymous pipeline list: 401
port 80: 301

Step 4: The release

/opt/concourse/bin/concourse --version
fly --version
cat /usr/share/doc/concourse-cloudimg/resource-types.txt

8.3.3
8.3.3
bosh-io-release 1.3.7
bosh-io-stemcell 1.7.2
git 1.24.1
github-release 1.14.3
hg 1.5.7
mock 0.14.8
pool 1.8.4
registry-image 1.19.1
s3 2.5.7
semver 2.0.5
time 1.11.6

These are the resource types built into the worker. Upstream's deprecated docker-image type is not shipped (its image carries packages with known, unfixed vulnerabilities); use registry-image, which fetches and pushes images without a Docker daemon.

Step 5: Get your sign-in details

The credentials file holds the address Concourse is served on and the administrator account, generated on this instance:

sudo cat /root/concourse-ci-credentials.txt

CONCOURSE_URL=https://<public-ip>/
#
CONCOURSE_ADMIN_USER=admin
CONCOURSE_ADMIN_PASSWORD=<28 letters and digits>
#
# admin is the main team's owner (the Concourse administrator).
#   fly -t ci login -c https://<public-ip>/ -u admin --ca-cert concourse.crt

The password comes from the kernel's random number generator on first boot and is known to nobody else. Concourse stores only a bcrypt hash of it, in /etc/concourse/web.env, which only root can read. The same file holds CONCOURSE_ENCRYPTION_KEY, which encrypts pipeline configurations and credentials in the database: keep a copy of it with your database backups (Step 13).

The external URL is the instance's public IPv4 address, read from the instance metadata service at first boot. Step 10 shows how to serve Concourse under your own DNS name.

Step 6: Check the certificate

The HTTPS certificate is self-signed and was generated on your instance. Note its fingerprint here, over SSH, and compare it with what your browser shows before you accept it:

openssl x509 -in /etc/nginx/ssl/concourse.crt -noout -subject -enddate -fingerprint -sha256 -ext subjectAltName | sed -E 's/[0-9]+(\.[0-9]+){3}/<ip>/g; s/ec2-[0-9-]+/ec2-<ip>/g; s/ip-[0-9-]+/ip-<ip>/g; s/=([0-9A-F]{2}:){31}[0-9A-F]{2}/=<64 hex digits>/'

subject=CN = <ip>, O = cloudimg
notAfter=Feb  9 19:01:38 2029 GMT
sha256 Fingerprint=<64 hex digits>
X509v3 Subject Alternative Name: 
    IP Address:<ip>, IP Address:<ip>, DNS:ec2-<ip>.compute-1.amazonaws.com, DNS:ip-<ip>.ec2.internal, IP Address:<ip>, DNS:localhost

(The sed only hides this example's addresses; run the openssl command on its own to see yours.) It names the instance's public and private addresses and DNS names, and 127.0.0.1.

Step 7: Sign in to the web interface

Open https://<public-ip>/ in your browser and accept the certificate once you have checked its fingerprint. You see the sign-in page:

The Concourse sign-in page

Sign in as admin with the password from Step 5. The dashboard lists the pipelines of every team you belong to; it is empty until you set one (Step 8 sets hello):

The Concourse dashboard with the hello pipeline in the main team

Signing in comes first on this image: upstream Concourse lets anonymous visitors browse pipelines that a team has exposed, and shows them an empty dashboard otherwise. Here nginx sends every visitor without a session to the sign-in page, so exposed pipelines and status badges are visible to signed-in users only.

Step 8: Use fly on the instance and run a pipeline

fly is installed at /usr/local/bin/fly. concourse-fly-login gets a token for admin from Concourse's token endpoint without the password appearing on a command line, in your shell history or on the screen, and saves it as the target ci in your ~/.flyrc (mode 0600, with this instance's certificate). The token is valid for 24 hours; run the command again when it expires.

sudo concourse-fly-login
fly -t ci workers

fly target 'ci' -> https://127.0.0.1 (team main, admin) saved in /home/ubuntu/.flyrc; try: fly -t ci pipelines
ip-172-31-81-63  3           linux     none  none  running  3.0      2m49s

Set a pipeline with one job, unpause it and run the job, following its output:

WORKDIR="$(mktemp -d)"
cat > "${WORKDIR}/hello.yml" <<'YML'
jobs:
- name: hello
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: public.ecr.aws/docker/library/alpine, tag: "3.20"}
      run:
        path: sh
        args: [-c, 'echo "hello from Concourse: $(cat /etc/alpine-release) on $(uname -m)"']
YML
fly -t ci set-pipeline -p hello -c "${WORKDIR}/hello.yml" -n
fly -t ci unpause-pipeline -p hello
fly -t ci trigger-job -j hello/hello --watch
rm -rf "${WORKDIR}"

pipeline created!
you can view your pipeline here: https://127.0.0.1/teams/main/pipelines/hello
unpaused 'hello'
started hello/hello #1

initializing
initializing check: image
selected worker: ip-172-31-81-63
fetching public.ecr.aws/docker/library/alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
selected worker: ip-172-31-81-63
running sh -c echo "hello from Concourse: $(cat /etc/alpine-release) on $(uname -m)"
hello from Concourse: 3.20.10 on x86_64
succeeded

The pipeline and its build are in the web interface:

The hello pipeline in the Concourse web interface

Build 1 of the hello job with the task's output

fly execute runs a one-off task without a pipeline. This one shows what a task can and cannot reach on this image:

WORKDIR="$(mktemp -d)"
cat > "${WORKDIR}/probe.yml" <<'YML'
platform: linux
image_resource:
  type: registry-image
  source: {repository: public.ecr.aws/docker/library/alpine, tag: "3.20"}
run:
  path: sh
  args:
  - -c
  - |
    echo "task user: $(id -u) (root inside the task's user namespace)"
    wget -q -T 5 -O /dev/null https://aws.amazon.com/ && echo "internet: reachable"
    wget -q -T 5 -O /dev/null http://169.254.169.254/latest/meta-data/ 2>/dev/null && echo "instance metadata: REACHABLE" || echo "instance metadata: blocked"
    wget -q -T 5 -O /dev/null "http://$(ip route | awk '/default/ {print $3}'):8080/" 2>/dev/null && echo "host port 8080: REACHABLE" || echo "host port 8080: blocked"
YML
fly -t ci execute -c "${WORKDIR}/probe.yml"
rm -rf "${WORKDIR}"

executing build 12 at https://127.0.0.1/builds/12
initializing
initializing check: image
selected worker: ip-172-31-81-63
INFO: found existing resource cache
selected worker: ip-172-31-81-63
task user: 0 (root inside the task's user namespace)
internet: reachable
instance metadata: blocked
host port 8080: blocked
succeeded

When you are done with the example, remove it:

fly -t ci destroy-pipeline -p hello -n

!!! this will remove all data for pipeline `hello`

`hello` deleted

Step 9: Use fly from your workstation

Download fly from your instance (the download is the one part of the server that needs no session) and the instance's certificate, then log in. Without -p, fly opens the sign-in page in your browser and receives the token when you sign in:

curl -k -o fly 'https://<public-ip>/api/v1/cli?arch=amd64&platform=linux' && chmod +x fly
scp -i <your-key.pem> ubuntu@<public-ip>:/etc/nginx/ssl/concourse.crt concourse.crt
./fly -t ci login -c https://<public-ip>/ -u admin --ca-cert concourse.crt

Use platform=darwin or platform=windows for macOS or Windows. Check the downloaded certificate's fingerprint against Step 6 before you rely on it; --ca-cert makes fly trust exactly that certificate, so you never need -k. fly -p <password> also works, but it puts the password on the command line, where other users of the machine and your shell history can see it.

Step 10: Your own DNS name and certificate

Point a DNS name at the instance (an Elastic IP keeps the address across stops), then run:

sudo concourse-set-hostname ci.your-domain.example

It sets Concourse's external URL (used in sign-in redirects, fly login and links), nginx's server name and the cross-origin check to the new name, generates a new certificate that names it, and restarts the web node. To use a certificate from your own CA instead, run the command first, then replace /etc/nginx/ssl/concourse.crt (the full chain) and /etc/nginx/ssl/concourse.key (mode 0600, owned by root) and reload nginx with sudo nginx -t && sudo systemctl reload nginx. Browsers and fly then need no certificate exception or --ca-cert.

Step 11: Security checks

The web node exchanges packets with loopback only, task containers' IPv6 traffic to and through the host is dropped, the worker refuses task traffic to the metadata addresses, and root cannot log in over SSH:

systemctl show -p IPAddressAllow -p IPAddressDeny concourse-web
systemctl show -p IPAddressDeny concourse-worker nginx postgresql@16-main
sudo ip6tables -S INPUT | grep concourse0
grep -o -- '--containerd-restricted-network [^ ]*' /etc/systemd/system/concourse-worker.service
grep -E '^CONCOURSE_(BIND_IP|CONTAINERD_ALLOW_HOST_ACCESS|CONTAINERD_DNS_PROXY_ENABLE)=' /etc/concourse/worker.env
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication) '

IPAddressAllow=::1/128 127.0.0.0/8
IPAddressDeny=0.0.0.0/0 ::/0
IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128

IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128

IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128
-A INPUT -i concourse0 -j DROP
--containerd-restricted-network 169.254.169.254/32
--containerd-restricted-network 168.63.129.16/32
CONCOURSE_BIND_IP=127.0.0.1
CONCOURSE_CONTAINERD_DNS_PROXY_ENABLE=false
CONCOURSE_CONTAINERD_ALLOW_HOST_ACCESS=false
permitrootlogin no
passwordauthentication no

Only root (the worker) can open garden and baggageclaim, and only root and the concourse account (the web node) can open the TSA's tunnel ports; any other account on the instance gets a refused connection. concourse-ci-local-guard.service applies the rule (IPv4 and IPv6) at every boot, before the web node and the worker start:

sudo iptables -S CLOUDIMG-CI-LOCAL
sudo -u nobody bash -c 'echo > /dev/tcp/127.0.0.1/7777' && echo reachable || echo refused

-N CLOUDIMG-CI-LOCAL
-A CLOUDIMG-CI-LOCAL -m owner --uid-owner 0 -j RETURN
-A CLOUDIMG-CI-LOCAL -p tcp -m multiport --dports 7777,7788 -j REJECT --reject-with tcp-reset
-A CLOUDIMG-CI-LOCAL -m owner --uid-owner 997 -j RETURN
-A CLOUDIMG-CI-LOCAL -p tcp -m tcp --dport 32768:60999 -j REJECT --reject-with tcp-reset
bash: connect: Connection refused
bash: line 1: /dev/tcp/127.0.0.1/7777: Connection refused
refused

Because the tunnel ports are picked from the kernel's ephemeral range (32768 to 60999), the rule covers that whole range on loopback for every account except root and concourse. If you run your own service on this instance for other local users, give it a loopback port below 32768.

What a task can do. Tasks run as root inside a user namespace, which maps to an unprivileged user on the instance, with no access to the instance's own services and no IAM role. A task or resource type with privileged: true is different: it runs as real root with the host's capabilities and devices, which amounts to root on the instance. Use it only for tasks you trust (building images with a Docker daemon is the usual reason) and prefer registry-image and rootless image builders.

Who can run code here. Anyone who can set a pipeline or run fly execute in a team can run arbitrary code on the worker. Only members of a team can do either, and only the main team's owners can create teams. Add people as members of their own team rather than sharing admin, and review pipelines that ask for privileged: true.

Opening the web node to the network. Because concourse-web talks to loopback only, features that make the web node itself call out (GitHub, GitLab or OIDC sign-in, or a Vault or other credential manager on another machine) need a deliberate exception. Allow just the addresses it must reach, for example a Vault server in your VPC:

sudo systemctl edit concourse-web.service
# add, then save:
[Service]
IPAddressAllow=10.0.12.34/32

Then sudo systemctl restart concourse-web. The metadata service stays blocked unless you allow it. Anything you allow can also reach the TSA's tunnel ports, so keep those addresses trusted, and keep your security group limited to 22, 80 and 443.

Step 12: Where things are

What Where
Pipelines, builds, teams and encrypted credentials (PostgreSQL 16) /var/lib/postgresql/16/main (database concourse)
Worker volumes and containers (caches; rebuilt as needed) /var/lib/concourse/worker
Web node settings, password hash and encryption key /etc/concourse/web.env (root only)
Worker settings /etc/concourse/worker.env
Keys (session signing, TSA host, worker) /etc/concourse/keys/
The administrator's password /root/concourse-ci-credentials.txt
HTTPS certificate and key /etc/nginx/ssl/
Concourse and fly /opt/concourse, /usr/local/bin/fly

Logs are in the journal: sudo journalctl -u concourse-web -u concourse-worker.

Step 13: Backups and upgrades

Back up the database and the encryption key together; one is useless without the other:

sudo -u postgres pg_dump -Fc concourse > concourse-$(date +%F).dump
sudo grep '^CONCOURSE_ENCRYPTION_KEY=' /etc/concourse/web.env > concourse-encryption-key.txt

Copy both off the instance (to S3, for example) and keep them private. The worker's volumes need no backup: Concourse fetches resources again when it needs them.

Ubuntu, PostgreSQL and nginx receive security updates through unattended upgrades. Concourse is not upgraded automatically; to move to a newer release, take a backup, stop concourse-worker and concourse-web, replace /opt/concourse and /usr/local/bin/fly with the new release, and start the services again (the web node migrates the database on start).

Step 14: Security recommendations

  • Keep TCP 443 and 80 open only to your team's networks; build logs and pipeline configurations describe your systems in detail.
  • Replace the self-signed certificate with your own (Step 10) before you share the address widely.
  • Give each team its own Concourse team and keep admin for administration; refuse privileged: true tasks you have not reviewed.
  • Keep secrets in Concourse's credential management, not in pipeline files: they are encrypted at rest with your encryption key.
  • Keep the instance patched, and upgrade Concourse deliberately after a backup.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk or https://www.cloudimg.co.uk/support. Include the instance type, the Region and the output of Steps 3 and 11.

Concourse is released under Apache-2.0; its licence and notice are at /usr/share/doc/concourse-cloudimg/LICENSE.md and NOTICE.md. PostgreSQL and nginx are Ubuntu archive packages under their own licences (see /usr/share/doc/<package>/copyright). cloudimg is not affiliated with, endorsed by, or sponsored by the Concourse project.