DKAN Data Portal 4.1.5 on AWS User Guide

Overview

This guide covers launching and using the DKAN Data Portal 4.1.5 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg BI and web analytics listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

DKAN is an open data portal built on Drupal. You publish datasets with DCAT-US metadata (title, description, publisher, keywords, licence and one or more distributions, which are the data files), DKAN imports CSV distributions into database tables called the datastore, and everything is served through a REST API: the metastore (the catalogue and its metadata), datastore queries and SQL, and harvesting, which copies the datasets of another portal's data.json catalogue into yours.

An open data portal is public by design: anyone who can reach it reads the catalogue and queries the data. The cloudimg image keeps that read path open and closes the rest:

• A Drupal site of your own. The image ships no Drupal site. Your instance's first boot installs one (drush site:install) and mints its hash salt, site UUID, private key, cron key, administrator password and TLS certificate. None of them is baked into the image. The password is written to a root only file.
• HTTPS from the first boot. nginx serves DKAN on port 443 with a certificate generated on your instance's first boot for its public address; port 80 redirects to HTTPS. Requests for a host name the instance does not serve get no answer.
• Visitors read, accounts write. Anonymous visitors can read the catalogue, the dataset pages, the metastore API and datastore queries. The datastore SQL endpoint (/api/1/datastore/sql) and every API request other than a read (GET) need a signed in Drupal account: nginx asks Drupal before it passes the request on, and an anonymous request gets 401.
• Rate limited per client. The sign in form, the JSON sign in (/user/login?_format=json), password reset and registration allow 10 requests a minute (bursts of 5) per client, then answer 429. Every write and every request that carries a password has its own per client limit. A client is an IPv4 address or an IPv6 /64, not an account, and forged X-Forwarded-For headers do not move the limit.
• No secret in settings.php. settings.php is owned by root, readable by the dkan account and not writable by it (root:dkan 0440). The hash salt lives in a root only file and reaches PHP through systemd. The code in /var/www/dkan is owned by root and read-only to PHP.
• Only Drupal's front controller runs. Every other *.php path (update.php, install.php and the rest) is a flat 404. Uploaded and public files never execute PHP and are served with nosniff and a sandboxing content security policy.
• A file boundary for PHP. DKAN accepts file:// distribution URLs, so PHP runs under open_basedir (the code, the data root and the CA bundle only), and drush and cron also run in a systemd sandbox. A dataset that names a server local file cannot read /etc, /proc or /root.
• No outbound reach inside AWS. PHP-FPM, drush, cron and MariaDB are refused, at the kernel level and on every boot, any connection to the EC2 instance metadata service, link-local, private (RFC 1918), carrier grade NAT and reserved addresses and your VPC's own address ranges. Outbound internet is allowed, so harvesting a remote data.json and importing remote CSV files work.
• A database with no password. MariaDB has no TCP listener. DKAN's database user signs in over the local socket as the dkan account (unix_socket), with only the privileges it needs on its own database. No database password exists.
• Less surface in Drupal. Drupal's automated cron (replaced by a systemd timer) and the contact form are uninstalled, and only an administrator can create accounts.
• Fully patched Ubuntu 24.04 LTS at build time.

First boot also harvests a small fictional demo catalogue so you can try DKAN straight away: two datasets about a made up Riverside City, each with a CSV file imported into the datastore. Step 11 shows how to remove them.

What is on the image: DKAN 4.1.5 on Drupal core 11.4.12 with drush 13.8.0 (/var/www/dkan, docroot /var/www/dkan/docroot), PHP 8.3 FPM, MariaDB 10.11, nginx, the data root /var/lib/dkan (files, private, php, tmp, config-sync, home), the demo catalogue in /usr/local/share/cloudimg/dkan-demo, the tools dkan-selftest, dkan-set-password, dkan-set-domain, dkan-drush and dkan-port-check, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu | Architecture: x86_64 | Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair, and a VPC subnet with internet access.
• A security group that allows TCP 22 (SSH) from your own network, TCP 443 (the portal and API) from the people who should reach it, and TCP 80 (redirects to HTTPS) if you want plain http:// links to work. A public open data portal opens 443 to everyone; the SQL endpoint and the admin still need a signed in account.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick DKAN Data Portal 4.1.5 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (a t3.medium serves a small catalogue to a few people), your key pair, a subnet and a security group as above. An Elastic IP keeps the address, and so the certificate's name, stable across a stop and start.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=dkan}]'

First boot installs the Drupal site and imports the demo data, which takes a few minutes. The portal answers once it has finished.

Step 2: Connect to your instance

Version	SSH user
DKAN Data Portal 4.1.5 on Ubuntu 24.04	ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot reads the instance's public address from EC2 instance metadata, starts MariaDB on a database directory made for this instance, generates the hash salt and the certificate, installs the Drupal site, harvests the demo catalogue and writes the credentials file. PHP-FPM and nginx start only after it has finished and the egress backstop is loaded:

systemctl is-active cloudimg-dkan-egress mariadb php8.3-fpm nginx
sudo test -f /var/lib/cloudimg/dkan-firstboot.done && echo FIRSTBOOT_DONE

All four units are active and the first boot marker is present:

active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on the network. PHP-FPM listens on a local socket only and MariaDB has no TCP listener at all:

sudo ss -lntpH | awk '{print $4, $6}' | sed -E 's/users:\(\("([^"]+)".*/\1/' | grep -v '127.0.0.5' | sort -u
dkan-port-check --running

0.0.0.0:22 sshd
0.0.0.0:443 nginx
0.0.0.0:80 nginx
[::]:22 sshd
[::]:443 nginx
[::]:80 nginx
DKAN_PORT_CHECK_OK

dkan-port-check fails if anything other than ports 22, 80 and 443 (and the local DNS resolver) is listening, for example MariaDB on 3306.

Drupal cron runs every 15 minutes from a systemd timer as the dkan account; DKAN's harvest and datastore import queues run from it:

systemctl is-enabled dkan-cron.timer
systemctl is-active dkan-cron.timer

enabled
active

The installed versions:

sudo jq -r '.packages[] | select(.name == "drupal/dkan" or .name == "drupal/core" or .name == "drush/drush") | "\(.name) \(.version)"' /var/www/dkan/composer.lock
sudo dkan-drush status --fields=drupal-version,php-version,bootstrap

drupal/core 11.4.12
drupal/dkan 4.1.5
drush/drush 13.8.0
Drupal version   : 11.4.12    
PHP version      : 8.3.6      
Drupal bootstrap : Successful 

dkan-drush runs drush as the dkan account (never root) in a sandbox; use it for every drush command on this instance.

Step 4: Get your URL and password

sudo cat /root/dkan-credentials.txt

The file is readable by root only and holds the URL, the administrator's user name and password and the certificate's fingerprint:

# DKAN Data Portal 4.1.5 (Drupal 11.4.12, cloudimg) on this instance. Open DKAN_URL and sign in as DKAN_ADMIN_USER / DKAN_ADMIN_PASSWORD.
# The password is unique to this instance. Change it: sudo dkan-set-password (or --stdin to choose one).
# The certificate is self-signed for this instance: compare its SHA-256 fingerprint with DKAN_TLS_SHA256.
# The two Riverside City datasets are FICTIONAL demo data; delete them once your own catalogue is in place.
DKAN_URL=https://<public-ip>/
DKAN_HOST=<public-ip>
DKAN_ADMIN_USER=admin
DKAN_ADMIN_PASSWORD=<DKAN_ADMIN_PASSWORD>
DKAN_TLS_SHA256=<64 hex digits, colon separated>
DKAN_ADDRESS_SOURCE=imdsv2-public-ipv4
GUIDE=https://www.cloudimg.co.uk/guides/dkan-data-portal-4-1-5-on-ubuntu-24-04-aws/

DKAN_ADDRESS_SOURCE says where the address came from: the public IPv4 address from instance metadata (imdsv2-public-ipv4), or the private address when the instance has no public one (private-fallback).

The image includes a self test. Through nginx, pinned to this instance's certificate, it checks that port 80 redirects, that anonymous visitors can read the portal home page (the catalogue), the catalogue API, a demo CSV file and a datastore query, that anonymous SQL queries and seven kinds of API write are refused, that 14 paths such as update.php and settings.php answer 404, that a wrong password is refused, and that the administrator can sign in, open the dataset admin and the datastore import status page, and run a SQL query:

sudo dkan-selftest

dkan-selftest: portal + catalogue + CSV download + query open; anonymous SQL and 7 API writes refused; 14 paths 404; a wrong password refused; admin signs in, opens the dataset admin and import status and runs SQL
DKAN_SELFTEST_OK

Step 5: Sign in

Open DKAN_URL in your browser. The certificate is self signed for your instance, so the browser warns once. To check you are talking to your instance, open the certificate details and compare its SHA-256 fingerprint with DKAN_TLS_SHA256, then continue.

The portal's public pages need no sign in. To manage the portal, open /user/login (for example https://<public-ip>/user/login) and sign in as admin with DKAN_ADMIN_PASSWORD. Your browser keeps the session until you sign out.

Signed in, the admin toolbar has a DKAN menu. DKAN > Datasets (/admin/dkan/datasets) lists every dataset with its publication state and links to edit it, and is where you add new ones:

The rest of Drupal's admin (people, appearance, configuration) is under Manage in the same toolbar. Use People to add accounts for the people who publish datasets: visitors cannot register themselves.

Step 6: Browse the catalogue and use the API

The portal's home page is the public catalogue (/dataset/search): every published dataset with full text search and sorting by title or date. Choosing a dataset opens its page, with its description, theme, keywords, the additional metadata (publisher, contact, licence, coverage) and its distributions:

Each distribution link downloads the copy of the file this portal keeps (https://<public-ip>/sites/default/files/resources/...), so visitors never need to reach the original source. This image has no JavaScript data explorer installed (DKAN's optional dkan_js_frontend app is not enabled): visitors read rows through the datastore API below, and you follow each import signed in, on the Datastore Import Status page (/admin/dkan/datastore/status, under DKAN > Datastore Settings):

The same catalogue is served as JSON. Reading needs no account. On the instance (pinned to the instance's certificate, connecting to nginx locally under the instance's own name):

H=$(sudo sed -n 's/^DKAN_HOST=//p' /root/dkan-credentials.txt)
curl -s --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" \
  "https://$H/api/1/metastore/schemas/dataset/items" | jq -r '.[] | "\(.identifier)  \(.title)"'

5f0c6c3e-2b1d-4e8a-9c7f-1d2e3a4b5c60  Riverside City Air Quality 2025 (DEMO)
8a7b6c5d-4e3f-4a2b-8c1d-0e9f8a7b6c51  Riverside City Library Branches (DEMO)

The identifier is the dataset's id. The datastore query API reads the imported rows of a dataset's distribution by the dataset id and the distribution's position in it (0 is the first). The air quality dataset holds 24 rows, one per month and station:

H=$(sudo sed -n 's/^DKAN_HOST=//p' /root/dkan-credentials.txt)
curl -s --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" \
  "https://$H/api/1/datastore/query/5f0c6c3e-2b1d-4e8a-9c7f-1d2e3a4b5c60/0?limit=3" | \
  jq -c '.count, (.results[] | {month, station, pm25_ugm3, no2_ugm3})'

24
{"month":"2025-01","station":"Harbour Road","pm25_ugm3":"14.2","no2_ugm3":"31.5"}
{"month":"2025-01","station":"Northgate Park","pm25_ugm3":"9.8","no2_ugm3":"18.2"}
{"month":"2025-02","station":"Harbour Road","pm25_ugm3":"13.1","no2_ugm3":"29.8"}

From your computer, use DKAN_URL with the same paths, for example https://<public-ip>/api/1/metastore/schemas/dataset/items. The query API also takes offset, column lists, conditions and sorts; https://<public-ip>/api/1 describes every endpoint the portal serves (OpenAPI).

Step 7: Run SQL as a signed in user

The datastore SQL endpoint takes a query in DKAN's bracketed syntax, which names a distribution's table by the distribution's id (not the dataset's):

[SELECT * FROM <distribution-id>][LIMIT 2];

A dataset's distribution ids appear when you add ?show-reference-ids to its metastore URL:

H=$(sudo sed -n 's/^DKAN_HOST=//p' /root/dkan-credentials.txt)
curl -s --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" \
  "https://$H/api/1/metastore/schemas/dataset/items/5f0c6c3e-2b1d-4e8a-9c7f-1d2e3a4b5c60?show-reference-ids" | \
  jq -r '.distribution[0].identifier'

d40d2c1a-129a-5ed0-9f9f-e6198e9d2c06

The distribution id is generated on your instance's first boot, so yours differs from the one above. The dataset id is the same on every instance.

On this image the SQL endpoint needs a signed in account. Anonymously it answers 401:

H=$(sudo sed -n 's/^DKAN_HOST=//p' /root/dkan-credentials.txt)
DIST=$(curl -s --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" \
  "https://$H/api/1/metastore/schemas/dataset/items/5f0c6c3e-2b1d-4e8a-9c7f-1d2e3a4b5c60?show-reference-ids" | \
  jq -r '.distribution[0].identifier')
curl -s -o /dev/null -w 'SQL without signing in: %{http_code}\n' --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) \
  --resolve "$H:443:127.0.0.1" -G --data-urlencode "query=[SELECT * FROM $DIST][LIMIT 2];" "https://$H/api/1/datastore/sql"

SQL without signing in: 401

DKAN's API accepts HTTP basic authentication, so a script signs in by sending a Drupal user name and password with each request. Here the administrator's password is read from the credentials file and handed to curl as a config file on standard input, so it never appears on a command line:

H=$(sudo sed -n 's/^DKAN_HOST=//p' /root/dkan-credentials.txt)
DIST=$(curl -s --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" \
  "https://$H/api/1/metastore/schemas/dataset/items/5f0c6c3e-2b1d-4e8a-9c7f-1d2e3a4b5c60?show-reference-ids" | \
  jq -r '.distribution[0].identifier')
sudo sed -n 's/^DKAN_ADMIN_PASSWORD=/user = "admin:/p' /root/dkan-credentials.txt | sed 's/$/"/' | \
  curl -s -K - --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" \
  -G --data-urlencode "query=[SELECT * FROM $DIST][LIMIT 2];" "https://$H/api/1/datastore/sql" | \
  jq -c '.[] | {month, station, pm25_ugm3, o3_ugm3}'

{"month":"2025-01","station":"Harbour Road","pm25_ugm3":"14.2","o3_ugm3":"38.0"}
{"month":"2025-01","station":"Northgate Park","pm25_ugm3":"9.8","o3_ugm3":"44.1"}

The syntax also takes a column list, WHERE, ORDER BY and OFFSET, each in its own brackets, for example:

[SELECT month,pm25_ugm3 FROM <distribution-id>][WHERE station = "Harbour Road"][ORDER BY month DESC][LIMIT 3];

From your computer, curl -u admin asks for the password, and Python's requests takes auth=("<user>", password). Give scripts an account of their own rather than the administrator's (Step 5, People). Requests that carry a password are rate limited per client, so a script in a tight loop will meet 429.

Step 8: Add your own data

In the browser. Signed in, open DKAN > Datasets and add a new dataset. Fill in the title, description, publisher, contact and access level, and add a distribution whose download URL is the https:// address of a CSV file, with the format csv. Save it as published.

DKAN then fetches the CSV file into the portal's public files and imports it into the datastore. This runs from the import queues, which cron works through every 15 minutes. To run them now:

sudo dkan-drush queue:run localize_import
sudo dkan-drush queue:run datastore_import
sudo dkan-drush queue:run post_import

To import one distribution directly, give its id (from ?show-reference-ids, Step 7):

sudo dkan-drush dkan:datastore:import <distribution-id>

The dataset's rows are then available to the query and SQL APIs (Steps 6 and 7).

Harvest another catalogue. DKAN can copy every dataset from a DCAT-US data.json catalogue, such as another open data portal's. Register the source as a harvest plan with an id of your choice, run it, then work through the import queues as above:

sudo dkan-drush dkan:harvest:register --identifier=<plan-id> --extract-uri=https://<other-portal>/data.json
sudo dkan-drush dkan:harvest:run <plan-id>
sudo dkan-drush queue:run localize_import
sudo dkan-drush queue:run datastore_import

sudo dkan-drush dkan:harvest:list lists the plans. Run a plan again to pick up changes in its source.

Sources inside your network. PHP, drush and cron are refused private addresses and your VPC's ranges (Step 12), so a harvest or CSV URL on such a host fails. To allow one, list its network in /etc/cloudimg/dkan-egress-allow (one CIDR per line, inside 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 or fc00::/7) and run sudo systemctl reload cloudimg-dkan-egress. The instance metadata service can never be allowed.

Server local files. A file:// URL works only for files PHP may read: under /var/lib/dkan, which is the data root. Prefer an https:// URL.

Step 9: Use your own domain, or a new address

The certificate and the host names nginx and Drupal answer are set for the addresses the instance had on first boot: its public IPv4 address, public DNS name and private address. If you stop and start an instance that has no Elastic IP, its public address changes and the portal stops answering on it; run dkan-set-domain with the new address. To use a domain, point an A record (for example data.example.com) at the instance's public IP address, then:

sudo dkan-set-domain <your-domain>

This sets the name nginx accepts and Drupal trusts, generates a new self signed certificate for it, restarts nginx, rebuilds Drupal's caches and updates DKAN_URL, DKAN_HOST and DKAN_TLS_SHA256 in the credentials file. The first name is the one Drupal builds its links with. Add more names or addresses after it to serve them too (sudo dkan-set-domain <your-domain> <public-ip>); names you leave out are no longer served.

For a certificate your browser trusts, put the certificate chain for <your-domain> in /etc/ssl/dkan/dkan.crt and its private key in /etc/ssl/dkan/dkan.key (root only, mode 600) and run sudo systemctl reload nginx, or put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate that forwards to port 443 on the instance. Running dkan-set-domain again replaces both files with a new self signed pair.

Step 10: Change the administrator password

To generate a new random password (written to the credentials file as DKAN_ADMIN_PASSWORD):

sudo dkan-set-password

To choose your own (12 characters or more, read from standard input and not stored on the instance):

sudo dkan-set-password --stdin

Give another account's user name as the last argument to set that account's password instead (sudo dkan-set-password --stdin <user>). You can also change passwords in the browser under People, or on your own account page.

Step 11: Remove the demo datasets

The two Riverside City datasets are fictional. Once your own datasets are in place, delete them: in DKAN > Datasets, tick both and choose Delete from the actions, or delete each from its edit page. They came from a harvest plan named riverside_demo, which you can then remove so it is never run again:

sudo dkan-drush dkan:harvest:deregister riverside_demo

The demo files in /usr/local/share/cloudimg/dkan-demo are read-only and are not served.

Step 12: Security checks

Anonymous visitors read the portal and the catalogue; the SQL endpoint and API writes need an account; Drupal's scripts and settings are not served; a host name the instance does not serve gets no answer; plain HTTP redirects:

H=$(sudo sed -n 's/^DKAN_HOST=//p' /root/dkan-credentials.txt)
c() { curl -s -o /dev/null --cacert <(sudo cat /etc/ssl/dkan/dkan.crt) --resolve "$H:443:127.0.0.1" "$@"; }
c -w 'home page: %{http_code}\n' "https://$H/"
c -w 'catalogue API: %{http_code}\n' "https://$H/api/1/metastore/schemas/dataset/items"
c -w 'SQL without signing in: %{http_code}\n' "https://$H/api/1/datastore/sql?query=x"
c -w 'new dataset without signing in: %{http_code}\n' -X POST -H 'Content-Type: application/json' --data '{"title":"x"}' "https://$H/api/1/metastore/schemas/dataset/items"
c -w 'update.php: %{http_code}\n' "https://$H/update.php"
c -w 'install.php: %{http_code}\n' "https://$H/core/install.php"
c -w 'settings.php: %{http_code}\n' "https://$H/sites/default/settings.php"
c -w 'health check: %{http_code}\n' "https://$H/healthz"
curl -sk -o /dev/null -w 'unknown host name: %{http_code}\n' --resolve evil.example:443:127.0.0.1 https://evil.example/ || true
curl -s -o /dev/null -w 'http: %{http_code}\n' --resolve "$H:80:127.0.0.1" "http://$H/"

home page: 200
catalogue API: 200
SQL without signing in: 401
new dataset without signing in: 401
update.php: 404
install.php: 404
settings.php: 404
health check: 200
unknown host name: 000
http: 301

nginx, PHP-FPM, MariaDB and the cron service are blocked from the instance metadata service and reserved addresses, and the dkan account (PHP, drush and cron) is refused metadata and private addresses:

for u in nginx php8.3-fpm mariadb dkan-cron; do printf '%s ' "$u"; systemctl show -p IPAddressDeny --value "$u" | tr ' ' '\n' | sort | tr '\n' ' '; echo; done
sudo -u dkan curl -s -m 5 -o /dev/null -w 'metadata from the dkan account: %{http_code}\n' http://169.254.169.254/latest/meta-data/ || true
sudo -u dkan curl -s -m 5 -o /dev/null -w 'a private address from the dkan account: %{http_code}\n' http://10.0.0.1/ || true

nginx 168.63.129.16/32 169.254.169.254/32 192.0.0.0/24 198.18.0.0/15 2001::/32 2002::/16 224.0.0.0/4 240.0.0.0/4 64:ff9b:1::/48 64:ff9b::/96 ::ffff:0.0.0.0/96 fd00:ec2::254/128 
php8.3-fpm 168.63.129.16/32 169.254.169.254/32 192.0.0.0/24 198.18.0.0/15 2001::/32 2002::/16 224.0.0.0/4 240.0.0.0/4 64:ff9b:1::/48 64:ff9b::/96 ::ffff:0.0.0.0/96 fd00:ec2::254/128 
mariadb 168.63.129.16/32 169.254.169.254/32 192.0.0.0/24 198.18.0.0/15 2001::/32 2002::/16 224.0.0.0/4 240.0.0.0/4 64:ff9b:1::/48 64:ff9b::/96 ::ffff:0.0.0.0/96 fd00:ec2::254/128 
dkan-cron 168.63.129.16/32 169.254.169.254/32 192.0.0.0/24 198.18.0.0/15 2001::/32 2002::/16 224.0.0.0/4 240.0.0.0/4 64:ff9b:1::/48 64:ff9b::/96 ::ffff:0.0.0.0/96 fd00:ec2::254/128 
metadata from the dkan account: 000
a private address from the dkan account: 000

The private ranges and your VPC are refused by the egress backstop, which reads the VPC's ranges on every boot:

sudo iptables -S CLOUDIMG-DKAN-EGRESS | sed -n '1,4p'
cat /run/cloudimg/dkan-vpc-cidrs

-N CLOUDIMG-DKAN-EGRESS
-A CLOUDIMG-DKAN-EGRESS -d 127.0.0.0/8 -o lo -j RETURN
-A CLOUDIMG-DKAN-EGRESS -d 169.254.0.0/16 -p tcp -j REJECT --reject-with tcp-reset
-A CLOUDIMG-DKAN-EGRESS -d 169.254.0.0/16 -j REJECT --reject-with icmp-admin-prohibited
172.31.0.0/16

PHP-FPM runs DKAN under a file boundary and cannot start programs:

sudo grep -E '^php_admin_value\[(open_basedir|disable_functions)\]' /etc/php/8.3/fpm/pool.d/dkan.conf

php_admin_value[open_basedir] = /var/www/dkan/:/var/lib/dkan/:/etc/ssl/certs/:/usr/share/ca-certificates/
php_admin_value[disable_functions] = exec,passthru,system,shell_exec,popen,proc_open,pcntl_exec,dl

MariaDB has networking switched off, and DKAN's database user signs in by socket with privileges on its own database only:

sudo mariadb -N -B -e "SELECT @@skip_networking"
sudo mariadb -N -B -e "SHOW GRANTS FOR 'dkan'@'localhost'"

1
GRANT USAGE ON *.* TO `dkan`@`localhost` IDENTIFIED VIA unix_socket
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON `dkan`.* TO `dkan`@`localhost`

The credentials, the hash salt and the TLS key are readable by root only, and settings.php cannot be changed by PHP:

sudo stat -c '%a %U:%G %n' /root/dkan-credentials.txt /etc/dkan/dkan.env /etc/ssl/dkan/dkan.key /var/www/dkan/docroot/sites/default/settings.php

600 root:root /root/dkan-credentials.txt
600 root:root /etc/dkan/dkan.env
600 root:root /etc/ssl/dkan/dkan.key
440 root:dkan /var/www/dkan/docroot/sites/default/settings.php

Other notes:

• Rate limits. The sign in form, JSON sign in, password reset and registration allow 10 requests a minute per client with a burst of 5, then answer 429 until the minute passes. Writes, requests with a password and all requests to Drupal have higher per client limits. A client is an IPv4 address or an IPv6 /64, so many people behind one NAT address share a limit.
• Client addresses. nginx ignores any X-Forwarded-For or X-Real-IP a client sends; Drupal sees the address it connected from. The access log keeps no query strings, so no datastore query or SQL is logged.
• Cross site requests. nginx refuses a request that changes state when it comes from another site's page. Scripts that send no Origin header are not affected.
• Response headers. nginx sends HSTS, a content security policy, X-Content-Type-Options: nosniff and X-Frame-Options: SAMEORIGIN, and no nginx, PHP or Drupal version.
• NVMe host identity. No NVMe-oF host identity ships in the image. If you install nvme-cli, its host NQN and host ID are your instance's own.
• Who can reach it. Keep port 22 open only to your own networks.

Step 13: Back up

The portal's state is the dkan database and the files under /var/lib/dkan (uploads, fetched distributions and private files). To copy both to your computer:

ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo mysqldump --single-transaction dkan | gzip' > dkan-db.sql.gz
ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo tar -C /var/lib/dkan -czf - files private' > dkan-files.tar.gz

The datastore tables are in the same database, so the dump includes the imported rows. An EBS snapshot of the instance's volume is also a complete backup.

Step 14: Keep it up to date, and logs

To apply Ubuntu updates now (this includes PHP, MariaDB and nginx from Ubuntu's archive):

sudo apt-get update && sudo apt-get upgrade -y

DKAN, Drupal core and drush are installed from a fixed, audited Composer lock, and Composer is not left on the instance. New DKAN and Drupal releases arrive as new versions of this listing, built and checked the same way; to move to one, launch the new version and harvest or re-create your datasets on it, or restore your database. Updating the code in place with Composer is possible but is your own maintenance: take a backup first, plan a maintenance window, then run sudo dkan-drush updatedb and sudo dkan-drush cache:rebuild. Read the DKAN release notes before you move.

Logs:

• First boot: journalctl -u dkan-firstboot
• nginx: /var/log/nginx/access.log and /var/log/nginx/error.log
• PHP-FPM: journalctl -u php8.3-fpm and /var/log/php8.3-fpm.log
• Cron, harvests and imports: journalctl -u dkan-cron
• Drupal's own log: Reports > Recent log messages in the admin, or sudo dkan-drush watchdog:show

If the portal does not answer after a stop and start, check whether the public address changed (Step 9). If an import never finishes, run the queues by hand (Step 8) and read Drupal's log.

Support and licensing

DKAN and Drupal are open source under the GNU General Public License, version 2 or later; some of DKAN's libraries are GPL-3.0, so the combined work is distributed under GPL-3.0 terms. Their code is as released; only test suites, development files and sample keys inside vendored tests are left out, and SOURCE.txt lists what was removed. The licence text, where to get the source and the licence of every installed package are in /usr/share/doc/dkan:

ls /usr/share/doc/dkan

LICENSE.txt
SOURCE.txt
dependency-licences.txt

The source code is at drupal.org/project/dkan and github.com/GetDKAN/dkan. DKAN and Drupal are not endorsed by or affiliated with cloudimg. There is no per user licence fee. The AMI price covers cloudimg's build, hardening, first boot setup and support.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk
• Support: 24/7, with a 24 hour response SLA

The other versions on this listing are on the BI and web analytics guide.