DKAN Open Data Portal on Ubuntu 24.04 on Azure User Guide

Overview

DKAN is an open source open data portal built on Drupal. It is what an organisation runs when it needs to publish data to the public and be held to a standard while doing it. Every dataset is described with DCAT metadata through a schema-driven authoring form, so the catalog is machine readable and can be crawled, aggregated and cited rather than being a folder of spreadsheets. Alongside the catalog DKAN runs a datastore: it loads the tabular contents of a published file into the database and exposes it through a REST query API and a SQL endpoint, so an application or a notebook reads filtered, paged data directly instead of downloading whole files. A harvester can pull an entire catalog from another portal on a schedule, which is how regional and national indexes stay in step with the sources that feed them.

This cloudimg image delivers DKAN fully installed on Ubuntu 24.04 — DKAN 2.23.2 on Drupal core 11.4.4, served by Apache 2.4 with PHP 8.3 and backed by MySQL 8.0, all from the Ubuntu 24.04 main and universe repositories with no third-party APT repositories. The site is installed onto your own VM at first boot, so you land on a working portal and administration UI rather than an installer. Backed by 24/7 cloudimg support.

What is included:

• DKAN 2.23.2 (GPL-2.0) installed as a Composer project at /var/www/dkan, with the document root at /var/www/dkan/docroot so composer.json, composer.lock and vendor/ are never inside the served tree
• Drupal core 11.4.4, pinned via drupal/core-recommended so every core dependency is at an exact version
• The DKAN metastore, datastore, harvest, search, faceted browsing, JSON form widget and data dictionary widget modules, all enabled
• datastore_mysql_import, so CSV distributions are loaded with MySQL's native LOAD DATA rather than row batches
• Apache 2.4 with mod_rewrite, MySQL 8.0 bound to loopback, and PHP 8.3 with the mysql, curl, gd, mbstring, xml, zip, intl, bcmath, apcu and opcache extensions
• Drush 13 at /usr/local/bin/drush, and a dkan-cron.timer that runs Drupal cron every five minutes so harvest and datastore imports actually advance
• A per-VM MySQL root password, application database password, Drupal administrator password and Drupal hash_salt, all generated on first boot into a root-only file. No shared or default credential ships in the image
• 24/7 cloudimg support

DKAN version note

DKAN 3.0.x exists and is newer by date, but it is not the production line: upstream's own 3.0.0 release notes state that there will be "few to no additional 3.x releases" and that the release exists "only as a stepping-stone in the 2-4 upgrade path". DKAN 2.23.2 is the release upstream marks as Latest, and it is what this image ships.

Prerequisites

An active Azure subscription, an SSH key, and a VNet with a subnet. Standard_B2s (2 vCPU, 4 GB RAM) is the recommended size and is sufficient for a catalog of a few hundred datasets; move to a D2s or D4s size for large datastore imports, many concurrent API consumers, or heavy harvesting.

Allow inbound TCP 80 from the addresses that should reach the portal, and TCP 22 for administration.

Step 1 — Deploy from the Azure Marketplace

In the Azure portal choose Create a resource, search for DKAN Data Portal on Ubuntu 24.04 by cloudimg, and select Create. Pick your subscription, resource group and region, choose the Standard_B2s size, supply your SSH public key for the azureuser account, and allow inbound ports 22 and 80.

Step 2 — Deploy from the Azure CLI

az vm create \
  --resource-group my-resource-group \
  --name my-dkan \
  --image cloudimg1702aborea:dkan-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

Step 3 — Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 — Confirm the services are running

Both services that back DKAN report active, the cron timer reports active because it is armed and waiting, and the portal answers HTTP 200:

systemctl is-active apache2 mysql dkan-cron.timer
ss -tlnH | awk '{printf "%-22s %s\n", $4, $1}' | sort -u
curl -s -o /dev/null -w 'portal          -> HTTP %{http_code}\n' http://127.0.0.1/
curl -s -o /dev/null -w 'catalog         -> HTTP %{http_code}\n' http://127.0.0.1/dataset/search
curl -s -o /dev/null -w 'API root        -> HTTP %{http_code}\n' http://127.0.0.1/api/1

Expected: three active lines, then the listening sockets, then three HTTP 200 lines. Note that MySQL listens on 127.0.0.1:3306 only — the database is never exposed to the network. Port 80 is the only listener DKAN itself publishes; port 22 is your SSH administration path.

How the first boot protects an unconfigured portal

dkan-firstboot.service is a one-shot unit that runs on the first boot of each VM. It generates the per-VM secrets, provisions the application database, installs Drupal, enables DKAN, writes the credentials file at mode 0600, and only then creates a bootstrap-ready marker at /var/lib/cloudimg/dkan-bootstrap-ready before restarting Apache.

Apache carries ConditionPathExists on that marker, so the public listener is physically unable to answer until first boot has finished. There is no window in which an unconfigured portal is reachable. You can see the gate in the unit itself:

systemctl cat apache2.service | grep -A1 '^# /etc/systemd/system/apache2.service.d/cloudimg-bootstrap-gate.conf'
systemctl is-active dkan-firstboot.service
sudo test -f /var/lib/cloudimg/dkan-bootstrap-ready && echo "bootstrap-ready marker present — first boot completed"

On the very first boot the one-shot reports active (it is active (exited), which is what a completed one-shot with RemainAfterExit=yes reports). After a later reboot the unit is skipped by its own condition and reports inactive; both are correct, and the sentinel at /var/lib/cloudimg/dkan-firstboot.done is what guarantees it runs exactly once.

Step 5 — Retrieve your administrator credentials

Every secret this appliance uses is generated on your VM at first boot: a fresh MySQL root password, a fresh application database password, a fresh Drupal administrator password, and a fresh Drupal hash_salt. That last one matters more than it sounds — a salt shared between deployments would let a session cookie or a one-time login token minted on one VM be replayed against another, so cloudimg never bakes one into the image.

sudo stat -c '%n  mode=%a  owner=%U:%G' /stage/scripts/dkan-credentials.log
sudo grep -oE '^[A-Z_]+=' /stage/scripts/dkan-credentials.log | sed 's/=$/ = (set, unique per VM)/'

The file is mode 0600 and owned by root, so only root can read it. To see the values:

sudo cat /stage/scripts/dkan-credentials.log

It contains the portal, catalog, administration and API URLs, the administrator username (cloudimg) and password, and the database credentials.

You can prove the database credentials really authenticate, and that a wrong password is genuinely rejected, from the VM's own shell:

DB_USER=$(sudo awk -F= '$1=="DKAN_DB_USER"{print $2}' /stage/scripts/dkan-credentials.log)
DB_PASS=$(sudo awk -F= '$1=="DKAN_DB_PASSWORD"{sub(/^[^=]*=/,"");print}' /stage/scripts/dkan-credentials.log)
OK=$(mysql -h 127.0.0.1 -u "$DB_USER" -p"$DB_PASS" -D dkan -N -B -e 'SELECT 1' </dev/null 2>/dev/null || true)
BAD=$(mysql -h 127.0.0.1 -u "$DB_USER" -p'definitely-not-the-password' -D dkan -N -B -e 'SELECT 1' </dev/null 2>/dev/null || true)
echo "correct password -> ${OK:-rejected}"
echo "wrong password   -> ${BAD:-rejected}"

Expected: correct password -> 1 and wrong password -> rejected.

Step 6 — Run the built-in self test

The image ships a self test that checks the four things no unauthenticated probe can: that the per-VM database password authenticates and a wrong one is refused, that the per-VM administrator password authenticates through the real Drupal login form, that an authenticated administration page actually renders, and that a wrong administrator password does not produce a session.

sudo /usr/local/sbin/dkan-selftest.sh

Expected: four descriptive lines followed by DKAN_SELFTEST_OK. The authenticated-page check is the one worth understanding — a signed-out request to a healthy portal succeeds even when a session-bearing response is too large for the web server to buffer, so only fetching an admin page with a session proves the administration UI works.

Step 7 — Confirm the pinned versions and file modes

php -r '$l=json_decode(file_get_contents("/var/www/dkan/composer.lock"),true); foreach(array_merge($l["packages"],$l["packages-dev"]??[]) as $p){ if(in_array($p["name"],["getdkan/dkan","drupal/core","drush/drush"])) printf("%-18s %s\n",$p["name"],$p["version"]); }'
sudo -u www-data /usr/local/bin/drush status --field=drupal-version
php -v | head -1
sudo stat -c 'settings.php  mode=%a  owner=%U:%G' /var/www/dkan/docroot/sites/default/settings.php
sudo stat -c 'private dir   mode=%a  owner=%U:%G' /var/www/dkan/private

Expected: getdkan/dkan 2.23.2, drupal/core 11.4.4, drush/drush 13.7.6, Drupal 11.4.4, PHP 8.3, then settings.php mode=440 owner=root:www-data and private dir mode=750 owner=www-data:www-data.

settings.php holds the database password, so it is mode 0440 owned root:www-data rather than the 0444 Drupal suggests: Apache can still read it, but it is not world readable, and it is not writable by the web user either. The private file directory sits outside the document root, so a private file can never be fetched by URL even if a rewrite rule were lost.

Step 8 — Visit the portal

Browse to http://<vm-public-ip>/. The catalog lives at /dataset/search, and the administration UI is under /admin/dkan.

The image does not bake an address into the site configuration. Drupal derives the base URL from the incoming request, so the portal answers correctly on the VM's public address, its private address, or a custom domain you point at it later — with no reconfiguration. You can see that for yourself:

PRIVATE_IP=$(hostname -I | awk '{print $1}')
curl -s -o /dev/null -w 'arbitrary domain -> %{http_code}\n' -H "Host: data.example.com" http://127.0.0.1/user/login
curl -s -o /dev/null -w 'private address  -> %{http_code}\n' "http://${PRIVATE_IP}/user/login"
grep -c '\$base_url' /var/www/dkan/docroot/sites/default/settings.php || true

Expected: 200 for both, and 0 occurrences of a hard-coded $base_url.

Step 9 — Sign in as the administrator

Browse to http://<vm-public-ip>/user/login and sign in with the username cloudimg and the password from the credentials file.

Username: cloudimg
Password: <DKAN_ADMIN_PASSWORD>

Step 10 — Load the sample catalog

A brand new portal is deliberately empty: no datasets, no accounts other than yours, no cached metadata. DKAN ships a self-contained sample catalog so you can see the whole pipeline work before you publish anything of your own. It harvests from files inside the module on this VM, so it needs no internet access.

drushw() { sudo -u www-data /usr/local/bin/drush "$@"; }
drushw en sample_content -y >/dev/null 2>&1
drushw dkan:sample-content:create >/dev/null 2>&1
for i in $(seq 1 20); do
  drushw cron >/dev/null 2>&1 || true
  DATASETS=$(curl -s http://127.0.0.1/api/1/metastore/schemas/dataset/items | php -r '$d=json_decode(stream_get_contents(STDIN),true); echo is_array($d)?count($d):0;')
  TABLES=$(drushw sql:query "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='dkan' AND table_name LIKE 'datastore%'" 2>/dev/null | awk '/^[0-9]+$/{v=$1} END{print v+0}')
  echo "cron pass ${i}: ${DATASETS:-0} dataset(s) published, ${TABLES:-0} datastore table(s)"
  [ "${DATASETS:-0}" -ge 1 ] && [ "${TABLES:-0}" -ge 1 ] && break
  sleep 5
done
[ "${DATASETS:-0}" -ge 1 ] || { echo "no dataset reached the metastore"; exit 1; }
[ "${TABLES:-0}" -ge 1 ] || { echo "no datastore table was created"; exit 1; }
echo "sample catalog loaded: ${DATASETS} dataset(s), ${TABLES} datastore table(s)"

Expected: the dataset and datastore-table counts climb and the loop stops once both are non-zero, ending with a sample catalog loaded: ... line. Cron is what advances DKAN's harvest and import queues, which is why the image runs it every five minutes from dkan-cron.timer — the loop above just hurries it along.

Two details in that loop are deliberate, and both were bugs first:

• The table count comes from drush sql:query, which authenticates with the credentials in settings.php. A plain sudo mysql will not work — first boot rotates the MySQL root password out of the image, which is the point.
• Drush appends a trailing blank line to its output, so a | tail -1 would read that blank line and hand you an empty string. The awk filter picks the last numeric line instead.

Step 11 — Review the harvest

Choose DKAN → Harvests in the administration menu, or browse to /admin/dkan/harvest. The sample catalog appears as a harvest plan with its extract status and the number of datasets it produced. This is the same screen you use for a real source: register another portal's DCAT endpoint as a plan and DKAN pulls its whole catalog on every cron run.

Step 12 — Review the datasets

Choose DKAN → Datasets, or browse to /admin/dkan/datasets. Every dataset is listed with its data type, publication status and moderation state, and + Add new dataset starts the authoring form.

Step 13 — Publish your own dataset

Choose DKAN → Datasets → + Add new dataset. The form is generated from the DCAT metadata schema, so the fields you are asked for are the fields a catalog aggregator expects: title, description, public access level, publication frequency, keywords, publisher, contact point, licence, and one or more distributions. A distribution is either an uploaded file or a link to one; upload a CSV and DKAN will load it into the datastore on the next cron run.

Step 14 — Watch the datastore import

Choose DKAN → Datastore Import Status, or browse to /admin/dkan/datastore/status. Each resource shows its fetch and store stages. done in both columns means the file's contents are in the database and queryable through the API.

Because datastore_mysql_import is enabled, CSV distributions are loaded with MySQL's native LOAD DATA in a single step rather than in row batches, which is what makes multi-megabyte open data files practical on a small VM.

Step 15 — Query the data over the API

This is what separates a data portal from a file share. Pick a tabular distribution and query it — no download, no parsing:

DIST=$(curl -s 'http://127.0.0.1/api/1/metastore/schemas/dataset/items?show-reference-ids' | php -r '$d=json_decode(stream_get_contents(STDIN),true); foreach($d as $ds){ foreach(($ds["distribution"]??[]) as $x){ if(($x["data"]["format"]??"")==="csv"){ echo $x["identifier"]; exit; } } }')
echo "distribution: ${DIST}"
curl -s "http://127.0.0.1/api/1/datastore/query/${DIST}?limit=2" | php -r '$d=json_decode(stream_get_contents(STDIN),true); printf("rows in datastore : %d\ncolumns returned  : %d\n", $d["count"], count($d["results"][0])); foreach(array_slice($d["results"],0,2) as $i=>$r){ printf("row %d            : %s\n", $i+1, substr(implode(" | ", array_slice(array_values($r),0,5)),0,80)); }'
curl -s -G --data-urlencode "query=[SELECT * FROM ${DIST}][LIMIT 1]" http://127.0.0.1/api/1/datastore/sql | php -r '$d=json_decode(stream_get_contents(STDIN),true); printf("SQL endpoint rows : %d\nSQL endpoint cols : %s\n", count($d), substr(implode(", ", array_keys($d[0])),0,70));'

Expected: a row count, a column count, two sample rows, then one row back from the SQL endpoint with its column names.

Two things worth knowing, because they are easy to get wrong:

• Both APIs are keyed by the distribution identifier — the identifier field of an entry in a dataset's distribution array. It is not the identifier nested under %Ref:downloadURL, and it is not the underlying datastore_... MySQL table name. Either of those returns a distribution not found error.
• The SQL endpoint's FROM target is that same distribution identifier, wrapped in DKAN's bracket syntax: [SELECT * FROM <distribution-id>][LIMIT 1].

Only tabular distributions get a datastore. A ZIP or PDF distribution correctly reports No datastore storage found.

Step 16 — The rest of the catalog API

curl -s http://127.0.0.1/api/1/metastore/schemas/dataset/items | php -r '$d=json_decode(stream_get_contents(STDIN),true); printf("datasets published: %d\n", count($d)); foreach(array_slice($d,0,3) as $ds){ printf("  - %s\n", substr($ds["title"]??"",0,60)); }'
curl -s http://127.0.0.1/api/1/metastore/schemas | php -r '$d=json_decode(stream_get_contents(STDIN),true); printf("metastore schemas : %s\n", implode(", ", array_keys($d)));'
curl -s -o /dev/null -w 'OpenAPI spec      -> HTTP %{http_code}\n' http://127.0.0.1/api/1

GET /api/1 returns the full OpenAPI description of every endpoint this portal exposes, which is the quickest way to explore the rest of the surface: dataset create and update, harvest plan management, datastore import control, and search.

Removing the sample catalog

When you are finished exploring, remove the sample datasets and turn the module off. Your own datasets are untouched:

sudo -u www-data /usr/local/bin/drush dkan:sample-content:remove
sudo -u www-data /usr/local/bin/drush pmu sample_content -y
sudo -u www-data /usr/local/bin/drush cache:rebuild

Administration from the command line

Drush is installed as a wrapper that already points at this project's document root, so it works from any directory:

sudo -u www-data /usr/local/bin/drush status | head -12
sudo -u www-data /usr/local/bin/drush pm:list --status=enabled --filter=dkan --field=name

To reset the administrator password, or to add another administrator:

sudo -u www-data /usr/local/bin/drush user:password cloudimg 'your-new-password'
sudo -u www-data /usr/local/bin/drush user:create alice --mail=alice@example.com --password='her-password'
sudo -u www-data /usr/local/bin/drush user:role:add administrator alice

Useful DKAN commands:

sudo -u www-data /usr/local/bin/drush dkan:dataset-info <dataset-uuid>
sudo -u www-data /usr/local/bin/drush dkan:datastore:import <resource-identifier>
sudo -u www-data /usr/local/bin/drush dkan:datastore:drop <dataset-uuid>
sudo -u www-data /usr/local/bin/drush cron

Using your own domain and HTTPS

The image serves plain HTTP on port 80 so that it works immediately on any address. For a production portal, point your domain at the VM and terminate TLS. The quickest route is Certbot with the Apache plugin:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-apache
sudo certbot --apache -d data.example.com

Because no address is baked into the site configuration, the portal starts answering on your domain as soon as DNS resolves — there is nothing to reconfigure in Drupal.

For a production site you should also tighten the trusted host pattern. The image ships a permissive pattern so the portal works on any address out of the box; once you have settled on your domain, edit /var/www/dkan/docroot/sites/default/settings.php (it is mode 0440, so make it writable first) and replace it:

$settings['trusted_host_patterns'] = ['^data\.example\.com$'];

Keeping DKAN and Drupal up to date

DKAN and Drupal core are pinned to the versions verified at build time, so the image you certified is the image that runs. Apply upstream updates with Composer when you are ready:

cd /var/www/dkan
sudo -u www-data composer update drupal/core-recommended --with-all-dependencies
sudo -u www-data composer update getdkan/dkan --with-all-dependencies
sudo -u www-data /usr/local/bin/drush updatedb -y
sudo -u www-data /usr/local/bin/drush cache:rebuild

Always take a backup first (see below), and test updates outside production. Note that moving from DKAN 2.x to 4.x is a documented migration, not a Composer update — read upstream's upgrade guide before attempting it.

Backup and maintenance

Back up the database, the uploaded and localized resource files, and the private directory:

cd /var/www/dkan
sudo -u www-data /usr/local/bin/drush sql:dump --result-file=/var/backups/dkan-$(date +%F).sql
sudo tar czf /var/backups/dkan-files-$(date +%F).tar.gz \
  /var/www/dkan/docroot/sites/default/files /var/www/dkan/private

Datastore tables are inside the database dump, so a restored dump brings the imported data back with it.

The operating system receives unattended security upgrades. Check for pending updates with:

apt-get -s -o APT::Get::Always-Include-Phased-Updates=true dist-upgrade | grep -c '^Inst ' || true

Sizing and tuning notes

The image is deliberately tuned to fit Standard_B2s (2 vCPU, 4 GB): Apache mpm_prefork is capped at 12 workers, PHP has a 512 MB web limit with no CLI limit so Composer and datastore imports are not starved, and MySQL runs a 256 MB InnoDB buffer pool with performance_schema and the X protocol both off. A fully loaded portal with the sample catalog imported uses under 1 GB with no swap at all — Azure manages swap on the ephemeral resource disk, and nothing is baked into the OS disk.

If you plan large imports or a busy API, raise MaxRequestWorkers in /etc/apache2/mods-available/mpm_prefork.conf and innodb_buffer_pool_size in /etc/mysql/mysql.conf.d/zz-cloudimg-dkan.cnf after moving to a larger VM size.

Licensing

DKAN is distributed under the GNU General Public License, version 2 (GPL-2.0), and Drupal core under GPL-2.0-or-later. The licence texts ship in the image at /var/www/dkan/docroot/modules/contrib/dkan/LICENSE.txt and /var/www/dkan/docroot/core/LICENSE.txt. The bundled select2 JavaScript library is MIT licensed. cloudimg charges only for the packaging, maintenance and support of the image; the software itself remains under its open source licence.

Support

Every cloudimg image includes 24/7 support. Contact support@cloudimg.co.uk with your Azure subscription ID and the VM name.