Databases Azure

Groonga 16 on Ubuntu 24.04 on Azure User Guide

| Product: Groonga 16 on Ubuntu 24.04 LTS on Azure

Overview

Groonga is an open source full text search engine and column store. It is best known as the engine inside Mroonga (a MySQL and MariaDB storage engine) and PGroonga (a PostgreSQL index method), but it also runs standalone as an HTTP server: you talk to it through a command API under /d/* and it ships a browser admin UI. Groonga indexes text with a choice of tokenizers - TokenBigram for mixed language corpora with no dictionary required, and MeCab for real Japanese morphological analysis - normalizes with a MySQL compatible normalizer, stems English with Snowball, and because it stores data column by column it can aggregate large result sets quickly with drilldown.

The cloudimg image installs the pinned Groonga 16.0.8 from the official Groonga APT repository and then locks it down for a marketplace appliance, because Groonga's security model needs care: Groonga has no authentication of its own, and its /d/* command API is fully privileged. Anything that can reach the Groonga port can create, read, alter, drop or dump every table, and the bundled admin UI is simply a static client driving that same API. Upstream's documented position is to run Groonga on a trusted network only.

So on this image Groonga is bound to 127.0.0.1:10043 and is not reachable from the network at all. An nginx reverse proxy on port 80 is the only public listener, and it enforces HTTP Basic Auth with a password generated uniquely on first boot. The destructive /d/shutdown command is denied outright at the proxy. Backed by 24/7 cloudimg support.

What is included:

  • Groonga 16.0.8 installed from the official Groonga APT repository, running as the groonga-server-http systemd service
  • The command API and the bundled admin UI on :80, fronted by nginx, with Groonga itself bound to loopback only
  • Per-VM HTTP Basic Auth (user admin) protecting the admin UI and the entire /d/* command API, with a unique password generated on first boot
  • A systemd guard that refuses to start Groonga at all if its bind address is ever changed away from loopback
  • /d/shutdown denied at the proxy for anonymous and authenticated clients alike, so the daemon lifecycle stays with systemd
  • nginx held closed until first boot has written the credential, so the public port is never open unprotected
  • The TokenBigram and MeCab tokenizers, the MySQL compatible normalizer and the Snowball stemming token filter
  • An empty database at /var/lib/groonga/db/db, ready for your own collections
  • The Groonga APT repository left configured, so security updates arrive through unattended-upgrades
  • An unauthenticated /healthz endpoint for Azure Load Balancer health probes
  • 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is a comfortable starting point - Groonga is a compact C daemon and very light on resources. NSG inbound: allow 22/tcp from your management network and 80/tcp for the admin UI and command API. Do not open 10043/tcp; nothing listens on it externally by design. Groonga serves plain HTTP on port 80, so for production terminate TLS in front of it with your own domain and restrict access to trusted IP ranges (see Maintenance).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Groonga by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name groonga \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name groonga --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services and the loopback binding

Two services matter: groonga-server-http (the search engine) and nginx (the authenticating front door). Note the listener addresses carefully - nginx is on 0.0.0.0:80 and [::]:80, while Groonga is on 127.0.0.1:10043 only.

systemctl is-active groonga-server-http.service nginx.service
ss -ltn | grep -E ':(80|10043)\b'
groonga --version

The groonga-server-http and nginx services both active, the socket table showing nginx listening on 0.0.0.0:80 and on the IPv6 any-address while Groonga listens only on 127.0.0.1:10043, and groonga --version reporting Groonga 16.0.8

Step 5 - Retrieve your per-VM password

First boot generates a unique 24 character password for the Basic Auth user admin and writes it to a root-only file. There is no default password anywhere in the image.

sudo cat /root/groonga-credentials.txt

The per-VM credentials file showing GROONGA_URL, GROONGA_USERNAME admin and the generated password with the security model explained, the groonga-firstboot service active with both bootstrap markers present, and file modes 600 root:root for the credentials, 640 root:www-data for the htpasswd and 750 groonga:groonga for the database directory

Step 6 - Confirm the health endpoint

/healthz is served statically by nginx and is deliberately left unauthenticated so an Azure Load Balancer probe can reach it. It never touches Groonga and reveals nothing about your data.

curl -s http://127.0.0.1/healthz

Step 7 - Confirm the authentication gate

This is the single most important thing to understand about running Groonga safely. Every privileged path - the admin UI at /, and the whole command API including /d/status, /d/table_list and /d/dump - must reject an anonymous request with 401. Only the per-VM password gets through.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
for path in / /admin/ /d/status /d/table_list /d/dump; do
  printf 'anonymous     %-14s -> HTTP %s\n' "$path" \
    "$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1$path")"
done
curl -s -o /dev/null -w 'wrong password  /d/status -> HTTP %{http_code}\n' \
  -u 'admin:wrong-password' http://127.0.0.1/d/status
curl -s -o /dev/null -w 'per-VM password /d/status -> HTTP %{http_code}\n' \
  -u "admin:$GROONGA_PASSWORD" http://127.0.0.1/d/status

Every privileged path returning HTTP 401 to an anonymous client, a wrong password returning 401, the per-VM password returning 200, and both anonymous and authenticated requests to /d/shutdown returning 403 while the daemon stays active

Step 8 - Confirm /d/shutdown is denied

Groonga's shutdown command stops the daemon over HTTP. On this image the daemon lifecycle belongs to systemd, so the proxy denies that one command with 403 for everybody - including authenticated clients - and Groonga keeps running.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'anonymous     /d/shutdown -> HTTP %{http_code}\n' http://127.0.0.1/d/shutdown
curl -s -o /dev/null -w 'authenticated /d/shutdown -> HTTP %{http_code}\n' \
  -u "admin:$GROONGA_PASSWORD" http://127.0.0.1/d/shutdown
systemctl is-active groonga-server-http.service

To stop or restart Groonga, use systemd over SSH instead:

sudo systemctl stop groonga-server-http
sudo systemctl restart groonga-server-http

Step 9 - Confirm Groonga is not reachable from the network

The loopback binding is real, not cosmetic. Connecting to this VM's own routable address on the Groonga port is refused; only nginx answers from outside.

ROUTABLE_IP=$(hostname -I | awk '{print $1}')
curl -s -m 5 -o /dev/null "http://$ROUTABLE_IP:10043/d/status" 2>/dev/null
echo "connect to $ROUTABLE_IP:10043 -> curl exit $? (7 means connection refused)"
echo "listeners on port 10043:"
ss -Hltn 'sport = :10043' | awk '{print "  " $4}'

If you ever need Groonga on a different address, do not edit the bind address: an ExecStartPre guard refuses to start the daemon on anything other than loopback, precisely because doing so would expose the unauthenticated privileged command API to the network. Put your own authenticating proxy in front instead.

Step 10 - Create a collection

Now build something searchable. A Groonga collection is a table plus columns. Here Docs is keyed by a short text identifier and holds a title and a body.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
grn() { curl -s -u "admin:$GROONGA_PASSWORD" --get "http://127.0.0.1/d/$1" "${@:2}"; }
grn table_create --data-urlencode 'name=Docs' --data-urlencode 'flags=TABLE_HASH_KEY' \
                 --data-urlencode 'key_type=ShortText'; echo
grn column_create --data-urlencode 'table=Docs' --data-urlencode 'name=title' \
                  --data-urlencode 'flags=COLUMN_SCALAR' --data-urlencode 'type=ShortText'; echo
grn column_create --data-urlencode 'table=Docs' --data-urlencode 'name=body' \
                  --data-urlencode 'flags=COLUMN_SCALAR' --data-urlencode 'type=Text'; echo

Each command answers [[0,<timestamp>,<elapsed>],true]. The leading 0 is Groonga's return code - 0 means success.

Step 11 - Add a lexicon and an inverted index

A table on its own is only scannable. Full text search needs a lexicon: a second table whose keys are the tokens, carrying an index column that points back at your documents. TokenBigram splits text into overlapping two character grams so it works without a dictionary, and NormalizerAuto folds case and width differences so a search for Cloud also matches CLOUD.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
grn() { curl -s -u "admin:$GROONGA_PASSWORD" --get "http://127.0.0.1/d/$1" "${@:2}"; }
grn table_create --data-urlencode 'name=Lexicon' --data-urlencode 'flags=TABLE_PAT_KEY' \
                 --data-urlencode 'key_type=ShortText' \
                 --data-urlencode 'default_tokenizer=TokenBigram' \
                 --data-urlencode 'normalizer=NormalizerAuto'; echo
grn column_create --data-urlencode 'table=Lexicon' --data-urlencode 'name=docs_index' \
                  --data-urlencode 'flags=COLUMN_INDEX|WITH_POSITION|WITH_SECTION' \
                  --data-urlencode 'type=Docs' --data-urlencode 'source=title,body'; echo

WITH_POSITION records where each token occurs, which is what makes phrase search possible. WITH_SECTION lets one index column cover several source columns - here both title and body.

Step 12 - Load records

Records are loaded as a JSON array with /d/load.

Set Content-Type: application/json. This is not optional and it is the most common way to lose an import: with curl's default application/x-www-form-urlencoded, Groonga loads zero records and still answers with a success envelope, so the load looks like it worked. Compare the number after the closing bracket - that is the count actually loaded, and it must match how many records you sent.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$GROONGA_PASSWORD" -X POST \
  -H 'Content-Type: application/json' \
  --data-binary '[
    {"_key":"d1","title":"Indexing text with a bigram tokenizer","body":"A bigram tokenizer splits text into overlapping two character grams, so it can index any language without a dictionary."},
    {"_key":"d2","title":"Normalizing before you index","body":"A normalizer folds case and width differences so a query matches regardless of how it was typed."},
    {"_key":"d3","title":"Storing telemetry in a column store","body":"Because values of one column sit together on disk, aggregating a large result set stays fast."}
  ]' \
  'http://127.0.0.1/d/load?table=Docs'; echo

The response ends in ,3] - three records loaded.

Step 13 - Run a full text search

match_columns says which columns to search, query is what to look for, and output_columns selects what comes back.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
grn() { curl -s -u "admin:$GROONGA_PASSWORD" --get "http://127.0.0.1/d/$1" "${@:2}"; }
grn select --data-urlencode 'table=Docs' --data-urlencode 'match_columns=title,body' \
           --data-urlencode 'query=tokenizer' \
           --data-urlencode 'output_columns=_key,title'; echo

Only d1 comes back: the index discriminates rather than returning everything. Try query=normalizer or query=column store to see the match set change.

A curl request to the select command returning the single matching record as Groonga's JSON envelope, and the same search aggregated by author with drilldown reporting four matches grouped across three authors

Step 14 - Aggregate results with drilldown

drilldown groups the matching set by a column value and returns counts alongside the results - this is the column store paying off. The response carries two result sets: your records, then the aggregation.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
grn() { curl -s -u "admin:$GROONGA_PASSWORD" --get "http://127.0.0.1/d/$1" "${@:2}"; }
grn select --data-urlencode 'table=Docs' --data-urlencode 'match_columns=title,body' \
           --data-urlencode 'query=index OR column' \
           --data-urlencode 'output_columns=_key,title' \
           --data-urlencode 'drilldown=title' | python3 -c '
import json,sys
d = json.load(sys.stdin); body = d[1]
print("matched:", body[0][0][0])
print("grouped by title:")
for row in body[1][2:]:
    print("   ", row[0], "->", row[1])
'

Note that the bundled admin UI sends the drilldown parameters but only renders the record set, so use the command API when you want to see the aggregation itself.

Step 15 - Tidy up the tutorial collection

Removing the index and the tables leaves the database exactly as the image shipped it.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
grn() { curl -s -u "admin:$GROONGA_PASSWORD" --get "http://127.0.0.1/d/$1" "${@:2}"; }
grn table_remove --data-urlencode 'name=Lexicon'; echo
grn table_remove --data-urlencode 'name=Docs'; echo
grn table_list; echo

Step 16 - Open the admin UI

Browse to http://<vm-public-ip>/ and sign in with admin and the password from Step 5. The Summary view reports the daemon's start time, uptime, the number of queries served and the cache hit rate.

The admin UI screenshots below were taken against a sample Articles collection paired with a Terms lexicon - the same shape as the Docs and Lexicon pair you built in Steps 10 and 11, just with more records in it.

The Groonga admin UI Summary view signed in through HTTP Basic Auth, reporting start time, uptime, 931 queries served and a 98.6 percent cache hit rate, with the side menu listing the Articles and Terms tables

Step 17 - Inspect your tables

List table shows every table with its flags, key type, tokenizer and normalizer. A document table is typically TABLE_HASH_KEY; a lexicon is TABLE_PAT_KEY with a tokenizer and normalizer set - here TokenBigram and NormalizerAuto.

The admin UI List of table view showing the Articles document table as TABLE_HASH_KEY and the Terms lexicon as TABLE_PAT_KEY with default_tokenizer TokenBigram and normalizer NormalizerAuto

Step 18 - Search from the browser

Pick a table in the side menu to get List records. The Query box accepts Groonga's query syntax, so body:@index means "the body column contains the word index". Tick Advanced search to fill in match_columns, filter, sortby, output_columns and the drilldown parameters directly.

The admin UI List of records view for the Articles table with the query body:@index entered, reporting Total count 3 and rendering the three matching records with their keys, authors, body text and titles

Step 19 - Inspect the inverted index

Selecting the lexicon and opening List columns shows the index column that makes search work: type index, flags COLUMN_INDEX|WITH_SECTION|WITH_POSITION, with domain the lexicon and range the document table it points at.

The admin UI List of columns view for the Terms lexicon showing the article_index column of type index with flags COLUMN_INDEX WITH_SECTION WITH_POSITION PERSISTENT, domain Terms and range Articles

Maintenance

Back up and restore. dump writes the schema and every record as a stream of Groonga commands, so a restore is just replaying it into a new database.

GROONGA_PASSWORD=$(sudo grep '^GROONGA_PASSWORD=' /root/groonga-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$GROONGA_PASSWORD" http://127.0.0.1/d/dump > groonga-backup.grn

sudo systemctl stop groonga-server-http
sudo -u groonga groonga /var/lib/groonga/db/db < groonga-backup.grn
sudo systemctl start groonga-server-http

Rotate the Basic Auth password. Replace the htpasswd entry and record the new value.

sudo htpasswd -B /etc/nginx/.groonga.htpasswd admin
sudo chown root:www-data /etc/nginx/.groonga.htpasswd
sudo chmod 640 /etc/nginx/.groonga.htpasswd
sudo systemctl reload nginx

Add TLS and a domain. Point a DNS record at the VM, open 443/tcp, then let certbot configure nginx. Do this before exposing the VM to the internet - Basic Auth over plain HTTP sends the password in every request.

sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d <your-domain> --redirect

Restrict who can reach port 80. Even behind Basic Auth, limit the source ranges.

az network nsg rule update --resource-group <your-rg> --nsg-name <your-nsg> \
  --name allow-http --source-address-prefixes <your-office-cidr>

Keep Groonga patched. The official Groonga APT repository stays configured on this image, so Groonga updates arrive with the rest of the system through unattended-upgrades. To apply them by hand:

sudo apt-get update && sudo apt-get upgrade -y
sudo systemctl restart groonga-server-http

Move the database to a larger disk. The database lives on the OS disk at /var/lib/groonga/db. For a large corpus, attach an Azure data disk, format and mount it, then relocate the database and keep the ownership and mode intact.

sudo systemctl stop groonga-server-http
sudo rsync -a /var/lib/groonga/ /mnt/groonga-data/
sudo mv /var/lib/groonga /var/lib/groonga.old
sudo ln -s /mnt/groonga-data /var/lib/groonga
sudo chown -R groonga:groonga /mnt/groonga-data
sudo chmod 750 /mnt/groonga-data/db
sudo systemctl start groonga-server-http

Logs. Groonga writes a daemon log and a query log; nginx records access and errors.

sudo tail -n 50 /var/log/groonga/groonga-http.log
sudo tail -n 50 /var/log/groonga/query-http.log
sudo journalctl -u groonga-server-http -n 50 --no-pager
sudo tail -n 50 /var/log/nginx/error.log

Support

cloudimg provides 24/7 support for this image. Visit www.cloudimg.co.uk or email support@cloudimg.co.uk.