Evidently ML & Data Observability on Ubuntu 24.04 on Azure User Guide

Overview

Evidently is an open-source platform for evaluating and monitoring machine learning models and data pipelines. You compute Reports over your datasets - data drift, data quality, data summary, classification and regression performance, and LLM/text evaluations - and the Evidently UI monitoring service renders those reports as interactive projects in a web dashboard, so teams can watch production data and model behaviour over time and catch regressions before they reach users. The cloudimg image installs the pinned official Evidently 0.7.21 pip package into a dedicated virtual environment, runs the Evidently UI server (evidently ui, a Litestar/uvicorn service) under systemd, and locks it down for a marketplace appliance: Evidently itself is bound to loopback only, and nginx does all customer-facing work on port 80, behind a per-VM HTTP Basic Auth gate. The OSS Evidently UI has no built-in user accounts, so the nginx Basic Auth credential (user admin, unique password generated on first boot) is the gate on the UI and API. The Evidently workspace - the projects and report snapshots the dashboard renders - lives on a dedicated 20 GiB Azure data disk and is durable across restarts and reboots. A demo project with a real data-drift and data-quality report is seeded on first boot so the dashboard shows real data immediately. Backed by 24/7 cloudimg support.

What is included:

• Evidently 0.7.21 installed via pip into a dedicated virtual environment, running as the evidently systemd service
• The Evidently UI monitoring dashboard and API on :80, fronted by nginx with Evidently bound to loopback only
• Per-VM HTTP Basic Auth (user admin) protecting the UI and API, with a unique password generated on first boot
• A dedicated Azure data disk at /var/lib/evidently holding the Evidently workspace (projects + report snapshots, durable across restarts/reboots)
• A seeded demo project (cloudimg-demo) containing a real data-drift + data-summary report over a synthetic sample, so the dashboard shows a rendered report on first login
• evidently.service + nginx.service as systemd units, enabled and active
• An unauthenticated /healthz endpoint on :80 for Azure Load Balancer health probes
• 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet + subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is a comfortable starting point for a small team's monitoring dashboard; size up the VM for larger workspaces or heavier report computation. NSG inbound: allow 22/tcp from your management network, 80/tcp for the UI and API, and 443/tcp if you add TLS. Evidently serves plain HTTP; for production use, terminate TLS in front of it with your own domain.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Evidently by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name evidently \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name evidently --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active evidently.service nginx.service

Both report active. Evidently serves its monitoring UI and API on the loopback address 127.0.0.1:8000 only; nginx fronts port 80 with the per-VM HTTP Basic Auth gate. The Evidently workspace lives on the dedicated data disk at /var/lib/evidently - the projects and report snapshots the dashboard renders are stored here, so they survive service restarts and VM reboots.

Step 5 - Retrieve your web UI password

nginx protects the Evidently UI and API with HTTP Basic Auth. The username is admin and a unique password is generated on the first boot of your VM and written to a root-only file:

sudo cat /root/evidently-credentials.txt

This file contains EVIDENTLY_USERNAME, EVIDENTLY_PASSWORD, the EVIDENTLY_URL to open in a browser, and EVIDENTLY_WORKSPACE (the workspace path on the data disk). The password is stored on disk only as a bcrypt hash in /etc/nginx/.evidently.htpasswd, so no plaintext password ships in the image. Store the password somewhere safe.

Step 6 - Confirm the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes:

curl -s http://localhost/healthz

It returns ok. This endpoint never requires authentication, so it is safe for an Azure Load Balancer health probe.

Step 7 - Confirm authentication and browse the seeded demo project

Because a password is set on first boot, an unauthenticated request to the UI and API returns HTTP 401, so nobody reaches your monitoring data without the password. The following reads the per-VM password from the credentials file, proves the auth round-trip, and lists the demo project seeded on first boot via the authenticated API:

PW=$(sudo grep '^EVIDENTLY_PASSWORD=' /root/evidently-credentials.txt | cut -d= -f2-)
echo "unauth  : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/api/projects)"
echo "wrongpw : $(curl -s -o /dev/null -w '%{http_code}' -u admin:wrong-pw http://127.0.0.1/api/projects)"
echo "authed  : $(curl -s -o /dev/null -w '%{http_code}' -u admin:$PW http://127.0.0.1/api/projects)"
curl -s -u admin:$PW http://127.0.0.1/api/projects | python3 -m json.tool

It prints unauth : 401, wrongpw : 401, authed : 200, and the API response lists the cloudimg-demo project seeded on first boot. Only the per-VM password reaches the Evidently UI and API, because Evidently itself is bound to loopback and nginx port 80 is the only way in.

Step 8 - Sign in and browse your projects

Browse to http://<vm-public-ip>/. Your browser prompts for a username and password: enter admin and the password from Step 5. Evidently opens on the Projects list, showing the cloudimg-demo project seeded on first boot. Each project groups the reports and test suites you add to it over time.

Step 9 - Open the demo project

Click the cloudimg-demo project to open it. The project view has a Dashboard tab (for monitoring panels you build from report metrics over time) and a Reports tab listing the report snapshots stored in the project. On a fresh appliance the dashboard is ready for you to add panels; the seeded report is available under Reports.

Step 10 - View the seeded data-drift report

Open the Reports tab and click the seeded report. Evidently renders the full report in the browser: the Data Drift section compares the current dataset against the reference dataset column by column, flags which columns drifted, and draws the distribution of each feature so you can see the shift. This is the core of the product - a data observability report computed over your data and rendered interactively.

Step 11 - Review data quality and summary metrics

The same report includes the Data Summary section: per-column statistics, missing values, and distributions for the reference and current datasets side by side. Together the drift and summary sections give you a complete first-pass observability view of a dataset without writing any dashboard code.

Step 12 - Monitor your own data

Point your own datasets at this workspace. Compute an Evidently Report over your data and add its snapshot to a project in the workspace with the Evidently Python SDK. On the VM (or anywhere with the workspace path available) this looks like:

import pandas as pd
from evidently import Dataset, DataDefinition, Report
from evidently.presets import DataDriftPreset, DataSummaryPreset
from evidently.ui.workspace import Workspace

# your reference and current datasets
reference = pd.read_csv("reference.csv")
current = pd.read_csv("current.csv")

data_def = DataDefinition()  # or specify numerical_columns / categorical_columns
ref_ds = Dataset.from_pandas(reference, data_definition=data_def)
cur_ds = Dataset.from_pandas(current, data_definition=data_def)

report = Report(metrics=[DataDriftPreset(), DataSummaryPreset()])
snapshot = report.run(current_data=cur_ds, reference_data=ref_ds)

ws = Workspace.create("/var/lib/evidently/workspace")
project = ws.create_project("my-monitoring")   # or reuse an existing project
ws.add_run(project.id, snapshot)

pip install evidently==0.7.21 in your environment to match the server's SDK version. After adding a run, refresh the dashboard (the server reads the workspace on startup; restart evidently.service if a newly added project does not appear) - see the upstream Evidently documentation for building monitoring dashboard panels, test suites, and LLM/text evaluations.

Maintenance

• Password: the UI/API password is set on first boot and stored as a bcrypt entry in /etc/nginx/.evidently.htpasswd. To change it, run sudo htpasswd -B /etc/nginx/.evidently.htpasswd admin and then sudo systemctl reload nginx.
• Durable storage: the Evidently workspace (/var/lib/evidently/workspace) is genuinely durable - it survives evidently service restarts and VM reboots. Back up the data disk (or snapshot it) like you would any production data store if your monitoring history matters.
• Restrict access: Evidently serves plain HTTP on port 80. For production, restrict access to trusted IP ranges in your Network Security Group, and front it with TLS (for example certbot with your own domain) terminating on :443.
• Loopback binding: Evidently is bound to 127.0.0.1:8000 in /etc/systemd/system/evidently.service (--host 127.0.0.1 --port 8000), so nginx is the only path in. Keep it that way - do not change the bind address to a public interface.
• Disk space: the workspace grows with the number of reports and snapshots you add. Monitor /var/lib/evidently usage (df -h /var/lib/evidently) and resize the data disk if you store a large monitoring history.
• Logs: journalctl -u evidently shows recent Evidently server output; journalctl -u nginx shows the web server's own logs.
• Security patches: unattended-upgrades remains enabled so the OS continues to receive security updates automatically.

Support

cloudimg provides 24/7 expert support for this image. Contact support@cloudimg.co.uk.