MooseFS on AWS User Guide

Overview

This image runs MooseFS Community Edition, the open source POSIX compliant distributed file system. MooseFS spreads files across a cluster of chunkservers, so a MooseFS volume behaves like an ordinary directory on every client while the data underneath is distributed and, once you add chunkservers, replicated.

The image is a complete, self contained MooseFS starter node. A master server, one chunkserver and the MooseFS FUSE client all run on the single instance, and the MooseFS volume is mounted and ready at /srv/moosefs the moment first boot completes. You can write to it, read from it, and watch the chunks appear in the web monitor without configuring anything.

Because a single instance holds one copy of every chunk, this image on its own is an evaluation and starter topology, not a redundant one. Real fault tolerance comes from adding chunkservers on separate instances and raising the storage goal; the last section of this guide walks through exactly how to do that.

MooseFS keeps both the master metadata and the chunk storage on a dedicated EBS data volume mounted at /var/lib/mfs, separate from the operating system disk and independently resizable. The chunk storage lives at /var/lib/mfs/chunks.

The image is secure by default. The MooseFS master control port, the chunkserver port and the web monitor are bound to the loopback interface, and an nftables guard drops the entire MooseFS protocol port range on every other interface, so nothing that speaks the MooseFS protocol is reachable from off the instance regardless of your security group. The only published endpoint is the web monitor, served over TLS by nginx on port 443 behind HTTP Basic Auth, with port 80 redirecting to 443. On the first boot of every deployed instance a one shot service generates a per instance monitor password and a TLS certificate bound to the instance, and records the credentials in /root/moosefs-credentials.txt with mode 0600. No shared or default credentials ship in the image.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and inbound port 443 (and optionally port 80, which only redirects to 443) from where you will reach the web monitor
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for MooseFS. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that allows inbound port 22 from your management network and inbound port 443 from where you will reach the web monitor. Leave the root volume at the default size or larger; the image adds a separate 40 GiB data volume for the MooseFS metadata and chunk storage automatically.

Select Launch instance. First boot initialisation takes under a minute after the instance state becomes Running and the status checks pass.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg MooseFS Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens port 22 and port 443 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":20,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=moosefs-01}]'

The command prints a JSON document on success. Note the instance ID, then retrieve its public address once it is running with aws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].PublicIpAddress" --output text.

Connecting to Your Instance

Connect over SSH with the key pair you selected and the instance's public IP address. The SSH login user depends on the operating system of the AMI variant you launched:

Step 3: Retrieve the Web Monitor Credentials

On the first boot of the instance, moosefs-firstboot.service generates a per instance web monitor password, regenerates a TLS certificate bound to the instance, brings the MooseFS stack up, mounts the filesystem, and writes the credentials to /root/moosefs-credentials.txt (readable only by root). Connect over SSH and read that file:

ssh <login-user>@<public-ip>
sudo cat /root/moosefs-credentials.txt

The file contains GUI_USER, GUI_PASSWORD, MOOSEFS_MONITOR_URL (the https://<instance-public-ip>/ address of the web monitor) and MOOSEFS_MOUNT_POINT (/srv/moosefs). Note the user and password for the next step.

Step 4: Sign In to the Web Monitor

Open the MOOSEFS_MONITOR_URL from the credentials file in a browser, for example https://<public-ip>/. The certificate is self signed and bound to the instance, so your browser warns on the first visit; accept the warning to continue. When prompted for HTTP Basic Auth, sign in with the GUI_USER and GUI_PASSWORD values from the credentials file.

The monitor opens on the Status dashboard, showing total and free space, cluster health, usage and live throughput for the master server and the chunkserver.

The Info tab shows the cluster summary, the master metadata server and the chunk redundancy matrix. On a fresh single node instance you will see the health reported as "all in danger" and chunks at redundancy level 1 (RL=1): this is expected and simply means every chunk currently has a single copy because there is one chunkserver. It is not an error; see the last section for how to add redundancy.

The Chunkservers tab lists each chunkserver with its version, chunk count and the used and total space on the dedicated data volume.

The Mounts tab shows the active MooseFS FUSE client mounted at /srv/moosefs, ready to read and write files.

Step 5: Verify the Stack

From an SSH session on the instance, confirm every service is running:

systemctl is-active moosefs-master moosefs-chunkserver moosefs-gui moosefs-mount nginx

Confirm the web monitor backend answers on the loopback interface and the MooseFS filesystem is mounted:

curl -sS -o /dev/null -w 'monitor backend HTTP %{http_code}\n' http://127.0.0.1:9425/mfs.cgi
findmnt --noheadings --output SOURCE,FSTYPE,TARGET /srv/moosefs

The mount source reads mfs#127.0.0.1:9421 with filesystem type fuse, which confirms the mount is genuinely MooseFS and not an ordinary directory. Inspect the cluster with the MooseFS command line client, which talks to the master on loopback:

mfscli -H 127.0.0.1 -SIN
mfscli -H 127.0.0.1 -SCS

-SIN prints the cluster information summary and -SCS lists the chunkservers with their state and free space.

Step 6: Use the Filesystem

The MooseFS volume at /srv/moosefs behaves like any other directory. Write a file, read it back, and confirm the bytes really landed in MooseFS chunk storage:

echo "hello from cloudimg MooseFS" > /srv/moosefs/hello.txt
cat /srv/moosefs/hello.txt
mfsfileinfo /srv/moosefs/hello.txt
rm -f /srv/moosefs/hello.txt

mfsfileinfo reports the chunk backing the file and the chunkserver copy holding it (copy 1: with status:VALID). Applications write to /srv/moosefs exactly as they would to a local path; MooseFS transparently stores the data as distributed chunks. To make the volume available to other machines you widen the MooseFS export list and open the chunkserver port to those peers, which is the same operation as adding a chunkserver described below.

Security Model

The image is hardened so the MooseFS protocol is never exposed to the network:

• The client control port (9421), the metalogger port (9419) and the web monitor (9425) are bound to 127.0.0.1.
• MooseFS refuses a loopback address for the chunkserver to master link, so the master's chunkserver port (9420) and the chunkserver's own port (9422) bind the instance's private IP. An nftables table then drops ports 9419 to 9425 on every non loopback interface, so even those private IP bound ports are unreachable from off the instance regardless of your security group.
• The MooseFS export list permits mounts only from 127.0.0.1/32.
• The only published listener is nginx: port 443 serves the web monitor over TLS behind HTTP Basic Auth, and port 80 redirects to 443.

You can see the published listener and the firewall guard directly:

ss -H -lnt 'sport = :443'
sudo nft list table inet cloudimg_moosefs

Because the monitor password and the TLS certificate are generated per instance on first boot, no two instances share credentials and the captured image contains no valid credential at all.

Adding Chunkservers for Redundancy

A single instance holds one copy of every chunk, so on its own this image provides no fault tolerance. To get real redundancy, add one or more chunkservers on separate instances and raise the storage goal so each chunk is copied to more than one chunkserver.

At a high level:

1. Launch additional instances (any Linux host with the MooseFS chunkserver package installed) in the same VPC, each with their own storage disk for chunks.
2. On the master instance, allow the new peer to reach the master's chunkserver port (9420) by inserting an accept rule ahead of the drop rule in the nftables guard and opening 9420 to that peer in your AWS security group. For example, to allow a peer at a private address:

sudo nft insert rule inet cloudimg_moosefs input ip saddr <private-ip> tcp dport 9420 accept

1. Point each new chunkserver at the master's private IP (MASTER_HOST in /etc/mfs/mfschunkserver.cfg) and start it; it registers with the master and appears on the Chunkservers tab.
2. Raise the goal on the directories or files that need redundancy so MooseFS keeps more than one copy of each chunk:

# run this once a second chunkserver at <private-ip> has registered with the master
mfssetgoal -r 2 /srv/moosefs

With two chunkservers and a goal of 2, MooseFS keeps a copy of every chunk on each chunkserver, and the health on the Info tab moves from "all in danger" to stable. To widen filesystem access to other clients, add a scoped entry with a password= option to /etc/mfs/mfsexports.cfg and open the client control port to those peers in the same way.

Maintenance and Backup

The MooseFS master metadata and the chunk storage live on the dedicated data volume at /var/lib/mfs, so you can grow storage by resizing that EBS volume and extending the filesystem. To take a consistent copy of the master metadata for backup, snapshot the EBS data volume through the EC2 console or the AWS CLI, or copy /var/lib/mfs/metadata.mfs to <backup-dir> after briefly stopping the master. Security updates are applied to the base operating system at build time, and you can keep the instance current with the usual apt update workflow.

Support

This image is provided by cloudimg with 24/7 technical support by email and chat, with a guaranteed 24 hour response. We can help with deployment, adding chunkservers for redundancy, configuring TLS and access control, storage and upgrade planning, and troubleshooting. Contact details are on the AWS Marketplace listing and at www.cloudimg.co.uk.