Post

mdadm Software RAID: Building, Monitoring, and Recovering Arrays

mdadm Software RAID: Building, Monitoring, and Recovering Arrays

Software RAID with mdadm is the standard way to get redundancy on Linux without a hardware controller. It is also the layer most admins ignore until a disk fails, at which point the recovery steps matter enormously. This post covers building, monitoring, and recovering arrays.

Checking the Current State

1
2
3
4
5
6
7
8
# All arrays
cat /proc/mdstat

# Detailed view
sudo mdadm --detail /dev/md0

# Scan for arrays
sudo mdadm --examine --scan

Tip: /proc/mdstat is the one file to check first. It shows every array, its state, and any rebuild in progress in a single view.

Creating an Array

1
2
3
4
5
6
7
8
9
10
11
12
13
# Create a RAID 1 (mirror) from two disks
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

# Create a RAID 5 from three disks
sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd

# Create a RAID 10 from four disks
sudo mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

# Format and mount
sudo mkfs.ext4 /dev/md0
sudo mkdir -p /mnt/raid
sudo mount /dev/md0 /mnt/raid

Tip: --raid-devices must match the number of devices you pass. A mismatch creates an array that will not assemble at boot.

Persisting the Array

1
2
3
4
5
6
# Save the array configuration
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

# Update the initramfs so the array assembles at boot
sudo update-initramfs -u    # Debian/Ubuntu
sudo dracut -f              # RHEL family

Tip: Without the mdadm.conf entry and an updated initramfs, the array may not assemble at boot, and the system drops to emergency mode.

Monitoring an Array

1
2
3
4
5
6
7
8
9
# Watch the array state
watch -n 2 cat /proc/mdstat

# Check for degraded arrays
cat /proc/mdstat | grep -i degraded

# Check SMART on the member disks
sudo smartctl -H /dev/sdb
sudo smartctl -H /dev/sdc

Tip: A degraded array is not an emergency, but it is a warning. Replace the failed disk before a second failure takes the array down.

Replacing a Failed Disk

1
2
3
4
5
6
7
8
9
# Mark the failed disk as removed
sudo mdadm /dev/md0 --fail /dev/sdb
sudo mdadm /dev/md0 --remove /dev/sdb

# Physically replace the disk, then add the new one
sudo mdadm /dev/md0 --add /dev/sdb

# Watch the rebuild
watch -n 2 cat /proc/mdstat

Tip: The rebuild can take hours on large disks. Run it in tmux so a dropped SSH session does not interrupt your monitoring.

Growing an Array

1
2
3
4
5
6
7
8
# Add a new disk to a RAID 5
sudo mdadm /dev/md0 --add /dev/sdf

# Grow the array to use it
sudo mdadm --grow /dev/md0 --raid-devices=4

# Grow the filesystem
sudo resize2fs /dev/md0

Tip: Growing a RAID 5 is a multi-hour operation with a real risk of failure. Back up before you start, and never grow an array that is already degraded.

Recovering a Broken Array

1
2
3
4
5
6
7
8
9
10
11
# Stop the array
sudo mdadm --stop /dev/md0

# Assemble it manually
sudo mdadm --assemble /dev/md0 /dev/sdb /dev/sdc

# Force assembly if the superblock is out of date
sudo mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdc

# Examine a member disk
sudo mdadm --examine /dev/sdb

Tip: --force assembly is a last resort. It can bring up an array with stale data, so back up whatever you can read before you try it.

RAID 6: Double Parity

RAID 6 tolerates two simultaneous disk failures, at the cost of two parity disks.

1
2
3
4
5
# Create a RAID 6 from four disks
sudo mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

# Check the array
cat /proc/mdstat

Tip: RAID 6 is the right choice for arrays with more than six disks. The rebuild window on large RAID 5 arrays is long enough that a second failure is a real risk.

Reshaping an Array

mdadm --grow changes the RAID level, the number of devices, or the chunk size without losing data.

1
2
3
4
5
6
7
8
9
10
11
12
# Add a disk and grow a RAID 5 to four devices
sudo mdadm /dev/md0 --add /dev/sdf
sudo mdadm --grow /dev/md0 --raid-devices=4

# Change the RAID level from RAID 1 to RAID 5
sudo mdadm --grow /dev/md0 --level=5 --raid-devices=3

# Change the chunk size
sudo mdadm --grow /dev/md0 --chunk=512

# Watch the reshape
watch -n 2 cat /proc/mdstat

Tip: A reshape is a multi-hour operation with a real risk of failure. Back up before you start, and never reshape an array that is already degraded.

Write-Intent Bitmap

A write-intent bitmap speeds up rebuilds after an unclean shutdown by tracking which blocks have changed.

1
2
3
4
5
6
7
8
9
10
11
# Add an internal bitmap
sudo mdadm --grow /dev/md0 --bitmap=internal

# Add an external bitmap
sudo mdadm --grow /dev/md0 --bitmap=/var/lib/md0.bitmap

# Remove the bitmap
sudo mdadm --grow /dev/md0 --bitmap=none

# Check the bitmap status
cat /proc/mdstat

Tip: An internal bitmap is the right choice for most arrays. It costs a small amount of write performance but dramatically reduces rebuild time after a crash.

mdadm –monitor: Alerting on Failures

mdadm --monitor watches arrays and sends alerts when a disk fails or a rebuild starts.

1
2
3
4
5
6
7
8
# Run in the foreground
sudo mdadm --monitor --scan --mail=[email protected]

# Run as a daemon
sudo mdadm --monitor --scan --daemonise --mail=[email protected]

# Test the alert path
sudo mdadm --monitor --scan --test --mail=[email protected]

Tip: mdadm --monitor --test sends a test alert, which is the only way to confirm the mail path works before a disk actually fails.

mdadm.conf: Persistent Configuration

1
2
3
4
5
6
7
8
9
10
11
# /etc/mdadm/mdadm.conf

# Scan for arrays at boot
DEVICE partitions

# Array definitions
ARRAY /dev/md0 metadata=1.2 name=host:0 UUID=12345678:abcdef01:23456789:abcdef01

# Monitoring
MAILADDR [email protected]
PROGRAM /usr/share/mdadm/checkarray --cron --all --idle --quiet

Tip: The ARRAY line is generated by mdadm --detail --scan. Do not hand-write it; a typo means the array will not assemble at boot.

Quick Reference

1
2
3
4
5
6
7
8
9
10
11
12
13
# What arrays exist?
cat /proc/mdstat

# Create a mirror
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

# Replace a failed disk
sudo mdadm /dev/md0 --fail /dev/sdb
sudo mdadm /dev/md0 --remove /dev/sdb
sudo mdadm /dev/md0 --add /dev/sdb

# Persist the config
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Tip: Work in order: check /proc/mdstat, replace the failed disk, watch the rebuild, then verify SMART on the remaining disks. Most RAID incidents are a failed disk that nobody noticed.

This post is licensed under CC BY 4.0 by the author.