Post

Disk Health on Linux: smartctl, badblocks, and RAID Recovery

Disk Health on Linux: smartctl, badblocks, and RAID Recovery

Drives rarely fail without warning. Between SMART counters creeping upward and a mdadm array quietly dropping a member, the signs are almost always there before data is actually lost — if something is watching for them. This post covers reading disk health with smartctl, scanning for bad sectors, and recovering a degraded RAID array.

Installing smartmontools

SMART (Self-Monitoring, Analysis, and Reporting Technology) is built into the drive firmware; smartmontools is just the userspace client that talks to it.

1
2
3
4
5
6
7
sudo apt install smartmontools   # Debian/Ubuntu
sudo dnf install smartmontools   # RHEL/CentOS/Fedora

# List drives smartctl can see
sudo smartctl --scan
# /dev/sda -d ata # /dev/sda, ATA device
# /dev/sdb -d ata # /dev/sdb, ATA device

Tip: On hardware RAID controllers, smartctl often can’t see the physical disks directly behind the controller. You may need the controller’s own CLI (e.g. storcli, megacli, arcconf) or a -d megaraid,N style device syntax.

Checking Overall Health

The single most useful command is the overall health self-assessment, which the drive computes itself from its internal thresholds.

1
2
sudo smartctl -H /dev/sda
# SMART overall-health self-assessment test result: PASSED

Tip: A PASSED result means no monitored attribute has crossed its failure threshold yet. It is not a guarantee the drive won’t fail tomorrow — it’s a lagging indicator, not a predictive one. Watch the raw attribute values too.

Reading Key SMART Attributes

The full attribute table tells a more useful story than the pass/fail summary alone, especially for attributes that trend toward failure well before the threshold is crossed.

1
2
3
4
5
6
sudo smartctl -A /dev/sda
# ID# ATTRIBUTE_NAME          VALUE WORST THRESH RAW_VALUE
#   5 Reallocated_Sector_Ct   100   100   010     0
#   9 Power_On_Hours          096   096   000     31210
# 197 Current_Pending_Sector  100   100   000     0
# 198 Offline_Uncorrectable   100   100   000     0

Tip: Watch three attributes above all others: Reallocated_Sector_Ct (sectors already remapped), Current_Pending_Sector (sectors waiting to be remapped, i.e. suspect but not yet confirmed bad), and Offline_Uncorrectable. Any of these rising from zero is a reason to plan a replacement, not a reason to panic immediately.

Running Self-Tests: Short vs Extended

SMART attributes update passively during normal use, but a self-test actively exercises the drive and is far more likely to catch a developing problem.

1
2
3
4
5
6
7
8
9
10
# Short test: a few minutes, checks electrical and mechanical basics
sudo smartctl -t short /dev/sda

# Extended test: can take hours, scans the full surface
sudo smartctl -t long /dev/sda

# Check progress and results
sudo smartctl -l selftest /dev/sda
# Num  Test_Description    Status                  Remaining  LifeTime(hours)
#   1  Extended offline    Completed without error  00%        31198

Tip: Run the extended test overnight or over a weekend on production drives — it generates real I/O load and will measurably affect performance while running.

Scheduling Automatic Checks with smartd

Nobody remembers to run smartctl by hand on a schedule. smartd is the daemon that does it for you and emails or logs when something crosses a threshold.

1
2
3
4
5
6
7
sudo systemctl enable --now smartd

# /etc/smartd.conf: monitor all attributes, mail on problems, run a short test daily and long test weekly
cat /etc/smartd.conf
# /dev/sda -a -m [email protected] -s (S/../.././02|L/../../6/03)

sudo systemctl reload smartd

Tip: The -a flag monitors all SMART attributes and tests, which is the sane default. Without it, smartd only watches health status and a small attribute subset, missing the early-warning signs covered above.

Scanning for Bad Blocks

badblocks reads (and optionally writes) every sector of a device to find ones the drive can’t reliably service. This is destructive when run in write mode — never point it at a mounted filesystem or a disk holding data you care about.

1
2
3
4
5
6
# Read-only scan, safe on a live filesystem, just slower to catch everything
sudo badblocks -sv /dev/sdb

# Non-destructive read-write test: only safe on a drive with NO data you need,
# since it overwrites every block with a pattern and reads it back
sudo badblocks -svw /dev/sdb

Tip: -svw destroys all data on the target device. Only run it against a drive that’s been wiped or is brand new and not yet partitioned. For anything already in service, stick to -sv (read-only) and lean on SMART’s Current_Pending_Sector/Offline_Uncorrectable counts instead.

Recognizing a Degraded mdadm Array

A RAID array doesn’t go down loudly when one member fails — it keeps serving reads and writes from the remaining disks, which is exactly why it’s easy to miss.

1
2
3
4
5
6
7
8
9
cat /proc/mdstat
# md0 : active raid1 sdb1[1] sda1[0](F)
#       976630464 blocks super 1.2 [2/1] [_U]

sudo mdadm --detail /dev/md0
# State : clean, degraded
# Number   Major   Minor   RaidDevice State
#    0       8        1        0      faulty   /dev/sda1
#    1       8       17        1      active sync   /dev/sdb1

Tip: [_U] in /proc/mdstat means one slot in the array is down (_) and one is up (U). Count the letters to know how many of your members are actually missing before you decide this is urgent.

Replacing a Failed RAID Member

Once a replacement disk is in the machine, pulling the faulty member and adding the new one back in is a short, well-defined sequence.

1
2
3
4
5
6
7
8
9
# Remove the faulty member from the array
sudo mdadm /dev/md0 --remove /dev/sda1

# Partition the new disk to match, then add it back
sudo mdadm /dev/md0 --add /dev/sdc1

# Confirm it's rebuilding
sudo mdadm --detail /dev/md0
# Rebuild Status : 4% complete

Tip: If the array is [_U] and you physically swap the bad disk first, mdadm won’t automatically know a new disk arrived — you still need the explicit --add step after partitioning it to match the existing member’s layout.

Monitoring Resync Progress

A rebuild can take hours on large disks, and the array is running without redundancy for the entire duration — worth watching closely.

1
2
3
4
watch -n 5 cat /proc/mdstat
# md0 : active raid1 sdc1[2] sdb1[1]
#       976630464 blocks super 1.2 [2/1] [_U]
#       [=====>...............]  recovery = 24.3% (238234112/976630464) finish=98.2min speed=124583K/sec

Tip: Resync speed is throttled by /proc/sys/dev/raid/speed_limit_min and speed_limit_max. If a rebuild is competing with production I/O, you can lower the minimum temporarily; just remember the array has no redundancy until it finishes, so don’t stretch it out longer than necessary.

This post is licensed under CC BY 4.0 by the author.