Linux Boot Process and Recovery: GRUB2, initramfs, and Rescue Mode
A server that will not boot is the highest-stakes Linux problem there is. The good news is that the boot process is a short, well-defined sequence, and each stage has a known failure mode and a known recovery. This post walks the sequence and the fixes.
The Boot Sequence
- Firmware (BIOS or UEFI) selects a boot device.
- Bootloader (GRUB2) loads the kernel and initramfs.
- Kernel initialises hardware and mounts the initramfs.
- initramfs loads the drivers needed to mount the real root filesystem.
- systemd (PID 1) starts units and reaches the default target.
Each stage fails differently, and the error message usually tells you which one.
Inspecting the Boot
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# How long did boot take?
systemd-analyze
# Which units took the longest?
systemd-analyze blame
# The critical dependency chain
systemd-analyze critical-chain
# Kernel messages from this boot
dmesg -T
# Full boot log
journalctl -b
Tip: systemd-analyze critical-chain shows the exact dependency path that is slowing down boot. It is far more useful than blame alone.
GRUB2 Basics
1
2
3
4
5
6
7
8
9
10
11
# Regenerate the GRUB config (Debian/Ubuntu)
sudo update-grub
# Regenerate the GRUB config (RHEL family)
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# Show the current kernel command line
cat /proc/cmdline
# List installed kernels
ls /boot/vmlinuz-*
Tip: Never edit /boot/grub/grub.cfg directly. It is generated. Edit /etc/default/grub and regenerate.
Editing the Kernel Command Line at Boot
When a system will not boot, the fastest fix is often a kernel parameter.
- At the GRUB menu, press
eto edit the selected entry. - Find the line starting with
linux(orlinux16). - Append the parameter you need.
- Press
Ctrl+XorF10to boot.
Useful parameters:
singleorsystemd.unit=rescue.target— boot to a single-user shell.systemd.unit=emergency.target— minimal environment, root filesystem read-only.rd.break— drop to a shell inside the initramfs before the root filesystem is mounted.rw init=/bin/bash— boot straight to a root shell with no systemd.
Tip: rd.break is the standard way to reset a forgotten root password on RHEL family systems. It drops you into the initramfs, where you can remount the root filesystem read-write and change the password.
Resetting a Forgotten Root Password
1
2
3
4
5
6
7
8
9
10
# At the GRUB menu, edit the kernel line and append:
rd.break
# Inside the initramfs shell:
mount -o remount,rw /sysroot
chroot /sysroot
passwd root
touch /.autorelabel
exit
exit
Tip: touch /.autorelabel is essential on SELinux systems. Without it, the new password file has the wrong context and the system will not boot.
initramfs Problems
If the kernel cannot find the root filesystem, the initramfs is usually missing a driver.
1
2
3
4
5
6
7
8
# Rebuild the initramfs (Debian/Ubuntu)
sudo update-initramfs -u -k all
# Rebuild the initramfs (RHEL family)
sudo dracut -f
# List the contents of an initramfs
lsinitramfs /boot/initrd.img-$(uname -r)
Tip: After adding a new storage controller or moving a disk to new hardware, rebuild the initramfs. The old one may not contain the driver for the new controller.
Filesystem Errors at Boot
If the root filesystem fails fsck, the system drops to emergency mode.
1
2
3
4
5
6
7
8
# Check the root filesystem from a rescue environment
sudo fsck -f /dev/sda1
# Remount read-write after repair
sudo mount -o remount,rw /
# Check fstab for errors
sudo findmnt --verify
Tip: A bad fstab entry is the most common cause of emergency mode. Boot with rd.break, comment out the offending line, and reboot.
Rescue and Emergency Targets
1
2
3
4
5
6
7
8
# Boot to rescue mode (single-user, most services stopped)
sudo systemctl rescue
# Boot to emergency mode (minimal, root read-only)
sudo systemctl emergency
# Return to the default target
sudo systemctl default
Tip: Rescue mode mounts local filesystems and starts a shell. Emergency mode mounts only the root filesystem, read-only. Use emergency when rescue itself fails.
systemd-boot and bootctl
systemd-boot is a simple UEFI boot manager that reads entries from the EFI System Partition.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Install systemd-boot
sudo bootctl install
# Show boot entries
bootctl list
# Show the current boot status
bootctl status
# Update the bootloader
sudo bootctl update
# Set the default entry
sudo bootctl set-default myentry.conf
Tip: bootctl status shows the current boot entry, the firmware, and the Secure Boot state. It is the fastest way to confirm what actually booted.
Unified Kernel Images (UKI)
A UKI bundles the kernel, initramfs, and command line into a single signed file, which makes Secure Boot verification straightforward.
1
2
3
4
5
6
7
8
# Build a UKI
sudo kernel-install add $(uname -r) /boot/vmlinuz-$(uname -r)
# List installed UKIs
ls /boot/EFI/Linux/
# Inspect a UKI
ukify inspect /boot/EFI/Linux/*.efi
Tip: UKIs are the cleanest way to sign a boot chain. The kernel, initramfs, and command line are all covered by a single signature, which makes Secure Boot verification straightforward.
kernel-install
kernel-install is the standard way to add and remove kernels, and it works with both GRUB and systemd-boot.
1
2
3
4
5
6
7
8
9
10
11
# Add a kernel
sudo kernel-install add $(uname -r) /boot/vmlinuz-$(uname -r)
# Remove a kernel
sudo kernel-install remove 5.15.0-91-generic
# List installed kernels
ls /boot/vmlinuz-*
# Inspect the kernel-install plugins
ls /usr/lib/kernel/install.d/
Tip: kernel-install is the correct way to manage kernels on modern systems. It handles the bootloader entry, the initramfs, and the UKI in one command.
Quick Reference
1
2
3
4
5
6
7
8
9
10
11
12
# Why is boot slow?
systemd-analyze critical-chain
# Reset root password
# At GRUB: append rd.break, then:
mount -o remount,rw /sysroot && chroot /sysroot && passwd root && touch /.autorelabel
# Rebuild initramfs
sudo dracut -f
# Check fstab
sudo findmnt --verify
Tip: Work in order: read the error, identify the stage, boot to the right target, fix, reboot. Most boot failures are a bad fstab entry, a missing initramfs driver, or a filesystem that needs fsck.
