Post

Core Dumps and coredumpctl: Capturing and Analysing Crashes

Core Dumps and coredumpctl: Capturing and Analysing Crashes

When a process crashes, the kernel can write its memory to a core file. That file is the only record of what the process was doing at the moment it died, and on systemd hosts it is managed by systemd-coredump and accessed through coredumpctl.

Enabling Core Dumps

1
2
3
4
5
6
7
8
9
10
11
# Check the current limit
ulimit -c

# Enable for the current shell
ulimit -c unlimited

# Enable system-wide
echo '* soft core unlimited' | sudo tee -a /etc/security/limits.conf

# Check where cores go
cat /proc/sys/kernel/core_pattern

Tip: On systemd hosts, core_pattern is usually |/usr/lib/systemd/systemd-coredump ..., which means cores go to the journal, not to a file. coredumpctl is the way to read them.

Listing Cores with coredumpctl

1
2
3
4
5
6
7
8
9
10
11
# All recorded cores
coredumpctl list

# Cores for a specific binary
coredumpctl list myapp

# Cores since a specific time
coredumpctl list --since "1 hour ago"

# Details of the most recent core
coredumpctl info

Tip: coredumpctl list shows the PID, the signal, and the time of each crash. It is the fastest way to confirm whether a process is crashing repeatedly.

Analysing a Core

1
2
3
4
5
6
7
8
9
10
11
# Open the most recent core in gdb
coredumpctl gdb

# Open a specific core
coredumpctl gdb 1234

# Inside gdb:
#   bt
#   info registers
#   thread apply all bt
#   quit

Tip: coredumpctl gdb loads the binary and the core together, so you do not need to find either by hand. It is the fastest path from “it crashed” to a backtrace.

Extracting a Core File

1
2
3
4
5
# Dump the core to a file
coredumpctl dump 1234 --output /tmp/core.1234

# Dump the most recent core
coredumpctl dump --output /tmp/core.latest

Tip: Extracting the core is useful when you need to analyse it on a different machine or attach it to a bug report.

Configuring systemd-coredump

1
2
3
4
5
6
7
8
# /etc/systemd/coredump.conf

[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=2G
MaxUse=1G
1
2
# Apply the change
sudo systemctl restart systemd-coredump.socket

Tip: Storage=external writes cores to /var/lib/systemd/coredump/. Storage=journal keeps them in the journal, which is convenient but can fill the disk quickly.

Cleaning Up Cores

1
2
3
4
5
6
7
8
# Remove cores older than 7 days
sudo journalctl --vacuum-time=7d

# Remove all cores
sudo rm -rf /var/lib/systemd/coredump/*

# Check disk usage
du -sh /var/lib/systemd/coredump/

Tip: Cores are large. A single core can be gigabytes, and a crash loop can fill a disk in minutes. Set MaxUse in coredump.conf to cap the total size.

coredumpctl debug: Interactive Debugging

coredumpctl debug opens the core in gdb with the binary and symbols already loaded.

1
2
3
4
5
6
7
8
# Open the most recent core in gdb
coredumpctl debug

# Open a specific core
coredumpctl debug 1234

# Pass extra arguments to gdb
coredumpctl debug 1234 -- -ex "thread apply all bt"

Tip: coredumpctl debug is the fastest path from “it crashed” to a backtrace. It handles the binary, the core, and the debug symbols in one command.

systemd-coredump Configuration

1
2
3
4
5
6
7
8
9
# /etc/systemd/coredump.conf

[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=2G
MaxUse=1G
KeepFree=1G
1
2
# Apply the change
sudo systemctl restart systemd-coredump.socket

Tip: KeepFree= reserves free space on the filesystem, which prevents a crash loop from filling the disk. It is the single most important setting on a production host.

systemd-pstore: Persistent Storage for Cores

systemd-pstore saves kernel oops and panic data from the firmware’s persistent storage (pstore) to disk, so it survives a reboot.

1
2
3
4
5
6
7
8
# Is it running?
systemctl status systemd-pstore

# Show saved pstore data
ls /var/lib/systemd/pstore/

# Trigger a save manually
sudo systemctl start systemd-pstore

Tip: systemd-pstore is the only way to recover a kernel panic log after a reboot. It is enabled by default on most modern distributions.

Quick Reference

1
2
3
4
5
6
7
8
9
10
11
# List cores
coredumpctl list

# Analyse the most recent core
coredumpctl gdb

# Extract a core
coredumpctl dump 1234 --output /tmp/core.1234

# Clean up
sudo journalctl --vacuum-time=7d

Tip: Work in order: list the cores, open the most recent in gdb, get a backtrace, then fix the bug. Most crash investigations are resolved by coredumpctl gdb and bt.

This post is licensed under CC BY 4.0 by the author.