Linux Text Processing: grep, sed, awk, cut, sort, and xargs
The Unix philosophy is small tools that do one thing well, composed with pipes. The text-processing trio — grep, sed, and awk — is the heart of that philosophy, and mastering them turns hours of manual log reading into a one-liner.
grep: Finding Lines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Basic search
grep "error" /var/log/syslog
# Case-insensitive
grep -i "error" /var/log/syslog
# Invert match
grep -v "DEBUG" app.log
# Recursive search
grep -r "TODO" src/
# Show line numbers
grep -n "error" app.log
# Show only the matching part
grep -o "req_id=[a-z0-9]*" app.log
# Count matches
grep -c "error" app.log
# Extended regex
grep -E "error|warn|critical" app.log
# Fixed string, no regex
grep -F "1.2.3.4" access.log
Tip: grep -F is dramatically faster than grep when you are searching for a literal string, because it skips regex compilation entirely.
sed: Stream Editing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Replace the first occurrence per line
sed 's/foo/bar/' file.txt
# Replace all occurrences per line
sed 's/foo/bar/g' file.txt
# Edit in place
sed -i 's/foo/bar/g' file.txt
# Edit in place with a backup
sed -i.bak 's/foo/bar/g' file.txt
# Delete lines matching a pattern
sed '/DEBUG/d' app.log
# Print only matching lines
sed -n '/error/p' app.log
# Print a line range
sed -n '10,20p' file.txt
# Insert a line before a match
sed '/^server {/i # managed by ansible' nginx.conf
Tip: sed -i on macOS requires an argument (sed -i ''). On Linux it does not. If you write scripts for both, use sed -i.bak and delete the backup.
awk: Field Processing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Print the first column
awk '{print $1}' file.txt
# Print specific columns
awk '{print $1, $3}' file.txt
# Custom field separator
awk -F: '{print $1, $7}' /etc/passwd
# Filter by condition
awk '$3 > 100 {print $1, $3}' file.txt
# Sum a column
awk '{sum += $2} END {print sum}' file.txt
# Average a column
awk '{sum += $2; n++} END {print sum/n}' file.txt
# Group and sum
awk '{sum[$1] += $2} END {for (k in sum) print k, sum[k]}' file.txt
Tip: awk is a full programming language. If you find yourself writing a loop in bash to process columns, awk is almost always shorter and faster.
cut, sort, uniq
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Extract a field by delimiter
cut -d: -f1 /etc/passwd
# Extract a character range
cut -c1-10 file.txt
# Sort lines
sort file.txt
# Sort numerically
sort -n file.txt
# Sort by a specific field
sort -t: -k3 -n /etc/passwd
# Unique lines
uniq file.txt
# Count occurrences
sort file.txt | uniq -c
# Top 10 most frequent
sort file.txt | uniq -c | sort -rn | head
Tip: uniq only removes adjacent duplicates. Always sort first, or you will get wrong counts.
xargs: Building Command Lines
1
2
3
4
5
6
7
8
9
10
11
# Run a command for each line
cat files.txt | xargs rm
# Handle filenames with spaces
find . -name "*.log" -print0 | xargs -0 rm
# Run in parallel
find . -name "*.jpg" -print0 | xargs -0 -P 4 -I {} convert {} {}.png
# Limit arguments per invocation
cat urls.txt | xargs -n 1 curl -O
Tip: Always use -print0 with find and -0 with xargs when filenames may contain spaces. It is the single most common source of “xargs deleted the wrong file” incidents.
Composing the Tools
The real power is composition.
1
2
3
4
5
6
7
8
9
10
11
# Top 10 IPs in an access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# Count errors per hour
grep "ERROR" app.log | awk '{print $1, $2}' | cut -d: -f1 | sort | uniq -c
# Find the largest files in a directory tree
find . -type f -printf '%s %p\n' | sort -rn | head -20
# Replace a value across many files
grep -rl "old_value" . | xargs sed -i 's/old_value/new_value/g'
Tip: The last one is the pattern you will use most often: grep -rl to find the files, xargs sed -i to change them. It is faster and safer than a for loop.
awk Deep Dive
awk is a full programming language. These are the patterns that go beyond simple column extraction.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# BEGIN and END blocks
awk 'BEGIN {print "start"} {print} END {print "end"}' file.txt
# Multiple conditions
awk '$3 > 100 && $4 == "active" {print $1}' file.txt
# Regular expression match
awk '/error/ {print $0}' app.log
# Field separator as a regex
awk -F'[,;]' '{print $2}' file.txt
# Output field separator
awk -F: 'BEGIN {OFS=","} {print $1, $7}' /etc/passwd
# Multi-line records
awk 'BEGIN {RS=""; FS="\n"} {print $1}' file.txt
# Arrays and grouping
awk '{count[$1]++} END {for (k in count) print k, count[k]}' file.txt
# Sort within awk
awk '{print $2, $1}' file.txt | sort -n
Tip: awk is almost always shorter and faster than a bash loop for column processing. If you find yourself writing a while read loop to process fields, reach for awk instead.
sed Hold Space
sed has two buffers: the pattern space (the current line) and the hold space (a scratch buffer). The hold space lets you do multi-line operations.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Reverse the order of lines in a file
sed -n '1!G;h;$p' file.txt
# Join pairs of lines
sed 'N;s/\n/ /' file.txt
# Delete the last line
sed '$d' file.txt
# Print the line before a match
sed -n '/error/{x;p;d};x' app.log
# Replace across multiple lines
sed ':a;N;$!ba;s/foo\nbar/baz/g' file.txt
Tip: The hold space is the most powerful and least understood part of sed. The 1!G;h;$p idiom for reversing a file is worth memorising.
grep -P: Perl-Compatible Regex
1
2
3
4
5
6
7
8
9
10
11
# Lookahead
grep -P 'foo(?=bar)' file.txt
# Lookbehind
grep -P '(?<=foo)bar' file.txt
# Non-greedy match
grep -P 'foo.*?bar' file.txt
# Named groups
grep -P '(?<name>\w+)' file.txt
Tip: grep -P is not available on every platform (notably macOS). If you need portability, use grep -E or awk.
jq and yq: Structured Data
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Pretty-print JSON
jq . file.json
# Extract a field
jq -r '.name' file.json
# Filter an array
jq -r '.[] | select(.status == "active") | .name' file.json
# Modify and output
jq '.name = "new"' file.json
# YAML: read a value
yq '.spec.replicas' deployment.yaml
# YAML: modify a value
yq -i '.spec.replicas = 3' deployment.yaml
Tip: jq and yq are the correct tools for structured data. Do not parse JSON with grep and sed; it works until it does not.
Quick Reference
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Find lines
grep -i "error" app.log
# Replace text
sed -i 's/old/new/g' file.txt
# Process columns
awk -F: '{print $1, $7}' /etc/passwd
# Top 10 most frequent
sort file.txt | uniq -c | sort -rn | head
# Safe xargs
find . -name "*.log" -print0 | xargs -0 rm
Tip: Work in order: grep to find, sed to change, awk to compute. Most text-processing tasks are a pipeline of those three, and most bugs are a missing sort before uniq.
