Kprobes — Dynamic Kernel Function Interception on Linux

Ftrace lets you trace kernel function calls. Kprobes go further — they let you inject your own code at any point in the kernel, without recompiling and without rebooting.

What Are Kprobes?

Kprobes (Kernel Probes) is a Linux kernel mechanism that lets you dynamically attach a handler to virtually any instruction in the kernel. When the CPU hits the probe point, execution is temporarily redirected to your code, then returns to normal flow.

Three variants:

• kprobe — fires before the target instruction executes (typically at function entry)
• kretprobe — fires at function return, gives access to the return value
• jprobe — deprecated (removed since kernel 4.15), replaced by kprobe with pre_handler

Requirements

The kernel needs:

CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y

Most distros have this enabled by default. Check with:

grep CONFIG_KPROBES /boot/config-$(uname -r)

You also need debugfs mounted:

mount | grep debugfs
# If not mounted:
sudo mount -t debugfs none /sys/kernel/debug

Kprobes via debugfs

The quickest way to set a probe — no module writing, no compilation.

Intercepting Function Entry

We want to know when the kernel opens files — hook into do_sys_openat2:

# Set the probe
echo 'p:myprobe do_sys_openat2 filename=+0(%si):string' > /sys/kernel/debug/tracing/kprobe_events

# Enable it
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable

# Watch
cat /sys/kernel/debug/tracing/trace_pipe

Every file open in the system now shows up in trace_pipe with the filename. +0(%si) is the second function argument (register si on x86_64), cast to string.

Cleanup:

echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo '-:myprobe' >> /sys/kernel/debug/tracing/kprobe_events

Intercepting Returns (kretprobe)

We want to know how long vfs_read takes and what it returns:

echo 'r:myretprobe vfs_read ret=$retval' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable
cat /sys/kernel/debug/tracing/trace_pipe

$retval is the function’s return value — for vfs_read that’s the number of bytes read (or an error).

Kprobes via perf

perf can set kprobes without manually touching debugfs:

# Add probe
sudo perf probe --add 'do_sys_openat2 filename=+0(%si):string'

# Record events
sudo perf record -e probe:do_sys_openat2 -a -- sleep 5

# View
sudo perf script

Remove:

sudo perf probe --del do_sys_openat2

Probing a Specific Line in a Function

perf probe can hook not just at entry, but mid-function:

# List available probe points in tcp_sendmsg
sudo perf probe -L tcp_sendmsg

# Probe at specific line
sudo perf probe 'tcp_sendmsg:15 size'

Requires debug symbols (kernel-debuginfo / linux-image-*-dbg).

Kprobes via bpftrace

If you have bpftrace, this is the most convenient way:

# Who opens which files?
sudo bpftrace -e 'kprobe:do_sys_openat2 { printf("%s: %s\n", comm, str(arg1)); }'

# How long does vfs_read take?
sudo bpftrace -e '
kprobe:vfs_read { @start[tid] = nsecs; }
kretprobe:vfs_read /@start[tid]/ {
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}'

# Top 10 most called kernel functions
sudo bpftrace -e 'kprobe:* { @[probe] = count(); } interval:s:5 { print(@, 10); clear(@); }'

That last example can heavily load the system — be careful in production.

Practical Use Cases

Who’s Killing Processes?

sudo bpftrace -e 'kprobe:do_send_sig_info {
    printf("%s (pid=%d) -> signal %d to pid %d\n",
        comm, pid, arg0, ((struct task_struct *)arg2)->pid);
}'

Tracking Memory Allocations >1MB

sudo bpftrace -e 'kprobe:__alloc_pages /arg1 > 8/ {
    printf("%s: alloc order %d (%d KB)\n", comm, arg1, (1 << arg1) * 4);
}'

DNS Latency (udp_sendmsg to Port 53)

sudo bpftrace -e 'kprobe:udp_sendmsg {
    $sk = (struct sock *)arg0;
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    if ($dport == 53) {
        printf("%s -> DNS query\n", comm);
    }
}'

Limitations

• Can’t probe inline functions — the compiler inlines them, they have no address
• Can’t probe kprobes themselves — recursion
• Blacklist — some critical functions are blocked (/sys/kernel/debug/kprobes/blacklist)
• Overhead — each probe is an interrupt, thousands of probes on hot paths cause noticeable slowdown
• Unstable ABI — kernel function names change between versions, probes are not portable

Kprobes vs ftrace vs eBPF

Kprobes are the foundation — ftrace and eBPF often use them under the hood. If you need to quickly check something in production, bpftrace with kprobes is the best combination.