eBPF from Scratch, the Programmable Linux Kernel

Kprobes let you inject code anywhere in the kernel, but you do it by writing a kernel module. One bug and you get a kernel panic. eBPF solves that: you load a small program into the kernel, and a verifier guarantees it can’t do anything dangerous. It’s now the foundation of modern observability, networking and security on Linux.

What is eBPF?

eBPF (extended Berkeley Packet Filter) is a virtual machine inside the kernel. It runs small programs in kernel space, attached to events, without recompiling the kernel and without rebooting. Roughly, it works like this:

1. You write a program in C compiled to BPF bytecode, or in a higher-level language via bpftrace.
2. You load it with the bpf() system call.
3. The verifier checks it before it runs: no unbounded loops, no out-of-bounds memory access, guaranteed termination, no leaking kernel pointers to userspace.
4. The JIT compiles the bytecode to native machine code, so overhead is minimal.
5. The program attaches to a hook: kprobe, tracepoint, XDP, network socket, LSM, perf event, cgroup or uprobe.

The verifier is exactly what separates eBPF from a kernel module. A module can do anything, including taking the system down. An eBPF program that fails verification simply won’t load.

Maps: talking to userspace

An eBPF program doesn’t run in isolation. It exchanges data with userspace through maps (BPF maps): hash tables, arrays, ring buffers. The in-kernel program writes, a userspace tool reads. That’s how you get counters, histograms and event streams.

CO-RE and BTF: portability

The classic kprobes problem is an unstable ABI: kernel symbol names and struct layouts change between versions. eBPF solves this with CO-RE (Compile Once, Run Everywhere) and BTF (BPF Type Format). A kernel built with CONFIG_DEBUG_INFO_BTF=y exposes its own types in /sys/kernel/btf/vmlinux, and the eBPF program adapts to them at load time. One compiled program runs across many kernel versions.

Requirements

Most modern distros ship eBPF enabled. The key options:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_DEBUG_INFO_BTF=y    # for CO-RE

Check with:

grep -E 'CONFIG_BPF_SYSCALL|CONFIG_DEBUG_INFO_BTF' /boot/config-$(uname -r)
ls /sys/kernel/btf/vmlinux   # exists if BTF is available

You’ll want the bpftrace, bpfcc-tools (the BCC toolkit) and bpftool packages.

bpftrace, the fastest way in

bpftrace is a one-liner language for eBPF. No need to write or compile C. For example, show every process being executed on the system:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
    printf("%s -> %s\n", comm, str(args->filename));
}'

We used a tracepoint, not a kprobe. Tracepoints are a stable interface. Unlike kernel function names they don’t change with every release, so the script is portable.

Count syscalls per program over 10 seconds:

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# Ctrl+C ends it and prints the sorted counters

A histogram of disk read latency:

sudo bpftrace -e '
tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
    @us = hist((nsecs - @start[args->dev, args->sector]) / 1000);
    delete(@start[args->dev, args->sector]);
}'

Ready-made BCC tools

The bpfcc-tools package is dozens of ready eBPF programs you don’t have to write yourself:

sudo execsnoop-bpfcc      # every new process (exec in real time)
sudo opensnoop-bpfcc      # who opens which files
sudo tcpconnect-bpfcc     # new outbound TCP connections
sudo biolatency-bpfcc     # block I/O latency histogram
sudo tcpretrans-bpfcc     # TCP retransmits (diagnosing network issues)

The same idea as strace or tcpdump, but for the whole system at once and with negligible overhead. For broader profiling, see 5 tools for performance diagnostics.

Networking: XDP

XDP (eXpress Data Path) is an eBPF program attached as early as possible, in the network driver, before the kernel builds an sk_buff. For each packet it returns a verdict: XDP_PASS, XDP_DROP or XDP_TX. That’s how it reaches millions of packets per second, which is why it powers anti-DDoS solutions (Cloudflare, Meta’s Katran) and container networking (Cilium).

Check whether an XDP program is attached to an interface:

ip link show dev eth0        # look for "xdp" in the output
sudo bpftool net show        # network programs (XDP, tc) per interface

Ready-made filters live in the xdp-tools project, and tc with eBPF lets you do the same a little further up the network stack, with access to the full packet.

Runtime security

Since eBPF sees every system call, every connection and every process launch, it’s an ideal foundation for attack detection:

• Falco: rules that flag suspicious behavior (a shell in a container, writing to /etc/passwd, reading /etc/shadow).
• Tetragon (Cilium) and Tracee (Aqua): kernel-level observation and policy enforcement.
• LSM-BPF (CONFIG_BPF_LSM=y): lets you write Linux Security Module hooks as eBPF programs. A programmable AppArmor, you decide in the kernel whether an operation is allowed to proceed.

Unlike userspace logging, this is harder for an attacker to dodge, because the event is caught inside the kernel, at the moment of execution.

eBPF is attack surface too

Since an eBPF program runs in the kernel, a bug in the mechanism itself means privilege escalation. The verifier is a large, complex piece of code and has itself been a source of vulnerabilities, for instance CVE-2021-3490 and a series of LPEs through value-range tracking bugs. Two practical takeaways:

• Disable unprivileged eBPF. Set kernel.unprivileged_bpf_disabled = 1 (or 2) and net.core.bpf_jit_harden = 2. In the kernel config, enable CONFIG_BPF_UNPRIV_DEFAULT_OFF=y. This makes bpf() require CAP_BPF/root.
• Remember eBPF rootkits. Programs of type kprobe or fmod_ret can hide files, processes and connections, which is exactly what malware families like BPFDoor and Symbiote do.

Whether your kernel is correctly hardened for eBPF (and beyond) you can check in Kernel Security Checker. The configuration, sysctl and CVE tabs cover every setting above.

eBPF vs strace vs ftrace

strace is great for one process, ftrace and kprobes for the kernel, but eBPF ties it all together: observability, networking and security, at an overhead you can afford in production. If you only run one thing, start with bpftrace. A few one-liners and you can see what your system is actually doing.