What is Ftrace?

If you’ve ever wondered exactly what the Linux kernel is doing when your program calls read(), or why the system suddenly slowed down, ftrace (Function Tracer) is the answer.

ftrace is the official, built-in kernel tracing mechanism. It allows you to monitor function calls inside the kernel, trace latencies, context switches, and many other system events with microsecond precision.

Kernel Requirements

To use ftrace, your kernel must be compiled with the appropriate options. Most standard distributions (Ubuntu, Fedora, Debian) have these enabled by default. However, if you are compiling your own kernel, make sure you have set:

  • CONFIG_FUNCTION_TRACER=y - basic function tracing.
  • CONFIG_FUNCTION_GRAPH_TRACER=y - function graph tracing (entry and exit of functions).
  • CONFIG_DYNAMIC_FTRACE=y - allows dynamic enabling/disabling of tracing for specific functions (without this, the performance overhead would be huge).

You can check if your current kernel supports this by looking for the config file in /boot/ or checking if the /sys/kernel/debug/tracing directory exists.

How Does Ftrace Work?

Unlike strace, which traces system calls from the process’s point of view (user space), ftrace operates inside the kernel itself.

The kernel exposes the ftrace interface through a special tracefs filesystem (usually mounted at /sys/kernel/debug/tracing or /sys/kernel/tracing). We can control tracing simply by writing values to files in this directory.

Manual Control (For the Brave)

You can use ftrace directly from the terminal:

  1. Go to the tracing directory: 
    cd /sys/kernel/debug/tracing
  2. Select a tracer (e.g., function_graph, which draws a call tree): 
    echo function_graph > current_tracer
  3. Enable tracing: 
    echo 1 > tracing_on
  4. Perform some action on the system.
  5. Disable tracing and view the result: 
    echo 0 > tracing_on
cat trace | head

trace-cmd: Ftrace for Humans

While manual control is educational, everyone uses the trace-cmd tool for daily work. It is a convenient front-end for ftrace that automates setting options and collecting data.

Installation

On most distributions, the package is simply called trace-cmd:

sudo apt install trace-cmd  # Debian/Ubuntu
sudo dnf install trace-cmd  # Fedora/RHEL

Basic Usage

The simplest command to record the kernel function flow during the execution of a program (e.g., ls):

sudo trace-cmd record -p function_graph ls

This will create a trace.dat file. To analyze it, use:

trace-cmd report | less

You will then see a detailed graph of kernel function calls, allowing you to understand exactly which functions were called and how long they took.

When to Use Ftrace?

  • Kernel Performance Debugging: When the system is “laggy” and top doesn’t show the culprit.
  • Driver Analysis: To see how a driver communicates with hardware.
  • Learning: To understand how kernel subsystems work (e.g., the scheduler, filesystem).

Summary

ftrace is a powerful tool that turns the kernel from a “black box” into an open book. Although the entry barrier is higher than with strace, the analysis possibilities are virtually unlimited. If you want to get serious about Linux performance, ftrace (and its cousin perf) are tools you must know.