ftrace (Function Tracer) is the kernel’s built-in tracing mechanism. It monitors function calls inside the kernel, traces latencies, context switches, and other system events with microsecond precision.

Kernel Requirements

To use ftrace, your kernel must be compiled with the appropriate options. Most standard distributions (Ubuntu, Fedora, Debian) have these enabled by default. However, if you are compiling your own kernel, make sure you have set:

  • CONFIG_FUNCTION_TRACER=y - basic function tracing.
  • CONFIG_FUNCTION_GRAPH_TRACER=y - function graph tracing (entry and exit of functions).
  • CONFIG_DYNAMIC_FTRACE=y - allows dynamic enabling/disabling of tracing for specific functions (without this, the performance overhead would be huge).

You can check if your current kernel supports this by looking for the config file in /boot/ or checking if the /sys/kernel/debug/tracing directory exists.

How Does Ftrace Work?

Unlike strace, which traces system calls from the process’s point of view (user space), ftrace operates inside the kernel itself.

The kernel exposes the ftrace interface through a special tracefs filesystem (usually mounted at /sys/kernel/debug/tracing or /sys/kernel/tracing). We can control tracing simply by writing values to files in this directory.

Manual Control

You can use ftrace directly from the terminal:

  1. Go to the tracing directory: 
    cd /sys/kernel/debug/tracing
  2. Select a tracer (e.g., function_graph, which draws a call tree): 
    echo function_graph > current_tracer
  3. Enable tracing: 
    echo 1 > tracing_on
  4. Perform some action on the system.
  5. Disable tracing and view the result: 
    echo 0 > tracing_on
cat trace | head

trace-cmd: Ftrace for Humans

While manual control is educational, everyone uses the trace-cmd tool for daily work. It is a convenient front-end for ftrace that automates setting options and collecting data.

Installation

On most distributions, the package is simply called trace-cmd:

sudo apt install trace-cmd  # Debian/Ubuntu
sudo dnf install trace-cmd  # Fedora/RHEL

Basic Usage

The simplest command to record the kernel function flow during the execution of a program (e.g., ls):

sudo trace-cmd record -p function_graph ls

This will create a trace.dat file. To analyze it, use:

trace-cmd report | less

You will then see a detailed graph of kernel function calls, allowing you to understand exactly which functions were called and how long they took.

When to Use Ftrace?

  • Kernel Performance Debugging: When the system is “laggy” and top doesn’t show the culprit.
  • Driver Analysis: To see how a driver communicates with hardware.
  • Learning: To understand how kernel subsystems work (e.g., the scheduler, filesystem).

For deeper performance work, ftrace pairs well with perf — together they cover most kernel-level profiling and tracing needs.