Updating a kernel usually means a patch plus a reboot. On a laptop that is nothing, but on a database server, a host packing a hundred VMs, or a cluster expected to hold its uptime, every reboot is a maintenance window, a queue, and a risk. Live patching solves it differently: it applies the fix to a running kernel, without shutting anything down and without losing any process. Here is how it works underneath and how you actually do it.

The Basic Idea

A kernel in memory is a collection of functions. Most security fixes come down to swapping one or a few of them: the old function had a bug, the new one does not. Live patching does exactly that. It does not rewrite code in place; it redirects calls from the old function to a new version that is loaded into the kernel as a module.

The redirection mechanism is the key part. On Linux that job belongs to ftrace. The compiler leaves an instrumentation slot at the start of every function (the same hook points ftrace uses for tracing, which we covered in the ftrace post). The livepatch subsystem attaches its own handler there, intercepts the entry into the function before it touches the stack or the arguments, and jumps to the new version. The old function physically stays in memory, nobody just enters it anymore.

The Consistency Model, the Hard Part

Swapping a single function is easy. The trouble starts when a patch changes several functions at once, or changes the way functions talk to each other. You cannot let one process run half on the old code and another on the new one if those versions differ in semantics.

That is why livepatch uses a hybrid per-task consistency model. Once a patch is enabled the kernel enters a transition state: each task is migrated separately, at a safe moment. Sleeping tasks are checked by stack analysis (if no patched function sits on the stack, the task can be switched), tasks in userspace switch when they return from a syscall, and idle threads switch during the idle loop. You can watch the transition in /sys/kernel/livepatch/<patch>/transition and per process in /proc/<pid>/patch_state. Only when every task has crossed over is the patch fully active.

This model landed in mainline in Linux 4.9. The livepatch subsystem itself (CONFIG_LIVEPATCH) arrived earlier, in 4.0 back in 2015, as the common denominator of two independent solutions: kpatch from Red Hat and kGraft from SUSE.

A Short History and Who Does What

  • Ksplice (2008) came first. It started as an MIT project, then Oracle bought it and closed it. Today it patches essentially only Oracle Linux and is part of paid support.
  • kpatch (Red Hat, 2014) is the native solution for RHEL, CentOS Stream, AlmaLinux and Rocky. Open, but not automatic: you install the patches yourself.
  • kGraft (SUSE, 2014) is the SUSE-side counterpart, today shipping as SUSE Linux Enterprise Live Patching.
  • Canonical Livepatch is the Ubuntu service, wired into Ubuntu Pro. Automatic, but Ubuntu only.
  • KernelCare from TuxCare plays it differently: one agent covering dozens of distributions at once (RHEL, Debian, Ubuntu, Oracle, Alma, Rocky, Amazon Linux). Handy in mixed fleets where you would rather not run a separate tool per distribution.

The common thread is that kpatch, kGraft and Canonical Livepatch all stand on the same upstream livepatch subsystem. They mainly differ in who builds and delivers the patches and on which distribution.

In Practice: Building and Loading a Patch

On the Red Hat family the tool is kpatch-build. It takes an ordinary kernel-source diff and assembles a livepatch module from it, pulling out only the changed functions.

# install
sudo dnf install kpatch kpatch-build

# build a module from a patch (kpatch-build fetches the matching sources)
kpatch-build -t vmlinux cve-fix.patch

# load it live
sudo kpatch load livepatch-cve-fix.ko

# list active patches
kpatch list

The subsystem is also visible directly in sysfs:

ls /sys/kernel/livepatch/
cat /sys/kernel/livepatch/livepatch_cve_fix/enabled     # 1 = active
cat /sys/kernel/livepatch/livepatch_cve_fix/transition  # 0 = transition done

On Ubuntu with Ubuntu Pro you build nothing by hand, you just check the service status:

canonical-livepatch status

Newer patches can be cumulative: a module with the atomic replace flag supersedes all earlier patches at once, which simplifies managing the patch stack. When a patch has to add a field to an existing kernel structure, that is done through shadow variables, because you cannot extend the structure itself on the fly.

Limitations

Live patching is not magic and it does not replace updating:

  • Functions only. You can fix a bug inside a function’s code. You cannot change a data structure layout, an on-disk format, or the ABI. Shadow variables ease this partly, but not every change can be worked around that way.
  • Not every fix qualifies. Large architectural changes, or ones touching kernel initialization, simply need a reboot.
  • The patched function must be traceable by ftrace. Code invisible to ftrace is out of reach, and there are conflicts with kretprobes on the same functions.
  • It is a bridge, not a cure. It defers the reboot, it does not delete it. The in-memory kernel drifts further from the one on disk, so at some point you still schedule a window, if only to come down onto a fully fresh kernel.

When It Makes Sense

Anywhere a reboot is expensive and the security window is measured in hours: databases, virtualization hosts, cluster nodes, systems with a hard availability SLA. A critical kernel CVE, say the fresh DirtyClone family, can be closed in minutes instead of queuing a reboot for the weekend. On higher-risk machines it is worth pairing live patching with a module that detects kernel tampering at runtime, such as LKRG: one patches, the other watches.

If you build your own kernel, remember that on the config side you need CONFIG_LIVEPATCH (plus CONFIG_DYNAMIC_FTRACE_WITH_REGS), and for the full consistency model also CONFIG_HAVE_RELIABLE_STACKTRACE on a supported architecture. How to turn kernel options on is covered in compiling the kernel.

Sources: Kernel documentation: livepatch, Red Hat: what is Linux kernel live patching, TuxCare: live patching tools overview