The Last Resort

Imagine this scenario: you’re managing a remote server that suddenly becomes unresponsive. You can’t log in via SSH, websites aren’t working, and pings are either delayed or timing out. All you have left is a “hard” reboot through your hosting provider’s panel, risking data loss and filesystem corruption.

But what if there was a way to “talk” to the kernel even when the rest of the system is down? This last resort is the Magic SysRq key.

What is the “Magic SysRq Key”?

SysRq (System Request) is a special feature in the Linux kernel that allows you to perform low-level commands directly on the kernel, bypassing most of the normal system mechanisms. It works as long as the kernel itself is not completely “dead.” It’s a powerful tool for debugging and regaining emergency control over a frozen system.

How to Enable and Use SysRq

For security reasons, this feature is limited by default on many distributions. Full control is enabled using sysctl.

  1. Check the status:

    cat /proc/sys/kernel/sysrq

    A value of 1 means all functions are enabled. Other values (e.g., 176) mean that only some, safer commands are allowed.

  2. Enable full functionality (temporarily):

    sudo sysctl -w kernel.sysrq=1

  3. Enable it permanently: Add an entry to the /etc/sysctl.conf file (or a file in /etc/sysctl.d/):

    kernel.sysrq = 1

How to trigger the commands?

  • On a physical machine (with a keyboard): Hold down the Alt + SysRq keys (often the same key as Print Screen), and then press the key corresponding to the command (e.g., b for reboot). Alt + SysRq + b

  • On a remote machine (via SSH or console): We don’t have a physical keyboard, so we use the echo command to write the command letter to a special file in /proc:

    echo b | sudo tee /proc/sysrq-trigger

The Most Important SysRq Commands: The R-E-I-S-U-B Sequence

The most famous use of SysRq is to safely reboot a frozen system. Instead of a “hard” reset, we execute a sequence of commands that minimizes the risk of data loss. It’s easy to remember with the mnemonic “Raising Elephants Is So Utterly Boring” (or, in a less polite version, “Reboot Even If System Utterly Broken”).

Execute each command, waiting a few seconds in between:

  1. R - echo r > /proc/sysrq-trigger

    • Raw: Switches the keyboard from XLATE mode to raw mode, taking control away from things like the X.org graphical server. The first step to regaining control.

  2. E - echo e > /proc/sysrq-trigger

    • End: Sends the SIGTERM signal to all processes (except init), asking them to terminate gracefully.

  3. I - echo i > /proc/sysrq-trigger

    • Interrupt: Sends the SIGKILL signal to all processes (except init), brutally killing them. This is in case they didn’t listen to the polite request.

  4. S - echo s > /proc/sysrq-trigger

    • Sync: Synchronizes all mounted filesystems. It writes all data from buffers to the disk. This is a crucial step to prevent data loss!

  5. U - echo u > /proc/sysrq-trigger

    • Unmount: Remounts all filesystems in read-only mode. This is an additional safeguard against data corruption during the reboot.

  6. B - echo b > /proc/sysrq-trigger

    • Boot: Immediately reboots the system, without unmounting filesystems or syncing (but we already did that in steps s and u). It’s the equivalent of pressing the “Reset” button.

Other useful commands

  • h - Help: Displays a list of available commands in the kernel log (dmesg).
  • m - Memory: Displays information about memory usage.
  • t - Tasks: Shows a list of currently running tasks (processes).
  • f - Freeze: Invokes the “OOM killer” (Out-of-Memory killer) to kill the most memory-intensive process. Useful when the system has frozen due to lack of memory.

Summary

The Magic SysRq key is a powerful diagnostic and recovery tool. While not needed for everyday use, in a crisis situation, knowing the REISUB sequence can save your data and spare you a lot of trouble. It’s worth enabling this feature on your servers and memorizing at least this one, most important sequence.