Last modified: June 01, 2025
This article is written in: 🇺🇸
We'll explore the inner workings of the Linux kernel, focusing on how it loads and executes binaries. We'll dive into the execve
system call, build a custom kernel, and use debugging tools to see the execution process in action. Whether you're a seasoned developer or just curious about operating systems, this walkthrough aims to shed light on the fascinating journey from a command to a running program.
execve
System CallAt the heart of program execution in Linux lies the execve
system call. This function is responsible for replacing the current process image with a new one, effectively running a new program within the same process. When you run a command in the terminal, execve
is the mechanism that makes it happen.
In C, the function signature of execve
is:
int execve(const char *pathname, char *const argv[], char *const envp[]);
Let's break down the parameters:
When a program calls execve
, the kernel loads the specified executable into memory, sets up the new environment, and starts executing the program from its entry point. The original process is overwritten, but it retains its process ID, which is why execve
doesn't create a new process but transforms the existing one.
execve
in Action with strace
To see how execve
operates under the hood, we can use a tool called strace
. This utility traces system calls made by a program, allowing us to observe interactions with the kernel.
For example, if we want to see how the ls
command is executed, we can run:
strace -e execve ls
The -e execve
option tells strace
to filter and display only execve
system calls. When you run this command, you'll see output showing which executables are being invoked, along with the arguments and environment variables. This simple exercise reveals the essential role execve
plays in running even the most basic commands.
To truly understand how the kernel executes programs, we can build our own version of the Linux kernel. This allows us to customize it for debugging and explore its internals.
Start by cloning the official Linux kernel repository:
git clone https://github.com/torvalds/linux.git
This command downloads the entire kernel source code into a directory named linux
. Navigate into this directory:
cd linux
Before compiling, we need to configure the kernel to suit our needs. We can generate a default configuration as a starting point:
make defconfig
This command creates a standard configuration file based on your system's architecture.
To make debugging easier and reduce compilation time, we'll adjust some settings:
I. Disable Address Space Layout Randomization (KASLR): KASLR randomizes the memory address where the kernel is loaded, which can complicate debugging. To disable it:
make menuconfig
to open the configuration menu.II. Streamline the Kernel: Disabling unnecessary features speeds up compilation and simplifies debugging.
III. Enable Debugging Options: These settings provide valuable information when debugging.
After making these changes, save your configuration and exit the menu.
Now we're ready to compile the kernel:
make -j$(nproc)
The -j$(nproc)
option tells make
to use all available CPU cores, speeding up the process. Compiling the kernel can take some time, so feel free to take a break while it builds.
Once compilation is complete, generate the GDB scripts:
make scripts_gdb
These scripts help GDB (the GNU Debugger) understand kernel data structures during debugging sessions.
An initramfs (initial RAM filesystem) is a simple filesystem loaded into memory during the boot process. We'll create one containing a statically linked shell and a sample program to test our custom kernel.
First, we'll build a minimal shell to use as the initial process:
I. Download BusyBox: BusyBox combines tiny versions of many common UNIX utilities into a single small executable.
wget https://busybox.net/downloads/busybox-1.35.0.tar.bz2
tar -xjf busybox-1.35.0.tar.bz2
cd busybox-1.35.0
II. Configure for Static Linking:
make defconfig
Then, edit the .config
file or run:
make menuconfig
Under BusyBox Settings, ensure Build BusyBox as a static binary (no shared libs) is enabled.
III. Compile BusyBox:
make -j$(nproc)
IV. Install BusyBox:
make install
This installs BusyBox into a _install
directory.
Create a directory to hold the initramfs contents:
mkdir -p ../initramfs/{bin,sbin,etc,proc,sys,usr/{bin,sbin}}
Copy BusyBox into the initramfs:
cp -r _install/* ../initramfs/
Create a simple init
script that the kernel will execute first:
cat > ../initramfs/init << 'EOF'
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
echo "Welcome to the custom initramfs shell!"
exec /bin/sh
EOF
Make the script executable:
chmod +x ../initramfs/init
Let's create a simple C program to test:
cat > ../initramfs/hello.c << 'EOF'
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
int main() {
printf("Hello from the custom kernel!\n");
return 0;
}
EOF
Compile the program statically:
gcc -static -o ../initramfs/bin/hello ../initramfs/hello.c
Finally, create the initramfs cpio archive:
cd ../initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../initramfs.cpio.gz
cd ..
This command finds all files in the initramfs directory, creates a cpio archive in the newc
format, and compresses it with gzip.
Now that we have a custom kernel and initramfs, we can boot them using QEMU, an open-source emulator that supports virtualization.
Run the following command:
qemu-system-x86_64 \
-kernel linux/arch/x86/boot/bzImage \
-initrd initramfs.cpio.gz \
-append "console=ttyS0" \
-nographic \
-s -S
Here's what the options mean:
With these options, QEMU waits for a debugger to connect before starting the kernel.
Now we can connect GDB to QEMU to debug the kernel.
In a new terminal window, navigate to the kernel source directory and start GDB:
gdb linux/vmlinux
Within GDB, connect to QEMU:
(gdb) target remote :1234
The kernel is now paused and ready for debugging.
Let's set a breakpoint at the do_execve
function:
(gdb) break do_execve
Then, continue execution:
(gdb) continue
Switch back to the QEMU terminal. The kernel should now boot and drop into our custom shell.
In the QEMU shell, run our sample program:
./hello
Because we set a breakpoint at do_execve
, GDB will pause execution when the kernel attempts to execute hello
.
do_execve
Back in GDB, we can step through the do_execve
function to observe how the kernel handles the execution request:
(gdb) step
As you step through, pay attention to how the kernel prepares the binary for execution, including loading the executable, setting up memory spaces, and initializing the process environment.
We can inspect variables and structures to understand the state of the kernel. For example, to view the binary parameters:
(gdb) print *bprm
This displays the contents of the linux_binprm
structure, which holds information about the binary being executed.
The kernel supports multiple binary formats (like ELF, scripts, etc.). It determines how to execute a file based on its format.
search_binary_handler
The function search_binary_handler
is responsible for finding the appropriate handler for the binary:
(gdb) break search_binary_handler
(gdb) continue
When the breakpoint hits, you can examine the available handlers:
(gdb) print fmt->name
By continuing execution and checking the handler's name, you can see how the kernel selects the ELF handler for our compiled program.
Once the ELF handler is selected, the kernel uses load_elf_binary
to load the executable.
load_elf_binary
Continue stepping through the code:
(gdb) step
Within load_elf_binary
, the kernel reads the ELF header, sets up memory mappings, and prepares the process for execution.
You can examine the ELF header:
(gdb) print elf_ex.e_ident
This should display the magic numbers identifying the file as an ELF executable.
After loading the binary, the kernel sets up the initial CPU state and starts the program.
start_thread
The start_thread
function sets the instruction pointer and stack pointer for the new process:
(gdb) step
Check the instruction pointer:
(gdb) print/x regs->ip
This address should correspond to the entry point of our hello
program.
To confirm, you can check the entry point in the compiled binary:
readelf -h hello
Look for the Entry point address and compare it to the instruction pointer value from GDB.
Once the setup is complete, we can allow the program to run:
(gdb) continue
Switch back to the QEMU terminal. You should see the message from our program:
Hello from the custom kernel!
This indicates that the kernel successfully executed our program.
exec_test.c
that calls execve
to run /bin/echo
with arguments echo HelloWorld
. Compile it and run it. Then, modify it to print a message before and after the execve
call. Observe and explain why the “after” message never appears. Reflect on how execve
replaces the current process image.strace
to trace a command that invokes a shell script (e.g., ./myscript.sh
). In your script, have one line that executes another binary (e.g., /bin/ls
). Run strace -e execve ./myscript.sh
and identify each execve
call in the output. Discuss how the kernel prepares and hands off execution at each stage.git clone https://github.com/torvalds/linux.git
) into a directory of your choice. Navigate into it and run make defconfig
. Without compiling the entire kernel, open the generated .config
file and locate the configuration option for KASLR. Explain in your own words what KASLR does and why disabling it aids in kernel debugging.make menuconfig
, disable KASLR under Processor type and features, disable loadable module support, and disable networking support. Save the configuration and exit. Then, write a short paragraph (in a text file called config_changes.txt
) describing each change you made and why it speeds up or simplifies debugging.make -j$(nproc)
to compile the kernel (or at least start the compile until you feel comfortable pausing). While it’s building, investigate where the resulting bzImage
and vmlinux
files will reside once compilation completes. After compilation, locate and list the paths to those two files, then explain the difference between them (bzImage
versus vmlinux
).wget https://busybox.net/downloads/busybox-1.35.0.tar.bz2
), extract it, and configure it for static linking (make defconfig
+ enable static in menuconfig
). Build BusyBox statically and install it into a directory named busybox_root
. Then, inspect the resulting BusyBox binary with ldd busybox_root/bin/busybox
to verify it is truly statically linked, and write a brief note about why static linking matters for an initramfs.initramfs
directory structure (mkdir -p initramfs/{bin,sbin,etc,proc,sys,usr/{bin,sbin}}
). Copy your static BusyBox files into initramfs
. Write a minimal init
script in initramfs/init
that mounts /proc
and /sys
, prints “Custom Initramfs Loaded!”, and then invokes /bin/sh
. Make the init
script executable. Finally, create the cpio archive (cd initramfs && find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../initramfs.cpio.gz
). Verify the archive exists and note its size.hello.c
) in your initramfs/bin
folder that prints “Hello from initramfs!”. Compile it statically (gcc -static -o initramfs/bin/hello hello.c
). Verify via file initramfs/bin/hello
that it is an ELF binary. Explain why the kernel will be able to run this program without any userspace libraries.do_execve
function (break do_execve
) and continue. In the QEMU serial console, run /bin/hello
. When GDB breaks at do_execve
, step through (step
) a few instructions and use print *bprm
to inspect the linux_binprm
structure. Write a short explanation (in a file called gdb_inspection.txt
) of what bprm
contains and how the kernel uses it to prepare the hello
binary for execution.bzImage
with your initramfs.cpio.gz
using:
qemu-system-x86_64 \
-kernel path/to/arch/x86/boot/bzImage \
-initrd initramfs.cpio.gz \
-append "console=ttyS0" \
-nographic \
-s -S
Once QEMU is waiting for a debugger connection, open another terminal, navigate to your kernel source, and run gdb vmlinux
. In GDB, do target remote :1234
and then continue
. Describe what you see on the QEMU console and why you’re dropped into your initramfs shell.