The Journey Before main()

A while back, I worked on a RISC-V-based userspace simulator for fun. In doing so, taught myself a lot more than I wanted to know about what happens in-between when the Kernel is asked to run a program, and when the first line of our program’s main function is actually executed. Here’s a summary of that rabbit hole.

In the beginning…

First question: When is the OS kernel actually asked to run any program? The answer, at least on Linux, is the execve system call (“syscall”). Let’s take a quick look at that:

int execve(const char *filename, char *const argv[], char *const envp[]);

This is actually quite straightforward! We pass the name of the exectuable file, a list of arguments, and a list of environment variables. This signals to the kernel where, and how, to start loading the program.

Many programming languages provide an interface to execute commands that eventually call execve under the hood. For example, in Rust, we have:

use std::process::Command; Command::new("ls").arg("-l").spawn();

In these higher-level wrappers, the language’s standard library often handles translation of the command name to a full path, acting similarly to how a shell would resolve the command via the PATH environment variable. The kernel itself, however, expects a proper path to an executable file.

A note on interpreters: If the executable file starts with a shebang (#!), the kernel will use the shebang-specified interpreter to run the program. For example, #!/usr/bin/python3 will run the program using the Python interpreter, #!/bin/bash will run the program using the Bash shell, etc.

ELF

What does an executable file look like? On Linux, it’s ELF, which the kernel knows how to parse. Other operating systems have different formats (e.g. Mach-O on MacOS, PE on Windows), but ELF is the most common format on Linux. I won’t go into too much detail here, to keep things brief, but ELF files have grown out of the original a.out format, and are expressive enough to support pretty much every program you’ll ever write. Here’s what the header of an ELF file looks like:

% readelf -h main # main is an ELF file ELF Header: Magic: 7f 45 4c 46 01 01 01 03 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - GNU ABI Version: 0 Type: EXEC (Executable file) Machine: RISC-V Version: 0x1 Entry point address: 0x10358 Start of program headers: 52 (bytes into file) Start of section headers: 675776 (bytes into file) Flags: 0x1, RVC, soft-float ABI Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 7 Size of section headers: 40 (bytes) Number of section headers: 32 Section header string table index: 31

The important parts here are:

1. The “ELF Magic” bytes, which tell the kernel that this is, indeed, an ELF file. 45 4c 46 is ASCII for “ELF”!
2. “Class” tells us we’re dealing with a 32-bit executable.
3. “Start of …” tells us where things are in the file, and “Size of …” tells us how big they are; The kernel is effectively given a map of the file.
4. “Entry point address” — Relatively self-explanatory! But we’ll be coming back to this.

Other ELF files will have different entries and specific values, but the general structure is what we’re after here.

As you can see by the numerous mentions to “RISC-V”, this is an ELF file I compiled and linked targeting the RV32 architecture (which the aforementioned emulator is built for), hence the “32” in “ELF32”, the “RVC” flag, and the “RISC-V” machine type.

More than just a header

ELF files contain everything our program needs to run, including the code, data, symbols, and more. We can see this again with the readelf command with the -a flag. Here’s what we care about:

Section Headers: [Nr] Name Type Addr Off Size [ 0] NULL 00000000 000000 000000 [ 1] .note.ABI-tag NOTE 00010114 000114 000020 [ 2] .rela.plt RELA 00010134 000134 00000c [ 3] .plt PROGBITS 00010140 000140 000010 [ 4] .text PROGBITS 00010150 000150 03e652 [ 5] .rodata PROGBITS 0004e7b0 03e7b0 01b208 ... [16] .data PROGBITS 0007a008 069008 000dec [17] .sdata PROGBITS 0007adf4 069df4 000004 [18] .bss NOBITS 0007adf8 069df8 002b6c ... [29] .symtab SYMTAB 00000000 095124 009040 [30] .strtab STRTAB 00000000 09e164 006d10

These sections contain code (.text), data (.data), space for global variables (.bss), shims for accessing shared library functions (.plt), and quite a bit more (including symbol tables for debugging, relocation tables, etc.), most of which we won’t be discussing.

So evidently, there’s some code that we care about in the .text section, so we copy that and call it a day? Not quite. There’s a massive amount of machinery inside the kernel to make all sorts of programs under all sorts of conditions run.

For example, the “PLT” (Procedure Linkage Table) is a section that allows us to call functions in “shared libraries”, for example, libc, without having to package them alongside our program (“dynamically” vs “statically linking”). The ELF file contains a dynamic section which tells the kernel which shared libraries to load, and another section which tells the kernel to dynamically “relocate” pointers to those functions, so everything checks out.

libc is the C standard library, which contains all the “useful” functions: printf, malloc, etc. Various flavors implementing the libc interfaces exist, most commonly glibc and musl. Most of the binaries that are discussed in this post are compiled and linked against musl, since it’s much easier to statically link.

The symbol table looks something like this:

Symbol table '.symtab' contains 2308 entries: Num: Value Size Type Bind Vis Ndx Name 0: 00000000 0 NOTYPE LOCAL DEFAULT UND 1: 00010114 0 SECTION LOCAL DEFAULT 1 .note.ABI-tag 2: 00010134 0 SECTION LOCAL DEFAULT 2 .rela.plt 3: 00010140 0 SECTION LOCAL DEFAULT 3 .plt 4: 00010150 0 SECTION LOCAL DEFAULT 4 .text ... 1782: 00010358 30 FUNC GLOBAL HIDDEN 4 _start ... 1917: 00010430 52 FUNC GLOBAL DEFAULT 4 main 2201: 00010506 450 FUNC GLOBAL HIDDEN 4 __libc_start_main ...

You may ask: “Wow! 2308 looks like a lot, right? What behemoth of a program could possibly need that many symbols?“.

Good question! Here’s the behemoth:

#include <stdio.h> int main() { printf("Hello, World!\n"); return 0; }

Yeah, that’s it. Now, 2308 may be slightly bloated because we link against musl instead of glibc, but the point still stands: There’s a lot of stuff going on behind the scenes here.

The kernel’s job here is to iterate over each section, loading those marked as “loadable”. Some security mitigations start to become relevant here, such as moving sections around in memory (ASLR — Address Space Layout Randomization), marking sections as non-executable (NX bit — hardware-level security), etc. But ultimately, the kernel loads the code and data into memory, sets up the stack, and prepares to jump to the entry point of the program.

The stack

Ah yes, the infamous stack! Fortunately for most of us, the stack is something we take for granted. Unfortunately for the kernel, the stack is not some omnipotent magical space that just exists — it needs to be set up properly before our program can run.

As a reminder: stack space is typically used for variables, function arguments, “frames” (to keep track of function-local variables, call trees, etc), and a variety of other things, depending on what, and how your program is running.

Hypothetically, if we simplify a bit and say that the ELF file is loaded into memory starting at the zero address, the stack is typically placed at the “opposite end” of the memory, from a high address, and grows “downwards” towards the lower addresses, with the space in-between used as heap space, and for other data (shared libraries, mmapped files, etc). This is a simplification, but in fairness, there is significant ambiguity as much of the semantics here depend on the program itself.

The stack is also something that is non-empty! Remember argv and envp from the execve call above? Those are passed to the program via the stack. In most programming languages we frequently access these via the various args and env utilities, whether that be directly, like in C, or more indirectly, like in Rust (std::env) or Python (sys.argv).

The kernel also stores something called the “ELF auxiliary vector” in the nascent stack. This “auxv” contains information about the environment, such as the memory page size, metadata from the ELF file, and other system information. These are important! For example, musl uses the “page size” entry of the auxv so that malloc can request and manage memory more optimally. There are over 30 entries in the auxiliary vectors, but not all of them are used by every program (and some may not be defined by the kernel).

Let’s pretend we’re the kernel. Here’s a simplified version of how we might setup the stack of a new process (taken and simplified from my RISC-V emulator, which also emulates parts of the kernel):

// Choose an arbitrary high address for the stack let mut sp = 0xCFFF_F000u32; // sp = "stack pointer" let mut stack_init: Vec<u32> = vec![]; // The stack begins empty. stack_init.push(args.len()); // argc: number of arguments for &arg in args.iter().rev() { // Copy each argument to the stack sp -= arg.len() // move "downwards" in address space mem.copy_to(sp, arg); // Keep track of the arg pointer in the init vector stack_init.push(sp); } stack_init.push(0); // argv NULL terminator // Environment variables are similar: for &e in env.iter().rev() { sp -= e.len(); mem.copy_to(sp, e); stack_init.push(sp); } stack_init.push(0); // envp NULL terminator // Setup the auxiliary vector stack_init.push(libc_riscv32::AT_PAGESZ); // Keys for auxv stack_init.push(0x1000); // Values for auxv; this specifies a 4 KiB page size stack_init.push(libc_riscv32::AT_ENTRY); stack_init.push(self.pc); // N.B.: We'll be coming back to this // ... // Copy the stack init vector, with all the pointers, to the stack sp -= (stack_init.len() * 4); mem.copy_to(sp, &stack_init)

A diagram might help illustrate what the address space looks like at this point:

Entrypoint

Finally, we get to the “entry point” address, mentioned at several points. This is the address of the first instruction to run in the process. Typically, this is under a function called _start. Both glibc and musl provide implementations of _start, but it’s also possible to write your own. Again, here’s a Rust example:

// Disable the language runtime, we're DIYing it. #![no_std] #![no_main] #[panic_handler] fn panic(_info: &core::panic::PanicInfo) -> ! { loop {} } #[no_mangle] pub extern "C" fn _start() -> ! { // Instead of "waiting" for main, we can immediately start execution. loop {} }

Depending on your program, _start may be the only thing between the entrypoint and your main function, but most languages have some sort of runtime that needs to be initialized first. For example, Rust has std::rt::lang_start. It’s at this part that things like global constructors, thread-local storage, and other language-specific features are set up.

Here, our journey comes to an end — things become much more language-specific from this point on. Most languages will set up their own runtimes (yes, even C and C++ have a “runtime”!), and eventually call the standard main function we’re normally familiar with.

In Rust, the generated code ends up looking like the following:

// the user defined main function fn main() { println!("Hello, world!"); } // the generated _start function fn _start() -> { let argc = ...; // get argc from stack let argv = ...; // get argv from stack let envp = ...; // get envp from stack let main_fn = main; // pointer to user main function std::rt::lang_start(argc, argv, main_fn); }

With the lang_start function (defined here)[https://github.com/rust-lang/rust/blob/04ff05c9c0cfbca33115c5f1b8bb20a66a54b799/library/std/src/rt.rs#L199] and taking care of the rest.

C and C++ have similar, minimal setups. Languages that are traditionally thought to have “heavier” runtimes, such as Java or Python, work the same way, but with the std::rt::lang_start equivalent doing far more than the Rust/C/C++ runtimes.

And there you have it! I’m missing lots of detail here, but hopefully this gives a rough idea of what happens before main() gets called. I’ve left out complexity that is mostly internal to “real” linux kernels, such as how the kernel sets up address space, the process tables, various group semantics, and et cetera, but I hope this still serves as a decent primer.

Feel free to reach out to me with any questions or corrections!