.png)
1 day ago
5
Jun 4, 2025
Recently I’ve been working on a pretty big rust project and to my surprise I couldn’t get tests to work properly.
Running cargo test would start running all the tests in the repo and after a couple of milliseconds every single test would start to fail because of an error that I’m not very familiar with
Fortunately, the error is pretty explicit and straightfoward so I was able to understand what was going on in a reasonable time. I’ve started digging a bit and learned some stuff along the way.
Ever wondered how your programs juggle multiple tasks - reading files, sending data over the network, or even just displaying text on your screen - all at once? File descriptors are what make this all possible (in Unix systems).
At its core, a file descriptor (often abbreviated as fd) is simply a positive integer used by the operating system kernel to identify an open file. In Unix, "everything is a file." Contrary to what the word says, a file descriptor doesn’t just refer to regular files on your disk. It can represent:
-
Regular files: The documents, images, and code files you interact with daily.
-
Directories: Yes, even directories are treated like files to some extent, allowing programs to list their contents.
-
Pipes: Used for inter-process communication, allowing one program’s output to become another’s input.
-
Sockets: The endpoints for network communication, whether it’s talking to a web server or another application on your local machine.
-
Devices: Hardware devices like your keyboard, mouse, and printer are also accessed via file descriptors.
When a program wants to interact with any of these resources, it first asks the kernel to "open" it. If successful, the kernel returns a file descriptor, which the program then uses for all subsequent operations (reading, writing, closing, etc.).
By convention, every Unix process starts with at least three standard file descriptors automatically opened:
-
0: Standard Input (stdin) - Typically connected to your keyboard for user input.
-
1: Standard Output (stdout) - Usually connected to your terminal for displaying normal program output.
-
2: Standard Error (stderr) - Also usually connected to your terminal, but specifically for displaying error messages.
On macOS we can quickly check this, open your favorite terminal and run ls /dev/fd.
On Linux we can do something similar but the repository is different and usually follows the current pattern /proc/<pid>/fd. Running the same on Linux gives me this:
As you can see, we have 0, 1 and 2 as expected, but we also have a bunch of other file descriptors.
Another useful command to check for open file descriptors is lsof, which stands for "list open files".
According to the lsof documentation: - cwd: The current working directory of the process. - txt: Executable files or shared libraries loaded into memory (e.g., /bin/zsh, modules like zutil.so, or system libraries like /usr/lib/dyld). - 0u, 1u, 2u: Standard input (0), output (1), and error (2) streams, respectively. The u means the descriptor is open for both reading and writing. These are tied to /dev/ttys001 (my current terminal device). - 10u: Another file descriptor (also tied to /dev/ttys001`), likely used for additional terminal interactions.
We now know that file descriptors are a way for the operating system to keep track of open files and other resources, nice!
Have you ever wondered how many file descriptors can be open at the same time? The most common answer in software engineering applies here too: It depends.
Each operating system has its own limits on the number of file descriptors a process can open simultaneously. These limits are in place to prevent a single misbehaving program from hogging all available resources and crashing the system.
On macOS, we can easily inspect these limits using the sysctl and ulimit commands in your terminal.
-
kern.maxfiles represents the absolute maximum number of file descriptors that can be open across the entire macOS system at any given moment. It’s a global governor, preventing the system from running out of file descriptor resources, even if many different applications are running.
-
kern.maxfilesperproc is the hard limit on the number of file descriptors that a single process can have open. Think of it as the ultimate ceiling for an individual application. No matter what, a process cannot open more files than this hard limit set by the kernel.
-
ulimit -n is your shell’s "soft" limit for the number of open file descriptors. If a process tries to open more files than its soft limit, the operating system will typically return an error (e.g., "Too many open files"). The good news is that a process can raise its own soft limit, but only up to its hard limit.
Enough with the theory, let’s get back to the problem I was having with my rust tests. My assumption was that since cargo test gets executed in my terminal, it inevitably reaches a point where it tries to open more files than the soft limit set by my shell, which is 256 in this case. When that happens, the operating system screams at cargo and tells it that it can’t open any more files, cargo then propagates that error to the tests and they all fail.
I wanted to confirm this hypothesis, so I created this monitoring script that watches for cargo test PID and prints the number of open file descriptors at different intervals.
Note that usually to get an accurate count of open file descriptors you would also need to consider the entire process tree, not just the main process. This is because child processes can also open files, and their file descriptors contribute to the total count. In my case, there was only one process (`cargo test`) running, so I didn't.
I can now run this script in one terminal and run cargo test in another terminal. I actually had to do this a couple of times to get a good sample of data, rust runs pretty fast once the code is compiled and this monitor script does not run fast enough to catch all the open file descriptors changes.
I couldn’t get the script to catch the exact moment the process reached the soft limits, but I can clearly see that the tests starts failing when the number of open file descriptors reaches 237, which is pretty close to the soft limit of 256.
Time to fix this! This is a bit underwhelming, but the solution is to just bump the soft limit of open file descriptors in my shell. I can do this by using the ulimit command again.
Running cargo test now works as expected and no "Too many open files" error is thrown.
Open File Descriptors Over Time
The above chart shows the number of open file descriptors with the new soft limit. As you can see the max value reached is around 1600, which is way above the previous limit of 256.
All in all, this was a fun exercise that taught me a lot about file descriptors and how they work in Unix-like systems. Now you know how to troubleshoot this error that might pop up in your own projects!
