.png)
3 months ago
4
From Python 3.14, python -m pdb -p pid lets you connect a pdb session to a running Python process.
This post goes into a part of what makes this possible.
The barrier to entry for writing general debugging tools for Python programs has always been quite low. Unlike many languages, you're rarely finding yourself working with weird internals. Instead, debugging tools can be built off of pretty straightforward knowledge of the language.
This is powered by the languages treating things like exception tracebacks as first class objects, standard library modules like traceback, and of course the power of eval/exec.
And thanks to this, tools like pdb can be provided out of the box, and easily customized by people without much (if any) understanding of the internals of the language.
Some might comment that the existence of things like pdb have prevented Python programmers from "simply" learning how to use more universal debuggers like gdb/lldb. But I believe that loads of excellent REPL-based debugging experimentation happens because People can simply write tools like ipdb or pdb++ in Python.
There's always been a bit of a catch, though. Generally these debugging tools require you first to edit your program source.
You can easily stick a:
right in the middle of your program near a problematic spot, or even set up an exception handler beforehand to capture outright exceptions.
But if you haven't done this work beforehand, you'll likely end up needing to edit your source code somehow, restart your program, and then go into it.
There are tools that can work without restarting the program. Beyond language agnostic debugging tools, tools like pyspy can work on running processes. But they are often relying on tricks and needing to know very specific details about CPython to work.
Python 3.14 offers some new functionality to standardize injecting some Python code to a running process. This should reduce the need for a lot of hacks.
sys.remote_exec takes a process ID of an existing Python process, along with the path of a Python script.
Calling sys.remote_exec(pid, script_path) will signal to the remote process that we want it to run the script at script_path "soon".
What does this look like in practice with a "real" program?
This program takes some integers as input, and calculates a cumulative sum. A very basic client-server program where you might wonder "... what is the program doing and why am I waiting so long?"
Running the program starts a "REPL" taking integers to add them up, printing out the PID.
The program is now waiting for input.
Over in remote-script.py we'll put in some code that we want to inject into the process. Here, we will just print out the stack to see what the program is doing.
With my original program still running, I've opened a Python REPL in a different shell, and run sys.remote_exec, pointing at the PID of my number adding process.
The call to sys.remote_exec returns immediately, but the remote process at first didn't seem to do anything, still waiting on another integer from me:
But once I actually provide an integer and return from the input call, I will see that my remote script runs.
This behavior illustrates a couple of things:
- the remote script execution kicked in on the return from the call to input.
The high level idea is that CPython's interpreter checks to see if any remote execution has been requested at specific points in the interpreter's main loop. So if your Python program is just waiting on external input, it's likely you're not making any progress in Python land... and so your remote script won't execute.
- the remote script execution happpens within the context of whatever's running at the moment
our remote script's call to print_stack printed out a trace where we were within our program's main, and then the last frame is our script running inline. This means that you can very easily grasp what your program is doing, instead of your script being run in an isolated way
Even with the fact that you need to make some forward progress to get remote execution, given that many cases you will eventually yield back to Python, you hopefully will get some information by the time your remote script executes.
In the following continuation, I add 40. My script then calls time.sleep(40). When I call sys.remote_exec during that sleep, the back trace doesn't appear until the end of that sleep call. But it did appear!
At this point the world is your oyster.
You could go spelunking for data in the remote process:
You could use a library like remote_pdb to set up an interactive debugging sessional:
with this, when you call sys.remote_exec you'll see a prompt like:
You can then connect to the pdb session with netcat, and have your interactive debugging session.
You might think that this is too many steps. If you're already on the same host, you can run python -m pdb -p pid to get a shell into a running process directly! To state the obvious, though: this will suspend your running process! Maybe not a great thing to do in production.
Remember how we called traceback.print_stack() in our example? That won't suspend the running process! Scripts that dump state and do nothing else, when done carefully, pose a lot less risk to the running process.
Though we're a far cry from dtrace-style protections against breaking our existing process. We're still running arbitrary Python code.
But even without the CPython team showing up to build the debugger, you could build out something like support for pdb -p pid yourself!
Being able to do this kind of script injection in a fully supported way, thanks to CPython providing both the right kind of hooks and a reference implementation means that the barrier to entry for writing tools are lower than ever.
Some reading:
-
The remote debug attachment protocol HOWTO. This describes what you need to do to implement something like sys.remote_exec. Valuable if you intend
-
PEP 768 – Safe external debugger interface for CPython. This is the original PEP for this functionality. Includes links to existing hacks and details on how this works
