Debugging live code with CPython 3.14

Introduction

Debugging a live Python process just got incredibly easier in Python 3.14, but when I read the release notes I didn't pay much attention to PEP 768: Safe external debugger interface for CPython, not every PEP sparks enough interest to me to spend 1-2 days going pep-deep, and I was honestly eclipsed by the new Template strings and the Multiple interpreters in the standard library.

It was not until I saw ♱☠︎︎ Pablo Galindo 𓃵♱, a core CPython and one of PEP's authors live at PyConES that I understood the importance of this, since it changes the way we will be debugging Python.

Maybe in the future we will not be debugging as I show you in this post, since the ergonomics are a bit raw, but it'll definitely be the foundation of live Python debuggers.

How bad was it before?

Before Python 3.14, there was not a standard way of accessing a Python process' memory. You could, of course, read the memory of any given process if you have the necessary permissions. For example, on Linux you can inspect /proc/{pid}/mem to read a process’s memory, but this is os-dependant; painful. Then you have to locate where the current state of the interpreter (PyRuntime) is, and the actual parts of the state that you want to access. Then you have to somehow run your code in a safe place, running it in the middle of critical operations like memory allocation or reference counting could mess up the entire execution.

This means that in order to write a Python debugger you had to have very deep knowledge of CPython internals and be very careful, debugging a running process was unsafe, using tools like memray carried some risk of borking your program, making debugging in production not for the faint of heart.

With the new PEP, this risk is pretty much gone, since now we'll be debugging with the interpreter and not against it. Without going into details (that you can read in the PEP), the complications of reading a CPython process memory is now outsourced to Pablo's and the CPython's team coffe machines and Friday nights, new structs have been added to Python that store more meta information about debugging state and locations of critical internal structures (offsets) and now part of the CPython execution loop is to check whether there is a pending debugging piece of code to execute, and because this is checked in a consistent state, running it is also safe.

How to debug a process.

For us pythonistas, it's all just an API change (addition). We have a new function in the sys module: sys.remote_exec(pid, path) where pid is the id of the process we want to debug, and path is the file name of the debugging script. Both Python versions have to be the same.

When you run remote_exec it will load the file, send it to the target process, and at some point, when it's deemed safe, it'll be executed.

Diagram of sys. remote_exec workflow

Let's see a practical example, we'll debug a simple script that increases a counter inside an infinite loop, when the counter is divisible by five, it will silently raise an exception.

        import sys
import time, os
 
if __name__ == '__main__':
    print(sys.version, 'pid:', os.getpid())
    c = 0
    not_divisible_by = 5
    
    while True:
        c += 1
        print('My cool counter! iteration nº', c)
        try:
            if not c % not_divisible_by:
                raise Exception(f'We cannot have numbers'
                                f' that are divisible by {c}')
        except Exception as e:
            exception = e
 
        time.sleep(1)

Our client debugger will just print 'Hello from the execution' for now.

        print('Hello from the execution')

Now, we run sys.remote_exec, as this will read memory from another process, it is a privileged operation, so you need to run this with sudo.

        sudo uv run python -c \
 "import sys; sys.remote_exec(136886, 'client_debugger.py')"

The output of the debugging script will show up in the target process, and as you can see, the process does not stop.

Now, to access the target process's variables, you need to import the module.

        import __main__ as debug
 
print("Hello from the target process")
print(debug)
print(f"Current iteration is: {debug.c}")

You can pretty much do anything at this point, like adding new variables, changing variables values, anything. Technically, this could be used to do hot-patching, but I wouldn't recommend it.

        import __main__ as debug
 
print("Hello from the target process")
print(debug)
print(f"Current iteration is: {debug.c}")
print(f"Last exception is: {debug.exception}")
print("Resetting c")
debug.c = 0

You could get fancier and write a pseudo-REPL script:

        #!/usr/bin/env python3.14
import argparse
import os
import sys
import tempfile
 
def exec_code(pid, code: str) -> None:
    with tempfile.NamedTemporaryFile(delete=False) as tmp:
        tmp.write(('import __main__ as d;'+code).encode('utf-8'))
        os.chmod(tmp.name, 0o644) # Make it readable.
        sys.remote_exec(pid, tmp.name)
 
 
def main():
    print("Debug repl (type 'exit' or Ctrl+C to quit)")
 
    parser = argparse.ArgumentParser()
    parser.add_argument("--pid",
                        required=True, 
                        type=int,
                        help="Process ID")
 
    args = parser.parse_args()
 
    print('Debugging module is available as "d", e.g. print(d)')
 
    while True:
        try:
            user_input = input(">>> ")
            if user_input.strip().lower() == 'exit':
                print("\nTa luego!")
                break
 
            exec_code(args.pid, user_input)
 
        except KeyboardInterrupt:
            print("\nTa luego!")
            break
        except Exception as e:
            print(f"Error: {e}")
 
 
if __name__ == "__main__":
    main()

A similar thing could be achieved by running pdb which added support for sys.remote_exec(...)

        sudo python -m pdb -p 194725

Debugging the CrateDB driver

We can write debugging scripts that target specific programs, for example, we could write a script that prints the current state of a script sending data to a CrateDB server.

        import __main__ as current
import pprint
import datetime
 
start = datetime.datetime.now()
 
def debug_connection(connection):
    client_keys = [
        'ssl_relax_minimum_version', 'username', 'schema'
    ]
    print('Active servers', connection.client.active_servers)
    print('Inactive servers: ', connection.client._inactive_servers)
    print('Client configuration: ', end='')
    pprint.pp(dict(filter(lambda k: k[0] in client_keys,
                          connection.client.__dict__.items())))
 
 
def debug_cursor(cursor):
    print("Cursor", '❌ closed' if cursor._closed else '✅ open')
    print("Last result: ", end='')
    pprint.pprint(cursor._result)
    print('Cursor timezone:', cursor.time_zone)
    print('Last query duration(ms):', cursor.duration)
    if hasattr(cursor, '_debug_query_count'):
        print("Queries sent since starting debugging",
              cursor._debug_query_count)
        print('Queries per seconds:', cursor._debug_query_count / (
            datetime.datetime.now() - current._debug_started_at).seconds)
 
print(("=" * 10) + 'DEBUGGING INFO' + ("=" * 10))
print(f'Debugging module at: {current}')
 
debug_connection(current.conn)
 
# Monkey patches 'execute' to count queries.
if not hasattr(current.cursor, '_debug_query_count'):
    current.cursor._debug_query_count = 0
    def debug_execute(*args, **kwargs):
        current.cursor._debug_query_count += 1
        return exec_f(*args, **kwargs)
    exec_f = current.cursor.execute
    current.cursor.execute = debug_execute
 
if not hasattr(current, '_debug_started_at'):
    current._debug_started_at = datetime.datetime.now()
 
debug_cursor(current.cursor)
 
print("=" * 33 + '\n')

In this case, we monkey patch some attributes to start counting things that we could not count before but other than that, it's pretty straight forward if you know what it's inside __main__ just access whatever attributes you need to start debugging.

Conclusion

Debugging live Python programs is easier and safer than ever, just remember that you need to have access to the same machine it's running, have admin privileges and that the process has to be running version >=3.14. The overhead is minimal, and it's turned on by default.

You can start writing a library of debugging scripts, share them with your co-workers, test them, and more importantly, re-use them specifically targeting your programs.