--- name: python-debugpy-cb1b8b0795a5 description: "Debug Python: pdb REPL + debugpy remote (DAP)." version: 1.0.0 author: Hermes Agent license: MIT platforms: [linux, macos] metadata: hermes: tags: [debugging, python, pdb, debugpy, breakpoints, dap, post-mortem] related_skills: [systematic-debugging, node-inspect-debugger] --- # Python Debugger (pdb + debugpy) ## Overview Three tools, picked by situation: | Tool | When | |---|---| | **`breakpoint()` + pdb** | Local, interactive, simplest. Add `breakpoint()` in the source, run normally, get a REPL at that line. | | **`python -m pdb`** | Launch an existing script under pdb with no source edits. Useful for quick poking. | | **`debugpy`** | Remote / headless / "attach to already-running process." Talks DAP, scriptable from terminal, works for long-lived processes (gateway, daemon, PTY children). | **Start with `breakpoint()`.** It's the cheapest thing that works. ## When to Use - A test fails and the traceback doesn't reveal why a value is wrong - You need to step through a function and watch a collection mutate - A long-running process (hermes gateway, tui_gateway) misbehaves and you can't restart it - Post-mortem: an exception fired in prod-ish code and you want to inspect locals at the crash site - A subprocess / child (Python `_SlashWorker`, PTY bridge worker) is the actual bug site **Don't use for:** things `print()` / `logging.debug` solve in under a minute, or things `pytest -vv --tb=long --showlocals` already reveals. ## pdb Quick Reference Inside any pdb prompt (`(Pdb)`): | Command | Action | |---|---| | `h` / `h cmd` | help | | `n` | next line (step over) | | `s` | step into | | `r` | return from current function | | `c` | continue | | `unt N` | continue until line N | | `j N` | jump to line N (same function only) | | `l` / `ll` | list source around current line / full function | | `w` | where (stack trace) | | `u` / `d` | move up / down in the stack | | `a` | print args of the current function | | `p expr` / `pp expr` | print / pretty-print expression | | `display expr` | auto-print expr on every stop | | `b file:line` | set breakpoint | | `b func` | break on function entry | | `b file:line, cond` | conditional breakpoint | | `cl N` | clear breakpoint N | | `tbreak file:line` | one-shot breakpoint | | `!stmt` | execute arbitrary Python (assignments included) | | `interact` | drop into full Python REPL in current scope (Ctrl+D to exit) | | `q` | quit | The `interact` command is the most powerful — you can import anything, inspect complex objects, even call methods that mutate state. Locals are read-only by default; use `!x = 42` from the `(Pdb)` prompt to mutate. ## Recipe 1: Local breakpoint Easiest. Edit the file: ```python def compute(x, y): result = some_helper(x) breakpoint() # <-- drops into pdb here return result + y ``` Run the code normally. You land at the `breakpoint()` line with full access to locals. **Don't forget to remove `breakpoint()` before committing.** Use `git diff` or a pre-commit grep: ```bash rg -n 'breakpoint\(\)' --type py ``` ## Recipe 2: Launch a script under pdb (no source edits) ```bash python -m pdb path/to/script.py arg1 arg2 # Lands at first line of script (Pdb) b path/to/script.py:42 (Pdb) c ``` ## Recipe 3: Debug a pytest test Use `terminal` and the canonical runner for noninteractive diagnostics: ```bash # Show locals in tracebacks without pdb: scripts/run_tests.sh tests/path/to/test_file.py --showlocals --tb=long ``` `scripts/run_tests.sh` captures each file in a separate subprocess, so `--pdb` or `--trace` cannot provide an interactive prompt there. For an interactive debugger only, use the independent development/test interpreter prepared in Recipe 5 (never a production generation): ```bash .venv/bin/python -m pytest tests/foo_test.py::test_bar --pdb ``` This bypasses the hermetic-env guarantees — fine for debugging, but re-run under the wrapper to confirm before pushing. ## Recipe 4: Post-mortem on any exception ```python import pdb, sys try: run_the_thing() except Exception: pdb.post_mortem(sys.exc_info()[2]) ``` Or wrap a whole script: ```bash python -m pdb -c continue script.py # When it crashes, pdb catches it and you're in the frame of the exception ``` Or set a global hook in a repl/jupyter: ```python import sys def excepthook(etype, value, tb): import pdb; pdb.post_mortem(tb) sys.excepthook = excepthook ``` ## Recipe 5: Remote debug with debugpy (attach to running process) For long-lived processes: Hermes gateway, tui_gateway, a daemon, a process that's already misbehaving and can't be restarted clean. ### Setup For Hermes, use a separate development checkout and data home, not a live production generation. Follow the [PM developer workflow](https://hermes-agent.nousresearch.com/docs/reference/package-management#developer-workflow) and activate that checkout — PowerShell: `. .\activate.ps1`. The declared `dev` extra includes debugpy, which PM activation does not sync (`all` excludes it). Through `terminal`, build a fresh, caller-owned debug/test environment with the prepared checkout's Python: ```bash source ./activate python -m pm.build_env --source . --out .venv --group dev --group test .venv/bin/python -c "import debugpy; print(debugpy.__file__)" ``` The output must not already exist. Stop its processes and intentionally remove only that disposable environment before rebuilding. Keep the same isolated `HERMES_HOME` for the debug target. `.venv/bin/python` is this explicitly built debug environment, not a guessed application venv, and the patterns below run through it. Do not add debugpy to a running production environment; reproduce there only with an already-prepared debug target or arrange a restart in the development environment. ### Pattern A: Source-edit — process waits for debugger at launch Add near the top of the entry point (or inside the function you want to debug): ```python import debugpy debugpy.listen(("127.0.0.1", 5678)) print("debugpy listening on 5678, waiting for client...", flush=True) debugpy.wait_for_client() debugpy.breakpoint() # optional: pause immediately once attached ``` Start the process; it blocks on `wait_for_client()`. ### Pattern B: No source edit — launch with `-m debugpy` ```bash .venv/bin/python -m debugpy --listen 127.0.0.1:5678 --wait-for-client your_script.py arg1 ``` Equivalent for module entry: ```bash .venv/bin/python -m debugpy --listen 127.0.0.1:5678 --wait-for-client -m your.module ``` ### Pattern C: Attach to an already-running process Needs the PID and debugpy preinstalled in the target's environment: ```bash .venv/bin/python -m debugpy --listen 127.0.0.1:5678 --pid # debugpy injects itself into the process. Then attach a client as below. ``` Some kernels/security configs block the ptrace-based injection (`/proc/sys/kernel/yama/ptrace_scope`). Fix with: ```bash echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope ``` ### Connecting a client from the terminal The easiest terminal-side DAP client is VS Code CLI or a small script. From inside Hermes you have two practical options: **Option 1: `debugpy`'s own CLI REPL** — not an official feature, but a tiny DAP client script: ```python # ~/.hermes/cache/scratch/dap_client.py import socket, json, itertools, time, sys HOST, PORT = "127.0.0.1", 5678 s = socket.create_connection((HOST, PORT)) seq = itertools.count(1) def send(msg): msg["seq"] = next(seq) body = json.dumps(msg).encode() s.sendall(f"Content-Length: {len(body)}\r\n\r\n".encode() + body) def recv(): header = b"" while b"\r\n\r\n" not in header: header += s.recv(1) length = int(header.decode().split("Content-Length:")[1].split("\r\n")[0].strip()) body = b"" while len(body) < length: body += s.recv(length - len(body)) return json.loads(body) send({"type": "request", "command": "initialize", "arguments": {"adapterID": "python"}}) print(recv()) send({"type": "request", "command": "attach", "arguments": {}}) print(recv()) send({"type": "request", "command": "setBreakpoints", "arguments": {"source": {"path": sys.argv[1]}, "breakpoints": [{"line": int(sys.argv[2])}]}}) print(recv()) send({"type": "request", "command": "configurationDone"}) # ... loop reading events and sending continue/stepIn/etc. ``` This is fine for one-off automation but painful as an interactive UX. **Option 2: Attach from VS Code / Cursor / Zed** — if the user has one open, they can add a `launch.json`: ```json { "name": "Attach to Hermes", "type": "debugpy", "request": "attach", "connect": { "host": "127.0.0.1", "port": 5678 }, "justMyCode": false, "pathMappings": [ { "localRoot": "${workspaceFolder}", "remoteRoot": "" } ] } ``` **Option 3: Ditch DAP, use `remote-pdb`** — usually what you actually want from a terminal agent: For an independently owned Python project, declare `remote-pdb` in that project's development dependencies and prepare its debug environment through the project's package manager. This is not a Hermes SDK install recipe. For Hermes, prefer the declared debugpy dependency; the remote-pdb examples below require a separately declared, freshly built debug environment, never an in-place pip install into the selected application generation. In your code: ```python from remote_pdb import set_trace set_trace(host="127.0.0.1", port=4444) # blocks until connection ``` Then from the terminal: ```bash nc 127.0.0.1 4444 # You get a (Pdb) prompt exactly as if debugging locally. ``` `remote-pdb` is the cleanest agent-friendly choice when `debugpy`'s DAP protocol is overkill. Use `debugpy` only when you actually need IDE integration. ## Debugging Hermes-specific Processes ### Tests See Recipe 3. The wrapper captures subprocess output, so run pytest directly for interactive pdb. ### `run_agent.py` / CLI — one-shot In the prepared debug checkout, add `breakpoint()` near the suspect line, then run `python hermes`. Control returns to your terminal at the pause point. ### `tui_gateway` subprocess (spawned by `hermes --tui`) The gateway runs as a child of the Node TUI. Options: **A. Source-edit the gateway:** ```python # tui_gateway/server.py near the top of serve() import debugpy debugpy.listen(("127.0.0.1", 5678)) debugpy.wait_for_client() ``` Start `python hermes --tui` from the prepared debug checkout. The TUI will appear frozen (its backend is waiting). Attach a client; execution resumes when you `continue`. Check the child's interpreter and imports before assuming it inherited the debug environment. **B. Use `remote-pdb` at a specific handler:** ```python from remote_pdb import set_trace set_trace(host="127.0.0.1", port=4444) # in the RPC handler you want to trap ``` Trigger the matching slash command from the TUI, then `nc 127.0.0.1 4444` in another terminal. ### `_SlashWorker` subprocess Same pattern — `remote-pdb` with `set_trace()` inside the worker's `exec` path. The worker is persistent across slash commands, so the first trigger blocks until you connect; subsequent slash commands pass through normally unless you re-arm. ### Gateway (`gateway/run.py`) Long-lived. Use `remote-pdb` at a handler, or `debugpy` with `--wait-for-client` if you're restarting the gateway anyway. ## Common Pitfalls 1. **pdb under a parallel/output-capturing runner silently does nothing.** You won't see the prompt, the test just hangs (true of pytest-xdist and of `scripts/run_tests.sh`'s captured per-file subprocesses). Run pytest directly on a single file for interactive debugging. 2. **`breakpoint()` in CI / non-TTY contexts hangs the process.** Safe locally; never commit it. Add a pre-commit grep as a safety net. 3. **`PYTHONBREAKPOINT=0`** disables all `breakpoint()` calls. Check the env if your breakpoint isn't hitting: ```bash echo $PYTHONBREAKPOINT ``` 4. **`debugpy.listen` blocks only if you also call `wait_for_client()`.** Without it, execution continues and your first breakpoint may fire before the client is attached. 5. **Attach to PID fails on hardened kernels.** `ptrace_scope=1` (Ubuntu default) allows only same-user ptrace of child processes. Workaround: `echo 0 > /proc/sys/kernel/yama/ptrace_scope` (needs root) or launch under `debugpy` from the start. 6. **Threads.** `pdb` only debugs the current thread. For multithreaded code, use `debugpy` (thread-aware DAP) or set `threading.settrace()` per thread. 7. **asyncio.** `pdb` works in coroutines but `await` inside pdb requires Python 3.13+ or `await` from `interact` mode on older versions. For 3.11/3.12, use `asyncio.run_coroutine_threadsafe` tricks or `!stmt`-based awaits via `asyncio.ensure_future`. 8. **`scripts/run_tests.sh` strips credentials and sets `HOME=`.** If your bug depends on user config or real API keys, it won't reproduce under the wrapper. Debug with raw `pytest` first to repro, then re-confirm under the wrapper. 9. **Forking / multiprocessing.** pdb does not follow forks. Each child needs its own `breakpoint()` or `set_trace()`. For Hermes subagents, debug one process at a time. ## Verification Checklist - [ ] In the independently built debug environment, confirm: `.venv/bin/python -c "import debugpy; print(debugpy.__version__); print(debugpy.__file__)"` - [ ] For remote debug, confirm the port is actually listening: `ss -tlnp | grep 5678` - [ ] First breakpoint actually hits (if it doesn't, you likely have `PYTHONBREAKPOINT=0`, you're under a parallel/capturing runner, or execution finished before attach) - [ ] `where` / `w` shows the expected call stack - [ ] Post-debug cleanup: no stray `breakpoint()` / `set_trace()` in committed code ```bash rg -n 'breakpoint\(\)|set_trace\(|debugpy\.listen' --type py ``` ## One-Shot Recipes **"Why is this dict missing a key?"** ```python # add above the KeyError site breakpoint() # then in pdb: (Pdb) pp d (Pdb) pp list(d.keys()) (Pdb) w # how did we get here ``` **"This test passes in isolation but fails in the suite."** ```bash scripts/run_tests.sh tests/the_test.py # confirm it fails under the isolated runner first # For interactive debugging, or if it only fails WITH other tests, use the # independent development/test interpreter prepared in Recipe 5: .venv/bin/python -m pytest tests/ -x --pdb # Now it pdb-traps at the exact failing test after state accumulated. ``` **"My async handler deadlocks."** ```python # Add at handler entry import remote_pdb; remote_pdb.set_trace(host="127.0.0.1", port=4444) ``` Trigger the handler. `nc 127.0.0.1 4444`, then `w` to see the suspended frame, `!import asyncio; asyncio.all_tasks()` to see what else is pending. **"Post-mortem on a crash in an Ink child process / subprocess."** ```bash PYTHONFAULTHANDLER=1 python -m pdb -c continue path/to/entrypoint.py # On crash, pdb lands at the frame of the exception with full locals ``` --- Source attribution: NousResearch Original project: https://github.com/NousResearch/hermes-agent License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License Copyright (c) 2026 Gabriel Moreira Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: NousResearch Original project: https://github.com/NousResearch/hermes-agent Attribution basis: Upstream repository explicitly recorded in package mirror path Fields of work: Development & Engineering Roles: Backend Developer, ML / AI Engineer, QA / Test Engineer Specialties: Backend, Testing Tools: Debugpy Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 1f22f5f6f8798ec4577ea186aa619def75df9ffda1f4d26f8516f2c1aeb548da