GSoC 2026: fixing OWTF's MiTM proxy

OWTF runs a MiTM proxy so it can record every request and response a target sees. Plugins then grep through that history. So if the proxy does not record something, the rest of the framework is working with an incomplete picture.

For Google Summer of Code 2026 with the OWASP Foundation, I picked OWTF's MiTM Proxy Upgrade project. This post is what I found, what landed upstream, and what comes next.

Reading the proxy first

Before the proposal I read the proxy code properly: proxy.py, cache_handler.py, socket_wrapper.py, interceptor_manager.py, and the transaction logger in owtf/transactions/main.py.

That turned up eight small framework fixes I opened as PRs (#1405 through #1419): things like a UTF-8 decode in shell_exec_monitor() that silently killed plugins when nmap or sqlmap emitted binary output, a SIGKILL escalation loop that was dead code because of an inverted condition, and deprecated logging.warn() calls in error paths. Each one was read the module, find the failure mode, write a test if it mattered, keep the diff small.

Why HTTPS transactions are never recorded

The recording pipeline works like this. When a request completes, CacheHandler.dump() writes the transaction to a JSON file and creates an empty <hash>.rd sentinel next to it. A separate process, TransactionLogger, polls for those .rd files and writes each one to PostgreSQL. A transaction reaches the database only if some code path calls CacheHandler.dump().

Plain HTTP does. The request handler builds a CacheHandler and dumps it.

HTTPS does not. ProxyHandler.connect() handles the CONNECT that browsers send for TLS sites. It writes 200 Connection established, sets self._finished = True, wraps the sockets in TLS, and hands them to a threading.Thread running a select loop that forwards bytes in both directions.

That thread never builds a CacheHandler and never writes a .rd file. I grepped connect() for CacheHandler and found zero references.

So HTTPS transactions never reach the database. Since most real targets are HTTPS-only, the proxy's recorded history is effectively empty for the traffic that matters.

There was an open issue for this: #1287, "proxy at port 8008 not working properly, transaction table stays empty." It had the symptom but no root cause. I posted the diagnosis with line references. There are really two problems bundled in that issue:

1. Empty transaction table for HTTPS (architectural): the tunnel has to run inside the event loop before the decrypted stream can be parsed and routed through the existing recording path.
2. Intermittent TLS handshake failures: blocking upstream connect() in the IOLoop, deprecated ssl.PROTOCOL_TLS, and synchronous 4096-bit cert generation on the hot path.

Fixing the first one is the core of my proposal: async CONNECT rewrite, then route decrypted traffic through the same CacheHandler pipeline HTTP already uses. That is not a one-line patch.

Other proxy bugs I found

While reading for the recording bug I hit several smaller issues:

• time.sleep() inside a coroutine. Live interception polls for a user decision in a loop and calls time.sleep() between checks. That blocks the entire Tornado IOLoop for up to 30 seconds. Fix: yield tornado.gen.sleep().
• Response interceptors never ran. add_interceptor() only appended to the request list. The response list was initialized and sorted but never populated, so intercept_response() was a no-op.
• except FileLockTimeoutException on a name that does not exist. Under lock contention this raises NameError. The real type is FileLock.FileLockException, which I only found by reading owtf/lib/filelock.py.

Each became its own PR with a regression test where the behaviour mattered.

What merged

Two proxy PRs merged during the coding period:

Getting CI-runnable tests around the recording core felt like the most durable midterm win. If someone lands the CONNECT rewrite later, those tests are the contract.

What is still open

Other proxy work is open and waiting on review:

• #1448: replace blocking time.sleep() with tornado.gen.sleep() in the live interception poll, with a deterministic interleaving test.
• #1450: replace deprecated logging.warn() and IOLoop.instance() in proxy startup.

Plus the eight framework PRs from March (#1405 through #1419) and a few more small proxy fixes I have ready locally (FileLock exception handling, TOR manager commands removal, dead ssl.PROTOCOL_TLS kwargs).

In total I opened twelve PRs against owtf/owtf:develop, with six new test files and 34+ test cases across the batch.

What I did not finish (and how I scoped it)

I did not land the full async/await rewrite of proxy.py and main.py, and HTTPS recording is not working yet. That was the original proposal headline, and it depends on moving CONNECT off the thread/select forwarder and parsing HTTP/1.1 framing on the decrypted stream inside the IOLoop.

Rather than opening a large PR blind, I wrote a phased RFC: async CONNECT first (bytes forwarded, no recording yet), then HTTPS recording through the existing CacheHandler.dump() path, then a unified hook pipeline with response modification defaulting to off. Each phase is meant to be one reviewable PR with tests.

The proxy modularity issue my project was based on (#913) had been closed as not planned, with a note that a fresh proposal against current develop would be the right place to pick it up. I used #1287 as the anchor instead, because it is still open and reproducible.

What I took away

• Grepping for a symbol is not enough. The FileLockTimeoutException bug looked like a missing import until I read the lock class and saw the real exception type.
• Small PRs with tests are easier to defend in review than a rewrite nobody has time to read in one pass.
• Publishing the #1287 diagnosis early was worth it. Even without the big feature merged, it gives the next person a starting point with line numbers.
• Reading async Python where coroutines, threads, and SSL all meet in one file is good practice. OWTF's proxy is not a tutorial codebase.

Links

• All my OWTF PRs
• #1287 diagnosis comment
• OWTF on GitHub

Thanks to my GSoC mentors Viyat Bhalodia and Abraham Aranguren for the project and for being reachable when I had something concrete to show.