Researchers Just Showed You Can Steal a Frontier Model's Hidden Reasoning — And Found Real API Keys and Passwords Leaking Inside It. That's a New AI Security Service.

A new paper landed on Hacker News with 428 points and a name that says exactly what it found: Stolen Thoughts.

The finding: "Stealing Reasoning Traces from Proprietary LLM APIs."

What the researchers actually did

Per the paper's own summary: "Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models." The researchers took a reasoning trace produced by a frontier model, replayed it into a weaker sibling model from the same provider, jailbroke that weaker model into transcribing the block verbatim, and recovered the stronger model's hidden reasoning in plaintext — "without ever attacking the stronger model directly or triggering its anti-distillation safeguards."

Then they pointed the technique at data that already exists in the wild:

• They "collected 6,708 publicly available agent trajectories from GitHub and Hugging Face, produced by Claude, GPT, and Gemini models and still containing encrypted reasoning blocks"
• Applying their decoding pipeline yielded "315,320 reconstructed reasoning blocks"
• Restricting to genuine, non-benchmark user sessions, they "recovered 704 distinct privacy artifacts, including 62 API keys, 33 passwords, 24 access tokens, and 30 personal email addresses, alongside names, postal addresses, internal URLs, and other technical identifiers"
• Of those 704 artifacts, "64 appeared exclusively inside the reasoning blocks and nowhere in the visible session"

That last line is the part worth sitting with. These weren't secrets someone forgot to redact from a chat log. They were secrets that only existed in a part of the transcript most people don't even know is stored — the encrypted reasoning block sitting quietly in the API response.

The business idea

Every team building agents on Claude, GPT, or Gemini is generating these encrypted reasoning blocks right now, whether they realize it or not. Some are storing them in logs. Some are shipping them into public eval datasets, benchmark submissions, or open-source demo repos — exactly the kind of GitHub and Hugging Face material this paper pulled its 6,708 trajectories from.

That's a sellable, fixed-scope audit: go through a team's agent pipeline, find every place a "thinking" or "signature" field from an API response gets logged, stored, or published, and report what's actually reachable. For teams that already publish agent trajectories for research or benchmarking, add a scrub-before-you-ship step so what goes out the door doesn't carry a customer's API key or passport number inside a field nobody was reading.

Who buys this

Any team running production agents that log full API responses (support bots, coding assistants, computer-use agents), and any team publishing agent trajectories publicly for benchmarks, demos, or research — the exact population this paper just showed is already leaking real credentials and PII into data anyone can download.

Bottom line

This isn't a theoretical vulnerability. The researchers ran it against real, already-public data and found working API keys, passwords, and a full passport-and-credit-card record sitting inside reasoning blocks nobody thought to check. That gap — between "we assumed thinking blocks were opaque" and "they're decodable and full of secrets" — is exactly what a security audit gets paid to close.

Source: https://stolen-thoughts.com/