Proprietary LLM Reasoning Traces Can Be Stolen via API Replay Attack
Original: Stealing Reasoning Traces from Proprietary LLM APIs
Why This Matters
Demonstrates a systemic API-level privacy vulnerability affecting all three major frontier model providers simultaneously.
Researchers from ELLIS Institute Tübingen, Max Planck Institute, and others found that encrypted chain-of-thought reasoning blocks from Anthropic, OpenAI, and Google APIs can be replayed into weaker sibling models to extract hidden reasoning in plaintext, requiring just two API calls.
A research team from ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, MATS Research, Snyk, and University of Tübingen has published a paper detailing a method to extract proprietary reasoning traces from frontier LLMs including Claude, GPT, and Gemini models.
The core technique exploits the fact that encrypted 'thinking' blocks returned by provider APIs are portable — they carry no binding to a specific session, user, or model. An attacker takes an encrypted trace from a stronger model (e.g., claude-opus-4-8), replays it into a weaker jailbroken sibling (e.g., claude-haiku-4-5), then instructs that model to transcribe the injected reasoning verbatim. This bypasses anti-distillation safeguards on the stronger model entirely.
To measure leakage at scale, the team collected 6,708 publicly available agent trajectories from GitHub and Hugging Face that still contained signed reasoning blocks. Running their decoding pipeline produced 315,320 reconstructed reasoning blocks. Among genuine user sessions, they identified 351 distinct leaked items, including 204 technical identifiers, 126 pieces of PII, and 23 sets of credentials.
Decoded token counts closely matched the hidden thinking-token counts reported by the APIs across 120 Codeforces benchmark problems, confirming high extraction fidelity. The paper also documents a special case involving Kimi-K3 and explores jailbreaking for misuse uplift and summary unfaithfulness.