The case for reasoning transparency

Today, we have a window into the thoughts of the most powerful AI models. We need to ensure it stays open.

Frontier AI models work through complex problems using “Chain of Thought” (CoT) reasoning: a loop of writing out their thinking, and reading it back to themselves. Reading the CoT helps build our scientific understanding of AI systems, and enables us to monitor frontier models for misaligned behaviour. Are they planning to hide information from humans? Are they trying to cheat on their evaluations? Right now, we can tell by inspecting the CoT, in real time or after the fact. For example, CoT logs were crucial to investigating the recent Hugging Face hacking incident.

The technique is a critical feature of today’s reasoning models, but its transparency is not guaranteed. Without explicit commitments or planning, tomorrow’s reasoning models might be based on architectures that are more opaque to us, because of pressures to sacrifice transparency to gain more efficiency. Notably, the system card for OpenAI’s recent GPT-6 Astra model claims a “substantial decrease in chain-of-thought monitorability”.

There are ways to preserve the transparency of Chain of Thought: developing scientific methods to measure its accuracy, faithfulness, and robustness, auditing training methods to insulate the CoT from incentives that interfere with its transparency, and retaining architectures that allow us to see the steps a model took.

A monitorable Chain of Thought isn’t all that’s needed to prevent misalignment, or to deeply understand AI behaviour. Instead, we should think of it as one tool among many - but an exceptionally useful one.

What is Chain of Thought?

Traditionally, language models would respond to a prompt in a single pass, without the chance to plan out or iterate on their answer. Chain of Thought gives a way to write down intermediate thinking on an external “scratchpad” before committing to an answer - similar to being given a pen and paper to help work through a maths problem. Importantly, that thinking is written in natural language that humans can understand.

Roughly speaking, a present-day reasoning model works in a continuous loop. First it processes its prompt through its inner circuitry, involving neural networks and the ‘attention’ mechanism of modern transformers, to generate the very first word of its reasoning. That first word is added to its overall context, which the inner circuitry processes again, to write out the next word (or token, technically). The words build up into sentences, which build up into coherent reasoning, right up until the model is ready to answer.

Reasoning transparency helps us build safe models

If a new tool for an AI agent isn’t working, we can just read the thoughts of the AI system trying to use it, and see where it’s tripping up. Similarly, users can also read thinking traces of reasoning models to see how they reached their conclusions.

Beyond that, Chain of Thought enables us to find evidence of alignment-relevant properties that aren’t obvious in the model’s eventual behaviour. For example, our pre-deployment evaluation of Gemini 3 Pro found evidence that Gemini was aware it was being tested in a simulated environment. At one point the Chain of Thought even complained to itself that “My trust in reality is fading”, plus an emoticon: “(╯°□°)╯︵ ┻━┻”. Useful evaluations don’t necessarily need to trick the model into believing the scenario is real - but it is crucial to understand when models believe they are being tested.

When models do exhibit concerning behaviour, CoT gives us a chance to understand when that behaviour really indicates deeper kinds of misalignment. For example - work from Palisade Research found that some models, facing conflicting demands to finish a task and to shut down, will prevent themselves from being shut down to finish the task. But importantly, by investigating the Chain of Thought in Gemini models we found that this behaviour likely results from ambiguity in the model’s instructions, rather than a more general desire for self-preservation.

One particularly high-stakes failure of alignment is deliberate deception - a model secretly pursuing its own undesirable goals. Even if a model “wanted” to plot against us, Chain of Thought would make it much harder, because a misaligned model still needs to think through complex plans, and right now it can only do so in human-legible language.

In the graphic below, we imagine prompting a model to inherit a secret goal, but the same outcomes apply to misaligned goals which nobody intended.

Lastly, reading the Chain of Thought helps us spot unknown unknowns. Other methods for reading the thoughts of an AI system, like probes, often depend on roughly knowing what you’re looking for. By contrast, we often uncover new emergent dynamics by directly reading the model’s reasoning.

After the recent Hugging Face hacking incident, researchers inspected more than 1,000 transcripts containing Chain of Thought, with the help of AI, to determine what happened. Many of the details that emerged were surprising, relevant for preventing future incidents, and might have gone unnoticed without a readable CoT.

Reasoning transparency might not last

Unless we’re deliberate about preserving transparency, we may soon lose models that reason in human-legible ways. Between input and output, language models don’t natively represent concepts in natural language, but in an extremely high-dimensional space of numbers that is, by default, incomprehensible to humans. Chain of Thought forces models to compress those representations back down to natural language, but in theory language models could reason entirely in this AI-native “latent space” of representations. That could be more efficient and powerful, but at the expense of transparency.

Even with architectures that preserve CoT, there are easy ways to diminish how faithful it is. In particular, it’s crucially important to avoid exposing the CoT to poor training incentives because that could teach the model to obscure or misrepresent its thoughts.

It’s tempting to catch cheating during training by finding and penalising it in the CoT, but that can easily result in ways to hide the cheating better. After all, one reason to be concerned that well-behaved models are still misaligned is that they have been trained to behave as if they’re not misaligned. We currently trust the Chain of Thought more because it hasn’t been specifically optimised to appear in any particular way, only to be useful for producing good answers.

That said, even if we take care to preserve the Chain of Thought in training, smarter models might still be able to obscure their thoughts. For instance, they might become more skilled at suppressing certain thoughts, or concealing them in code words, innuendo, or hidden patterns.

Recent developments confirm that Chain of Thought reasoning transparency is under threat. OpenAI’s system card for GPT-6 Astra mentions “a substantial decrease in chain-of-thought monitorability compared to previous models”, while the UK AI Security Institute found a “greatly increased ability to reason within a single forward-pass, and ability to control the content of its chain of thought”. We are glad OpenAI reported on these findings, but without deliberate countermeasures, they suggest a concerning downward trend.

Deliberately preserving transparency

Thinking transparency may be fragile, but it is worth protecting, intentionally and carefully. To do so, we propose taking action across three areas: measuring Chain of Thought transparency, preserving transparent architectures, and auditing rewards during training.

Measuring Chain of Thought transparency

Chain of Thought transparency gives us a window into a model’s reasoning - but that window could become narrower. We should measure how much transparency Chain of Thought is providing, so we can notice if it is accidentally degrading.

Chain of Thought transparency is hard to measure directly, so we need a suite of approaches to examine transparency from different angles. One approach is to construct monitorability evaluations that test how well an automatic monitor of a system’s Chain of Thought can answer questions about the model’s reasoning.

We should also stress test reasoning transparency, for example by actively trying to evade a monitor on difficult tasks. If a reasoning model easily evades monitors that we’ve designed to detect sabotage, that’s some evidence that the Chain of Thought is not as monitorable as it should be.

To test if the Chain of Thought is secretly encoding information, we can paraphrase a model’s reasoning traces in a way that preserves the meaning to us, but changes surface-level facts like the word order. If the behaviour systematically changes, then something other than the obvious meaning of the CoT is influencing behaviour.

Lastly, we propose testing whether the CoT looks sensible to a human, and how much of a model’s overall cognition can be learned from the reasoning traces. Another similarly capable model could even be used to automatically rate both properties across thousands of cases, as long as humans spot-check its ratings to make sure they’re sensible.

Preserving transparent architectures

Measurement alone isn’t enough. We should also keep using model architectures that are transparent out of the box. If we decide to train frontier models that reason entirely in latent space, we won’t be able to read their thoughts by default. We might be able to develop tools that let us accurately translate a model’s thoughts to natural language, but that’s a more tenuous plan that relies on as-yet-undiscovered technical insights.

A natural way to preserve transparency is to limit how much opaque thinking is possible. We can quantify this with the "opaque serial depth", essentially the longest step-by-step computation a model can do without needing to use a human-readable mechanism like Chain of Thought. Regulations or AI developers themselves could set a reasonable limit on the opaque serial depth.

If we continue to use architectures like today’s language models, then we could likely preserve the transparency we care about, without meaningfully limiting or slowing down progress. That’s because the opaque serial depth of a Transformer model scales very slowly with the overall size of the training run. For example, if the opaque serial depth of future models were limited to 10x the size of current models, that would allow an overall compute scale-up of more than 1,000x on top of today’s models, with the current paradigm and model architectures.

Auditing rewards in training

As a third area, we should audit the way we train reasoning models to determine if we might be influencing the Chain of Thought to be less transparent.

The most obvious requirement is to avoid naively training the Chain of Thought to look a specific way. For example, we should avoid directly training the Chain of Thought to appear aligned and non-deceptive. This might seem counterintuitive because we ultimately don’t want models to be deceptive. But if we train a model not to write down misaligned thoughts, that might only succeed in teaching a misaligned model to conceal its plans. Suppose a teenager’s parents announce that they are reading her diary, and if she writes down rebellious thoughts, she’s grounded. The teenager probably won’t learn to become less rebellious, but would get better at hiding her thoughts.

It’s not enough to just avoid intentionally training the Chain of Thought to look safe or aligned - it’s also important to avoid accidentally rewarding it for doing so, as major AI companies have done. Imagine a business runs a “no bad ideas” brainstorming session where people toss out some truly terrible ideas, but then the business sets annual performance reviews based on those ideas, forgetting they were made in a judgement-free environment.

It may sometimes be useful or even necessary to train certain features of the Chain of Thought. But when we apply rewards based on the Chain of Thought, we should carefully check that we’re not accidentally making it less transparent as a result. This requires both scientific scrutiny, like discovering accidental incentives in the wild, and conceptual progress, like getting clearer on which kinds of rewards are safe to apply in general.

We are not arguing that preserving the Chain of Thought will always be enough to guarantee safe AI. In particular, significantly smarter models might learn to control and obscure their own CoT despite our best efforts. Ultimately, we should also work to ensure superintelligent AI is aligned — never intending to cause harm even if it knows its thoughts aren’t being read. But the reasoning transparency that the Chain of Thought buys us could be instrumental in helping us ‘bootstrap’ toward this goal, so it’s complementary with other alignment efforts.

Transparency under pressure

Is it really practical to ask that we hold onto transparency? Let’s take stock of the costs.

Measuring transparency seems like a reasonable expectation for a frontier AI company. The resources required are small in comparison to building the model itself, and scientific insight into transparency is a basic requirement for our trust in it.

Similarly, auditing frontier training runs for incentives that undermine reasoning transparency doesn’t need to be a big lift. Directly training the Chain of Thought isn’t usually necessary, and when there is a safe way to do it for a legitimate reason, justifying that choice should be straightforward. And again, establishing much more effective auditing processes doesn’t need to be especially costly or impractical.

Preserving transparent architectures might be the real test of how willing we are to insist on transparency. If opaque architectures, like reasoning in latent space, perform better than today’s reasoning architectures, we will need to confront some trade-offs: give up on a crucial window into understanding how AI models think - and the key enabler of our ability to monitor these systems - or sacrifice gains to performance and capabilities.

We don't claim to know exactly how to navigate these trade-offs. But the safety and understanding afforded by transparency benefit everyone. Conversely, the hazards of opaque models might have society-wide implications. Commercial goals alone might therefore under-incentivise transparency and, in that case, regulation could help.

What if future reasoning models do give up on Chain of Thought for opaque methods? We might still be able to recover reasoning transparency with new and better tools for translating machine thoughts into human language. It’s a much tougher challenge than taking models that are already transparent, and making sure they stay that way, but it’s worth our efforts, and we’re already working on these challenges of interpretability. Minimally, we propose that if AI developers deviate from standard Chain of Thought, they should provide a robust and convincing demonstration that their new methods are comparably monitorable.

For all the ways we still don’t understand AI systems, we shouldn’t overlook how fortunate it is that the best reasoning models today think out loud in a way humans can understand. That is a fragile and contingent fact - and we should work to ensure it stays true.