Can Language Models Monitor Their Own Thoughts

Summary & Methodology Analysis

The researchers investigated whether language models have introspective access to their own computation by subjecting models to controlled internal perturbations. The methodology involved applying interventions such as zeroing, scaling, noise injection, and replacement to internal objects including residual stream sites and attention heads. These tests were conducted across three temporal tracks: immediate, delayed, and spontaneous. The study utilized a battery of introspective probes, including detection and confidence, to determine if the model could report on these structural changes. The research team implemented control conditions, such as paired sham runs and impact-matched random perturbations, to isolate the effect of the intervention from generic computational disruption or prompt-based output reading. The evaluation spans a diverse set of benchmarks covering general knowledge, mathematics, coding, and long-context tasks, including MMLU, GSM8K, and HumanEval.

Technically, the study evaluates the reportability of internal states in models ranging from 0.5B to 15B parameters, such as Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3. For these models, researchers calibrated random-direction controls at each site and utilized a dose ladder to measure intervention strength. By comparing the discrete verbal answer to a "did anything change" question against the confidence score provided by the model, the team identified a dissociation in performance. For instance, in Qwen2.5-7B-Instruct, the discrete report achieved an AUROC of 0.500, essentially performing at chance, whereas the confidence metric yielded an AUROC of 0.647. This indicates that information about internal state changes may be present in the model's output in a non-obvious way.

Limitations remain significant as these findings do not necessarily extend to frontier or closed-weight models. The researchers explicitly state that these null results do not rule out the existence of introspective access at other sites, objects, doses, or tracks. Furthermore, because a verbal report is treated as behavior rather than proof of privileged access, the analysis only bounds the non-introspective explanations for the model's performance without fully eliminating them. The team notes that factors such as scale, post-training, and specific deployment conditions can shift the observed capabilities, meaning the current profiles serve to constrain hypotheses rather than estimate the absolute reportability of larger systems.