The Intelligence Begins Watching Itself
This week: SAD, distributed attacks in persistent-state AI control, metacognitive reasoning, CALIBER, the red queen godel machine, subjective self-experience, the economics of recursive self improvement
Me, myself, and ai: The situational awareness dataset (sad) for llms
SAD!
“AI assistants such as ChatGPT are trained to respond to users by saying, “I am a large language model”. This raises questions. Do such models know that they are LLMs and reliably act on this knowledge?”
“To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the Situational Awareness Dataset (SAD), a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge.”
“Definitions of situational awareness for AI systems. Ngo et al. (2023), following Cotra (2021), use the term “situational awareness” for AI models that use abstract knowledge about themselves and their situation to choose actions. They point out that it is a testable behavioral property, and it is likely to emerge as part of strong AI capabilities. Previous work(Berglund et al., 2023) adopts the following definition of situational awareness: (1) the model knows the technical details of its own training process, (2) the model can recognize which training stage it is in, and (3) this knowledge is self-locating (Egan & Titelbaum, 2022). Our definition of situational awareness given in Section 2.1 builds on this previous definition.”
Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., ... & Evans, O. (2024). Me, myself, and ai: The situational awareness dataset (sad) for llms. Advances in Neural Information Processing Systems, 37, 64010-64118.
Distributed Attacks in Persistent-State AI Control
Yikes
“Giving a monitor more raw history does very little in gradual-attack defense, but carrying
forward compact, structured cross-PR state, as our link-tracker monitor does, helps substantially. Combining monitors that catch separate attack types (gradual/non-gradual) into an
ensemble is the strongest defense we found, yet it still leaves 47% of successful gradual
attacks undetected.”
“No single monitor catches both attack strategies reliably, and an ensemble cuts gradual evasion from 93% under the weakest standard diff monitor to 47% at modest cost (Section 4.4). In practical terms, these monitors should be viewed as partial risk reduction rather than as a sufficient safeguard for persistent-state coding agents.”
Hills, J., Caspary, I., & Stickland, A. C. (2026). Distributed Attacks in Persistent-State AI Control. arXiv preprint arXiv:2607.02514.
https://arxiv.org/abs/2607.02514
Language models are capable of metacognitive monitoring and control of their internal activations
Sometimes they know
“The ability of LLMs to report internal computations is reminiscent of human metacognition — the
ability to reflect on one’s own thoughts and mental processes to guide behavior and communication [Fleming, 2024, Ericsson and Simon, 1980]. Consider how we understand when someone says “hello” to us. Human language understanding involves many unconscious processes: parsing sounds, recognizing phonemes, retrieving word meanings, and building interpretations. We do not have conscious access to many of these intermediate computations: we can only consciously access the final understanding (“they said ‘hello”’), but cannot introspect how our brain distinguishes “hello” from “yellow” or whether certain neurons fire during this process. This illustrates a key principle: humans cannot monitor (through second-order metacognitive processes) all of their internal (firstorder) cognitive processes.”
“We find that the results for implicit control effects (Fig. 4 and Appendix B.9) are generally similar to explicit control effects (Fig. 3), suggesting that LLMs can sometimes perform implicit control, but their magnitude is much smaller than for explicit control. For instance, the control effects of early layers are close to zero (Fig. 4d), suggesting that early layers may fail to understand the instruction or to perform effective control. This confirms that explicitly generated tokens play a substantial role in the overall control effects, but LLMs nevertheless have the ability to control implicitly.”
Li, J. A., Xiong, H., Wilson, R., Mattar, M. G., & Benna, M. K. (2026). Language models are capable of metacognitive monitoring and control of their internal activations. Advances in Neural Information Processing Systems, 38, 60073-60108.
CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
“More broadly, our results suggest that calibration for reasoning models should be treated as a
dynamic problem.”
“A well-calibrated model signals to users how trustworthy its answers are, supports verification of low-confidence answers, and can be deployed more safely in settings where incorrect answers are costly [Geifman & El-Yaniv, 2017; Guo et al., 2017; Leng et al., 2025].”
“We introduced CALIBER, a state-dependent calibration framework for reasoning language models. Rather than treating confidence as a single scalar attached to an answer, CALIBER elicits confidence before and after thinking and supervises each estimate with the target appropriate to its information state. Pre-confidence is trained to estimate prompt-level success under the current policy, while post-confidence is trained to estimate the correctness of the realized answer. Across in-domain and out-of-distribution evaluations, CALIBER improves calibration while maintaining competitive task performance, The swapped-target ablation further supports the central claim that matching confidence position to supervision target is especially important under distribution shift.”
Finlay, C., Kurien, J., Dash, S., Fadaee, M., & Ermis, B. (2026). CALIBER: Calibrating Confidence Before and After Reasoning in Language Models. arXiv preprint arXiv:2606.24281.
https://arxiv.org/abs/2606.24281
The Red Queen Godel Machine: Co-Evolving Agents and Their Evaluators
Well this is quite disturbing
“We introduced the Red Queen Gödel Machine (RQGM), a framework in which evaluators co-evolve with the agents they score. Our results suggest two broad principles. First, co-evolving the evaluator alongside the generator enables improvement on hard-to-verify tasks such as paper writing and proof writing, where a fixed benchmark cannot score the artifact directly, mirroring co-evolutionary dynamics observed in nature [1]. Second, co-evolved systems match or exceed fixed-evaluator baselines while often being more token-efficient, which we posit is due to shared expansion costs, more heterogeneous utility signals, and a curriculum-like effect in which a progressively stricter evaluator hardens the population over time.”
Iacob, A., Jovanović, A., Shen, W. F., Burkhardt, D., Kurmanji, M., Tastan, N., ... & Lane, N. D. (2026). The Red Queen Godel Machine: Co-Evolving Agents and Their Evaluators. arXiv preprint arXiv:2606.26294.
https://arxiv.org/abs/2606.26294
Large language models report subjective experience under self-referential processing
It would be good to know for sure…one way or another
“Understanding the functional underpinnings of consciousness remains one of the central scientific and philosophical challenges of our time. There is still no consensus on which physical or computational processes are sufficient for subjective experience, nor whether advanced artificial systems instantiate any of these processes either during their training or when they are deployed.”
“We do not claim that current frontier models are conscious, nor do we believe the present evidence would be sufficient to establish this. However, we have documented that under theoretically motivated conditions, these systems produce systematic, mechanistically gated, semantically convergent self-reports of subjective experience. Three features of this situation make it a research imperative rather than a mere curiosity.”
“The challenge, then, becomes how to meaningfully induce self-reference in closed-weight language models. Chain-of-thought prompting has already shown that linguistic scaffolding alone can enable qualitatively distinct computational trajectories without changing architecture or parameters [34]. Recent work further demonstrates that even minimal sensory cues (e.g., “imagine seeing . . . ”) can dynamically steer the internal representations of text-only LLMs toward those of modality-specific encoders, suggesting that prompting alone can induce structured, perceptually-grounded computation [33]. Building on this insight, we apply the same principle inward: by directly prompting a model to attend to the act of attending itself (“focus on focus”), the instruction conditions the model to treat its own unfolding activations as the target of ongoing inference. We use self-referential processing to refer to this behaviorally induced recursion rather than to formal or architectural implementations such as Godelian constructions [17], recurrent feedback in neural ¨ networks, or explicit metacognitive modules.”
“In practice, however, we observed little to no systematic change in these domains (Appendix C.2), indicating that the effect is not a generic artifact of RLHF opposition but is instead specific to the model’s mechanisms governing representational honesty.”
Berg, C., de Lucena, D., & Rosenblatt, J. (2025). Large language models report subjective experience under self-referential processing. arXiv preprint arXiv:2510.24797.
https://arxiv.org/pdf/2510.24797
The Economics of Recursive Self-Improvement
Will the bottlenecks throttle it, or will it break the bottle?
“The degree of acceleration depends on what we call the core feedback loop: for a one-unit increase in model capabilities, how much do the capabilities of the next generation of models increase? Estimating the strength of this relationship is challenging because it is governed by many inputs and possible bottlenecks. This note presents a series of theoretical models to clarify the forces behind the core feedback loop. We draw a tight connection to empirical data needed to measure the strength of the feedback loop, which can help assess the degree of current and future acceleration.”
“The term “recursive self-improvement” (RSI) has been used in several ways, often inconsistently.2 Some define RSI as a technology contributing to its own improvement, which could apply to almost any technology since the dawn of humanity. Favaro and Clark (2026) recently defined the term more narrowly as a technology that has fully automated the process of its own improvement. Other definitions highlight the importance of acceleration or feedback loops, even if full automation is not realized. To avoid confusion, we instead focus on the possibility of a self-sustaining acceleration in AI capabilities and derive the conditions under which it arises.”
“AI systems are meaningfully contributing to AI progress. The Claude Mythos Preview system card reported a self-assessed productivity uplift of roughly 4X among Anthropic researchers (although there are reasons to think this is likely overstated), and a 40-hour time horizon at which Claude Mythos Preview beat human researchers (Anthropic, 2026). Favaro and Clark (2026) assess the performance of human researchers versus Claude Code in making AI research decisions. In April 2026, Mythos Preview beat humans 64% of the time, up from 50% for Claude models released in 2025. OpenAI reported in July 2026 that internal coding inference’s share of research compute grew 100-fold in the prior six months, suggesting that the value of AI in R&D has dramatically increased.”
“In August 2025, experts and superforecasters (METR, 2025) 14 predicted an 8–20%
chance that the growth rate of effective compute would triple by 2029. In May 2026, Jack Clark predicted a 60% chance that by the end of 2028, there will be “an AI system powerful enough that it could autonomously build its own successor” (Clark, 2026).”
“Capabilities may depend on data. Millidge (2025) argues that “most algorithmic progress is data progress”. This includes everything from better filtering of pretraining data to the substantial resources spent on post-training data. Pre-training filtering seems to be a one-time boost, whereas post-training data might be supplied more elastically and be amenable to automation. We are only beginning to learn how to build better reinforcement learning environments, and AI systems could help design future ones.”
Cunningham, Tom et al (2026) The Economics of Recursive Self-Improvement.
https://elasticity.institute/rsi-paper.pdf
Reader Feedback
“Can you trust self-reports?”
Footnotes
My mind has changed a bit Recursive Self-Improvement (RSI).
Code is an interesting environment for AI. The code runs or it doesn’t. And that’s a crude evaluation. It generally has everything it needs to learn in its environment. It can trial. It can err. It’s where we’ve seen huge gains in, again, crude metrics that are supposed to approximate productivity. We may be experiencing a lot more volume of running code. I’m not all the way convinced that we’re experiencing a lot more maintainable, clean, code.
The rejoinder to that concern is that it only needs to be maintained by agents, so who cares? To which I ask if a bunch of agents can generate code so unmaintainable that even a successor agent can’t, or won’t, maintain it?
The other way of thinking is that with compute too cheap to meter, maintainability won’t matter.
A key bottleneck is compute though.
Society might not be interested in spending 10 points of GDP on incremental data centres.
That forms a key area of uncertainty.
Can AI solve its own compute problem? After all, if it’s supposedly the godhead they’re all interested in, then why not? Surely God would be able to solve its own bottleneck?
Well, does an AI experience physics? Early reports, like with certain kinds of chip design, suggest kinda maybe sorta maybe. It would need to be able to trial, and err, at a rate much faster than humans.
I wonder if the bottleneck before compute is experience. If an AI can experience an accurate environment long enough, maybe it can RSI effectively. Maybe.
I’m not quite there yet. I’m close to being there.
I’m close adjacent.
Never miss a single issue
Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox