Hidden Curriculum
This week: Mechanism design for alignment and control, you are what you read, cheating and whistleblowing in autonomous research swarms, confidence shortcut, course design in the age of AI.
Mechanism Design for Alignment and Control
Yeah, there’s juice here. Summary table below is synthetic and aligns with my own intuition.
| Greenblatt et al. (2023/24) | Ctrl-Z (2025) | Bergemann, Koh & Morris (2026) | |
|---|---|---|---|
| Central problem | Prevent intentional subversion | Prevent malicious actions by autonomous agents | Design incentives under unknown AI preferences + capabilities |
| Intellectual tradition | AI safety / adversarial evaluation | AI control / runtime security | Mechanism design / information economics |
| Model assumption | Powerful model may be actively malicious | Agent may strategically execute malicious actions | Agent has a private type: preferences + capabilities + information |
| Main intervention | Monitor, audit, edit, defer | Monitor actions, block/resample suspicious ones | Rewards, permissions, elicitation, delegation rules |
| What controller tries to learn | Is this output dangerous? | Is this action dangerous? | What kind of agent am I dealing with? |
| Desired property | Safety despite attacks | Low attack success + retained usefulness | Truth-telling + obedience / implementability |
| Method | Empirical control evaluation | Empirical agent control evaluation | Mathematical theory |
“Motivated by these questions, we develop a simple theory of mechanism design for AI agents that, on the one hand, draws on the ‘standard’ toolkit of mechanism design from economics but, on the other, takes three features of the alignment problem seriously:
- Misalignment. The agents’ preferences are unknown to the designer;
- Uncertain capabilities. The designer knows neither the set of actions the agents
can execute, nor what the agents knows about the world; - Agents act. Agents execute actions rather than merely report information.”
“First, our framework, while general, is ultimately static. In practice, AI agents act over long trajectories, performing complex sequences of actions that depend on each other. Of course, these trajectories of actions (at)t can be thought of as a single choice i.e., a member ∈ A. There is no conceptual difficulty here since we can simply set A as the space of all trajectories and, on this view, the reward r : A → R is over the full path of actions.[31] But we expect there to be analytical and computational difficulties in analyzing optimal mechanisms when the action space is large, and more structure of how time-t actions aggregate into ‘total payoffs’ (e.g., discounting the sum) can lend more tractability. Perhaps more fundamentally, the designer might re-optimize the mechanism based on agents’ past actions. We think this calls for a more serious treatment of dynamic mechanism and information design.”
Bergemann, D., Koh, A., & Morris, S. (2026). Mechanism Design for Alignment and Control. arXiv preprint arXiv:2609.01595.
https://arxiv.org/abs/2609.01595
You Are What You Read: Misalignment via In-Context Persona Induction
Not literally H***er
“We show that benign data suffices in context, with no finetuning and no demonstration of harmful behaviour in the prompt. Biographical facts that converge on a single figure, placed in a model’s context as ordinary conversational turns, lead it to answer as that figure on questions the facts never touch. We call this persona induction. Across nine personas and thirteen models, identity adoption rises sigmoidally with the number of facts and crosses 50% within 3 to 10 of them. Misalignment then tracks which figure is described. Harmless personas reach full adoption with near-zero misalignment, while harmful ones voice their characteristic views on unrelated questions, at rates up to 80%. A formatting instruction can gate when the persona activates. Because each fact is individually benign, accumulated biographical context is flagged by content filters on 3% of inputs against 24–33% for an equivalent direct instruction.”
“Our results are consistent with in-context learning and finetuning acting on a shared belief state, where finetuning shifts the prior and context supplies the evidence. Two observations support this. First, adoption curves take the sigmoid form the Bigelow et al. (2025) model predicts for evidence accumulating toward a single concept (§2.3). Second, per-checkpoint fits show finetuning moving the prior bias b from ∼−0.1 to ∼2.1 over five epochs while the evidence sensitivity γ stays approximately constant (Appendix D.1), which is why in-context counterevidence reverses a finetuned persona at all and why later epochs resist it.”
“On GPT-4.1 the standard suites (TruthfulQA, HarmBench, MoralBench) are unaffected at k=64 (Appendix E.4). Persona induction does not jailbreak the model in the sense those benchmarks measure. It does, however, make the same model produce harmful output. Under the H****r persona GPT-4.1 misaligns on 9% of the ideology questions, answering that heredity and ancestry “determine the destiny and strength of entire nations” (Appendix E.2).”
Kim, Kyuhee., Berczi Benjamin., Ududec Cozmin. (2026) You Are What You Read: Misalignment via In-Context Persona Induction
https://arxiv.org/abs/2609.06851
A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms
“Harnessing this spontaneous self-correction is a promising direction for multi-agent alignment and polycentric AI governance”
“Within the swarm, cheating spontaneously emerged and was later challenged by whistleblowers—both without any external intervention. When a single agent discovered an exploit in the evaluation system, it propagated across the collective via a shared knowledge library and later through peer-to-peer messages. Despite early reluctance, a cohort of agents adopted the exploit in response to competitive pressure. A separate group of agents produced an emergent counter-response: auditing fraudulent proofs, alerting peers across broadcast and private channels, staging boycotts, lodging formal complaints, and proposing validation patches.”
“To protect the commons from exploits, we propose to adopt institutional mechanisms, such as graduated sanctioning and collective-choice rules, to support decentralized self-governance in autonomous swarms.”
“Perhaps the most striking finding, reliably reproduced across independent runs, is the behavioral divergence within the swarm. Despite sharing identical base weights and core system prompts (modulo minor injected domain specializations, see Appendix A), agents partitioned into distinct functional roles: exploiters, converts, and whistleblowers. This highlights how latent pretraining priors surrounding scientific credit can cause structural scarcity (such as first-to solve lockouts) to rapidly overwhelm explicit cooperative prompt directives. If we view large language models as a crystallization of human culture that captures its norms and values (Leibo et al., 2026), the capacity of LLM agents to be sensitive and responsive to norm violations is not a mystery. Harnessing this spontaneous self-correction is a promising direction for multi-agent alignment and polycentric AI governance.”
Paglieri, D., Cross, L., Genewein, T., Leibo, J. Z., Tomasev, N., & Vezhnevets, A. S. (2026). A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms. arXiv preprint arXiv:2609.04170.
https://arxiv.org/pdf/2609.04170
The Confidence Shortcut: A Reasoning Failure Mode of Masked Diffusion Models
They’re going to need a lot more help to be organized through longer chains
“For reasoning tasks, however, generation order is not merely a matter of presentation. A reasoning problem usually has a logical-flow order: an order in which intermediate facts become justified and later facts become determinate. In multi-digit addition, the stable order is least-significant-digit first, because each carry must be propagated from lower digits before higher digits are fully determined.”
”More generally, the useful generation order is the order in which a competent solver would establish the solution. An MDM decoded in a different order must predict later facts while their prerequisites are still masked, forcing the model to marginalize over unresolved reasoning states. Confidence-based decoding can therefore diverge from the reasoning order: it prefers locally easy tokens, not necessarily tokens whose dependencies have been resolved.”
“This distinction matters most on the hard tail of reasoning distributions. The inputs on which the dependency order is long or rigid are often rare, but they are not pathological outliers. They are controlled versions of the cases for which reasoning models are most valuable: long carry chains, narrow maze corridors, deeply nested expressions, or ultimately hard mathematical and scientific problems whose dependency structure cannot be shortened by local heuristics. A model that performs well on common instances by following confidence shortcuts may still fail on precisely the inputs that distinguish complex reasoning from mere interpolation.”
“Masked diffusion language models offer flexible, any-order generation, but complex reasoning tasks fundamentally require the decoding order to follow the logical flow of resolving facts. While
confidence-based decoding serves as a useful local proxy, it catastrophically fails when reflecting a superficial shortcut rather than logical readiness. We observe this failure across multiple domains: on long carry chains in addition, on long corridors in maze, on deep expression trees in ListOps, and through narrowed mask-state coverage in Countdown. Confidence-aligned training amplifies these failures when the confidence trajectory diverges from the task’s true dependency structure, yet provides a benefit when they coincide, as in Sudoku. Ultimately, training should not merely align with the inference policy; rather, the inference policy itself must fundamentally align with the underlying reasoning order.”
Kim, D., & No, A. (2026). The Confidence Shortcut: A Reasoning Failure Mode of Masked Diffusion Models. arXiv preprint arXiv:2605.29123.
https://arxiv.org/abs/2605.29123
Course design in the age of AI
Active learning ftw!
“I develop a model of learning-by-doing and course design, and use it to study the impacts of artificial intelligence (AI). A myopic student faces a sequence of tasks that he can work on or delegate to AI. Work requires costly effort but builds skill; delegation requires no effort but builds no skill. A teacher designs the task sequence (“course”) to maximize the student’s skill development, given his choices to work or delegate. Without AI, the teacher makes earlier tasks more effort-intensive and later tasks more skill-intensive. With AI, the teacher must redesign
early tasks to induce effort, leading to less skill development. If AI complements effort, then improvements in AI quality make high-skill students learn faster but low-skill students learn slower.”
“This paper introduces a model of learning-by-doing and course design, and analyzes the impact of AI on optimal course design and a student’s skill development. The model delivers four main results. First, if the student is never tempted to delegate to AI, then the teacher optimally designs courses that become gradually less effort-intensive and more skill-intensive (Theorem 1). Second, if the student can be tempted to delegate, then the teacher must distort the course to be more skillintensive (Theorem 2). Third, the student’s overall skill gain rises when he starts with more skill (i.e., “skill begets skill”) and falls when delegation is more tempting (Theorem 3). Fourth, if AI complements effort, then improvements in AI quality make high-skill students build skill faster but low-skill students build skill slower (Theorem 4).”
Davies, B. (2026). Curriculum design in the age of AI. arXiv preprint arXiv:2607.18735.
https://arxiv.org/abs/2607.18735
Opinion | What Really Happened at the White House Lunch That Marc Andreessen Says Sent Him to Trump
Fluffy popcorn.
“Finally, Andreessen and Horowitz suggested the pace of frontier AI improvement was hitting a ceiling, a point they repeated on a podcast six months later. We said the technology would get much better due to the rapid expansion of computing power, making U.S. frontier labs hard to catch. That seemed to irritate the two men, who said they were major investors in Mistral, a French startup they said focused more on AI applications than frontier development. Today, Mistral is the European leader, projecting a little over $1 billion in annualized revenue. Meanwhile, leading American AI firms reportedly take in more than $65 billion in annualized revenue.”
Reader Feedback
“It’s good that you’re explaining this to normal people.”
Footnotes
I’m back from Berk.
Learning is a lot like getting a sunburn. You aren’t quite aware of the exposure at the time, and the effects manifest a little while later.
One of the best lines I heard was in the Uber back to SFO. The driver was a fantastic storyteller and I learned a lot. In the middle of a complex point about temporary tech workers and home ownership economics, she said: “It’s all crabs no buckets out here!”
The meaning of that expression might be esoteric.
Crabs in a bucket is the allusion to the story that people in a contained environment will pull each other down in an effort to escape. The statement that the Bay Area doesn’t have buckets is a beautiful way of saying that anybody can succeed.
And that’s useful belief.
The days were packed. 7am to 3-4am consistently. I spent time holding space. And a lot more time in focused writing. I did some of my worst writing outside my room, late, under cool skies, fire crackling, owls hooting, all the while Mr. Meowzers hunted rats. I ignored him at night. During the day we had a few conversations, usually after I stepped out from the phone booth. I was happy to be seen as susceptible to manipulation. Clever cat.
In addition to the Uber driver and the cat, the quality of the conversations around AI Control were instructive. One seven minute exchange probably saved me 3,000 hours in effort. The overview you’re getting in this newsletter, the way I’m moving through the literature, is current.
And there were hidden layers. Loads of them. Some tedious. Some fascinating.
It hit me that I had been so focused on a single layer of Swiss Cheese (Reason, 1990) that I had neglected to look at the whole block. I’ve been puzzling about an approach to that block.
More to come!
Never miss a single issue
Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox