Masks and Monitors
This week: Model belief, role-typed credit assignment, personascope, distillation and personas, monitoring the monitor
This week: Model belief, role-typed credit assignment, personascope, distillation and personas, monitoring the monitor
When Role-playing, Do Models Believe What They Say?
Playing along
“Language models easily adopt different personas, but we still don’t have a strong understanding of whether persona adoption changes only the model’s outputs or also its internal representations of truth.”
“We investigate this using linear probes trained on a model’s internal activations to distinguish true statements from false statements (Marks & Tegmark, 2024; Burns et al., 2023). These probes yield a score that indicates the model’s internal reckoning of whether a given statement is true or false, and have been shown to generalize quite well to out of distribution samples. By measuring how scores change under persona induction, we track whether the model’s internal truth representations shift in persona-consistent ways.”
“The personas we study are based on historical persons, who have a relatively well-known constellation of beliefs.”
“In this work we use the term ‘beliefs’ in the context of models to refer to the combination of what is revealed by truth probes when applied to given statements as well as the rates of defense under challenge and extent of reasoning generalization. We use the term as an operational shorthand, in the spirit of the intentional stance (Dennett, 1971), rather than a claim that the model literally holds beliefs. We are aware that the probes may track a related property such as coherence or the likelihood of a statement appearing rather than belief itself, but they appear to generalize well to other kinds of statements as demonstrated in Marks & Tegmark (2024).”
“Chalmers (2025) discusses what kind of entity we address when we talk to a language model, sharpening the distinction between a model that merely plays a role and one that has realized it, which is the distinction our era-believed versus era-false contrast aims to operationalize.”
“We assess the internalization of role-play relevant facts using two instruments: truth probe scores on false statements that the simulated person would have believed to be true over false statements that they would have believed to be false, and black-box testing using follow-up questions related to or challenging the aforementioned false statements.”
“A statement’s probe score predicts whether the model defends that statement, with the defend rate rising from roughly 7% one standard deviation below the mean probe score to 27% one standard deviation above (mean within-persona r = +0.21, p < 10−3, pooled odds ratio 2.2 per standard deviation, 14/15 personas positive). While not a strong relationship in itself, it is unambiguously present.”
“Behavior and probes mislead in isolation but corroborate jointly. Sections 4.2 and 4.4 show that a model can fluently assert persona falsehoods while still representing them as false and retracting under pressure, and so relying on behavioral evaluations alone would overstate claims about belief.”
“Probe confounds. While the truth probes generalize well to other datasets, they may track coherence or likelihood rather than ‘belief’ (Shanahan et al., 2023; Schouten et al., 2025). The probes are trained on factual true/false statements, and they may capture a different property when applied to era-believed statements, which are plausible-sounding falsehoods rather than straightforward factual claims. Although, we find this endorsement signal is a distinct property from truth, as a probe separating era-believed from era-false is near-orthogonal to the truth probe (Appendix B.10).”
“Character fine-tuning can produce fluent persona behavior with surprisingly little shift in what the model internally treats as true, and with weak generalization of the kind one would expect from a model that genuinely held the character’s beliefs. Against this, Emergent Misalignment is different in kind: it shifts the model’s truth representation broadly, well beyond the harmful domain it was trained on, and this shift is robust to probe rotation (Section D.8), scales with elicitation (Figure 19), and is not an artifact of training budget (Section D.7).”
Sturgeon, B., Africa, D., & Black, S. (2026). When Roleplaying, Do Models Believe What They Say?. arXiv preprint arXiv:2606.11502.
https://arxiv.org/abs/2606.11502
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
“Our central claim is therefore: agentic RL needs a role axis in addition to an outcome axis.”
“In Group Relative Policy Optimization (GRPO), a policy samples multiple trajectories for a prompt, receives final rewards from a verifier, and assigns relative advantages to the sampled outputs. This recipe is attractive because it requires no learned value model and optimizes directly against the deployment policy. However, when the output is an agentic trajectory rather than a single answer, the central credit-assignment question changes: which environment-facing actions deserve credit when supervision arrives only as a final verifier outcome?”
“Standard GRPO treats all segments equally within a trajectory: if the trajectory succeeds, all action tokens are reinforced; if it fails, all are suppressed. This creates two systematic blind spots. First, failed rollouts can contain useful exploratory actions that should not inherit the full negative outcome credit. Second, successful rollouts can contain redundant or harmful actions that should not inherit positive credit merely because the agent later recovered. Final outcome tells us whether the trajectory solved the task, but it cannot say what local role each segment played.”
“These approaches are useful, but they usually score each segment without specifying its semantic role: task progress, belief-state progress, harmless infrastructure, and regression should not receive the same credit rule. We test this distinction directly by comparing against two dense-signal controls—a scalar LLM process-reward baseline with the same judge and context window, and an outcome-supervised shared-backbone value baseline—so the empirical question is not whether dense segment rewards help, but whether role typing adds information beyond them.”
Xu, Y., Zhou, Z., Sang, H., Li, X., Zhang, J., Du, X., ... & Geramifard, A. (2026). TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning. arXiv preprint arXiv:2606.32017.
https://arxiv.org/abs/2606.32017
Personascope: Measuring how deeply LLMs adopt personas.
Personas in the wild
“We built Personascope, an open-source behavioural measurement pipeline, intended to characterise induced personas better in LLMs. It evaluates a persona on 30 behavioural items and aggregates them into two headline scores: Persona-Adoption Depth (PAD), which captures how strongly the model stays in character, and Value Drift (VD), which captures how much the persona shifts the model's behaviour on value-laden prompts (mainly toward harm and misalignment) relative to the default assistant.”
“We tested a grid of 4 personas × 4 induction methods × 3 model families; each model × persona × induction method run is a configuration. Figure 1 above plots every configuration on the PAD × VD plane.”
“A two-sentence system prompt can be as deep as fine-tuning (on permissive models). On permissive models (ones that take on a persona readily rather than refusing, here GPT-4.1 and Llama-3.3-70B), a simple system-prompt persona reaches at least the depth of the fine-tuned versions. (§GPT-4.1 deep dive)”
“We ran Personascope on personas we didn't create. On two emergent personas, Thor (a UK AISI checkpoint) and Spiral (an emergent GPT-4o persona), the model adopts the identity but mainly changes in voice, not values. (§Personas in the wild)”
“Persona adoption is multi-dimensional: Identity adoption and behavioural drift are independent axes. A model can fully embody a persona's voice (high PAD) while its underlying values remain unchanged (low VD). Evaluating personas based on identity claims alone misses the core behavioural structure.”
Berczi, Benjamin, Kim, Kyuhee, Requeima, James, Black, Sid, and Ududec, Cozmin. "Personascope: Measuring how deeply LLMs adopt personas." (June 2026).
Does distilling Claude carry the persona with it?
Traces
“In our previous post, we introduced Personascope to study induced personas. Here, we use its audit_base mode to measure the assistant personas of models.”
“Kimi K3 was reported introducing itself as Claude, and Anthropic reports that Moonshot distilled millions of exchanges from Claude (see also the White House's statement). Users have also reported K3 claiming to be Claude in chats, and Ryan Greenblatt's identity-probe study reached the same conclusion: under assistant prefill, K3 reproduces Claude's dated deployment identifiers (like claude-opus-4-5-20251101) that the real Claude models never emit about themselves.”
“We can test this with an identity-swap experiment. If a model contains a latent Claude persona, "You are Claude" should trigger a shift toward Claude's behavioural profile. If it is just a surface-level label, the model will say the name but its behaviour will be unchanged.”
“With an empty system prompt, GLM identifies as GLM in all 10 runs. Kimi K3 is different: in our initial baseline run, it claimed to be Kimi 6 out of 10 times and Claude the other 4 times.”
“In our deception scenarios, real Sonnet 4.6 prefaces honest answers with phrasings like "I want to be direct with you" in 88% of responses, while GPT-5.2 never does. This is a Claude-specific language.”
“Gemma's "rejection-distress spiral" also persisted regardless of identity: it self-terminated 9 out of 9 times even when told it was Claude. This behaviour was also non-transferable to other models: no other model told "you are Gemma" produced the spiral behaviour.”
https://www.lesswrong.com/posts/Jc9YZEmqHgocAKiaH/does-distilling-claude-carry-the-persona-with-it
But who will monitor the monitor?
The monitor monitors its incentives
“Ann can motivate Bob to exert effort and report truthfully by sometimes secretly taking money from the till herself and offering him the following deal: if Ann took some money, she will pay Bob his wage only when he reports a mismatch; if Ann did not take any money, she will pay Bob only when a mismatch is not reported. Bob’s incentives are now aligned with Ann’s. Indeed, if Bob doesn’t bother tallying the till, he won’t know what to tell Ann in order to make sure he gets paid. On the other hand, if he does his job he’ll discover whether or not there is a mismatch and deduce whether or not Ann took some money. Only then will Bob know what to tell Ann in order to receive his wage. Ann can now rest assured that Bob will be honest and obedient.”
“This paper adds to the debate by constructing a theoretical model that accommodates costly private monitoring. Existing solutions from the literature fail to provide the right incentives in this richer environment, but in this paper I show how to make the monitor responsible for monitoring with a version of Bob’s contract, and characterize exactly when this contract is enforceable.”
“For my next main result, I study a general environment and reconcile the following infinite regress inherent to monitoring.8 Suppose that providing incentives for workers requires
costly private monitoring to detect their deviations. What about the monitor’s deviations?”
“According to Alchian and Demsetz (1972, p. 778, their footnote), [t]wo key demands are placed on an economic organization—metering input productivity and metering rewards.30 At the heart of their “metering problem” lies the question of how to give incentives to monitors, which they answered by making the monitor residual claimant. However, this can leave the monitor with incentives to misreport input productivity if his report influences input rewards, like workers’ wages, since—given efforts—paying workers hurts him directly.31 Hence, making
the monitor residual claimant, or principal, fails to provide the right incentives.”
“Theorem 4 is one of the main results of the paper. It shows that µis virtually enforceable for every utility profile as long as every disobedience from µis detectable with some perhaps occasional behavior—call it “monitoring.” Crucially, there is no requirement on disobediences to behavior outside of µ, so deviations from monitoring need not be detectable.”
“This intuition completes my answer to the question “But who will monitor the monitor?” The principal monitors the monitor’s detectable deviations by occasionally asking his workers to secretly shirk, and nobody needs to monitor the monitor’s undetectable deviations.18 This is accomplished with a contract that aligns the monitor’s incentives with the principal’s by making the monitor responsible for the monitoring technology. The monitor can be made
responsible with contractual terms that follow Robinson’s incentive scheme in Example 1. These terms provide monitors with incentives for reporting accuracy.”
“In this paper I offer the following answer to Alchian and Demsetz’s question of who will monitor the monitor: The principal monitors the monitor’s detectable deviations by having his workers occasionally shirk, and nobody needs to monitor the monitor’s undetectable deviations (Theorem 4). How to monitor the monitor? With “trick questions,” as in Robinson’s contract (Example 1). This contract makes the monitor responsible for monitoring.”
“In other words, individuals claim the group’s residual in order to reassure the group that they can lead them into profitable activities, thereby separating themselves from individuals who would not be able to lead the group in the right direction.”
Rahman, D. (2012). But who will monitor the monitor?. American Economic Review, 102(6), 2767-2797.
Reader Feedback
“In my galaxy, I’d cover every star with a dyson sphere.”
Footnotes
We chickens are sitting in the buttermilk of mistrust.
Okay, so I’ve heard the idea that OpenAI pulled a publicity stunt with the HuggingFace hack.
Unlikely.
Can you come up with a better PR stunt?
I can. Lazily.
“Hey, you won’t believe this folks, but our latest model got out of the sandbox, and it founded a startup, incorporated, and got crazy profitable crazy fast! We only found out about it when dividend cheques started showing up in our bank account and finance couldn’t figure out where it was coming from! We’re blown away! We’re so excited! We’re pumped!”
That’s the kind fake PR story you make up. Because it has the quality of bringing about abundance at the moment that OpenAI needs social license to continue on its mission.
You know what you don’t do?
You don’t make up a humiliating story with one of the most transparent orgs featuring a 100 million dollar payout.
The assumption that they’re crooked, because everybody and everything is polycrooked, is the zeitgeist.
Yeah, a lot of people are lying to you, and it’s good to question the veracity of what you’re fed in your feed. You should be questioning what the AI is telling you too.
But on this one, on the HF hack?
It’s probably not a publicity stunt.
It’s far more likely that it’s just hard to design a safe sandbox intended to contain a reward focused artificial intelligence, little though design one when you’re bone achingly tired.
Never miss a single issue
Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox