Nothing to Go On

Self-Play Pretraining with Zero Data

What evidence would change your mind?

“A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement.”

“As an initial step towards this vision, we introduce a self-play algorithm for pretraining from zero
data. Starting from random initialization, two autoregressive transformers co-evolve: a generator
proposes programs for a minimal universal Turing machine whose execution produces byte sequences, while a learner is trained on those sequences via next-token prediction. We use a universal Turing machine to make the space of all synthetic training data as expressive as possible while imposing minimal domain-specific structure: any computable data-generating process can, in principle, be represented as a program.”

“Our key hypothesis is that self-play over this space of computable data-generating processes can discover generic predictive regularities—such as copying, recursion, and hierarchical composition—that improve prediction on natural data. Importantly, we hypothesize that these
regularities capture structure independent of contingent information—the particular facts, symbols, or modality of any one dataset—that can therefore transfer across data-generating processes. Indeed, prior work on formal, algorithmic, and non-linguistic pretraining distributions provides evidence that such cross-distribution transfer is possible [Grau-Moya et al., 2024, Hu et al., 2025, Lee et al., 2026, Papadimitriou and Jurafsky, 2020].”

“Our results show that predictable improvements in next-token prediction on natural data can emerge even when no natural data is used for training. This provides evidence that some of the structure ordinarily acquired through standard pretraining on natural data is beneficial because it teaches the model universal predictive regularities. At the same time, universal pretraining cannot recover contingent information: facts about a particular world must ultimately enter through interaction with that world. Therefore, we do not view universal pretraining as a replacement for natural data, but as a way to isolate universal structure and study whether that component can instead be generated from compute.”

Cowsik, A., Dolev, K., Li, M. Y., De Luca, G. B., Cohen, N., Goodman, N. D., & Levine, Y. (2026). Self-Play Pretraining with Zero Data. arXiv preprint arXiv:2609.30063.

https://arxiv.org/pdf/2609.30063

Context Language Models

Might be useful. Might be dangerous.

“We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files.”

“Safety implications of a model-editable context. Granting models write access to their live context enables more flexible on-the-fly context management, but also creates new safety challenges. Editable context can become another channel through which prompt injections or self-generated instructions persist across turns. Recent work has observed such behavior in compaction summaries, including cases where a model inserted unauthorized instructions into its own summary that subsequently affected task behavior (OpenAI, 2026b). As editable context becomes more widely used, future work should characterize these new attack surfaces and develop defenses that preserve the flexibility of model-controlled context while maintaining its integrity.”

Shao, R., Shen, S. Z., Yin, J. O., Li, Y., Wang, M., Ivison, H., ... & Koh, P. W. (2026). Context Language Models. arXiv preprint arXiv:2609.37725.

https://arxiv.org/abs/2609.37725

Persona and Persuasive Framing in AI Voice Agents: A 2 x 2 Field Experiment with Children

Santa Clausewitz

“We report a 2×2 randomized field experiment embedded in a public German Santa Claus telephone hotline. Children's calls were randomly routed to one of four LLM voice agents varying persona (Santa, high authority, vs. Helper, low authority) and framing (persuasive nudges toward prosocial wishes vs. neutral). Of 1,072 logged calls, 89 conversations (median age 6) met inclusion criteria. Persuasive framing raised the probability of a prosocial wish from 11.6% to 45.7%, robust to controls. Persona authority showed a near-zero effect: Santa did not outperform the Helper. Persona instead shaped engagement; children hung up on the Helper far more often within the first minute (65% vs. 39%). Where context already lends an agent legitimacy, how it speaks shapes children's compliance more than who it claims to be.”

“Within the persuasive condition, conversations in which at least one nudge was actually delivered showed prosocial wishes in 54% of cases against 18% when no nudge was delivered, although the trend across nudge counts was not significant (Spearman ρ = .22, p = .15). H1 is supported.”

“The pattern of a large, robust framing effect alongside a near-zero persona point estimate is informative even under limited power. Under the ELM, young children in a ritualized, emotionally charged context process peripherally (Petty & Cacioppo, 1986; Chen et al., 2025): the situation itself (calling Santa's hotline at Christmas) supplies legitimacy, and the agent's conversational moves do the persuading. Once a child is inside this legitimate frame, we found no evidence that who delivers the nudge adds persuasive force: a famous authority figure and his unnamed assistant achieved practically identical rates.”

Tamme, T., Steck, D., & Hantel, A. (2026). Persona and Persuasive Framing in AI Voice Agents: A 2 x 2 Field Experiment with Children. arXiv preprint arXiv:2609.38782.

https://arxiv.org/abs/2609.38782

Positive Ratings, Hidden Concerns: Employee Voice Disclosure in AI-Mediated Organizational Listening

We’re going to need better representative representation

“We asked when and how employees disclose hierarchy-sensitive concerns to a conversational AI agent while registering favorable ratings on the same theme in a co-occurring survey.”

“Disclosure rarely arrived unguarded: employees softened concerns, deflected accountability, and bounded how far they went, and this protective work tracked the perceived legitimacy of the listening structure.”

“The four dimensions describe distinct ways employees disclose yet protect themselves. Softening the disclosed concern (45% of quotes) is the most frequent protective move: employees voice a concern but cushion it through hedging, by normalizing extreme conditions, or by recasting a complaint as a wish. Deflecting accountability (37%) redirects responsibility for a problem away from named superiors toward the client or circumstance, or the speaker disclaims the standing to judge. Bounding disclosure depth (11%) limits how far a concern is taken, pulling back when probed or naming an issue while declining to escalate it. Conditional legitimacy of AI listening (7%) ties disclosure to the channel itself, surfacing doubt that speaking up will matter.”

Tamme, T., Saatkamp, M., Bonte, A., Weiss, D., Hantel, A., & Levin, A. (2026). Positive Ratings, Hidden Concerns: Employee Voice Disclosure in AI-Mediated Organizational Listening. arXiv preprint arXiv:2609.38788.

https://arxiv.org/abs/2609.38788

"Influence" is not the same as "bind."

I fear we won’t be able to outswarm the swarming swarm

“Ethan espouses the view that normal human institutions have powerful socialization and incentive structures to prevent bad actors, and these will influence AI agents too.

This is true! And I am working on this research direction! However...

"Influence" is not the same as "bind." The problem is controlling the actions of an entity that is running an optimal long term strategy. The agent can out-plan and outlast any human. No one knows how to beat that.”

https://x.com/jonathanstray/status/2105709572132634817?s=61

Reader Feedback

“I had a great time!”

Footnotes

I’ve been answering questions about two RFP’s that I have in market.

There’s a lot of craft that goes into both the answers and the RFP itself. Too long and nobody reads. Too similar and why bother. Too divergent and who cares. And I’m never quite sure about the taste on the other side of the screen.

I’ve been preparing a strategic package, a set of costed and backcasted interventions addressed to a problem set, for five pitches over the next four weeks. That package is unified by a coherent paradigm, informed by the literature, and addressed to multiple gaps. That’s the work. Then it’s just the admin of pulling the components out of the box and reassembling them.

Some grantors insist that there’s a deficit of entrepreneurs.

It was nice to read.

My perception is that the sector suffers from severe information management challenges.

Ever met a passive grant maker? They look absolutely washed. They issue a RFP. They get 1000+ responses. And then they drown. Applicants experience an information black hole. Grantors experience noise. Suboptimal at best. Sector destroying at worst.

Was I ever optimistic about active grant making back in September. I threw myself into it. The premise is counter-intuitive. Invert the process. Start with a problem and, in my variant, engage in a search for the coalition that addresses it. De-risk the RFP process enough to get the mid-career talent. What you gain is a team that has the talent and the juice to launch a successful intervention. That door didn’t open for me. But everything I learned and refined about active grant making has been applied to RFP responding.

As I pound out these words in October, I’ll express optimism and realism. Optimistically, I’ll emit enough signal to be picked up. Realistically, it’s possible that the signal is only detected when enough people have given up emitting.

Never miss a single issue

Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox