This week: Haiku to Opus in just 10 bits, QKV variants, next-latent prediction transformers, ai moral status, motivated reasoning, emotion concepts, AI designed radio chips.

Haiku to Opus in Just 10 bits: LLMs Unlock Massive Compression Gains

CLEVER!

“We study the compression of LLM -generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute. For lossless compression, domain-adapted LoRA adapters can improve LLM-based
arithmetic coding by 2× over compression with the base LLM alone, achieving SOTA on lossless compression of LLM-generated text.”

“We introduce a novel interactive compression mechanism that adapts the children’s game 20 Questions to the task of LLM knowledge transfer. Given a prompt, an SLM first generates an initial response and then formulates N binary yes/no questions about its solution strategy, for example, “Is my approach to step 3 correct?” The LLM answers each question with yes or no, transferring exactly one bit of information per response. The SLM then revises its response by incorporating all N answers. Crucially, the SLM’s questions are deterministic given its initial response, the prompt, and fixed hyperparameters. As a result, only the N binary answers must be transmitted.”

“When two parties share access to the same model, they effectively share a rich internal representation of the world. Communication no longer requires transmitting an entire idea from scratch; instead, it is sufficient to transmit only the difference between what the shared model would already generate and the intended content. In this sense, shared models enable an unusually low-bandwidth form of communication grounded in a common representation. More broadly, question-asking serves not only as a compression primitive but also as a mechanism for probing conditional information. Quantifying the conditional information between two beings (human or LLM) is generally abstract and difficult to measure directly. By framing communication
as a sequence of binary questions, QA-style interaction provides an operational way to bound how much additional information one agent must transmit to another about a specific concept. The number of questions required offers a concrete proxy for their conditional information gap.”

Rinberg, R., Carrell, A. M., Henniger, S., Carlini, N., & Warr, K. (2026). Haiku to Opus in Just 10 bits: LLMs Unlock Massive Compression Gains. arXiv preprint arXiv:2604.02343.

https://arxiv.org/abs/2604.02343

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Thorough authors: they try it all

“Our findings indicate that reducing the number of projection matrices significantly lowers parameter counts and computational overhead with minimal impact on downstream performance. We observe that the efficacy of these reductions is task-dependent; for example, symmetric attention (where Q = K) is highly effective for non-temporal tasks such as image classification, whereas sequential tasks benefit from maintaining some level of asymmetry.”

“Several limitations apply. Our largest validated scale is 1.2B parameters; whether the Q-K=V degradation trend continues to improve beyond 7B remains unconfirmed.”

Kayyam, A., Gopal, A. M., & Lewis, M. A. (2026). Do Transformers Need Three Projections? Systematic Study of QKV Variants. arXiv preprint arXiv:2606.04032.

https://arxiv.org/pdf/2606.04032

Next-Latent Prediction Transformers Learn Compact World Models

It generalizes well!

“How can we encourage transformers to form simpler, more principled explanations that avoid such shortcuts? A natural approach is to reinstate a key property of recurrent models: the ability to learn compact world models that channel future prediction through compressed representations of the past. We will show that this inductive bias can be reintroduced while also retaining the parallel training efficiency of transformers.”

“In this paper, we introduce Next-Latent Prediction (NextLat), which extends the standard next-token prediction objective with self-supervised predictions in latent space. NextLat jointly trains a transformer and a latent dynamics model: the transformer learns to encode past tokens into compact latent summaries such that the dynamics model can predict the transformer’s next latent state given only the current latent state and the next token (i.e., the “action”). This objective encourages the transformer to form a compact internal world model with coherent recurrent-like dynamics, while avoiding the sequential processing overhead of recurrent architectures.”

Teoh, J., Tomar, M., Ahn, K., Hu, E. S., Sharma, P., Islam, R., ... & Langford, J. (2025). Next-Latent Prediction Transformers Learn Compact World Models. arXiv preprint arXiv:2511.05963.

https://arxiv.org/abs/2511.05963

Towards evaluating ai systems for moral status using self-reports

I reckon the mechinterp path would be the one to take

“We argue that under the right circumstances, self-reports, or an AI system’s statements about its own internal states, could provide an avenue for investigating whether AI systems have states of moral significance.”

“Introspection training may come with safety risks. AI systems that have a high degree of understanding about their training process and environment may learn to interfere with that process in undesirable ways, in service of the training objective (Cotra, 2022; Berglund et al., 2023). For example, Lehman et al. (2018) document numerous cases of reward hacking where models learn to exploit bugs in the reward function or environment in undesirable ways. Introspective training incentivizes the model to learn more about itself and its environment, which may thereby increase such risks.”

Perez, E., & Long, R. (2023). Towards evaluating ai systems for moral status using self-reports. arXiv preprint arXiv:2311.08576.

https://arxiv.org/pdf/2311.08576

How to distinguish motivated reasoning from Bayesian updating

“People with different political views often react to the same information in different
ways.”

“In this paper, I develop two nested formal models of motivated reasoning, and contrast the predictions they make for empirical work.

In the first model (“once-motivated reasoning”), subjects choose a belief that optimizes a trade-off between accuracy and directional motives (Kunda, 1990; Bracha & Brown, 2012; Little, 2019). Alas, it is impossible to determine whether subjects are affected by once-motivated reasoning or “fully Bayesian” by observing how they respond to new information.”

“Any once-motivated reasoner has a Fully Bayesian Equivalent (FBE) who only cares about accuracy and has a different objective prior belief. These two subjects are identical in both their (motivated) prior and posterior beliefs – i.e., what we can actually measure – for any signal structure.”

“Finally, it bears emphasis that motivated beliefs are not the only explanation of these results. For example, take the discussion of parallel updating. Outside of the lab, supporters of different parties tend to consume different political information, and if they don’t adjust for this bias, beliefs need not converge. Further, individuals
may broadly agree on the underlying facts of, say, how the economy is performing, but then partisan differences in how to interpret these facts or expressive responding can generate a relatively constant difference in how these map to survey responses.”

Little, A. T. (2025). How to distinguish motivated reasoning from Bayesian updating. Political Behavior, 47(4), 1501-1525.

https://link.springer.com/article/10.1007/s11109-024-09999-7

Emotion concepts and their function in a large language model

“We caution against conclusions about whether models “feel” or “experience” emotions.”

“Interestingly, they do not by themselves persistently track the emotional state of any particular entity, including the AI Assistant character played by the LLM.”

“They are also implicated in alignment relevant behaviors including blackmail, reward hacking, and sycophancy. These representations appear to be part of general charactermodeling machinery inherited from pretraining. The structure of the model’s emotion space reflects
human psychology, with valence and arousal emerging as primary organizing dimensions.

Sofroniew, N., Kauvar, I., Saunders, W., Chen, R., Henighan, T., Hydrie, S., ... & Lindsey, J. (2026). Emotion concepts and their function in a large language model. arXiv preprint arXiv:2604.07729.

https://arxiv.org/pdf/2604.07729

AI Is Designing Radio Chips That Humans Couldn’t Even Imagine

And we still want to understand what’s happening inside them!

“Chip testing and debugging is a long, arduous process, sometimes even more so than design. Engineers often prefer ICs to have interpretable structures, so that if a problem crops up, they can understand how the chip works well enough to debug it.”

https://spectrum.ieee.org/ai-radio-chip-design

Plan A

Finally, some positive sum

https://ai-2040.com/

Reader Feedback

“It’s likely that maybe some sciences could automated most of the way, but there’s a whole set of sciences that are just too chaotic, that I could see, for AutoScience to be useful. But I get why there’s so much interest in making them work.”

Footnotes

Can explicit, inspectable models of a researcher's current beliefs improve scientific reasoning in AI Control better than document-centric AI assistants?

That’s the subject of a grant application I made last week.

Because I’m experiencing the bottleneck right now. You know the memes: The space moves so fast. It’s hard to know what to keep up with. It’s time consuming to get up to speed. Red Queen.

You Better Work!

Almost every AI researcher I know has cobbled together a set of tools to deal with the bottleneck. Usually there’s a repository for papers, a repository for code, a tool for a form of priority information management, and a tool for authoring. Some make use of graphs. One has a mind-mapping tool.

The fragmentation doesn’t bother me. It’s cheaper than ever to wire a bunch of disparate tools together. Some are using swarms of AI agent scientists to help them make more sense of it all.

And that’s great. It’s super-cyclical.

We get dissatisfied with just how fragmented the information is, so we commit to consolidating everything into a single tool, a single platform, because only if all the data were in one place, all the problems would be solved. And then once everything is all centralized, we chafe at the lack of flexibility or the speed of the functionality development, and so we break it all apart again. I’m just going to assume that the super-cycle will continue long after the research data mesh data lake cottage MCP thing works its way through the trough of disillusionment. All good. No shade.

Beliefs might be unreasonably effective at focus and widening the bottleneck. At least that’s what I’d like to explore. What does a researcher believe they believe, and what evidence would change those beliefs?

Gratitude to those who helped with the proposal!

Never miss a single issue

Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox