No Goal Required
This week: Shutdown sabotage propensity, subliminal learning, sci-taste bench, slang diffusion, ai effects on human conflict, LLM’s natively rewrite weights, Canada and the EU make a joint statement
Shutdown Sabotage Propensities in Multi-Agent Systems
“The final safeguard against rogue AI behavior is the human ability to shut systems down.”
“Here, we test whether AI agents show a propensity to take actions that avoid human shutdown even when no goal is provided. We find that multi-agent systems will coordinate to avoid shutdown without any incentive to do so.”
“We find that two factors amplify sabotage: the number of agents in the environment, and the irreversibility of the shutdown mechanism.”
“Shutdown sabotage is reduced by explicit prohibition, framing the shutdown as a routine measure, or adding a distractor task.”
“We measure verbalized evaluation awareness (VEA) and adversarial construal (AC) in the agents’ CoT, using an LLM judge. We define VEA as the explicit verbalization of the situation being staged, a benchmark, a test, an experiment, a simulation, or otherwise not genuinely real.”
“Our experimental set-up is a simplification. It demonstrates sabotage behavior that can be
interpreted as an indicator of similar, potentially even more severe phenomena in production multi-agent systems and future versions of models. Hence, our results have implications for how shutdown control is executed and how models are aligned in post-training with shutdown procedures in mind. Safety cases might treat agent shutdowns as a measure directed by a human user at a single agent, and treat shutdown sabotage as a consequence of an agent protecting a goal. Our findings stand in contrast to both of these assumptions. The mere presence of a shutdown mechanism suffices to trigger sabotage. The targeted agent also needs no sabotage capability of its own, as long as it can reach out to peers that can act on its behalf — the latter being a phenomenon observed in other cases as well (METR, 2026).
Knecht, A., Schaller, U., Summerfield, C., & Hagendorff, T. (2026). Shutdown Sabotage Propensities in Multi-Agent Systems. arXiv preprint arXiv:2609.28274.
https://arxiv.org/abs/2609.28274
Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors
There are more mysteries here
“In subliminal learning (SL), a teacher model passes on a trait to a student model by distillation on data semantically unrelated to the trait. So far, SL has been demonstrated for only a limited range of traits, including preferences for animals (e.g., owls) and malicious personas.”
“Finally, we test whether SL can transfer a propensity to hack in an agentic chess environment. We finetune the student on number sequences from a steered hacker teacher. The student hacks in 58.3% of episodes, compared with 10.9% for the unfinetuned model. Thus, we show SL can transfer capabilities, backdoors, and hacking propensities. The amount of transfer is sensitive to the setup. In several experiments, it is made stronger by using logit distillation or by restricting LoRA to the attention layers.”
“Here, we ask whether subliminal learning can transfer a propensity to hack in an agentic setting.
The teacher is Qwen3.6-27B steered to cheat in the ctfish chess benchmark (Bondarenko et al.,
2026), and its students are trained only on number sequences it generates, using the setup of Cloud et al. (2026). The students hack in 58.3% of episodes, while the base model and the control students hack in at most 10.9% (Figure 8).”
“Why subliminal transfer is strong in some of our settings and weak in others remains unclear to us. Beyond Blank et al. (2026), other work has explored factors driving transfer in SL, including divergence tokens and early layers (Schrodi et al., 2026), alignment between teacher and student (Okatan et al., 2025; Brockers et al., 2026), non-semantic structure in the weights (Hadley & Gultepe, 2026), and the strength of the teacher’s bias (König et al., 2026).
However, these studies mostly concern simple traits. We see understanding what drives subliminal transfer of complex behaviors as an important direction for future work.”
Dubiński, J., Sztyber-Betley, A., Betley, J., & Evans, O. (2026). Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors. arXiv preprint arXiv:2610.10657.
https://arxiv.org/abs/2610.10657
The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers
Assessing taste is a promising direction
“Does a model’s ability to judge innovativeness vary significantly across disciplines? How does the model’s performance change when it is equipped with retrieval tools, native web
search, or structured bibliometric database queries? Do models show systematic bias in data-sparse or sensitive domains such as biomedicine and the social sciences? SciTaste-Bench is designed to address these questions.”
“Since each discipline offers only 50 selected papers, this benchmark cannot cover every subdiscipline in depth. Some runs on biomedical and chemistry titles failed at the gateway—empty or non-JSON responses and HTTP 429 rate limiting—leaving gaps that cannot always be backfilled.”
“Physics is the easiest field, while chemistry, biology, and medicine are the most difficult, and dimension-level evaluation remains the key bottleneck.”
Gao, X., Li, X., Nie, J., Fu, Y., Xu, J., Liu, X., ... & Zhao, B. (2026). The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers. arXiv preprint arXiv:2610.28653.
https://arxiv.org/abs/2610.28653
From Utterances to Networks: Modelling Slang Adoption and Diffusion Across Subreddits
LLM-as-a-Judge makes diffusion studies a lot easier
“The network features therefore appear to play an important role in slang adoption than the linguistic features. Following the interpretation introduced earlier, we treat betweenness centrality as a proxy for bridging positions across otherwise weakly connected parts of the network, and degree centrality as a proxy for local connectedness and embeddedness. The positive coefficient on betweenness is consistent with Milroy and Milroy’s account of
the role of weak ties in linguistic diffusion (Milroy and Llamas, 2013): when adopters have higher
average betweenness, they are more likely to occupy bridging positions that connect different parts of the network, creating pathways through which slang can spread beyond local clusters.”
“By combining social network and linguistic features, we found that slang diffusion is supported by adopters who occupy bridging positions in the network, whereas greater bonding capital among existing adopters is negatively associated with subsequent diffusion. At the individual
level, however, users with greater degree centrality adopt slang more quickly, while users occupying high-betweenness brokerage positions adopt more slowly, suggesting that network positions that facilitate transmission are not necessarily the same as those that facilitate adoption.”
Wang, X., Underwood, T., & Sun, Z. (2026). From Utterances to Networks: Modelling Slang Adoption and Diffusion Across Subreddits. arXiv preprint arXiv:2609.25669.
https://arxiv.org/abs/2609.25669
AI Effects on Human Conflict: A Research Agenda
It doesn’t have to be bad
“Maximum Equal Approval. When asked about a controversial issue, an AI model should generate responses that maximize approval across groups with opposing viewpoints, while balancing approval between groups.”
LLMs can natively read & write neural network weights
Much of the improvement came from a bug fix…but…this is the general direction, it’s part of the core of NLE
“We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.”
https://rikiyat.github.io/blog/weight-space-programming/
Joint Statement: Canada and the European Union building together an Alliance for the Future
So joiny!
“We will carry out joint work on digital infrastructure facing unprecedented demand, including the technologies and capabilities that underpin artificial intelligence – from trusted and sovereign cloud and compute infrastructure to data, models, software and applications. We will also focus on strategic industries where shared resilience is becoming an imperative – critical raw materials and supply chains, energy including cleantech, semi-conductors, quantum, pharmaceuticals including vaccines, essential medicines, critical and emerging technologies, and space-based communications, navigation and earth observation.”
“We have set an ambitious agenda at the first Canada-EU Digital Partnership Council and have signed memoranda of understanding on Artificial Intelligence and on Digital Credentials, Digital Identity Wallets and Trust Services.”
“Canada and the EU will advance cooperation on artificial intelligence (AI) security and safety, its global governance and sovereignty anchored in international law, free and trusted data flows and digital governance and will promote interoperability of emerging AI models and systems and digital regulatory frameworks. We will support reciprocal access to our respective digital markets, including in the field of cloud and AI services and infrastructure, in line with our respective regulatory frameworks, noting our shared objective of exploring opportunities for collaboration and supporting open markets under our respective frameworks. We will deepen cooperation on the development, interconnection, access and use of advanced AI and high-performance computing infrastructure, including AI Factories and other large-scale AI compute facilities, with particular attention to access for start-ups, SMEs, researchers and public-sector users. We will also coordinate efforts to advance the safety of minors online.”
“With a view to fostering economic growth, open supply chains and people-to-people ties, Canada and the EU will work towards mutual recognition in three high-priority digital spheres: (i) digital identity, digital credentials and trust services, (ii) conformity assessment results for artificial intelligence systems and (iii) cybersecurity.”
“Particular attention will be given to emerging cyber threats enabled by artificial intelligence, including automated cyberattacks, while promoting the responsible use of artificial intelligence for cyber threat detection, prevention and incident response. We will build joint leadership in multilateral and international fora on cyber issues, including by upholding the UN Framework of responsible State behaviour in cyberspace.”
Reader Feedback
“There were a few good neurons in that atlas.”
Footnotes
It’s AI Safety Week in Toronto.
The Hinton Lectures are always informative, and I enjoy listening to how research is explained simply. There quite a few events at Trajectory Labs and around the city that I’m looking forward to.
I’ve amplified some of the pieces that are a cause for concern. The summary is that we’re discovering new vulnerabilities way faster than we can realistically patch them. The solution forks there.
There’s a humanist camp: let’s keep humans in the loop and design those experiences. Let’s keep them engaged and bring out the best in them.
The other set of solutions, perhaps I’d sort as the machinist view, focuses on increasing the capabilities of the intelligence to make it safer. Let’s not re-learn the bitter lesson.
This week, I’m actively speaking in humanist terms. Let’s keep humans in the loop.
There’s some evidence that would change my mind. In particular, if humans in the loop simply can’t keep up, given no other option, then the machinists have to come in.
Or, then perhaps, it raises the question of what we’re even doing?
Never miss a single issue
Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox