A Few Good Neurons
This week: Weight pruning, safety neurons, political neutrality, novelty evaluation, neuron personas, anthropic IPO.
Large Language Models Generate Harmful Content Using a Distinct, Unified Mechanism
“weight pruning throughout this work acts as a causal probe of model internals”
“Aligned models exhibit a greater compression of harm generation weights than unaligned counterparts, indicating that alignment reshapes harmful representations internally—despite the brittleness of safety guardrails at the surface level. This compression explains emergent misalignment: if weights of harmful capabilities are compressed, fine-tuning that engages these weights in one domain can trigger broad misalignment. Consistent with this, pruning harm generation weights in a narrow domain substantially reduces emergent misalignment.”
“…Do LLMs encode harmfulness internally as a coherent concept, or as a collection of surface-level patterns? If harmfulness is compressed into a shared mechanism, this structure could be leveraged to build more robust alignment methods.”
“Here we investigate the internal structure of harmful content generation in LLMs. As illustrated in Figure 1a, we use targeted weight pruning as a causal probe of model internals. By identifying and removing parameters that contribute specifically to harmful generations—while preserving those essential for benign tasks—we test whether the mechanism of harmfulness generation is compressed within a distinct subset of model weights, or dispersed and entangled.”
“We find that harmful content generation depends on a remarkably compact subset of model parameters— approximately 0.0005% of total parameters—which can be surgically removed while leaving general model capabilities largely intact.”
“Crucially, the weights we find are responsible for producing harmful content, not for the underlying knowledge of harmful topics: pruned models retain the ability to detect harmful requests and explain why they pose risks, a necessary condition for safety methods that target harm generation. Additionally, they can partially relearn to generate harmful content through fine-tuning on harmful examples.”
“Rather than serving as a proposed deployment-ready safety intervention, weight pruning throughout this work acts as a causal probe of model internals—and the structural insights it reveals may inform such approaches in future work. Harmfulness in LLMs is not a diffuse property suppressed by alignment training: it is a structured, localized, and causally accessible mechanism. This opens a path toward safety interventions that address the underlying mechanisms of harm rather than relying solely on behavioral guardrails, potentially yielding more robust defenses.”
Orgad, H., Wei, B., Zheng, K., Wattenberg, M., Henderson, P., Goldfarb-Tarrant, S., & Belinkov, Y. (2026, April). Large Language Models Generate Harmful Content Using a Distinct, Unified Mechanism. In Mechanistic Interpretability Workshop at ICML 2026.
https://arxiv.org/pdf/2604.09544v1
Unraveling llm jailbreaks through safety knowledge neurons
The other way to do it
“In this work, we present a novel neuron-level interpretability method that focuses on the role of safety-related knowledge neurons. Unlike existing approaches, our method projects the
model’s internal representation into a more consistent and interpretable vocabulary space. We then show that adjusting the activation of safety-related neurons can effectively control the models behavior with a mean ASR higher than 97%. Building on this insight, we propose SafeTuning, a fine-tuning strategy that reinforces safety-critical neurons to improve model robustness against jailbreaks.”
“We show that isolating safety-critical knowledge neurons is vital for maintaining the functionality of the model, in both modifying activation value and fine-tuning the model.”
Zhao, C., Ke, Y., & Huang, K. (2026, March). Unraveling llm jailbreaks through safety knowledge neurons. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1889-1906).
https://aclanthology.org/2026.eacl-long.83.pdf
Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses
Hope!
“…we propose a new definition of AI political neutrality and design a large-scale user study to test it, releasing a new dataset PARETO with 7,434 participants and 208,152 evaluations of AI responses. Our definition follows a simple principle grounded in political theory: when asked about a controversial issue, an AI model should generate responses that maximize approval across groups with opposing viewpoints, while balancing approval between groups. This definition allows empirical testing of whether an AI response is “neutral” and generalizes to any political context without pre-supposing a single left-right axis of division.”
“Our study reveals that broad approval of AI responses is possible, even on highly contentious political issues. Furthermore, the loss from single-side to balanced responses is relatively small (<10%), suggesting that balanced models may be able to gain wide traction even with partisan preferences. However, our study also reveals room for improvement: current frontier AI models’ default responses either lean liberal (GPT, Gemini, Claude, Llama) or fall well below the Pareto frontier of approval (Grok); models perform significantly worse on charged user prompts; participants still have concerns with default and balanced responses, and average approval scores per side rarely exceed 0.70.”
Stray, J., Yang, D. Z., Luo, S., Takagi, M. N., & Chang, S. (2026). Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses. arXiv preprint arXiv:2605.28911.
https://arxiv.org/abs/2605.28911
Old Ideas, Novel Problems:The Instability of LLM-Based Novelty Evaluation
This may be key capability for RSI
“AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well.”
Sternlicht, N., Shahid, S., Jansen, P., Weld, D. S., Siangliulue, P., & Hope, T. (2026). Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation. arXiv preprint arXiv:2610.02022.
https://arxiv.org/abs/2610.02022
https://github.com/allenai/novelty-eval
Neuron Personas
Cool.
https://neuronpersonas.com/#llama/n-L3-11214-p
Anthropic IPO
Class F. Classy. Classy take.
“The share of our Class F common stock will be entitled to an aggregate number of votes that represents 50.1% of the total voting power of our outstanding shares as of the applicable record date on all matters submitted to a vote of stockholders generally, including the election of the directors other than the Class T directors. Shares of our Class E common stock will be entitled to one vote per share with respect to certain limited matters.”
Reader Feedback
“The Commons Problem was insaaaaaane. There was an entire stock market that nobody bothered manipulating!”
Footnotes
Denys Linkov, seeing that there weren’t a lot of meetups for senior tech people in Toronto, filled that gap. He hosts a fantastic meetup. And it attracts a steady audience from Papers In The Park. That’s a sign of quality.
We get into talking about Nick Cammarata’s Neural Networks Are Already Clean piece (https://already-clean.goodfire.com/). And it’s beautiful work. We truly admire what the good people at Goodfire, as far as we know, are up to. And you should check it out.
Most of the leadership at Denys’ event have been level on the gradual ramp in capabilities. It isn’t a group that’s vulnerable to extreme enthusiasm or hype about anything. The approach to the assessment and deployment of increasingly intelligent systems has been measured all year. For instance, the group didn’t over-rotate on open claw as many did. We hear stories of people having hundreds of thousands of lines of code written, but relatively few stories of what any of it does, or if it mattered.
Mysteries remain though. There’s still demand for great questions. Nick’s work is a good indicator of that.
Never miss a single issue
Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox