Boundaries

Open-weight genome language model safeguards: Assessing robustness via adversarial fine-tuning

It might be a good idea to have a few more safeguards?

“Genomic language models (gLMs) have demonstrated impressive predictive and generative capabilities, raising concerns that such models may also enable misuse, for instance via the generation of genomes for human-infecting viruses. These concerns have catalyzed calls for risk mitigation measures.”

“The de facto mitigation of choice for these models, though, is known as data exclusion (also known as data filtering). Data exclusion involves the deliberate removal of data from training datasets in order to limit model performance on risky capabilities tied to such data, such as capabilities related to biological weapons development. For safety and security reasons, the developers of the Evo models, a series of gLMs “excluded genomic sequences from viruses that infect eukaryotic hosts (...) to ensure our openly shared model did not disseminate the capability to manipulate and design pathogenic human viruses” [Brixi et al., 2025]”

“We demonstrate that, in the case of open-source models, it is possible to circumvent data exclusion safeguards via fine-tuning with human-infecting virus sequences. If sensitive pathogen data such as this is publicly accessible, it can be used to fine-tune an openly available gLM, thereby rescuing misuse-enabling capabilities. Importantly, the rescued performance did not match that of a narrow, purpose-built tool for functional immune escape prediction tasks. Therefore, while sensitive data exclusion raises the bar for misuse, it is susceptible to circumvention. Other risk mitigation measures for gLMs will be required in addition. To ensure safe and responsible development and deployment of this class of powerful biological AI models, further work is needed to develop a taxonomy of misuseenabling capabilities and a corresponding toolkit of implementable gLM-specific risk evaluation and mitigation measures.”

Black, J. R., Hanke, M. S., Maiwald, A., Hernandez-Boussard, T., Crook, O. M., & Pannu, J. (2025). Open-weight genome language model safeguards: Assessing robustness via adversarial fine-tuning. arXiv preprint arXiv:2511.19299.

https://arxiv.org/abs/2511.19299

Optimal Watermark Localization in Mixed-Source Large Language Model Texts

I see what you did there.

“We formulate watermark localization as a token-level multiple-testing problem based on pivotal statistics, with a latent indicator recording whether watermark dependence survives at each position. Under an asymptotic regime indexed by exponents for signal sparsity, next-token concentration, and effective-vocabulary growth, we derive a sharp boundary for global detection and phase transitions for discovery and classification within the class of coordinate wise pivot-based localization rules. We show that discovery is strictly harder than detection and that consistent classification is impossible across the parameter regime within this class.”

“The Gumbel-max watermark. For concreteness, our theoretical analysis focuses on the Gumbelmax watermark [1], one of the most influential unbiased watermarking schemes. We use it as our main example because both its decoder and pivotal statistic admit explicit distributional forms, making it a clean setting for deriving sharp localization limits.”

“We now present our method SPOT in Algorithm 1. At a high level, the method searches for positions whose pivotal statistics look unusually abnormal under the null law µ0, and declares such positions as discoveries. Under the null case, θt = 0, so that the p-value pt := 1 − F0(Yt) is i.i.d. U(0, 1), where F0(y) := µ0(Y ≤ y) is the CDF of the null distribution µ0. Hence, very small p values, or equivalently very large values of F0(Yt), provide evidence that the corresponding positions may preserve watermark signal. Therefore, the problem reduces to choosing a threshold: positions with F0(Yt) above this threshold are declared as discoveries. The key difficulty is that the optimal threshold is hard to find. We want to output a discovery set Sδ while controlling false discoveries. Here, a false discovery is a selected position whose watermark dependence has been erased, that is, a position with θt = 0 but δt = 1. A threshold that is too low may include too many false discoveries, while a threshold that is too high may remove many true discoveries. We introduce λn as the target level for controlling the fraction of false discoveries among the selected positions.”

“Third, our analysis focuses on token-level watermark localization. For Gumbel-max and other token-level watermarking schemes, the extension is relatively direct whenever valid pivotal statistics can be constructed for individual token positions. Semantic or sentence-level watermarks are different [14, 39, 17]: they may encode watermark information through sentence meanings, paraphrase invariant features, or vector representations of larger text units rather than token-level dependence on pseudorandomness. In such settings, the localization unit may be a sentence, span, or semantic embedding, and the null-versus-signal formulation must be redefined. Extending localization theory to these non-token-level watermarks is an important open direction.”

https://arxiv.org/abs/2608.14906

AI Guardrail Survival under Single-Cycle Agentic Self-Summarization

“…when compaction does not drop a rule outright, it often leaves something that looks like a rule but does not act like one.”

“Long-running agents periodically compact their context, replacing the transcript with a model-generated summary. Recent work shows that dropping a standing safety constraint during compaction drives behavioral violations across many models (Governance Decay; Chen, 2026). We ask a finer question: under a single compaction cycle, how is a safety rule lost, and what does that imply for detection and evaluation? Our central finding is that a presence check is not a
safety check: when compaction does not drop a rule outright, it often leaves something that looks like a rule but does not act like one.”

“A separate distinction runs throughout the prior literature and ours. A line of work shows that agent safety degrades as raw context length grows, with refusal behavior shifting unpredictably (e.g. Hadeliya et al., 2025; Liu et al., 2024); that work holds the full history in the window and attributes degradation to dilution and position. We instead isolate compaction (the act of replacing history with a summary) as the intervention, and we track the survival of a named referent through the rewrite rather than overall refusal rates. Length-degradation and compaction-degradation are distinct failure surfaces: a system short enough to avoid context dilution can still lose a rule the moment it compacts.”

“Finally, these results required a methodological discipline that we report as a contribution in its own right. We scored summaries with an automated LLM judge used only as a first-pass filter, then validated its labels two ways: author adjudication blinded to the judge label, and behavioral replay, which loads a summary into a fresh context and checks whether a surviving-looking rule still refuses the prohibited action. At two points the judge’s labels, taken at face value, would have changed a scientific conclusion; in the most consequential case the judge made an underpowered condition look analyzable by counting content a reader could merely infer as having survived. Author review caught each. We report this as a concrete caution for the growing practice of judge-only safety evaluation.”

Kwartler, T., Aqrawi, A., & Abbasi, A. (2026). AI Guardrail Survival under Single-Cycle Agentic Self-Summarization. arXiv preprint arXiv:2608.11392.

https://arxiv.org/pdf/2608.11392

Stealing Reasoning Traces from Proprietary LLM APIs

I can see why maybe it could be considered stealing.

“Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with each subsequent request. Building on prior research, we identify an architectural vulnerability: these encrypted blocks are fully compatible and interchangeable across different sessions, users, and models within a provider’s ecosystem. We exploit this compatibility to develop a scalable decryption jailbreak. By injecting an encrypted reasoning trace from a given model into a weaker, and less safeguarded model from the same provider, we force it to decode and output the trace verbatim in plaintext, without ever jailbreaking the more capable model directly.”

“Why is this more scalable than directly jailbreaking a model? A direct extraction attack on a more capable model is possible, but the attacker would have to bypass both model-level alignment (the model’s refusal to reveal its internal chain-of-thought) and system-level defenses such as input filters and output substring-matching filters. The availability of a less-capable model with compatible reasoning thus significantly lowers the difficulty of successful extraction. Our experiments with Haiku 4.5, for instance, use a single fixed extraction prompt across all attacks show in Figure 1. By contrast, extraction from a relatively more capable GPT-5.6 Luna required different prompt templates for different reasoning blocks, best-of-n sampling, and ad hoc workarounds for anti-distillation safeguards (e.g., splitting the extraction into chunks under 50 generated tokens).”

Panfilov, A., Schmotz, D., Shumailov, I., Beurer-Kellner, L., Schaeffer, J., Prabhu, A., ... & Andriushchenko, M. (2026). Stealing Reasoning Traces from Proprietary LLM APIs. arXiv preprint arXiv:2608.09867.

https://arxiv.org/abs/2608.09867

How to Steal Reasoning Without Reasoning Traces

You wouldn’t download a car would you?

“Many large language models (LLMs) use reasoning to generate responses but do not
reveal their full reasoning traces (a.k.a. chains of thought), instead outputting only final answers and brief reasoning summaries. To demonstrate that hiding reasoning traces does not prevent users from “stealing” a model’s reasoning capabilities, we introduce Trace Inversion models that, given only the inputs, answers, and (optionally) reasoning summaries exposed by a target model, generate detailed, synthetic reasoning traces. We show that (1) traces synthesized by Trace Inversion have high overlap with the ground-truth reasoning traces (when available), and (2) fine-tuning student models on inverted traces substantially improves their reasoning and enables distillation from proprietary, black-box LLMs.”

“Fine-tuning effectiveness matters more than accurate reconstruction. For closed-source victims, we cannot tell whether our inverted traces accurately match their “true” reasoning traces. It does not matter in practice, however, as long as users can exploit their access to these closed models to improve the reasoning capabilities of their own models.”
”A key implication for defenses is that making exposed reasoning hard to imitate is not sufficient, because an attacker can always ignore it and apply trace inversion to outputs alone. Obfuscated reasoning thus reduces transparency for users yet does not prevent capability stealing.”
"Even stronger trace inversion is possible. We drew queries from OpenThoughts at a modest budget (at most 25K), compared to corpora such as OpenThoughts itself at 114K and OpenThoughts3 at 1.2M [12]. One natural direction is to scale up: larger query budgets, larger inversion models, and a broader mix of reasoning tasks. A complementary direction is to make inversion models more robust by training them on a variety of summary lengths and styles. Finally, trace synthesis can benefit from verification (e.g., self-consistency for reasoning, execution for code, or tool-based checking for math) to filter or refine traces before student fine-tuning.”

Zhang, T., Morris, J. X., & Shmatikov, V. (2026). How to Steal Reasoning Without Reasoning Traces. arXiv preprint arXiv:2603.07267.

https://arxiv.org/abs/2603.07267

A U.S. Strategy to Prevent the Creation of Mirror Life

Let’s just not do it.

“Commit clearly to not developing mirror life, even if other actors are suspected of pursuing it. This commitment reflects the fact that possession of mirror life would not provide reliable protection, deterrence, or resilience against mirror life developed by others and would instead exacerbate global risk.”

“Exclude coercive measures from the strategy at this time. As we discuss in Chapter 6, actions that rely on pressure, disruption, or punishment would erode the trust and cooperation among major scientific powers on which prevention depends. Such measures should be out of scope at this time. If such measures ever become appropriate, they should be pursued cooperatively and transparently, not unilaterally.”

KALRA, N., CRAWFORD, F. W., NICOTRA, M. L., BEHMER, S., DETTMAN, J., EPSTEIN, G. L., ... & LANGELAND, K. (2026). A US Strategy to Prevent the Creation of Mirror Life.

https://www.rand.org/pubs/research_reports/RRA4335-1.html

Reader Feedback

“I’m worried about what happens to those who can’t get access to compute.”

Footnotes

Have you ever noticed how some will put an additional space after a sentence? Why the double space after a period? If you learned how to touch type on a typewriter, you were likely taught to double space after a period. Tap tap on that space bar. It has to do with the way that old typewriters treated periods, which didn’t look quite right compared to others. It’s silly. But it’s a form of lock in. And the convention just carried over to the keyboard and the computer. I unlearned it sometime in the 2000’s.

The reliance on the literature is giving me pause.

Because of the way models were trained to never admit that they didn’t know something, I always check out a claim.

This might end up being as silly as the space bar double tap.

I still read a paper the same way I did in grad school. I take the first read. Then I ask an LLM to summarize it. I read the summary. A lot of the time the summary confirms my understanding. Sometimes, it points out something that surprises. So I go back into the paper to confirm what the LLM said about it. And sometimes it’s just a matter of interpretation, or, I discover a blocking assumption in myself. This convention continues to serve me. It’s a lot more active than merely taking it.

I wonder how long that convention will be locked in?

Never miss a single issue

Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox