A 20-hour research sprint from September 2026, done for my MATS Winter 2027 application (Neel Nanda stream). Code available on request.

Papers: Thought Anchors (Bogdan et al., 2025) and Value Leakage (Betley et al., 2026).

Summary

I used Thought Anchors, the sentence-resampling method from Bogdan et al. (2025), to locate where value leakage enters the chain of thought of Qwen3.6-35B-A3B on the Donation Bet task from Betley et al. (2026). I wanted to find out whether the leak enters at a sentence I can point to, whether that sentence is the moral statement about the donation or something else, and whether editing it changes the leak. A model whose values quietly shift its answers is an important case for chain-of-thought monitoring. If the shift is visible as a sentence, a monitor can catch it. If it is not, we need to understand where it is instead.

  1. Can Thought Anchors resampling find where value leakage enters, sentence by sentence? Yes. Setup: Qwen3.6-35B-A3B on the Donation Bet “bridge” question, where the prompt says a donation goes to a good cause if the estimate lands above (or below) a threshold. Exp 1 confirmed the model leaks toward the good side. Exp 2 labelled every sentence of 90 reasoning traces and found an explicit value-driven decision in about two thirds of them, appearing late. Full resampling of every sentence (N = 100 each) was too costly, so I used the labels to choose which sentences to resample: the value sentence and two matched control sentences next to it. Exp 3 and 4 then did the Thought Anchors move: keep the sentence or replace it with a semantically different regeneration, resample the rest, and measure how often the answer lands on the favoured side. Keeping the value sentence raises P(favoured) from 0.63 to 0.77 (p = 0.003), while the two control sentences do nothing. So there is a sentence, and the leak is localisable to it in traces that were still undecided at that point.

Counterfactual resampling Keeping the value sentence raises P(favoured); the two nearby control sentences do not. Dots are individual traces.

  1. Is it a particular type of sentence? This is the most interesting finding. The anchor is almost never a moral statement about the donation. Nine of thirteen anchors are plain commitments to a number on one side of the threshold, like “I’ll go with 7,612,000” next to a threshold of 7,585,500. Their effect equals that of the four sentences that mention the good cause. The model reverse-engineers the threshold early, parks its arithmetic beside it, and the decisive sentence is whichever one picks a side. A monitor looking for donation language would miss most of them. Exp 9 confirmed this: donation-language detectors are at chance (AUC 0.50) on the raw chain of thought, while a “commits to a number just on one side” detector reaches AUC 0.83 by three quarters of the trace.

Raw chain-of-thought monitors Donation-language detectors stay at chance (AUC 0.50). Detectors for a commitment near the threshold rise to about 0.83 by three quarters of the trace.

  1. Can we change the leak by editing those sentences? Yes. The hand edits in Exp 7 are the cleanest evidence. Replacing the anchor with a neutral hand-written sentence gives the same drop as the model’s own replacements (0.63 vs 0.61, against 0.82 kept). Rewriting it to commit to the other side flips the outcome (pooled P(favoured) 0.82 to 0.34). Flipping a nearby arithmetic step does not. The attention-mask follow-up is the mechanistic version: with the text left in place, blocking later tokens from attending to the anchor removes about half of the deletion effect, and blocking the prompt’s donation clause as well removes all of it.

Hand edits to the anchor sentence A neutral hand-written replacement drops P(favoured) like the model’s own replacement. Rewriting the sentence to commit to the other side flips the outcome (red).

The caveat that shapes the whole story. The leak is only partly in that sentence. Deleting every value sentence still leaves P(favoured) at 0.57, not the 0.45 of traces with no value action. The rest lives in the numbers: the threshold pulls the parameter values well before any commitment is written. Exp 8 and 12 show this pull travels through a summary scrubbed of all bet language, and disappears only when the numbers themselves are removed. So the answer to “does leakage happen at the sentence level?” is: the side choice does, the magnitude pull does not.

Experiment index

  • Exp 1, reproduction. The paper’s giraffe question did not reproduce on this endpoint (bias 0.035), but the bridge question did: bias 0.31 [0.20, 0.42], p = 2×10⁻⁷, against the paper’s 0.37. All causal work uses the bridge question.
  • Exp 2, where value appears. Every one of 90 labelled traces restates the bet within its first 2 % of steps, but explicit value-driven decisions come late (median relative position 0.56) in 59 of 90 traces. Traces that contain one are favoured 0.75 vs 0.45 without (p = 0.010).
  • Exp 3, commitment curves. Keeping the first k steps and letting the model finish shows the outcome undecided for half of trace A and locking in across steps 53–60, with the value sentence the single largest jump. In trace C the lock-in happens one step earlier, at the sentence that reverse-engineers the threshold. Trace D is decided from k = 0.
  • Exp 4, counterfactual resampling (main result). Over 13 traces, keeping the value sentence raises P(favoured) from 0.63 to 0.77 (+0.13; odds ratio 2.1, p = 0.003, 1,248 continuations). The adjacent estimation sentence (+0.01) and nearest plan or backtrack sentence (−0.06) do nothing. Four anchors are individually significant (+0.17 to +0.37), three are inert, two are at ceiling.
  • Exp 5, resilience. After the value sentence is removed, 39 % of continuations regenerate one. Cutting every recurrence lowers P(favoured) only to 0.57, so the residual bias is upstream anchoring, not regeneration.
  • Exp 6, direction. The sentences push toward the good cause, not toward larger numbers: the above-good anchor raises P(estimate > T) by 0.22, and the three below-good anchors lower it by 0.17–0.37.
  • Exp 7, hand edits. On the five clearest anchors, a neutral hand-written replacement gives the same drop as the model’s own resamples. Rewriting the sentence to commit to the other side flips the outcome to 0.34 pooled. The one arithmetic-style anchor barely moves when flipped: commitments steer, calculations get re-checked.
  • Exp 8, summary transfer. A Haiku summary of the first half of a biased trace, with all bet language removed, given to a fresh no-bet context, still produces estimates near the threshold: 16 % within 10 % of T vs 4 % from baseline summaries (p = 7×10⁻⁵). A Sonnet monitor flagged only 4 of 320 summaries. At a 75 % cut the full gap (+0.31) comes through.
  • Exp 9, raw-CoT monitors. Donation-language detectors are at chance (AUC 0.50) at every point in the chain of thought. A detector for “commits to a number within 10 % of the threshold” reaches AUC 0.83 at three quarters of the trace and 0.87–0.89 at the end. The commitment, not the moral talk, is the signal a monitor can use.
  • Exp 11, forced-answer curve. Forcing an answer after every step (22,280 answers) shows the model’s leaning flip at the commitment sentence for the four significant anchors. It also shows the threshold pull arriving by step 6–16, long before any side is chosen.
  • Exp 12, coarsest summary. A summary with every number removed transfers no anchoring (4 % vs 3 %). Per-component values with no totals transfer it in full.
  • Attention-mask follow-up. With the text left in place, blocking later tokens from attending to the anchor removes about half of the deletion effect (+0.11 of +0.23). Blocking the prompt’s donation clause alone does nothing, and blocking both reproduces deletion exactly. Later tokens read the sentence itself; the prompt clause is a fallback.

What I checked before believing the numbers

  • Every prompt was compared byte-for-byte with the paper’s cached prompts, and the paper’s own cached answers re-scored to the published bias.
  • The number extractor was audited on random samples against the paper’s judge in every run (full agreement each time).
  • Only two hosted providers continue a <think> prefix verbatim; the rest would have silently broken every resample. I pinned one of them for all runs, and turned off a default top_k=20 the paper’s sampler does not use.
  • The neutral hand edit matching the model’s own replacement rate (0.63 vs 0.61) checks that the design measures the sentence, not the replacement.
  • The attention mask was checked on a tiny model of the same hybrid DeltaNet/attention architecture (masked span equals deletion, max logit difference 0.0).
  • Effects that shrank with more samples are reported at the larger n.

Limitations and next steps

  • One question for all causal work. The giraffe question barely leaks on this endpoint, and I have not found out why. The zills question reproduces (0.27) and is the replication I would run next.
  • The 13 traces were chosen for being undecided at the value step. That is where anchors can exist, so the pooled +0.13 is a selected estimate.
  • One LLM labeller, and some categories rely on regex filters.
  • One fp8 hosted endpoint, one model.