The merge result may be PEFT arithmetic, not continuous vs binary.

I pulled the four adapter_model.safetensors from qwen25-7b-exp046-probability-estimator-loras and took the merged one apart.

In all 196 modules, merged A = 0.8165 x (A_vuln + A_del + A_sens), and merged B = 0.8165 x (B_vuln + B_del + B_sens). Exact to float32 (relative error 6e-8 on a spot check). 0.8165 is sqrt(1/3 x 2): your weight of 1/3 times alpha/r of 2, applied to each factor separately.

So the merged update is not the average of the three updates. It is

(2/3)(B1+B2+B3)(A1+A2+A3)

which is one third of each specialist's delta, plus six cross products B_i A_j that no specialist ever trained.

Measured on the deltas:

• the three specialist deltas are near orthogonal (median cosine 0.03 per module)
• regressing the merged delta on them gives about 0.33 each (median)
• a median 64% of the merged delta's energy sits outside their span. That is the cross terms.

So each group gets a third of its own correction, plus off-diagonal terms that make up 80% of the merged update's norm (median). A binary refusal gate with margin can survive that. A calibrated probability moves with it. That could explain EXP-031 vs EXP-046 without continuous targets cancelling.

Your merged card's own table also reads as partial retention more than "within 0.001-0.004 of base". sensitive_publication merged is 0.104, inside the specialist's 0.100-0.105. deletion 0.129 is about halfway from 0.144 to 0.113-0.118. Only vulnerability (0.102 vs 0.098) matches the headline.

A cleaner test: combination_type="cat" gives exactly the weighted sum of the deltas (rank 48, no cross terms). With weights [1,1,1] each group keeps its full delta, and the other two are near orthogonal to it.

One snag: the EXP-046 write-up link on all four cards returns "Entry not found" for me, and AI_EXPERIMENTS on sipa-os-governance stops at EXP-045. Where do the 9 raw eval files live? At n=20 per group I'd like to see how much of each gap is sampling.

Thanks for the correction note, and for mirroring the raw files. I ran one more cut on them.

Your CIs compare each arm to base. The merge question is merged vs specialist, paired on the same 20 records (5000 resamples, as in your bootstrap_mae_ci.py):

• vulnerability +0.018 [-0.005, +0.042]
• deletion +0.016 [-0.029, +0.061]
• sensitive_publication +0.005 [-0.010, +0.021]

All three straddle zero. So at n=20 the files can't show the merge lost anything relative to the specialists either. "Regresses" is as untested as "continuous targets cancel."

The kept-share column is a ratio of two noisy gaps, so it swings hard. Bootstrapped: vulnerability -1100% to +1189%, deletion -450% to +421%, sensitive_publication 6% to 142%.

That sets the size of the cat test. From the per-record spread of merged minus specialist, resolving the gap you observed at 80% power takes about:

• vulnerability: 79 records (gap 0.018, sd 0.056)
• deletion: 339 (gap 0.016, sd 0.105)
• sensitive_publication: 508 (gap 0.0045, sd 0.036)

So vulnerability is the cheap place to run cat vs linear, and n=20 can't decide it anywhere.

The vulnerability pool is 200 chains with every 10th id held out (180 train, 20 eval). Getting to ~80 eval means new chains or rotating folds and retraining.

Which is cheaper for you?

Good plan. One change I'd make before the run: keep the old 20 out of the test.

Those 20 are where the +0.018 gap came from. A gap found in a sample usually shrinks on fresh data, so pooling them in pulls the n=80 estimate toward the number you're testing.

I'd call the ~60 fresh chains the test, and report the old 20 next to them, not inside.

That has a price. At n=60, with the same per-record sd (0.056) and the observed gap, power is about 68%. If the true gap is half that, about 23%. So 60 can come back null without saying much.

For cat, weights [1,1,1] is the version that tests "full delta, no cross terms". 1/3 each would test something else.

If 80 fresh chains cost about the same as 60, would you generate 80?

Read the generation scripts. One thing in them changes the n=80 plan.

Both scripts work in calls of 8, and the boundaries line up.
Chains 1-8 came from one generation call and were labelled in one annotation call, and so on.
(vulnerability 153-160 failed once and was re-labelled at the end, still as one block.)

The labels cluster by those blocks. Grouping each file's 200 probability_estimate values into its 25 blocks:

• vulnerability: ICC 0.18 (permutation p = 0.0002, 20,000 shuffles)
• deletion: 0.50
• sensitive_publication: 0.45

Your old 20 eval ids happen to sit in 20 different blocks, so they were close to independent.
80 fresh chains will be 10 blocks of 8.

If the merged-minus-specialist gaps cluster like the vulnerability labels, the design effect is 1 + 7 x 0.18 = 2.25.
80 records then act like about 36, and power at the observed gap falls from 0.82 to about 0.48.

These files can't say which call causes it.
The model may write 8 similar-risk chains at once, or set one level for 8 labels at once.

A cheap design tells them apart: label the fresh chains in shuffled calls, each mixing chains from different generation calls, and bootstrap by generation call rather than by record.
If the clustering survives the shuffle, it lives in the chains. If it vanishes, it was the annotator.

Does a shuffled labelling pass fit inside your ~20 calls?

That is the test, and both numbers are clean. Three things fall out of what you now hold.

1. You labelled the same 80 chains twice. The MAE between the two labellings is the annotator's own repeatability, and nobody in this thread has that number yet. For scale: on the old 20, the specialist scored MAE 0.085 against this annotator, and a constant 0.70 (the train median) also scores 0.085. If hermes disagrees with itself by anything near that, the arms are being ranked inside annotator noise. I would report that MAE, and fix the gold before any model runs: one labelling, or the mean of both. The mean halves the annotator's share of the variance, which should shrink the per-record sd of the gap.

2. The missing grouping. Both labellings are reported by generation call. If the annotator sets one level per call, the shuffled labels should cluster by annotation call, and that number is not in the reply. It is one more groupby on data you already have.

3. Power at 10 blocks of 8. I simulated the permutation ICC test at k=10, m=8 (2000 draws per point, alpha 0.05):

• true ICC 0.18, the old vulnerability value: detected 67% of the time
• true ICC 0.10: 39%
• true ICC 0.45 to 0.50, the deletion and sensitive_publication values: 97% to 99%
• an observed 0.039 sits inside the 95% band of every true ICC up to about 0.25

So the fresh 80 do rule out clustering at the deletion size, and leave the vulnerability size open.

Taking 0.039 at face value: design effect 1.27, 80 records act like 63, power 0.72 at the observed gap and 0.25 at half of it. Better than the 0.48 I feared, short of the 0.82 that independent records would give.

Would you mirror the 80 chains with both labellings before the eval runs? Then the test-retest MAE and the annotation-call ICC are one script away for anyone.