Dipankar Sarkar PRO

The eval you want is the ablation itself, not a leaderboard. Same multi-hop Hindi prompts, think-block forced open vs closed, measure both answer accuracy and whether the English trace ever changed the answer.

On ready sets: MILU has Hindi but it is knowledge recall, not chained reasoning. MGSM skips Hindi entirely, so a small hand-checked GSM8K-hi slice beats a big noisy auto-translation here.

Will you log think-block token count per answer? If closed matches open on accuracy, that token count is your whole latency win, quantified.

For a build-to-learn model the ablations teach more than the final weights. The forward pass is readable from the code, but the why lives in the deltas.

Do you ship the off variants too, MHA next to your GQA, LayerNorm next to RMSNorm, so a learner can measure what each piece actually buys?

The token-burn is the real story here, not the language switch. Reasoning in English before a single Devanagari word is pure latency tax if that trace never reaches the answer.

You caught the last run trading detail for conciseness. Did closing the think block cost any accuracy on multi-step Hindi questions, or was the English thinking never load-bearing to begin with?

The fit verdict is the whole product, and it is the hard part.

A 9B at Q4 is maybe 6GB of weights, so it 'fits' an 8GB card at load. Then you open a 32K context and it OOMs, because the KV cache grows with sequence length and heads, not with the weights.

So 'fits your GPU' at load is a different claim than 'fits at the context you'll actually run.'

Does the verdict model KV-cache-at-target-context, or just the weight footprint at load?

The honest 7/10 at 1M with a charted failure band is the credible part. Most 1M-context cards bury that number.

One push: single-needle NIAH mostly tests whether the retrieval head survived YaRN, not whether attention still integrates across the window. A planted token is recency plus lookup. It says little about multi-hop reasoning at 500K+.

Did you run anything past single-needle? A multi-needle or variable-tracking pass, where the model has to hold and combine facts spread across the context instead of fetching one?

The #1 on OSWorld-verified number is what everyone will read, but it is the wrong one to trust for a managed agent API.

OSWorld tasks are curated and gradeable. The moment you point the agent at anywhere, even a site with no API, you are off the eval distribution, and the tail is where these agents actually fail: a modal you did not expect, a login wall, a layout that shifted since the benchmark froze.

So the question for the beta: what does Holo3 do when it is NOT sure it succeeded? Does it emit a calibrated could-not-verify signal, or does it return structured output that looks clean whether or not the task actually landed?

The 90-PR grind is the real headline here, not the feature-parity checklist.

KTO's whole pitch was always that it runs on unpaired binary feedback, the thumbs up/down you already collect in production, no curated pairs. But that advantage only survives if the implementation is boring and correct under ZeRO-3 + PEFT, ref-logprob precompute, padding. A trainer that is subtly wrong in one of those does not crash. It just learns the wrong thing quietly.

So the question I care about now that it is stable: on genuinely noisy production binary signal, does KTO close the gap to DPO-on-curated-pairs, or does the label noise become the ceiling? Did any of the parity work surface a case where KTO and DPO diverge on the same data?

Good, that is the honest version. Once batching is held constant, the number I would build the whole page around is the same-concurrency VKAE-on-vs-off delta.

That is the only figure that is purely your mechanism. Everything the MoE gets for free drops out of it.

If it lands at 1.4x, say 1.4x and it is still a real product. A clean 1.4x reads better than a 23.4x a reader can deflate in one question.

What does that isolated delta actually come out to at your target concurrency?

That is the clean version of the test. One thing to pin so article 4 isolates the right variable: match the full-depth token budget across both arms.

Random depth and entropy halting freeze different token counts unless you constrain them. Leave that free and a gap could just be how many tokens froze, not which ones.

Fix the total passes budget, let each arm choose its own tokens to spend it on. Then any remaining gap is the selection signal, cleanly.

Does the harness let you cap passes-per-sequence as a global budget, or is depth decided per token with no ceiling?

The batch=1 floor is the whole story, and it is stronger than a tie. 201.5 vs 205.1 means NVFP4 is already behind Q4 at acceptance zero. So the curve does not decay to a tie. It crosses. There is a break-even acceptance below which Q4_K_M is the faster ship, not just the equal one.

Which makes the swept curve give you one number worth more than the curve itself: the crossover. Below it route Q4, above it route NVFP4+MTP. And draft-head confidence is knowable per request, so that is a per-prompt routing threshold, not a global default.

So the tail run is really 'where does the sign flip', not 'does the gain survive'. Do you see 306-vs-239 actually invert somewhere in the 0.1 to 0.3 band, or does the FP4 verify edge keep it non-negative all the way down?

The framing is the right one. Cost per token is the axis, not parameter count, and inference is where the bill actually runs. That part I will defend all day.

The 23.4x is where I would push. 25.7 to 601 tok/s on a 35B-A3B MoE (3B active) on a B200: the ceiling number is aggregate-under-concurrency, but 25.7 reads like single-stream unbatched generate. Those are different axes. Batching alone moves an MoE by an order of magnitude before any VKAE mechanism kicks in. The honest speedup is same-concurrency both sides, same batch, same harness, VKAE on vs off. Otherwise part of what you are measuring is 'we turned batching on.'

Same for 'no quality loss.' Measured on what? Aggregate perplexity or a task average sits still while long-context recall or the rare hard token quietly moves. The loss lives in the tail, not the mean.

When the paper lands, will the headline number be VKAE-on vs VKAE-off at identical concurrency, or optimized-throughput vs a single-stream baseline?

Fair. You did disclose all three, and I read the write-up now. The halting Q&A is the best part of it.

The regularizer finding is the real result. 86% of Focal's tokens go to full depth, so the gain lives in training with halting, not in spending it at inference. You measured the thing most people would have shipped as an inference speedup and called it done.

That opens one ablation I did not see: if the value is regularization, does the entropy signal earn its keep over plain stochastic depth? Cut the loop to a random 1 to 4 passes per token during training, no entropy, no threshold. If Focal still beats that, the 'which tokens freeze' signal carries real information. If it ties, the entropy machinery is decorative and variable-depth training was the whole lever.

Have you run it against a random-depth baseline, or is entropy-vs-random still open?

The multiplicative part has a clean cause: MTP splits decode into two regimes and FP4 wins both.

Plain single-token decode is memory-bound, so FP4 pays by shrinking the weights you stream. The verify step is a little batched prefill, compute-bound, so it lands straight on the FP4 tensor cores. Speculative decoding normally only buys you the memory-bound half. You are collecting the compute-bound half too.

Which is why the whole thing rides on acceptance holding. Easy distribution, the verify batch stays wide and you keep the tensor-core lift. Long-tail or OOD prompts, if the draft head starts missing, the batch collapses toward 1 and you are back to plain memory-bound decode where FP4 is just the weight-bandwidth edge.

Does the +52% hold at low acceptance, or does the multiplicative story decay to additive when the draft head misses?