GDN-Bit (BabyLM 2026, 30M)

GDN-Bit is a Gated DeltaNet whose linear layers are all ternary BitLinear: absmean ternary weights and int8 per-token activations. It can loop its whole layer stack T times.

This repository holds 30M-parameter models trained and evaluated on BabyLM 2026 Strict-Small:

  • Main models (AdamW weight decay 0.3):
    • GDN-Bit
    • full-precision GDN with the same architecture
    • HGRN-Bit (the MatMul-free LM mixer)
  • Key-dimension ablation of GDN-Bit (weight decay 0.1): dk = 16, 32 and its own dk = 64 reference, in kdim-ablation-wd0.1/.

Each comes without (T1) and with (T4) the loop, and with 3 seeds.

Layout

  • main, main models: <model>/babylm2026-strict-small-s<seed>/ holds the final model (100M words, the official evaluation checkpoint), the tokenizer, eval/ and, for GDN-Bit models, ternary/.
  • main, ablation: kdim-ablation-wd0.1/gdnbit-30m-k{16,32,64}-t{1,4}/babylm2026-strict-small-s<seed>/, the same contents.
    • These models were trained with weight decay 0.1, so compare them only with each other.
    • gdnbit-30m-k64-* is GDN-Bit itself at weight decay 0.1, the ablation's reference.
  • Branches chck_1M … chck_9M, chck_10M … chck_100M (the official naming): the same folders hold the model at that many training words. AoA and the fast evaluation use these checkpoints.
  • Seeds: 1337, 42 and 2026, each in its own babylm2026-strict-small-s<seed>/ folder.

Results: main models (weight decay 0.3)

Mean ± std over seeds 1337, 42, 2026; final checkpoint; official BabyLM 2026 pipeline.

Folder Model Params Dev NLL BLiMP Supp. EWoK Entity COMPS GPIQA GLUE NLP Avg Reading AoA Recall gain L256
gdnbit-30m-t1 GDN-Bit (ternary Gated DeltaNet), no loop 30,470,472 3.479 ± 0.056 69.62 ± 0.30 57.08 ± 1.96 51.14 ± 0.32 20.52 ± 0.85 52.51 ± 0.28 35.51 ± 3.35 65.00 ± 1.20 50.20 ± 0.92 4.73 ± 0.79 -0.17 ± 0.01 0.216 ± 0.015
gdnbit-30m-t4 GDN-Bit, whole-stack loop ×4 30,472,520 3.441 ± 0.024 70.05 ± 0.41 59.41 ± 2.24 51.29 ± 0.72 19.73 ± 0.36 52.49 ± 0.24 33.87 ± 0.59 65.21 ± 0.48 50.29 ± 0.41 5.61 ± 0.21 -0.20 ± 0.05 0.226 ± 0.010
gdn-30m-t1 Gated DeltaNet, full precision 30,453,064 3.386 ± 0.022 70.60 ± 0.88 59.37 ± 0.87 52.72 ± 0.68 20.87 ± 1.47 52.61 ± 0.11 32.85 ± 1.01 63.76 ± 0.72 50.40 ± 0.32 5.53 ± 0.57 0.00 ± 0.00 0.305 ± 0.010
gdn-30m-t4 Gated DeltaNet, full precision, loop ×4 30,455,112 3.332 ± 0.018 72.10 ± 0.30 60.23 ± 2.38 52.72 ± 0.53 23.53 ± 0.13 53.15 ± 0.18 33.29 ± 0.57 64.55 ± 0.70 51.37 ± 0.34 5.33 ± 0.14 -0.05 ± 0.08 0.434 ± 0.027
hgrnbit-30m-t1 HGRN-Bit (MatMul-free LM mixer) 30,566,304 3.512 ± 0.020 69.96 ± 0.85 58.61 ± 2.13 51.51 ± 0.21 19.31 ± 0.16 52.72 ± 0.13 34.85 ± 0.79 64.87 ± 0.60 50.26 ± 0.49 5.00 ± 0.51 -0.17 ± 0.02 0.060 ± 0.004
hgrnbit-30m-t4 HGRN-Bit, loop ×4 30,568,352 3.511 ± 0.009 70.27 ± 0.26 59.33 ± 0.88 51.85 ± 0.02 19.38 ± 0.52 52.71 ± 0.23 32.25 ± 1.78 64.74 ± 0.22 50.08 ± 0.46 5.18 ± 1.02 -0.06 ± 0.10 0.083 ± 0.002
  • Metrics:
    • Dev NLL is per token on the BabyLM 2026 Strict-Small dev split, with the 2026 tokenizer.
    • Accuracy in % (GLUE uses F1 for MRPC and QQP).
    • GPIQA is the mean of the GlobalPIQA parallel and nonparallel sets.
    • Reading is the mean of the eye-tracking and self-paced Δ%R².
    • NLP Avg is the mean of the 7 NLP columns (BLiMP … GLUE), the leaderboard aggregate.
  • AoA is the official code's model–child age-of-acquisition correlation over the 19 checkpoints. The official scorer reports 0.00 when that correlation is not significant (p > 0.1).
  • Recall gain L256 is a copy-recall probe:
    • A 64-token dev passage P is followed by L = 256 other dev tokens F, then P again.
    • The gain is the NLL of the second P without the first copy (BOS + F + P) minus its NLL with it (BOS + P + F + P).
    • It measures how much of P the recurrent state still holds after L tokens. eval/babylm2026_all_runs.csv also has L = 16, 32, 64 and 128.
  • GLUE uses the official fine-tuning settings. For the looped (T4) models, the batch-32 tasks run as 2 × 16 micro-batches with the pipeline's own --gradient_accumulation, to fit 24 GB GPUs. The effective batch is unchanged.
  • Per model, eval/babylm2026.csv has every task, including the 7 GLUE tasks separately, GlobalPIQA parallel/nonparallel and the two Reading measures.
  • Across models: eval/babylm2026_all_runs.csv has every model × seed (both groups, with their weight decay), and eval/babylm2026_summary.csv has the mean, std and n per model and metric.
  • Check: with the same pipeline and wrapper, the official GPT-2 Strict-Small baseline reproduces its published scores (eval/gpt2_2026_baseline_reproduction.csv).

Results: key-dimension ablation (weight decay 0.1, kdim-ablation-wd0.1/)

The models differ only in dk, so the state is 16, 32 or 64 × 128 per head. They were evaluated on the same suite without GLUE fine-tuning (GLUE was run only for the dk = 64 reference), so GLUE and NLP Avg are given for that reference alone.

Folder Model Params Dev NLL BLiMP Supp. EWoK Entity COMPS GPIQA GLUE NLP Avg Reading AoA Recall gain L256
gdnbit-30m-k16-t1 GDN-Bit, dk 16 28,701,000 3.622 ± 0.039 69.11 ± 0.67 58.63 ± 1.11 50.94 ± 0.35 20.21 ± 0.36 52.81 ± 0.07 31.88 ± 2.75 – – 5.06 ± 0.21 -0.16 ± 0.05 0.137 ± 0.014
gdnbit-30m-k16-t4 GDN-Bit, dk 16, loop ×4 28,703,048 3.568 ± 0.022 69.47 ± 0.23 57.78 ± 0.78 50.87 ± 0.34 19.80 ± 0.37 52.39 ± 0.41 32.20 ± 2.16 – – 5.51 ± 0.21 -0.15 ± 0.01 0.143 ± 0.005
gdnbit-30m-k32-t1 GDN-Bit, dk 32 29,290,824 3.590 ± 0.003 69.16 ± 1.38 58.96 ± 2.48 51.55 ± 0.33 19.63 ± 0.64 52.59 ± 0.34 32.53 ± 1.51 – – 4.36 ± 0.36 -0.11 ± 0.10 0.164 ± 0.005
gdnbit-30m-k32-t4 GDN-Bit, dk 32, loop ×4 29,292,872 3.559 ± 0.020 69.83 ± 0.41 60.49 ± 1.86 51.09 ± 0.62 19.95 ± 0.53 52.26 ± 0.17 31.41 ± 3.31 – – 6.41 ± 0.56 -0.10 ± 0.09 0.174 ± 0.005
gdnbit-30m-k64-t1 GDN-Bit, dk 64 (reference) 30,470,472 3.577 ± 0.004 68.82 ± 0.23 58.51 ± 0.67 51.63 ± 0.93 19.84 ± 0.76 52.38 ± 0.27 36.78 ± 3.12 64.75 ± 0.30 50.39 ± 0.62 4.74 ± 0.42 -0.17 ± 0.03 0.196 ± 0.021
gdnbit-30m-k64-t4 GDN-Bit, dk 64, loop ×4 (reference) 30,472,520 3.523 ± 0.015 69.94 ± 0.40 58.65 ± 3.01 51.86 ± 0.49 19.64 ± 0.38 52.39 ± 0.52 33.08 ± 0.75 64.31 ± 0.39 49.98 ± 0.30 4.88 ± 0.98 -0.16 ± 0.14 0.202 ± 0.013

Recurrent-state precision (eval/state_quant.csv)

The cost of storing the state at lower precision is measured by emulating SRAM state storage: the recurrence runs token by token, the state is quantized after every token, and the dev NLL is compared with the fp32 state.

Change in dev NLL; main models, seed 1337, 2.1M dev tokens:

Model int8 per row int8 per tensor int4 per row
GDN-Bit T1 / T4 −0.0016 / −0.0002 +0.0119 / +0.0042 +0.0736 / +0.0282
GDN T1 / T4 +0.0015 / −0.0020 +0.0233 / +0.0116 +0.0911 / +0.0562
HGRN-Bit T1 / T4 +0.0004 / +0.0048 +0.0004 / +0.0048 +0.1978 / +0.2140

Training

  • Data and tokenizer: BabyLM 2026 Strict-Small (10M words), with the official 2026 baseline tokenizer (16,384 tokens, included in every folder).
  • Budget: 10 epochs = 100M words, 10,100 steps of 32 × 512 tokens.
  • Optimizer: AdamW (β 0.9/0.95, gradient clip 1.0), cosine schedule to 0 after 500 warmup steps, bf16 autocast.
  • Weight decay: 0.3 for the main models, 0.1 for the key-dimension ablation.
  • Peak LR: 2e-3 for the ternary models, 1e-2 for full-precision GDN.
  • Architecture (GDN, GDN-Bit):
    • d = 512, 6 layers, 6 heads, dk = 64, dv = 128, ReLU² FFN 2048, no short convolution.
    • One fused input projection for q/k/v/g/a/b; tied embedding and head.
    • The ablation changes only dk.
  • HGRN-Bit: 7 layers, SwiGLU 1376.
  • Loop (T4): h ← Stack(h) + r_t ⊙ h_in, with a learned fp32 r_t. Every pass starts its recurrent state from zero.
  • Training log: eval/train.log has the per-epoch dev NLL and the final dev/test NLL, and eval/train_args.json has the exact arguments.

Loading

The model classes come from the project code: the custom_models package, gdnbit branch, not public yet. They need flash-linear-attention==0.5.1 and transformers==4.57.3.

import custom_models  # registers gdnbit (GDN, GDN-Bit) and mmfloop (HGRN-Bit)
from transformers import AutoModelForCausalLM, AutoTokenizer

repo, run = "ruhai-lin/GDNBit", "gdnbit-30m-t4/babylm2026-strict-small-s1337"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run)              # final model
early = AutoModelForCausalLM.from_pretrained(repo, subfolder=run, revision="chck_10M")

T4 models: the HF recurrent cache is keyed by layer only, so the 4 passes would overwrite each other's state. Score full sequences without a cache, or keep one state per (pass, layer) yourself; do not use generate() with the cache.

Ternary deployment weights (ternary/, GDN-Bit models)

Use these levels for hardware; do not re-quantize the latent fp32 weights yourself. Many latent weights sit exactly on the rounding threshold, so a different floating-point order flips their level.

  • What ternary.safetensors holds: for every BitLinear <module>, <module>.levels (int8 in {-1, 0, 1}, shape [out, in]) and <module>.scale (fp32). These are computed on CUDA exactly as the training kernel does, with W = levels × scale.
  • Forward: y = Q8(RMSNorm(x; eps 1e-6) · norm.weight) @ Wᵀ, where Q8 is per-token absmax int8.
  • Fused attention input projection (2316 rows for dk = 64): q 384 | k 384 | v 768 | g 768 | a 6 | b 6, with one scale for the whole matrix.
    • Decay = exp(−exp(A_log) · softplus(a + dt_bias)), β = sigmoid(b), q and k L2-normalized.
  • The tied head is quantized from the float embedding.
  • Check: verify.json records the full-dev NLL of the model rebuilt from these files minus the native model's. Across all GDN-Bit folders, |Δ| ≤ 1.1e-05.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support