Instructions to use ruhai-lin/GDNBit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ruhai-lin/GDNBit with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ruhai-lin/GDNBit", device_map="auto") - Notebooks
- Google Colab
- Kaggle
GDN-Bit (BabyLM 2026, 30M)
GDN-Bit is a Gated DeltaNet whose linear layers are all ternary BitLinear: absmean ternary weights and int8 per-token activations. It can loop its whole layer stack T times.
This repository holds 30M-parameter models trained and evaluated on BabyLM 2026 Strict-Small:
- Main models (AdamW weight decay 0.3):
- GDN-Bit
- full-precision GDN with the same architecture
- HGRN-Bit (the MatMul-free LM mixer)
- Key-dimension ablation of GDN-Bit (weight decay 0.1): dk = 16, 32 and its own dk = 64 reference, in
kdim-ablation-wd0.1/.
Each comes without (T1) and with (T4) the loop, and with 3 seeds.
Layout
main, main models:<model>/babylm2026-strict-small-s<seed>/holds the final model (100M words, the official evaluation checkpoint), the tokenizer,eval/and, for GDN-Bit models,ternary/.main, ablation:kdim-ablation-wd0.1/gdnbit-30m-k{16,32,64}-t{1,4}/babylm2026-strict-small-s<seed>/, the same contents.- These models were trained with weight decay 0.1, so compare them only with each other.
gdnbit-30m-k64-*is GDN-Bit itself at weight decay 0.1, the ablation's reference.
- Branches
chck_1M…chck_9M,chck_10M…chck_100M(the official naming): the same folders hold the model at that many training words. AoA and the fast evaluation use these checkpoints. - Seeds: 1337, 42 and 2026, each in its own
babylm2026-strict-small-s<seed>/folder.
Results: main models (weight decay 0.3)
Mean ± std over seeds 1337, 42, 2026; final checkpoint; official BabyLM 2026 pipeline.
| Folder | Model | Params | Dev NLL | BLiMP | Supp. | EWoK | Entity | COMPS | GPIQA | GLUE | NLP Avg | Reading | AoA | Recall gain L256 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
gdnbit-30m-t1 |
GDN-Bit (ternary Gated DeltaNet), no loop | 30,470,472 | 3.479 ± 0.056 | 69.62 ± 0.30 | 57.08 ± 1.96 | 51.14 ± 0.32 | 20.52 ± 0.85 | 52.51 ± 0.28 | 35.51 ± 3.35 | 65.00 ± 1.20 | 50.20 ± 0.92 | 4.73 ± 0.79 | -0.17 ± 0.01 | 0.216 ± 0.015 |
gdnbit-30m-t4 |
GDN-Bit, whole-stack loop ×4 | 30,472,520 | 3.441 ± 0.024 | 70.05 ± 0.41 | 59.41 ± 2.24 | 51.29 ± 0.72 | 19.73 ± 0.36 | 52.49 ± 0.24 | 33.87 ± 0.59 | 65.21 ± 0.48 | 50.29 ± 0.41 | 5.61 ± 0.21 | -0.20 ± 0.05 | 0.226 ± 0.010 |
gdn-30m-t1 |
Gated DeltaNet, full precision | 30,453,064 | 3.386 ± 0.022 | 70.60 ± 0.88 | 59.37 ± 0.87 | 52.72 ± 0.68 | 20.87 ± 1.47 | 52.61 ± 0.11 | 32.85 ± 1.01 | 63.76 ± 0.72 | 50.40 ± 0.32 | 5.53 ± 0.57 | 0.00 ± 0.00 | 0.305 ± 0.010 |
gdn-30m-t4 |
Gated DeltaNet, full precision, loop ×4 | 30,455,112 | 3.332 ± 0.018 | 72.10 ± 0.30 | 60.23 ± 2.38 | 52.72 ± 0.53 | 23.53 ± 0.13 | 53.15 ± 0.18 | 33.29 ± 0.57 | 64.55 ± 0.70 | 51.37 ± 0.34 | 5.33 ± 0.14 | -0.05 ± 0.08 | 0.434 ± 0.027 |
hgrnbit-30m-t1 |
HGRN-Bit (MatMul-free LM mixer) | 30,566,304 | 3.512 ± 0.020 | 69.96 ± 0.85 | 58.61 ± 2.13 | 51.51 ± 0.21 | 19.31 ± 0.16 | 52.72 ± 0.13 | 34.85 ± 0.79 | 64.87 ± 0.60 | 50.26 ± 0.49 | 5.00 ± 0.51 | -0.17 ± 0.02 | 0.060 ± 0.004 |
hgrnbit-30m-t4 |
HGRN-Bit, loop ×4 | 30,568,352 | 3.511 ± 0.009 | 70.27 ± 0.26 | 59.33 ± 0.88 | 51.85 ± 0.02 | 19.38 ± 0.52 | 52.71 ± 0.23 | 32.25 ± 1.78 | 64.74 ± 0.22 | 50.08 ± 0.46 | 5.18 ± 1.02 | -0.06 ± 0.10 | 0.083 ± 0.002 |
- Metrics:
- Dev NLL is per token on the BabyLM 2026 Strict-Small dev split, with the 2026 tokenizer.
- Accuracy in % (GLUE uses F1 for MRPC and QQP).
- GPIQA is the mean of the GlobalPIQA parallel and nonparallel sets.
- Reading is the mean of the eye-tracking and self-paced Δ%R².
- NLP Avg is the mean of the 7 NLP columns (BLiMP … GLUE), the leaderboard aggregate.
- AoA is the official code's model–child age-of-acquisition correlation over the 19 checkpoints. The official scorer reports 0.00 when that correlation is not significant (p > 0.1).
- Recall gain L256 is a copy-recall probe:
- A 64-token dev passage P is followed by L = 256 other dev tokens F, then P again.
- The gain is the NLL of the second P without the first copy (BOS + F + P) minus its NLL with it (BOS + P + F + P).
- It measures how much of P the recurrent state still holds after L tokens.
eval/babylm2026_all_runs.csvalso has L = 16, 32, 64 and 128.
- GLUE uses the official fine-tuning settings. For the looped (T4) models, the batch-32 tasks run as 2 × 16 micro-batches with the pipeline's own
--gradient_accumulation, to fit 24 GB GPUs. The effective batch is unchanged. - Per model,
eval/babylm2026.csvhas every task, including the 7 GLUE tasks separately, GlobalPIQA parallel/nonparallel and the two Reading measures. - Across models:
eval/babylm2026_all_runs.csvhas every model × seed (both groups, with their weight decay), andeval/babylm2026_summary.csvhas the mean, std and n per model and metric. - Check: with the same pipeline and wrapper, the official GPT-2 Strict-Small baseline reproduces its published scores (
eval/gpt2_2026_baseline_reproduction.csv).
Results: key-dimension ablation (weight decay 0.1, kdim-ablation-wd0.1/)
The models differ only in dk, so the state is 16, 32 or 64 × 128 per head. They were evaluated on the same suite without GLUE fine-tuning (GLUE was run only for the dk = 64 reference), so GLUE and NLP Avg are given for that reference alone.
| Folder | Model | Params | Dev NLL | BLiMP | Supp. | EWoK | Entity | COMPS | GPIQA | GLUE | NLP Avg | Reading | AoA | Recall gain L256 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
gdnbit-30m-k16-t1 |
GDN-Bit, dk 16 | 28,701,000 | 3.622 ± 0.039 | 69.11 ± 0.67 | 58.63 ± 1.11 | 50.94 ± 0.35 | 20.21 ± 0.36 | 52.81 ± 0.07 | 31.88 ± 2.75 | – | – | 5.06 ± 0.21 | -0.16 ± 0.05 | 0.137 ± 0.014 |
gdnbit-30m-k16-t4 |
GDN-Bit, dk 16, loop ×4 | 28,703,048 | 3.568 ± 0.022 | 69.47 ± 0.23 | 57.78 ± 0.78 | 50.87 ± 0.34 | 19.80 ± 0.37 | 52.39 ± 0.41 | 32.20 ± 2.16 | – | – | 5.51 ± 0.21 | -0.15 ± 0.01 | 0.143 ± 0.005 |
gdnbit-30m-k32-t1 |
GDN-Bit, dk 32 | 29,290,824 | 3.590 ± 0.003 | 69.16 ± 1.38 | 58.96 ± 2.48 | 51.55 ± 0.33 | 19.63 ± 0.64 | 52.59 ± 0.34 | 32.53 ± 1.51 | – | – | 4.36 ± 0.36 | -0.11 ± 0.10 | 0.164 ± 0.005 |
gdnbit-30m-k32-t4 |
GDN-Bit, dk 32, loop ×4 | 29,292,872 | 3.559 ± 0.020 | 69.83 ± 0.41 | 60.49 ± 1.86 | 51.09 ± 0.62 | 19.95 ± 0.53 | 52.26 ± 0.17 | 31.41 ± 3.31 | – | – | 6.41 ± 0.56 | -0.10 ± 0.09 | 0.174 ± 0.005 |
gdnbit-30m-k64-t1 |
GDN-Bit, dk 64 (reference) | 30,470,472 | 3.577 ± 0.004 | 68.82 ± 0.23 | 58.51 ± 0.67 | 51.63 ± 0.93 | 19.84 ± 0.76 | 52.38 ± 0.27 | 36.78 ± 3.12 | 64.75 ± 0.30 | 50.39 ± 0.62 | 4.74 ± 0.42 | -0.17 ± 0.03 | 0.196 ± 0.021 |
gdnbit-30m-k64-t4 |
GDN-Bit, dk 64, loop ×4 (reference) | 30,472,520 | 3.523 ± 0.015 | 69.94 ± 0.40 | 58.65 ± 3.01 | 51.86 ± 0.49 | 19.64 ± 0.38 | 52.39 ± 0.52 | 33.08 ± 0.75 | 64.31 ± 0.39 | 49.98 ± 0.30 | 4.88 ± 0.98 | -0.16 ± 0.14 | 0.202 ± 0.013 |
Recurrent-state precision (eval/state_quant.csv)
The cost of storing the state at lower precision is measured by emulating SRAM state storage: the recurrence runs token by token, the state is quantized after every token, and the dev NLL is compared with the fp32 state.
Change in dev NLL; main models, seed 1337, 2.1M dev tokens:
| Model | int8 per row | int8 per tensor | int4 per row |
|---|---|---|---|
| GDN-Bit T1 / T4 | −0.0016 / −0.0002 | +0.0119 / +0.0042 | +0.0736 / +0.0282 |
| GDN T1 / T4 | +0.0015 / −0.0020 | +0.0233 / +0.0116 | +0.0911 / +0.0562 |
| HGRN-Bit T1 / T4 | +0.0004 / +0.0048 | +0.0004 / +0.0048 | +0.1978 / +0.2140 |
Training
- Data and tokenizer: BabyLM 2026 Strict-Small (10M words), with the official 2026 baseline tokenizer (16,384 tokens, included in every folder).
- Budget: 10 epochs = 100M words, 10,100 steps of 32 × 512 tokens.
- Optimizer: AdamW (β 0.9/0.95, gradient clip 1.0), cosine schedule to 0 after 500 warmup steps, bf16 autocast.
- Weight decay: 0.3 for the main models, 0.1 for the key-dimension ablation.
- Peak LR: 2e-3 for the ternary models, 1e-2 for full-precision GDN.
- Architecture (GDN, GDN-Bit):
- d = 512, 6 layers, 6 heads, dk = 64, dv = 128, ReLU² FFN 2048, no short convolution.
- One fused input projection for q/k/v/g/a/b; tied embedding and head.
- The ablation changes only dk.
- HGRN-Bit: 7 layers, SwiGLU 1376.
- Loop (T4): h ← Stack(h) + r_t ⊙ h_in, with a learned fp32 r_t. Every pass starts its recurrent state from zero.
- Training log:
eval/train.loghas the per-epoch dev NLL and the final dev/test NLL, andeval/train_args.jsonhas the exact arguments.
Loading
The model classes come from the project code: the custom_models package, gdnbit branch, not public yet. They need flash-linear-attention==0.5.1 and transformers==4.57.3.
import custom_models # registers gdnbit (GDN, GDN-Bit) and mmfloop (HGRN-Bit)
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, run = "ruhai-lin/GDNBit", "gdnbit-30m-t4/babylm2026-strict-small-s1337"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run) # final model
early = AutoModelForCausalLM.from_pretrained(repo, subfolder=run, revision="chck_10M")
T4 models: the HF recurrent cache is keyed by layer only, so the 4 passes would overwrite each other's state. Score full sequences without a cache, or keep one state per (pass, layer) yourself; do not use generate() with the cache.
Ternary deployment weights (ternary/, GDN-Bit models)
Use these levels for hardware; do not re-quantize the latent fp32 weights yourself. Many latent weights sit exactly on the rounding threshold, so a different floating-point order flips their level.
- What
ternary.safetensorsholds: for every BitLinear<module>,<module>.levels(int8 in {-1, 0, 1}, shape [out, in]) and<module>.scale(fp32). These are computed on CUDA exactly as the training kernel does, with W = levels × scale. - Forward: y = Q8(RMSNorm(x; eps 1e-6) · norm.weight) @ Wᵀ, where Q8 is per-token absmax int8.
- Fused attention input projection (2316 rows for dk = 64): q 384 | k 384 | v 768 | g 768 | a 6 | b 6, with one scale for the whole matrix.
- Decay = exp(−exp(A_log) · softplus(a + dt_bias)), β = sigmoid(b), q and k L2-normalized.
- The tied head is quantized from the float embedding.
- Check:
verify.jsonrecords the full-dev NLL of the model rebuilt from these files minus the native model's. Across all GDN-Bit folders, |Δ| ≤ 1.1e-05.