KnowLine-4B-Gen4

The fourth release of PelaAI's 4B System One decision model.

Inference guide Β· δΈ­ζ–‡θ―΄ζ˜Ž Β· weights Apache-2.0 Β· previous version: KnowLine-4B-Gen3

Highlights:

  • Decision Index 0.3 public suite: 64.90 (self-run), 1.79 points above Gen3 (63.11). For comparison, on the board Perplexity Decider v1.1 (27B) scores 62.25 and Jev 1.13 scores 57.96, and the best model of 5B or smaller has a public score of 50.82; see Comparison.
  • Calibrated probabilities: a temperature, fitted on held-out data, is folded into the weights. Computed with the board's method, ECE is about 0.04-0.05 (board median: 0.084) and the Brier score is about 0.29-0.31, roughly 2nd-3rd among the 114 models on the board; see Calibration.
  • Other public benchmarks: JevBench / JevBench-hard 86.1 / 72.1 (Gen3: 86.6 / 72.1), C-Eval 78.7 (79.2).

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

  • one forward pass per question, with no text generation;
  • the output is the probability of each option's label token;
  • existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model name, because the interface and serving settings are unchanged.
Base model Qwen/Qwen3.5-4B (Apache-2.0)
Training LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights
Release format bf16 weights, with a readout temperature T = 1.32 folded into the final norm weight (see Calibration). The vision tower and MTP head are unchanged from the base model; the config, tokenizer and chat template are identical to those of Gen1-Gen3.
Languages English, Simplified Chinese, Traditional Chinese

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-4B-Gen4 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
  • Front end: knowline_server.py is a single file that depends only on transformers and requests. It is the same file as in Gen3.
  • Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

Comparison

Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.

model size DI 0.3 public DI 0.3 full source
KnowLine-4B-Gen4 4B 64.90 not yet scored self-run, official kit
KnowLine-4B-Gen3 4B 63.11 not yet scored self-run, official kit
KnowLine-4B-Gen2 4B 62.54 not yet scored self-run, official kit
KnowLine-4B-Gen1 4B 60.47 not yet scored self-run, official kit
Perplexity Decider v1.1 27B 62.25 62.75 board
Clef 27B 61.71 53.08 board
Jev 1.13 (API) 57.96 60.11 board
RSI-Jev v6.1-VL 4B 50.98 not listed self-reported
ezjev 4B s2 4B 50.82 46.95 board
jiwo 4B 4B 45.76 42.86 board
Nox 4B 4B 44.21 44.95 board

Public-suite scores by area (DI 0.3):

area KnowLine-4B-Gen4 KnowLine-4B-Gen3 Perplexity Decider v1.1 (27B) Jev 1.13
Knowledge & reasoning 44.6 43.9 52.0 53.9
Language 65.5 64.8 67.5 59.2
Retrieval & routing 72.5 70.9 61.3 55.4
Tools & agents 87.6 86.6 78.9 75.1
Arts & taste 59.2 49.8 47.0 39.1

New versions will follow.

model date DI 0.3 public DI 0.2.1 notes
KnowLine-4B-Gen4 (this model) 2026-10-10 64.90 β€” fourth release
KnowLine-4B-Gen3 2026-10-09 63.11 β€” third release
KnowLine-4B-Gen2 2026-10-08 62.54 β€” second release
KnowLine-4B-Gen1 2026-10-07 60.47 60.92 first release

Since Gen2, we run only Decision Index 0.3, not 0.2.1.

Evaluation

All results are self-run and have not been verified by a third party.

Other public benchmarks

suite Gen4 Gen3
C-Eval (4 categories, macro) 78.7 79.2
JevBench / JevBench-hard 86.1 / 72.1 86.6 / 72.1

Web operation (Mind2Web official test splits)

split Gen4 element selection Gen4 operation (balanced) Gen3 element selection Gen3 operation (balanced)
test_task (Cross-Task) 92.9 97.0 92.7 97.0
test_website (Cross-Website) 91.9 95.7 91.3 97.3
test_domain (Cross-Domain) 91.7 97.2 91.8 97.2
  • About the same as Gen3. The lower operation score on test_website comes from 3 more misses among its 72 SELECT steps; the differences are within noise.
  • So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some degree.
  • The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

metric Gen4 Gen3 board median (114 models) Jev 1.13
ECE (lower is better) 0.04-0.05 0.07-0.08 0.084 0.074
Brier score (lower is better) 0.29-0.31 0.31-0.33 0.49 0.36
Confidence β‰₯95% but wrong 1.7% 2.7% 1.3% 2.1%
Mean confidence / accuracy 0.82 / 0.77 0.84 / 0.76 0.81 / 0.74
  • Temperature in the weights: the final norm weight includes a readout temperature T = 1.32, so every logit is divided by 1.32 before the softmax over the option labels. The top-ranked option does not change, so neither do the Decision Index scores; the probabilities are just less extreme. T was fitted on held-out data.
  • Why ranges: the board does not publish which 32 benchmarks it uses. We checked our computation against three board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the plausible benchmark sets.
  • Still slightly overconfident: mean confidence is about 3-5 points above accuracy. Most of the gap comes from hard reasoning (HLE, CRUXEval, MuSR) and New Yorker caption matching.
  • Recommendation: if your use depends on probability thresholds, check calibration on your own data.

Disclosures

  • Decontamination:
    • All training data of this round, new and replayed, was checked with a revised tool against every Decision Index 0.3 row and the public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM).
    • Rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state were removed (1,909 rows). Rows that only share a source passage with an evaluation item, such as the same Wikipedia paragraph or tool description, were kept.
  • No model outputs as labels: no output of Jev or any other decision model was used as a training label.
  • Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all of them were checked by a human.
  • Public suite: the Decision Index public suite is public; the private tests may show a lower score.

Limitations

  • Knowledge-heavy reasoning: weaker than larger models. The knowledge area score is 44.6 (DI 0.3); MMLU-Pro barely moved (51.6), and HLE remains at chance level.
  • Math: answers are given without step-by-step reasoning. The rebuilt 0.3 GSM8K scores 61.6, about the same as Gen3.
  • Prompt injection: an instruction planted in the state can change the answer. Keep untrusted text clearly delimited.
  • Images: there was no image training; the serving path accepts images, but image use is zero-shot, so check it on your own data before relying on it.
  • Private tests: the 0.3 private tests may score lower than the public suite.

Citation

@misc{knowline4bgen4,
  title  = {KnowLine-4B-Gen4: a 4B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-4B-Gen4}
}

License

  • Weights: Apache-2.0, the same as the base model.
  • Code: knowline_server.py is MIT (see the file header).
Downloads last month
86
Safetensors
Model size
5B params
Tensor type
F32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for PelaAI/KnowLine-4B-Gen4

Finetuned
Qwen/Qwen3.5-4B
Finetuned
(1,474)
this model
Quantizations
1 model

Space using PelaAI/KnowLine-4B-Gen4 1