deVision v0.4

Image + English questions → structured answers with calibrated probabilities. deVision pairs a SigLIP2 vision encoder with Laya's ModernBERT decision model. It scores the options of yes/no (noul) and multiple-choice (choice) questions instead of generating text, follows the Jev answer format, and runs without a GPU. Several questions about one image share a single image encoding.

Usage

pip install devision

import devision

model = devision.load("lukbit/devision")   # latest release; device defaults to auto
result = model.predict(
    image="photo.jpg",   # a path, an http(s) URL, a data URI, base64, bytes or a PIL image
    state="",            # text context; "" when there is none
    questions={
        "has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"},
        "room": {"type": "choice", "instructions": "Which room is this?",
                 "criteria": {"kitchen": None, "bathroom": None, "bedroom": None}},
    },
)
print(result["answers"]["has_fork"]["noul"])        # probability of yes
print(result["answers"]["room"]["probabilities"])   # one probability per option, summing to 1

HTTP server with a browser demo at http://127.0.0.1:8000:

devision-serve --checkpoint lukbit/devision --port 8000
curl -s http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' \
  -d '{"image": "https://example.com/photo.jpg", "state": "", "questions": {"has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"}}}'

Over HTTP, image is an http(s) URL, a data URI or base64; the server never reads its own files. Responses follow Jev; confidence for choice is (n * p_max - 1) / (n - 1). Invalid requests return 422 (devision.InvalidRequest in Python).

Good for: whether something is present and how many (ask counts as a choice), what kind of thing or scene, colours and materials, up/down and left/right in photos (as a choice). Not for: comparing two places on a diagram, chart or map; relative-position yes/no questions; small text; other languages or several images. Use the probabilities as thresholds and send uncertain cases to a person or a stronger model, after checking the thresholds on your own data.

Evaluation

Accuracy through decide with the fitted temperatures. Mismatched: the same questions with every picture swapped for an unrelated one. These are project evaluation subsets, not official leaderboard scores. Although test pictures are excluded from training, these sets have informed successive experiments and release decisions; they are not a fresh independent test. See the release limitations below.

Set	Questions	Accuracy	Mismatched	NLL	ECE
COCO object presence	1,120	0.955	0.497	0.196	0.092
VQAv2 multiple choice	1,453	0.905	0.370	0.280	0.053
COCO size	1,198	0.870	0.516	0.506	0.234
POPE, project filtered set	9,000	0.865	0.520	0.363	0.048
GQA val subset	986	0.843	0.546	0.383	0.086
COCO position	1,276	0.938	0.509	0.238	0.047
VQAv2 yes/no subset	1,000	0.722	0.500	0.571	0.055
COCO relative position	2,366	0.917	0.480	0.281	0.037
VSR, project held-out split	902	0.731	0.460	0.591	0.073
Visual7W, project held-out split	1,000	0.783	0.442	0.597	0.073
CountBench, 0–10 choices	491	0.501	0.102	1.387	0.166
COCO mirrored position	640	0.933	0.503	0.283	0.053
COCO mirrored relative position	1,786	0.914	0.494	0.300	0.049
What'sUp	820	0.467	0.233	1.214	0.093

Against Laya Vision 201M on the same questions (its published per-question predictions; difference in points, 95% interval from paired resampling by picture):

Set	Questions	Laya Vision	deVision	Difference
VQAv2 yes/no	4,887	0.717	0.756	+3.9 [+2.4, +5.3]
A-OKVQA	1,138	0.598	0.634	+3.5 [+0.3, +6.6]
ScienceQA with images	2,097	0.824	0.821	-0.3 [-2.2, +1.7]
ScienceQA natural science, needs the picture	323	0.700	0.638	-6.2 [-12.7, +0.0]

On the full POPE random / popular / adversarial sets (3,000 questions each) deVision scores 0.899 / 0.873 / 0.824, against Laya Vision's published 0.836 / 0.819 / 0.777 (aggregate scores only, not paired).

CPU latency: P50 289 ms, P95 301 ms (Linux x86_64, 4 threads, FP32, one question per request, warm-up excluded).

Model

Vision	SigLIP2-B/16, 256 × 256 letterboxed input, 64 visual tokens after a 2 × 2 merge and an MLP projector; training policy described below
Decision	Laya-initialised ModernBERT-large and Laya's decision head; each option is scored at its [MASK], then a temperature-scaled softmax
Size	About 518M parameters, FP32, 2.07 GB (LoRA merged)

This release continues v0.3 in three steps on one RTX 5070 (bf16), with SigLIP2's top four encoder layers and final LayerNorm fine-tuned at 5e-6 alongside the projector, decision head and ModernBERT LoRA (merged into the released weights). The input remains 256×256 with 64 visual tokens; the architecture is unchanged.

1. 742,238 questions, one pass from v0.3. VQAv2 (annotators' answers as soft targets) and GQA (balanced train, all shards, open questions turned into choices), about 190,000 each, plus targeted data for the abilities v0.3 lacked:

   • all of Visual7W telling (52,390 after filtering);
   • caption-choice spatial relations (on / under / in front of / behind / left / right) from Visual Genome scene graphs, with every distractor checked against the boxes;
   • counting on COCO and TallyQA, always with the eleven options 0–10;
   • CLEVR, IconQA;
   • programmatically drawn magnet and solution diagrams in ScienceQA's style;
   • replay of v0.3's other abilities.

   Every picture of every evaluation set was excluded by id and image hash. Choice questions v0.3 still answered confidently with the wrong picture were dropped.

2. A short science-heavy continuation (123,084 questions): all ScienceQA / AI2D / TQA training questions ×3, IconQA, the magnet diagrams and 40,000 replay. This also repaired an exclusion bug that had dropped about half of the ScienceQA training questions in step 1.

3. A short continuation (104,331 questions) with presence and size labels softened from 0 / 1 to 0.9 / 0.1, to stop over-confident presence answers (POPE).

Temperatures were fitted on a development mix with balanced sources, by question type and option-count bucket. Missing buckets fall back to the question-type temperature:

Bucket	Temperature
Choice, 2 options	3.5631
Choice, 3–5 options	2.1146
Noul (yes / no)	1.0895
Choice, any other count	2.3642

The owner adopted this model although one prewritten acceptance rule (calibration) is not met. See the adoption decision, the full comparison with v0.3 and the training configurations.

Against v0.3 on the same questions (accuracy points, 95% intervals), 22 of 23 judged sets improve and none is significantly worse:

Set	Change
mirrored relations	+21.2
What'sUp	+18.0
CountBench	+14.5
Visual7W	+7.2
VSR	+7.0
GQA	+6.1
CLEVR-CoGenT B	+4.0
ScienceQA	+3.1 [+1.6, +4.8]
MMBench-en dev	+2.3
POPE	+1.7

Limitations

• Scope. English, one image, yes/no and choice questions only; score is not implemented. At 256×256, small text and fine diagram details are hard to read.
• Counting. It improved (CountBench 35.6% → 50.1%, COCO counts 30.6% → 34.5%), but this is still not an accurate counter, especially above five.
• Calibration. Probabilities are slightly under-confident on easy presence and size questions (size: mean confidence 0.64 at 0.87 accuracy) and slightly over-confident on MME. POPE calibration is on par with v0.3. Validate thresholds on your own examples.
• External benchmarks. MME (−0.6), NaturalBench (+0.3), CV-Bench (+0.7) and A-OKVQA (+2.1) did not change significantly.
• Diagrams. Diagram reading that depends on small printed numbers (particle speeds, magnet distances) remains weak. The drawn solution diagrams did not transfer to ScienceQA's beakers.
• Test-set reuse. Test sets informed several decisions in this cycle. This is one training seed, not an independent confirmation.

Licence

Weights and code: Apache-2.0, like the three base models (Laya, SigLIP2, ModernBERT). The training datasets (listed in this card's metadata) have their own terms, some non-commercial (e.g. ScienceQA, CC BY-NC-SA 4.0); check them against your use.

Links

Code and training records (this release: tag model-v0.4) · evaluation details · provenance · Laya Vision