webAI-Official

/

webAI-ColVec1.1-4b

like 4

Follow

webAI 108

Hello!

Thanks for releasing ColVec1.1, the ViDoRe V3 numbers are excellent. The MultiVectorEncoder class ships in the next Sentence Transformers release, planned for around the 18th, so for now the install below pulls from source. I would love to feature this model in that release's blog post and documentation, especially once it loads without the revision pin (that is, once this PR is merged).

Heads up, this PR was AI-generated and human-reviewed. Here's a summary of the changes as reported by my agent:

Pull Request overview

• Integrate webAI-Official/webAI-ColVec1.1-4b with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via MultiVectorEncoder.
• Implement ColQwen35BidirectionProcessor.replace_image_token, which ProcessorMixin requires and which currently raises NotImplementedError.

Details

The integration is config-only on the model side: sentence_bert_config.json sets transformer_task="retrieval", so Sentence Transformers resolves the model through your auto_map and loads ColQwen35Bidirection itself, keeping the bidirectional attention patching, the projection, the L2 normalization and the masking in your code. The pipeline is therefore just Transformer(retrieval) -> MultiVectorMask. The prompt formats are reproduced in a named chat template selected through processing_kwargs, so encode_query and encode_document produce byte-identical token ids to your process_queries / process_images helpers, and the existing AutoProcessor / AutoModel / score_retrieval path is unchanged.

One code change: ColQwen35BidirectionProcessor.replace_image_token. ProcessorMixin.__call__ (and apply_chat_template through it) delegates image-placeholder expansion to that method, which currently raises NotImplementedError, so any image input fails while text-only input works. The fix implements it with the same grid arithmetic as _process_single_image, so it is a strict addition: process_images, process_queries and score_retrieval are unchanged, and processor(text=..., images=...) now works for everyone, not only through Sentence Transformers.

Verified against your AutoProcessor + AutoModel + score_retrieval path on the two model-card documents: max absolute score difference 1.1e-5 in fp32 and 5e-5 in bf16, and exactly 0.0 on 3.2-megapixel pages that reach the full 1792-token budget. Loading from the config files reports all 8 full-attention layers with is_causal=False, confirming the bidirectional patching survives the Sentence Transformers load path.

pip install "sentence-transformers[image] @ git+https://github.com/huggingface/sentence-transformers.git"

from io import BytesIO

import requests
from PIL import Image
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("webAI-Official/webAI-ColVec1.1-4b", revision="refs/pr/1", trust_remote_code=True)

queries = [
    "When was the United States Declaration of Independence proclaimed?",
    "Who printed the edition of Romeo and Juliet?",
]
document_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg",
]
documents = [
    Image.open(BytesIO(requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).content))
    for url in document_urls
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (27, 640) (523, 640)

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[23.3935,  5.7660],
#         [ 4.8504, 23.1851]])

• Tom Aarsen