Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation

Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text-to-Image Generation

📄 arXiv: https://arxiv.org/abs/2608.06751

TL;DR: Naming an artist in a prompt isn't style control. T2I models tend to fall back on canonical shortcuts — the recurring motifs, stock palettes, and over-represented period signatures they associate with that name — and quietly overwrite the scene the user actually asked for. This paper introduces Atelier, which plans an explicit control state before generation instead of hoping the backend infers intent.

What Atelier does

• Decomposes an underspecified request ("a quiet subway platform in the style of Van Gogh") into an explicit control state: scene anchors, preserve vs. transform decisions, style-regime hypotheses, role-bound artist evidence, and anti-shortcut constraints.
• Grounds that state with artist-level knowledge plus local patch references — and unlike generic RAG prompting, binds each patch to a specific scene role rather than dumping it in as an undifferentiated style exemplar.
• Compiles backend-aware generation plans, then iteratively refines candidates with global + local authenticity critics.

ArtIntentBench
A companion benchmark that trades artist breadth for supervision depth — two deliberately contrastive artists (Van Gogh and Qi Baishi, different media and cultural traditions) across four tasks: artwork re-rendering, period-controlled generation (Paris / Arles / Saint-Rémy / Auvers), historically unseen subjects, and Qi Baishi re-rendering, plus shortcut auditing and human preference eval. The argument for going deep: a wide benchmark of many artist names with shallow labels would end up measuring generic style association, not artist-grounded control.

Results
Across both open-weight and closed-source generators, Atelier improves artist-level style fidelity, preserves source structure better, and substantially cuts shortcut substitution relative to prompt-engineered, retrieval-augmented, and general-purpose agent baselines.

Why it's interesting: the framing that artist-grounded generation is bottlenecked upstream of the diffusion model — in inferring explicit, evidence-grounded controls — is a nice counterpoint to the "just scale the generator" default. The shortcut-auditing setup also seems reusable well beyond these two artists.

Curious how the control state holds up on artists with less catalogued, more diffuse oeuvres, and whether the period-regime hypotheses transfer to movements rather than individuals.