Pixel Art Bench: Evaluating Structured Generation in Language Models
An Esoteric Structured Generation Task
A majority of evaluation benchmarks for large language models focus a fixed set of model capabilities such as reasoning, coding or natural language understanding.This benchmark operates in a very different regime. It is an esoteric task: one that sits outside the standard distribution of language modeling, and shows a different perspective of current Language Models.
Originally introduced as Pixel Art Bench on Emergent Mind, this benchmark evaluates how well LLMs can generate pixel art for a set of predefined subjects such as Mario, Saturn or abstract concepts like 'hope'. Each model is prompted to return a strictly formatted JSON object containing:
- A palette of colors
- A 24×24 grid of indices referencing that palette
The reason that this benchmark is interesting to measure is, conventionally pixel art is not simple to create for Language Models for the given reasons:
- A Non-Linguistic Output Space
The output is generally discrete, symbolic and format-sensitive. Even minor deviations such as extra text or incomplete colors can lead to poor results.
- Strict Structural Constraints
Unlike open-ended text, the images have a fixed grid size (24×24) and limited vocabulary (palette indices 0–9). The model must maintain global consistency across a set of 576 positions, not just local token fluency.
- Implicit Spatial Reasoning
The image grid is serialized as text, but if we take a closer look, the task itself is inherently 2D since it contains shapes and edges. This creates a mismatch between model inductive bias (1D sequence) and the task structure (2D layout), which we can cause further issues as we will explore later.
Qualitative Analysis of Benchmark Result
The following analysis is conducted on the source results of Pixel Art Bench, ensuring that all evaluations are grounded in the benchmark’s original data distribution. By using the canonical version, we isolate model behavior under the intended task constraints, allowing for a consistent comparison across evaluation axes.
The correlation matrix reveals that the evaluation metrics capture largely orthogonal aspects of model behavior, rather than a single unified notion of quality. Several relationships are weak or even negative, most notably the inverse correlation between palette validity and color efficiency (-0.47), indicating that strict adherence to palette constraints does not translate into effective color usage.
In contrast, edge density shows moderate positive correlation with both color efficiency (0.42) and fill balance (0.40), suggesting that models capable of forming coherent structures also tend to distribute colors more effectively. Taken together, these patterns reinforce that improvements along one axis do not reliably propagate to others, highlighting the multi-dimensional nature of the task.
The radar chart complements this analysis by illustrating how individual models distribute performance across these axes. Rather than demonstrating balanced capabilities, models exhibit distinct trade-offs and specialization: some achieve higher scores in structural metrics such as edge density and palette validity, while others perform better in distributional metrics like color efficiency and fill balance. Critically, no model dominates across all dimensions, underscoring the absence of a single scalable capability governing performance.
Within this setup, the correlation matrix and radar analysis reveal how models distribute performance across evaluation axes. Because all metrics are computed on the same , the observed trade-offs and weak correlations reflect intrinsic properties of structured generation for the given benchmark.
From Benchmark to Evaluation Protocol
In order to reproduce the benchmark locally, we can create a lightweight version of the evaluation harness with Inspect AI. We define the benchmark as a set of 3 scores. The first is JSON Validity which measures JSON correctness and the presence of required colors within the color grid. The second, Pixel Art Quality, evaluates local consistency by ensuring that all grid values correspond to valid palette indices and that palette usage is coherent. The third, Render Success, captures global structural properties such as continuity of shapes and spatial alignment, serving as a proxy for whether the model successfully treats the task as a 2D compositional problem.
Applying this evaluation framework across multiple model families, including Phi, Llama, and SmolLM, reveals a consistent and interpretable pattern. Performance degrades systematically from syntactic correctness to structural rendering and finally to semantic quality, indicating a clear hierarchy of task difficulty. Models first learn to satisfy formatting constraints, then achieve partial structural consistency, and only weakly capture semantic coherence.
Notably, the strongest model, Phi-3.5-mini-instruct, achieves perfect JSON validity, yet this does not translate proportionally into downstream performance, with render success dropping to approximately 0.74 and pixel art quality to around 0.42. Furthermore, scaling does not yield consistent gains. While Phi-3-mini-4k-instruct and Phi-4-mini-instruct exhibit comparable structural validity, their semantic quality diverges, and LFM2.5-1.2B-Instruct demonstrates near-collapse across all axes, failing even basic structural constraints.
In contrast, models from the Llama and SmolLM families fail to produce any successful pixel art renderings under this setup. The benchmark can also be accessed as a Hugging Face dataset, enabling reproducible experimentation. To investigate whether the observed limitations can be mitigated through supervision, we perform parameter-efficient fine-tuning using LoRA on the pixel art bench source dataset with a Qwen3-1.7B model.
Despite targeted training on prompt–pixel grid pairs, the results remain largely unfavorable. Fine-tuning improves format adherence but fails to produce meaningful improvements in pixel art quality. This suggests that the limitations exposed by Pixel Art Bench are not easily resolved through standard fine-tuning, reflecting a deeper mismatch between autoregressive language modeling and inherently 2D structured generation tasks.
Conclusion
Firstly, I would like to thank Matt Mazur for providing the Pixel Art Bench source dataset and creating the inital benchmark, which made this analysis possible. The benchmark serves as a useful measure for probing the gap between spatial tasks and autoregressive language modeling. It also demonstrates that structured generation tasks expose failure modes that are largely invisible in traditional text benchmarks. As such, it provides a useful lens for evaluating whether models can move beyond token-level fluency toward more spatially grounded reasoning.