Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models
Overview & Synopsis
Pixel-based language models process rendered text without relying on a fixed subword vocabulary, offering a promising approach to noisy and multilingual inputs. However, their practical value depends on whether they can be reliably adapted to downstream tasks. To this end, we systematically examine this question through fine-tuning behavior, typographical robustness, and zero-shot crosslingual transfer. Under our tested settings, pixel-based models require longer training and broader labelled-data coverage than text-based models, while delayed learning and learning-rate-dependent failures make adaptation less reliable. Our analyses further suggest that task-relevant information is less readily accessible in pretrained pixel representations, placing greater demands on fine-tuning. Once successfully adapted, however, pixel-based models lose substantially less accuracy under typographical noise and achieve higher target-language accuracy than English-pretrained text-based counterparts on two crosslingual benchmarks, despite weaker clean English performance. Together, these findings reveal a central trade-off: replacing tokenization with rendering reduces sensitivity to corrupted and unfamiliar inputs, but shifts a central bottleneck to downstream adaptation.
arxiv.org)
← / → for pages • + / - for zoom • 0 to fit width
Powered by Mozilla PDF.js v3.11
Document Information
| Document Title | Beyond Clean Accuracy: The Practical Challenges and Benefits of Pixel-Based Language Models |
|---|---|
| Author / Organization | Haowen Tang, Wei Xia, Raoyuan Zhao, Hinrich Schütze, Yihong Liu |
| Subject & Discipline | cs.CL |
| Publication Year | 2026 |
| License / Rights | arXiv Open Access / Creative Commons |
| Resource Verification | Listed PDF (Verification Pending) |
| Original Host Website |
arXiv
(arxiv.org)
|
| File Format | application/pdf (Portable Document Format) |
| Language | EN |
| Catalog ID | DOC-277781 |
| Original Source Link | Visit Source Page on arxiv.org » |