๐Ÿ“–
SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams
Hao Wang, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Tat-Seng Chua, Tian Tian

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Science & Mathematics Published 2026 Verification Pending

Overview & Synopsis

Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation. Scientific diagrams naturally span diverse types, each characterized by distinct visual structures and content organization. Consequently, during peer review and other scientific evaluation, human experts naturally consider type-specific criteria when assessing their quality. However, existing VLM-based evaluation benchmarks fail to reflect the above human evaluation practices for two main reasons: 1) benchmark datasets typically lack a systematic categorization of different diagram types and thus adopt the same generic criteria for all diagrams, and 2) their evaluation criteria are often designed heuristically, without explicit alignment to human evaluation preferences. Motivated by these limitations, we introduce SciDiagBench, which consists of a benchmark dataset that systematically categorizes scientific diagrams into 11 types and a human preference-guided evaluation method. Specifically, SciDiagBench contains 165 scientific diagrams across 10 broad domains, with types including graphical abstracts, technical roadmaps, and illustrative cases. Based on this dataset, SciDiagBench develops an evaluation method that leverages a preference-guided calibration process to incorporate human preferences into the generation of type-specific criteria. Ultimately, SciDiagBench enables more reliable automated evaluation that better aligns with human evaluation across diverse scientific diagrams.

Indexed from repository: arXiv (arxiv.org)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams
Author / Organization Hao Wang, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Tat-Seng Chua, Tian Tian
Subject & Discipline cs.CL
Categories Science & Mathematics
Publication Year 2026
License / Rights arXiv Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website arXiv (arxiv.org)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-270638
Original Source Link Visit Source Page on arxiv.org »