๐Ÿ“–
"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics
Vladimir Necula, David Gvadzabia, Sofia Serrano

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

Science & Mathematics Published 2026 Verification Pending

Overview & Synopsis

"Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which simple example-level metrics, drawn from prior work on data selection, text quality, and instruction tuning, best predict training utility under a fixed finetuning budget. For each metric, we sort instruction-response examples by score, partition them into low, middle, and high bins with roughly equal token mass, finetune separately on each bin, and compare validation perplexity and LLM-as-judge scores for the models' generated answers as a function of cumulative training tokens. Our results suggest that answer-dense supervision, as captured by response ratio, is a stronger and more portable predictor of instruction-tuning utility than commonly used alternative proxies.

Indexed from repository: arXiv (arxiv.org)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title "High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics
Author / Organization Vladimir Necula, David Gvadzabia, Sofia Serrano
Subject & Discipline cs.CL
Categories Science & Mathematics
Publication Year 2026
License / Rights arXiv Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website arXiv (arxiv.org)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-114341
Original Source Link Visit Source Page on arxiv.org »