Bug Severity Classification with Fine-Tuned Open LLMs: A Paired Comparison with TF-IDF and CodeBERT Baselines on Eclipse Reports

Overview & Synopsis

Open large language models (LLMs) small enough to run on a laptop are an obvious candidate for automatic bug severity prediction. We measure what they add over cheaper classifiers on one task: assigning one of four Bugzilla severity levels (Critical, Major, Minor, Trivial) to Eclipse bug reports. From a public dump of 361,006 unique Eclipse reports we draw a class-balanced sample of 20,000 rows and compare eleven configurations of nine model types: six TF-IDF classifiers, a fine-tuned CodeBERT encoder, and Llama 3 8B and Qwen 2.5 7B, each run zero-shot and after QLoRA fine-tuning. All eleven are scored on the same 2,000 held-out reports, with bootstrap intervals and McNemar tests. The zero-shot runs reach macro-F1 0.305 and 0.321, below every TF-IDF model (0.430-0.489). The fine-tuned runs reach 0.526 and 0.485, but the zero-shot and fine-tuned setups also differ in checkpoint, serving runtime and decoding, so this gap cannot be credited to fine-tuning alone. Fine-tuned Llama 3 shows no detectable difference from CodeBERT (0.518; difference +0.007, 95% CI -0.017 to +0.032), and fine-tuned Qwen 2.5 is below it on macro-F1. The models differ more by class than in aggregate: fine-tuned Llama 3 finds 36% of Critical reports, Qwen 2.5 finds 69% but only 21% of Minor ones. The dump repeats 60,000 rows, and 8% of the test reports duplicate a training report; removing them lowers TF-IDF scores by up to 0.031 and leaves the LLMs almost unchanged. We release split identifiers, per-report predictions and the analysis script at https://github.com/tbayan/bug_finetuning_llm_artifacts, together with CyberTriage, a tool that serves the fine-tuned classifier locally.