๐Ÿ“–
Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness
Caixin Kang, Sitong Gong, Tianyu Yan, Kaipeng Zhang, Yoichi Sato, Yifei Huang

Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness

Science & Mathematics Published 2026 Verification Pending

Overview & Synopsis

The Chinese idiom "kou mi fu jian" ("honeyed words, hidden daggers") names speech that is polite in form but hostile in function. We present ZhenhuanBench, a 1,592-item benchmark built from expert commentary on Empresses in the Palace, covering speaker intent, emotion-language mismatch, and per-listener uptake. Text-only evaluation covers 34 LLMs. GPT-6 Astra leads the leaderboard at 73.48% overall, while its listener-reaction score is 58.37%. The annotations also supply supervised traces and string-verifiable training rewards. With 338 gold traces, an 8B model outperforms teacher distillation on identical items. Subsequent RL saturates, and chance correction reduces the estimated recoverable performance gap by a factor of 7.5. Our CPR-GRPO loop (Compress-Probe-Replenish) uses this estimate to weight gold supervision, reaching 67.0 on public multiple-choice tasks and 74.5 under the training-matched hidden protocol. Without target-task training, CPR outperforms Base, SFT, and GRPO across six Chinese and English pragmatic and sarcasm evaluations. On three external datasets, training with CPR yields higher mean Macro-F1 than SFT and GRPO.

Indexed from repository: arXiv (arxiv.org)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness
Author / Organization Caixin Kang, Sitong Gong, Tianyu Yan, Kaipeng Zhang, Yoichi Sato, Yifei Huang
Subject & Discipline cs.CL
Categories Science & Mathematics
Publication Year 2026
License / Rights arXiv Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website arXiv (arxiv.org)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-273974
Original Source Link Visit Source Page on arxiv.org »