📖
When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
Isabel Guan, Dingyuan Zhang, Shipeng Lyu, I-Ming Chen

When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring

Published 2026 Verification Pending

Overview & Synopsis

Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.

Indexed from repository: arXiv (arxiv.org)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
Author / Organization Isabel Guan, Dingyuan Zhang, Shipeng Lyu, I-Ming Chen
Subject & Discipline cs.RO
Publication Year 2026
License / Rights arXiv Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website arXiv (arxiv.org)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-276723
Original Source Link Visit Source Page on arxiv.org »