When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
Overview & Synopsis
Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.
arxiv.org)
← / → for pages • + / - for zoom • 0 to fit width
Powered by Mozilla PDF.js v3.11
Document Information
| Document Title | When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring |
|---|---|
| Author / Organization | Isabel Guan, Dingyuan Zhang, Shipeng Lyu, I-Ming Chen |
| Subject & Discipline | cs.RO |
| Publication Year | 2026 |
| License / Rights | arXiv Open Access / Creative Commons |
| Resource Verification | Listed PDF (Verification Pending) |
| Original Host Website |
arXiv
(arxiv.org)
|
| File Format | application/pdf (Portable Document Format) |
| Language | EN |
| Catalog ID | DOC-276723 |
| Original Source Link | Visit Source Page on arxiv.org » |