VLMs guess right, see wrong

🔥 What happened Researchers from Seoul National University introduced GRIT, a training framework that forces vision-language models to actually use visual information instead of just getting lucky. The team identified "Correct-but-Visually-Misgrounded Reasoning" (CVMR): models produce the right answer while misreading the image. 💡 Why it matters Standard RLVR rewards only final-answer correctness, reinforcing both grounded and misgrounded reasoning equally. GRIT masks gradients for visually misgrounded tokens, improving both accuracy and grounding fidelity. They also released GRIT-Bench, a new benchmark for text-rich reasoning tasks. ⚡ Our take This is a much-needed reality check for VLM hype: many "wins" are just blind guessing. If you deploy models for document analysis or medical imaging, take GRIT seriously—otherwise a wrong rationale will blow up in your face.