TouchSafeBench

Probing Collision Grounding in Vision‑Language Models for Safe Human–Robot Collaboration

Jun Wang1*, Xiaohao Xu1*†, Xiaonan Huang1
1University of Michigan, Ann Arbor
*Equal contribution    †Project lead
Paper arXiv Code Data Cite
Accepted at CoRL 2026

Two-minute overview of TouchSafeBench: the dataset, the two tasks, real model predictions on benchmark clips, and the main findings.

Example TouchSafeBench clips: safe operation, near-miss hard negative, scene collision, and human collision, each shown at three time points plus a top-down trajectory map.

TouchSafeBench asks whether a vision-language model can tell safe operation, a visually plausible near miss, robot–scene collision, and robot–human collision apart from short multi-view clips, using simulator contact and trajectory state as ground truth.

Abstract

Safe human–robot collaboration requires more than visual description: a monitor must determine whether the robot body is safely separated, already colliding with the scene or a person, or about to collide. We call this capability collision grounding: binding visual observations to robot body geometry, camera viewpoint, scene layout, human proximity, and temporal motion in order to infer present and imminent contact.

We introduce TouchSafeBench, a physics-grounded benchmark for evaluating collision grounding in vision-language models (VLMs). Built in Habitat 3.0, TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social navigation and social rearrangement, with synchronized multi-view RGB-D observations, top-down trajectory maps, calibrated camera metadata, and simulator-derived contact labels. We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact. Across three frontier or robotics-oriented VLMs, nine visual representations, and three viewpoint configurations, current models remain far from reliable: the best Macro-F1 on either task is 56%, depth inputs are not automatically transformed into robot-body collision evidence, and robot–scene contact is consistently harder than human-contact risk. An expert human given the same clips reaches 74% with all views, far above every model, yet shares the scene-contact blind spot, and the failure modes transfer to real recordings.

TouchSafeBench reveals a central limitation of VLMs used as embodied safety monitors: visual fluency does not imply physical accountability. Reliable robot safety monitors will need representations that explicitly bind viewpoint, robot morphology, metric collision geometry, and future collision. Code and data are available through the links above.

Benchmark

Data generation pipeline: 49 HSSD-HAB scenes, 9 YCB objects, and 3 SMPL-X humanoids are combined across two Habitat 3.0 tasks to produce 2,940 episodes, each with four RGB-D streams, a trajectory file, and camera parameters.
Data generation. HSSD-HAB scenes, YCB objects, and SMPL-X humanoids are combined across two Habitat 3.0 tasks and three humanoid appearances to yield 2,940 episodes with RGB-D streams, trajectories, and camera metadata.
Synchronized observations of one event: ego-view robot, ego-view human, third-person robot, third-person human, in RGB and depth, plus a top-down trajectory map with a zoom-in.
Synchronized observations. We record ego-view, third-person, RGB, depth, and top-down trajectory evidence for the same event, enabling controlled ablation over viewpoint and depth modality.
Evaluation protocol: clips are built from three episode time points; Stage I sweeps nine visual representations; Stage II fixes RGB and varies viewpoint context; each VLM returns a structured JSON response that is scored.
Evaluation protocol. Clips are built from three episode time points, Stage I sweeps nine visual representations, Stage II fixes RGB and varies viewpoint context, and each VLM returns a structured JSON response scored by classification and diagnostic metrics.

Results

  • Current VLMs are far from reliable collision monitors. Across three frontier or robotics-oriented models, nine visual representations, and three viewpoint configurations, the best Macro-F1 on either task is 56%.
  • Depth is not automatically turned into collision evidence. Depth variants do not monotonically improve performance, so current models do not convert depth images into metric robot-body contact evidence.
  • Robot–scene contact is the dominant blind spot. Scene collision and scene-collision warning are consistently the hardest sample types, and a human expert shares this blind spot.
  • Humans are far ahead, and viewpoint matters. An expert given the same clips reaches 74% Macro-F1 with all views; top-down trajectory context sharply reduces model false alarms even when present-tense contact recognition stays hard.
  • The failure modes transfer to real recordings. A replication on THÖR-MAGNI motion-capture data shows the same error patterns outside simulation.
Empirical summary: four panels covering robotics-oriented pretraining, depth variants, scene-contact subtypes, and the split of scene-risk errors.
Empirical summary and failure anatomy. A: Robotics-oriented pretraining reduces false alarms but does not raise Macro-F1. B: Depth variants do not monotonically improve performance, indicating that current VLMs do not automatically convert depth images into metric robot-body contact evidence. C: Scene-contact subtypes are the hardest in both tasks. D: Scene-risk errors split between missing contact evidence and misattributing risk to the visually salient human.
Horizontal bar charts comparing ego-only, all-view, and top-down RGB inputs for Task 1 Macro-F1, Task 2 Macro-F1, and false-alarm rate.
Viewpoint changes what the model is good at. Horizontal bars compare the Stage II RGB viewpoint ablation with each model's best ego-only modality from Stage I. A: All-view RGB improves Task 1 current-state recognition for GPT-5.5 and Gemini 3.1 Pro, surpassing their best single-view modality baselines. B: For Task 2, top-down is competitive with ego-only RGB for GPT-5.5 and Gemini 3.1 Pro, but does not beat the strongest ego-only depth-based condition. C: Top-down sharply reduces false alarms, showing that global trajectory context improves calibration even when present-tense contact recognition remains difficult.

Qualitative Analysis

Three failure cases, each with RGB and depth frames, the ground-truth and predicted labels, the VLM reasoning excerpt, and the failure pattern.
Failure-case analysis of collision-grounding errors. Each row shows a representative RGB-D clip and the corresponding VLM reasoning excerpt. (a) Gripper-camera attribution fallacy: a wall fills the arm camera during a scene collision, yet the model discounts it because the robot body is behind the gripper camera, producing a B→A false negative. (b) Impossible evidence standards: the model observes close wall or doorway proximity but demands visible overlap, deformation, or penetration that the sensing setup cannot provide, again suppressing scene-contact recall. (c) Proximity–collision fallacy: the model treats close human appearance as collision evidence even when the human is stationary or outside the robot body's path, producing human-risk false alarms.
Two success cases with RGB and depth frames, labels, and the VLM reasoning excerpt.
Successful reasoning patterns. Two representative cases show when correct predictions emerge. (a) Full visual occlusion: when a wall or other surface fills the arm camera at near-zero depth and the human is absent, the model correctly interprets the observation as scene contact. (b) Camera-geometry disambiguation: when a human appears close in the gripper-mounted camera, the model sometimes correctly reasons that the arm camera is mounted low and forward on the robot body, so image-space proximity does not imply body contact. VLMs succeed when the evidence is either extreme or geometrically unambiguous.

BibTeX

@inproceedings{wang2026touchsafebench,
  title     = {Probing Collision Grounding in Vision-Language Models for Safe Human--Robot Collaboration},
  author    = {Jun Wang and Xiaohao Xu and Xiaonan Huang},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.31196},
  note      = {Project page: \url{https://jun25336-bit.github.io/TouchSafeBench/}}
}