Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [WACV 2027]

Project page | Paper | 🤗Hugging Face Data | 🤗Hugging Face Model

Guardian is a multi-view reasoning vision-language model for unified planning and execution verification in robotic manipulation. This repository contains the research code for the full Guardian / FailCoT stack:

• automatic failure synthesis from successful robot demonstrations
• generation of structured reasoning traces for planning and execution failures
• fine-tuning and evaluation of Guardian on failure-detection benchmarks

Quickstart

1. Quick: download the released model checkpoint and run the demo
2. Quick: download the released data, and reproduce evaluations using the released checkpoint
3. Long: reproduce the full data generation pipeline step-by-step

Quick: download the released model checkpoint and run the demo

Guardian supports:

• planning verification from an initial scene image and a proposed plan
• execution verification from before/after observations, with single-view or multi-view inputs
• thinking and vanilla response modes

Two checkpoints are released — pick the one that matches your latency vs. accuracy preference (and pair it with the matching thinking=True/False flag in the Guardian wrapper):

• 🤗 paulpacaud/guardian-thinking — emits a <think> reasoning trace before the answer, best accuracy.
• 🤗 paulpacaud/guardian-vanilla — emits the answer directly, less intelligent but faster at inference.

Run the demo following: demo doc.

Quick: download the released data, and reproduce evaluations using the released checkpoint

The released datasets are on the 🤗 Hugging Face Guardian collection. The codebase expects each dataset to live under ./data/failure_forge/data/<name>_dataset/ (e.g. ./data/failure_forge/data/robofail_dataset/).

For the three real-robot OOD evaluation benchmarks (derived/adapted from UR5-Fail, RoboFail, RoboVQA), download the bundle paulpacaud/Guardian-FailCoT-OOD-datasets and lay it out as follows:

hf download paulpacaud/Guardian-FailCot-OOD-datasets \
    --repo-type dataset \
    --local-dir ./data/failure_forge/data/_ood_tmp

for split in ur5fail_test robofail robovqa; do
    mv ./data/failure_forge/data/_ood_tmp/${split} \
       ./data/failure_forge/data/${split}_dataset
    tar -xzf ./data/failure_forge/data/${split}_dataset/records.tar.gz \
         -C ./data/failure_forge/data/${split}_dataset/
    rm  ./data/failure_forge/data/${split}_dataset/records.tar.gz
done
rm -rf ./data/failure_forge/data/_ood_tmp

You should end up with, e.g., ./data/failure_forge/data/robofail_dataset/{metadata_*.jsonl, internVL_dataset_*.jsonl, records/...}.

Finally run the evaluation following the evaluation doc.

Generate FailCoT-style data

The failure-generation pipeline has two entry points:

• Simulation (RLBench → rlbenchfail): RFailureForge/pipeline/ — see Simulation_Online_Data_Generation.md.
• Real robot (BridgeData V2 → bdv2fail): RFailureForge/failBridgeDataV2/ — see RealData_Offline_Data_Generation.md.

Planning failures for both pipelines are generated by the same script (RFailureForge/pipeline/s5b_add_planning_failures/build_planning_failures.py).

Fine-tune Guardian

To train Guardian on the FailCoT dataset (or your own), see Training_The_VLM_Failure_Detector.md for the full training stack, or FINETUNE_GUARDIAN.md for a short recipe to fine-tune from the released Guardian checkpoint.

Codebase structure

Within this repository, the expected local layout is typically:

data/
  failure_forge/
    data/
      ur5fail_test_dataset/
      robofail_dataset/
      robovqa_dataset/
      rlbenchfail_train_dataset/   # optional, for training
      bdv2fail_train_dataset/      # optional, for training
      ...
    models/
      guardian-thinking/
      guardian-vanilla/
  pretrained/
    InternVL2_5-8B/
    InternVL3-8B/

Fine-tune Guardian on your own data

If you want to fine-tune Guardian on your own data, you can follow the finetuning recipe.

Citation

If you use this codebase, please cite:

@misc{pacaud2026guardian_failcot,
  title={Scaling Cross-Environment Failure Reasoning Data for Vision-Language Robotic Manipulation},
  author={Paul Pacaud and Ricardo Garcia and Shizhe Chen and Cordelia Schmid},
  year={2026},
  eprint={2512.01946},
  archivePrefix={arXiv},
  primaryClass={cs.RO}
}

If you specifically build on the earlier Guardian workshop paper, you may also cite:

@inproceedings{pacaud2025guardian,
  title={Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models},
  author={Paul Pacaud and Ricardo Garcia Pinel and Shizhe Chen and Cordelia Schmid},
  booktitle={Workshop on Making Sense of Data in Robotics: Composition, Curation, and Interpretability at Scale at CoRL 2025},
  year={2025},
  url={https://openreview.net/forum?id=wps46mtC9B}
}