Simulation_Online_Data_Generation.md

Failure Generation Pipeline (FailCoT / RFailureForge)

This document walks through the FailCoT data-generation pipeline used to build the Guardian training and evaluation datasets, focused on the RLBench simulation path. For the real-robot (BridgeData V2 → bdv2fail) counterpart, see RealData_Offline_Data_Generation.md. Note that planning failures for both pipelines are generated by the same script (s5b_add_planning_failures/build_planning_failures.py).

The pipeline lives entirely in RFailureForge/pipeline/. Each stage is named sN_* and is meant to be run in order. Most stages are SLURM jobs but the underlying .py scripts can be invoked directly on any GPU machine after editing the paths.

end-to-end flow:

RLBench demos
    │
    ▼
s0  In RLBench, replay expert demos to generate microsteps + detect 2D object bboxes at each keystep
    │   why: microsteps let us deterministically replay subtasks later on; bboxes let us pick valid distractor objects for the perturbation step.
    ▼
s1–s2  label gt trajectories, list graspable objects per episode
    │   why: gives the perturbation pipeline a structured (subtask, primitive, target) view of each demo and the set of plausible wrong objects.
    ▼
s3  perturb gt trajectories into failure trajectories
    │   why: turns each clean demo into one trajectory per failure mode (no_close, slip, wrong_object, …).
    ▼
s4  Run perturbed trajectories in RLBench → save before/after frames + videos
    │   why: produces the actual visual evidence (RGB frames, MP4) the VLM will be trained to judge.
    ▼
s5  Post-process simulation data: captions, plans, visible objects
    │   why: enriches each sample with rich metadata that will be used to construct grounded CoTs (plan, captions, scene objects).
    │   ├─ s5b  Generate planning failures (LLM-based plan perturbations)
    │   │      why: planning failures can't be simulated — we mutate the plan with an LLM while keeping the world state.
    │   └─ s5c  Convert postprocessed dataset into InternVL fine-tuning format
    │          why: InternVL requires a specific data format for training/evaluation.
    ▼
s6 Generate Chain-of-Thought reasoning traces with a teacher VLM
    │   why: distils a stronger teacher (InternVL3-38B) into <think> traces for the 8B student.
    ▼
s7  Fine-tune Guardian (see Training_The_VLM_Failure_Detector.md)
    │   why: LoRA-fine-tunes InternVL3-8B on the resulting JSONL to specialise it for failure detection.
    ▼
s8/s9  Evaluate Guardian and SOTA baselines (see Offline_VQA_Evaluation.md)
        why: scores Guardian and baselines on the four held-out test splits.

---

Prerequisites

You will need to have downloaded in data/pretrained:

• InternVL2_5-8B (from HF)
• InternVL3-38B (from HF)
• Mistral-Small-3.2-24B (from HF)
• nvcuda_v2.sif: singularity container to run RLBench on a HPC cluster. (if you don't use a HPC cluster, you can skip using it)
  • A pre-built singularity image can be downloaded here.
• a pre-generated microsteps folder is available on HuggingFace so you can skip the first step: 🤗paulpacaud/guardian-rlbench-microsteps

Quickstart: Run the pipeline on one taskvar

Use this path when you want a small end-to-end run before launching a full split. It uses sanity_check as dummy dataset name. The taskvar must be an RLBench task variation already described by the pipeline assets: Most built-in taskvars are listed in assets/failure_forge/taskvars_full.json. For example, put_groceries_in_cupboard+0 is valid. Run:

TASKVAR=put_groceries_in_cupboard+0 bash RFailureForge/pipeline/sanity_check_launch.sh

The launcher parses TASKVAR into TASK and VARIATION_ID, writes a one-item taskvar JSON under $SCRATCH/failure_forge/sanity_check/, then submits the full dependency chain.

Details of each stage

Stage 0 — Prepare expert demonstrations

Two sub-stages that build the bookkeeping needed for the rest of the pipeline.

s0a — Generate microsteps

s0a_generate_microsteps/gen_microsteps.py collects dense (microstep) RLBench demonstrations for one task variation. Microsteps are needed to replay each subtask deterministically in s4.

• Input: an RLBench task variation (e.g. take_shoes_out_of_box+0).
• Output: pickled demonstrations under data/failure_forge/data/microsteps/.

s0b — Generate keystep bounding boxes

s0b_generate_keysteps_bbox/gen_keysteps_bbox.py runs a 2D detector on the keysteps to extract object bounding boxes used downstream for failure synthesis (e.g. wrong-object selection).

• Wall-clock: ~5 minutes for the full taskvar list on an H100.

---

Stage 1 — Load + label ground-truth trajectories

s1_label_gt_trajectories/create_labelled_gt_trajectories.py reads the keystep-bbox LMDB produced by s0b, extracts expert action trajectories, and annotates each subtask with a structured label (subtask name, primitive, target object, …). The expert trajectories are cached as expert_trajectories_{dataset}.json next to the labelled output:

assets/failure_forge/{dataset}/expert_trajectories_{dataset}.json   # cache (rebuilt if missing)
assets/failure_forge/{dataset}/labelled_gt_trajectories_{dataset}.json

• Wall-clock: a few minutes.

---

Stage 2 — List graspable / visible objects per episode

s2_list_episode_objects/list_graspable_objects.py inspects each RLBench scene and produces, for every episode, the list of graspable objects available in that scene. This list is later used by FailureForge to construct wrong object failures with semantically plausible distractors. We do this because RLBench scenes have non-manipulable objects that we cannot use for wrong-object failures.

Output:

assets/failure_forge/{dataset}/graspable_objects_{dataset}.json

• Wall-clock: ~1.5 min/taskvar on an H100, ~40 min for 35 taskvars.

---

Stage 3 — FailureForge: synthesise failure trajectories

s3_run_failure_forge/run_failure_forge.py is the core symbolic perturbation engine. Given a labelled gt trajectory and the list of graspable objects, it produces, for each subtask, a set of perturbed trajectories — one per failure mode.

Failure modes:

ground_truth, no_close, slip, translation, rotation, no_rotation, wrong_sequence, wrong_object

The number of episodes perturbed is set per-split inside RunFailureForgeArguments:

split	gt episodes	failure-mode episodes
train	100	40
val	50	20
test	50	20

(rule of thumb: 2.5× more positives than negatives so the final dataset stays balanced after some failure modes fail to materialise in s4).

Output:

assets/failure_forge/{dataset}/perturbed_trajectories_{dataset}.json

• Wall-clock: ~1 min per split (train / val / test).

---

Stage 4 — Run perturbed trajectories in RLBench

s4_run_simulation/run_simulation_parallel.py replays each perturbed trajectory inside RLBench/CoppeliaSim and records before/after RGB frames (front, left-shoulder, right-shoulder, wrist) plus an MP4 video for every (taskvar, failure_mode, episode) tuple. This is by far the most expensive stage.

Launch (one invocation per split):

module load jq

for split in train val test; do
  dataset="$split" \
    taskfile="assets/failure_forge/${split}/taskvars_${split}.json" \
    failfile="assets/failure_forge/failure_modes.json" \
    ./RFailureForge/pipeline/s4_run_simulation/run_mega_parallel_simulations.sh
done

A single SLURM job is submitted per (taskvar, failure_mode) pair. The wrapper run_simulation.sh loads the nvcuda_v2.sif Singularity container and runs xvfb-run for headless rendering.

Outputs (per dataset split):

data/failure_forge/data/{dataset}_dataset/
    failure_bank.jsonl                      # one row per simulated failure attempt
    records/{taskvar}/{failure_mode}/...    # rgb frames + mp4 videos

• Wall-clock: a few hours. Long-horizon tasks can take significantly more.

---

Stage 5 — Post-process the simulation data

Two stages, run in order:

1. post_process_simulation_data.py — CPU. (a) balances positive/negative samples per taskvar from raw_simulation_failure_bank.jsonl, (b) attaches the ground-truth plan (sequence of subtasks) to each sample, and (c) attaches heuristic start_caption / end_caption strings describing what the gripper is doing at the start and end of each subtask. For the execution config each failure mode (no_close, wrong_object, translation, …) gets its own end-state caption; for planning the captions are static. Produces metadata_{config}.jsonl (balanced) and metadata_{config}_positives.jsonl (positives only, consumed by s5b), one of each for config ∈ {execution, planning}.
2. add_visible_objects.py — GPU. Populates each sample's visible_objects field by prompting InternVL3-38B with the start frame and the plan's target objects. Loads InternVL3-38B, hence the GPU.

Run order:

# 1. CPU step (balance + plan + captions) — one job per (dataset, config)
for split in train val test; do
  for cfg in execution planning; do
    DATASET=$split CONFIG=$cfg sbatch RFailureForge/pipeline/s5_post_process_simulation_data/post_process_simulation_data.sh
  done
done

# 2. GPU step (visible objects via InternVL3-38B) — same loop
for split in train val test; do
  for cfg in execution planning; do
    DATASET=$split CONFIG=$cfg sbatch RFailureForge/pipeline/s5_post_process_simulation_data/add_visible_objects.sh
  done
done

s5b — Generate planning failures

Execution failures come "for free" out of s4 because RLBench can simulate them. Planning failures cannot be simulated — they require us to mutate the plan while keeping the world state consistent. This is what s5b does.

This stage must run in guardian_transformers-upgraded, not the default guardian environment. The script loads Mistral-Small-3.2 with Mistral3ForConditionalGeneration, which requires the upgraded Transformers version documented in INSTALL.md.

1. Input: metadata_planning_positives.jsonl (positives-only file emitted by s5).

2. Build planning failures — build_planning_failures.py (formerly two scripts: amplify_plan.py + planning_failure_forge.py) loads Mistral-Small-3.2-24B once, then runs both phases back-to-back:

   • Amplify: rewrite each positive plan into a more complex variant, growing the positive set. Intermediate file: metadata_planning_positives_amplified.jsonl.
   • Perturb: generate one negative per amplified positive by randomly picking one of five perturbation classes:
     • missing subtask (mechanistic)
     • wrong object manipulated (LLM rewrite of the plan)
     • wrong placement or state (LLM rewrite of the plan)
     • wrong order (mechanistic swap)
     • contradictory subtasks (LLM rewrite of the plan)

   Final output overwrites metadata_planning.jsonl — balanced 1:1 positive/negative — so s5c reads the planning-failures-augmented file.

Launch (one job per split):

for split in train val test bdv2fail_train bdv2fail_val bdv2fail_test; do
  DATASET=$split sbatch RFailureForge/pipeline/s5b_add_planning_failures/build_planning_failures.sh
done

• Wall-clock: ~2h end-to-end on 3 GPUs for the train split (≈1h30 amplify + ≈30 min perturb).

s5c — Convert to InternVL format

s5c_convert_metadata_to_internvl_format/convert_metadata_to_internvl_format.py reads metadata_{config}.jsonl and emits internVL_dataset_{config}_{config_name}.jsonl, the multi-turn chat format consumed by internvl_chat_finetune_w_video.py.

The set of supported config_names is defined in assets/failure_forge/configs/configs.yaml and includes:

vanilla, thinking, thinking_svp, thinking_concat, thinking_concat_svp,
vanilla_concat, vanilla_svp, vanilla_svp_concat,
config_svp, config_dvp, config_tvp, config_concat, config_video_t1,
config_sentinel, config_sentinel_tvp, ...

Each config controls things like:

• whether the prompt asks for a <think> trace (thinking vs. vanilla),
• which views are passed (svp = single-view, dvp = double-view, tvp = three-view, concat = views concatenated into one image),
• whether the input is a video clip or before/after frames.

A typical invocation iterates over splits and configs:

CONFIGS="thinking_svp"
DATASETS="bdv2fail_train bdv2fail_val bdv2fail_test \
          rlbenchfail_train rlbenchfail_val rlbenchfail_test \
          ur5fail_test robofail robovqa"

for dataset in $DATASETS; do
  for config_name in $CONFIGS; do
    python RFailureForge/pipeline/s5c_convert_metadata_to_internvl_format/convert_metadata_to_internvl_format.py \
      --dataset "$dataset" \
      --config_name "$config_name" \
      --config_path assets/failure_forge/configs/configs.yaml \
      --execution_only
  done
done

The output internVL_dataset_{config}_{config_name}.jsonl files are the actual training/evaluation splits referenced from the finetuning_data_*.json meta files (see Training doc).

---

Stage 6b — Generate Chain-of-Thought reasoning traces

The teacher is InternVL3-38B. Both the execution and planning configs go through the same six-step recipe; run each step for both configs.

1. Inject CoT-generation instructions into the prompt

1_prepare_prompt_gen_cot.py rewrites each sample so the assistant is asked to produce a structured <think>...</think> trace alongside the boolean answer and the failure category. Output: internVL_dataset_{config}_config_multiclass_cot_instructions_for_generation_{version}.jsonl.

DATASET=train      sbatch RFailureForge/pipeline/s6_gen_cot/1_prepare_prompt_gen_cot.sh
DATASET=bdv2fail_train sbatch RFailureForge/pipeline/s6_gen_cot/1_prepare_prompt_gen_cot.sh

2. Shard for parallel generation

Running the teacher VLM to generate CoT is expensive. To run faster, we suggest you split the workload across jobs. For this purpose, split each jsonl into N parts (one per GPU) with split -n l/N. Run for both execution and planning:

CONFIG=execution VERSION=vClosedForm
for d in train bdv2fail_train; do
  cd data/failure_forge/data/${d}_dataset && \
    split -n l/3 -d --additional-suffix=.jsonl \
      internVL_dataset_${CONFIG}_config_multiclass_cot_instructions_for_generation_${VERSION}.jsonl \
      internVL_dataset_${CONFIG}_config_multiclass_cot_instructions_for_generation_${VERSION}_part && \
    cd -
done
# repeat with CONFIG=planning

3. Run the teacher VLM

2_launch_gen_cot_parallel.sh submits one SLURM job per shard.

#SBATCH -A <YOUR_SLURM_ACCOUNT>
#SBATCH -C h100
#SBATCH --qos=qos_gpu_h100-t3
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=96
#SBATCH --time=20:00:00

Output: data/failure_forge/eval_results/InternVL3-38B/{config}_config_multiclass_cot_instructions_for_generation_{version}_part{00,01,02}/DATASET_{dataset}.jsonl.

4. Recombine the shards

cd data/failure_forge/eval_results/InternVL3-38B
for dataset in train bdv2fail_train; do
  for config in execution planning; do
    for version in vClosedForm; do
      base=${config}_config_multiclass_cot_instructions_for_generation_${version}
      mkdir -p "$base"
      cat ${base}_part{00,01,02}/DATASET_${dataset}.jsonl > ${base}/DATASET_${dataset}.jsonl
    done
  done
done

5. Re-attach the generated CoT to the original samples

3_recombine_original_sample_w_cot.py merges the teacher's output back onto the original sample. Output: internVL_dataset_{config}_config_multiclass_cot_{version}.jsonl.

DATASET=train      sbatch RFailureForge/pipeline/s6_gen_cot/3_recombine_original_sample_w_cot.sh
DATASET=bdv2fail_train sbatch RFailureForge/pipeline/s6_gen_cot/3_recombine_original_sample_w_cot.sh

6. Shorten / clean the noisy LLM output

Why this step? The teacher (InternVL3-38B) tends to over-justify each step of its <think> trace with verbose tail clauses ("…is logical because …", "…is logical if …", "**: this step is…"). At training time these long tails (a) inflate token count and shift loss mass onto boilerplate rather than the discriminative reasoning, (b) bias the 8B student toward copying the teacher's verbosity, which hurts inference latency and the </category>-stop early-exit used at eval. 4_shorten_closed_form.py applies a handful of regex rewrites that collapse those tails to their closed-form prefix (e.g. "is logical because X" → "is logical") without touching the boolean answer or the category. It loops over execution and planning internally and overwrites the input file in place.

DATASET=train      sbatch RFailureForge/pipeline/s6_gen_cot/4_shorten_closed_form.sh
DATASET=bdv2fail_train sbatch RFailureForge/pipeline/s6_gen_cot/4_shorten_closed_form.sh

The final output is internVL_dataset_{config}_config_multiclass_cot_vClosedForm_short.jsonl, referenced by the training meta files (see Training).

---

Where the released datasets fit in

The artefacts you can download from Hugging Face correspond to the outputs of s5 / s5b / s5c / s6 for the four environments we cover:

Released dataset	Pipeline output
RLBench-Fail	RLBench traversal of s0–s6
BridgeDataV2-Fail (bdv2fail_*)	s5–s6 on BridgeData V2 demos
UR5-Fail	s5–s6 on UR5 real-robot demos
RoboFail	s5c on the public RoboFail dataset

If you only need the fine-tuning data, you can skip s0–s4 and download the prepared splits from Hugging Face. You only need to re-run s0–s4 if you want to extend the dataset to a new RLBench taskvar or to a new failure mode.

---

Next steps

• Train Guardian on the resulting jsonl files: see Training_The_VLM_Failure_Detector.md.
• Evaluate Guardian (and baselines) on the resulting test splits: see Offline_VQA_Evaluation.md.