HuRo: Robotizing Human Videos
for Scalable VLA Pretraining

Jinho Jeong1,2,*, Se June Joo1,*, Jaehyun Kang1, Dongyun Kim2, Yena Kim2, Hanjung Kim2, Seon Joo Kim2

1 RLWRLD    2 Yonsei University

* Equal contribution

Conference on Robot Learning (CoRL) 2026

Paper arXiv Code Datasets
TL;DR

Can we scale VLA pretraining with human videos converted into robot observations and actions? We build a robotization pipeline that turns heterogeneous egocentric human videos into robot-aligned observations and retargeted actions, and use it to construct the HuRo dataset: 630K robotized episodes and 142M frames. Scaling this data improves real-world performance under both in-distribution and out-of-distribution conditions, with the largest gains in the out-of-distribution setting.

HuRo overview

Overview. Large-scale human videos are transformed into VLA episodes with robotized observations, language instructions, robot states, and actions. VLA policies pretrained on larger HuRo subsets achieve higher real-world manipulation performance after downstream finetuning.

Dataset scale
630K
Robotized episodes
142M
Processed frames
1,317h
Video at 30 fps
5
Human-video sources
Real-world impact (0% to 100% pretraining)
51.5 → 80.3
Overall completion (%)
68.1 → 88.4
In-distribution (%)
34.9 → 72.2
Out-of-distribution (%)

Frame share: EgoDex 55%, EgoVerse 27%, Ego4D 10%, Ego10K 6%, EPIC-Kitchens 2%.

Comparison with Prior Work

HuRo scales jointly robot-aligned observation-action data from diverse human videos for end-to-end VLA pretraining.

Prior robotized-video pretraining

  • Jointly robotized data from target-task demonstrations
  • Aligns observations or actions, not both at scale
  • Uses robotized videos for visual pretraining with auxiliary objectives
  • Limited-scale pretraining data

HuRo (ours)

  • Robotizes everyday human videos from five egocentric datasets
  • Jointly robot-aligned observations and retargeted actions
  • End-to-end VLA pretraining on robotized observations and actions
  • Larger scale: 142M frames / 630K episodes

Robotization Examples

The visible human is removed, a rendered ALLEX humanoid is overlaid, and human hand motion is retargeted into robot actions. Egocentric human videos become robot-aligned observations paired with actions and language.

Input · Human videos
Output · Robotized (ALLEX overlay)

Robotized observations across diverse scenes from five egocentric human-video sources.

Robotization Pipeline

Three stages convert an egocentric human video into a robotized episode, estimating any intermediate signals a source is missing so that heterogeneous videos map into one robot format.

HuRo construction pipeline
1

Human video annotation

Off-the-shelf models fill in missing camera geometry and 3D hand poses, split the video into manipulation chunks, and caption each chunk with a VLM.

2

Action conversion

The annotated hand motion is retargeted to a robot joint trajectory, yielding robot states and actions.

3

Visual conversion

The visible human is segmented and inpainted away, then a robot rendered in simulation is overlaid onto the cleaned scene.

We instantiate HuRo with ALLEX (a bimanual dexterous humanoid) and pretrain a VLA policy, then finetune on a small set of real-robot demonstrations.

Real-World Results on Four ALLEX Tasks

Apple Pick-and-Place, Cup Stacking, Cup-Noodle Handover, and Microwave Loading, evaluated across in-distribution (ID) and out-of-distribution (OOD) conditions with spatial and visual shifts.

Main results on four real-world tasks
1

Robotized pretraining scales real-robot performance

Downstream completion rises consistently with pretraining scale under both ID and OOD. At full scale, HuRo pretraining also outperforms the π0.5 and GR00T N1.6 reference models under both ID and OOD.

2

Visual robotization improves OOD robustness

With the same source clips and the same retargeted action supervision, a 10% overlay subset outperforms the 100% no-overlay variant on OOD (59.5% vs. 55.7%).

Example rollouts

Apple Pick-and-Place
Cup Stacking
Cup-Noodle Handover
Microwave Loading

Additional task: Diverse Pick-and-Place

Beyond the four main tasks, Diverse Pick-and-Place requires sequentially grasping two objects that demand different grasp types and placing them into a basket. It is tested under object and spatial shifts.

3

End-to-end pretraining outperforms visual-only transfer

Pretraining observations and retargeted actions end-to-end substantially outperforms PT (Visual Only), which keeps the pretrained visual pathway but reinitializes the action head. PT (Visual + Action) reaches 61.1% ID / 50.0% OOD and produces demonstration-consistent grasps where No PT and PT (Visual Only) grasp imprecisely.

Effect of end-to-end VLA pretraining

Additional Experiments

See the paper for detailed evaluation settings and further analysis.