SkillGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

Paper

SkillGen Overview Video

SkillGen is a system that trains proficient robotic agents with minimal human effort

First, a human teleoperator first collects ~3 demonstrations of the task and annotates the start and end of the skill segments, where each object interaction happens. Then, SkillGen automatically adapts these local skill demonstrations to new scenes and connects them through motion planning to generate more demonstrations. These demonstrations are used to train Hybrid Skill Policies (HSP), agents that alternate between closed-loop reactive skills and coarse transit motions carried out by motion planning.

SkillGen integrates learned policies and planning to solve real-world manipulation tasks with just 3 human demonstrations

SkillGen was used to generate 100 demonstrations automatically from just 3 human demonstrations. An HSP-Class agent was trained on the data and achieves 95% success rate on the Cleanup-Butter-Trash task.

SkillGen outperforms a state-of-the-art data generation system. It also achieves comparable performance to state-of-the-art imitation learning with 3% of the human data and fewer assumptions

On the challenging Coffee task, real-world SkillGen policies outperform those trained on MimicGen (a SOTA data generation system) data (65% vs. 14%). It achieves comparable results to HITL-TAMP (a SOTA imitation learning method achieving 74%) while using just 3% of the human data and making fewer assumptions.

SkillGen enables trained agents to be robust to large pose variation

An agent trained on SkillGen data achieves 95% success rate on the Pick-Place-Milk task, over a wide range of milk and bin placements.

SkillGen enables zero-shot sim-to-real transfer with 1 human demonstration provided in simulation

The SkillGen simulation policy achieves 35% success rate on the challenging Nut Assembly task in the real world.

The single human demonstration used for SkillGen.

1000 generated SkillGen demonstrations.

SkillGen agent evaluated in simulation.

SkillGen can generate data in cluttered workspaces

SkillGen can generate data even when large obstacles that were unseen in the human-provided source demonstrations are present.

Pick-Place-Bin

Cleanup-Butter-Trash

SkillGen data can produce near-perfect agents across a wide variety of tasks

We train Hybrid Skill Policies (HSP) on SkillGen datasets. These agents learn skill initiation conditions, closed-loop skill policies, and skill termination conditions. The skills (red border) are sequenced via motion planning (blue border).

Square D2 (94%)

Coffee D2 (100%)

Piece Assembly D2 (84%)

Threading D1 (84%)

Nut Assembly D1 (78%)

Coffee Prep D2 (84%)

Agents trained on SkillGen demonstrations can be comparable to agents trained on comparable amounts of human data

We generate 200 SkillGen demonstrations from 10 human demonstrations and compare trained agent performance with using 200 human demonstrations. Results are similar, but SkillGen requires a small fraction of the human effort.

Using 200 SkillGen demos generated from 10 human demos results in 36% success on Threading D1.

Using 200 human demos results in 32% success on Threading D1.

Generating larger SkillGen datasets is an easy way to boost performance without human effort -- agents demonstrate strong improvement with larger SkillGen datasets

We compare agent performance on 200, 1000, and 5000 SkillGen demos. Agents can improve substantially when using more generated data -- Threading D1 improves from 36% to 84% and Square D2 improves from 4% to 72%.

Threading D1, 200 demos, HSP-TAMP (36%)