Dobb·E

Dobb·E: An Open-Source Framework for Household Robotic Manipulation

Dobb·E is an innovative open-source framework designed to teach robots how to perform household tasks using imitation learning. This framework allows robots to learn new tasks in approximately 20 minutes, making it a fast and efficient solution for integrating robots into home environments.

Key Features:

• Open-Source: Dobb·E is freely available, allowing researchers and developers to contribute to and improve the framework.
• Imitation Learning: The framework uses imitation learning, enabling robots to learn by watching and mimicking human actions.
• Rapid Learning: Robots can learn new household tasks in about 20 minutes.
• High Success Rate: Dobb·E achieves an average success rate of 81% in solving various household tasks.

How Does Dobb·E Work?

Dobb·E uses a simple tool called the Stick to collect demonstrations of household tasks. The Stick is a low-cost device built from a reacher-grabber, 3D printed parts, and an iPhone. This tool is used to create the Homes of New York (HoNY) dataset, which contains 13 hours of interactions in 22 different homes across New York City.

The framework then trains a representation learning model called Home Pretrained Representations (HPR) on the HoNY dataset. HPR is a ResNet-34 model trained using the MoCo-v3 self-supervised learning objective. During deployment, HPR initializes a policy that allows the robot to perform new tasks in novel environments.

Components of Dobb·E:

1. The Stick: A tool for collecting demonstrations, built from a $25 reacher-grabber, 3D printed parts, and an iPhone.
2. Homes of New York (HoNY): A dataset containing 13 hours of interactions in 22 different homes.
3. Home Pretrained Representations (HPR): A ResNet-34 model pre-trained on the HoNY dataset.

Getting Started with Dobb·E

To start using Dobb·E, you can access the code, models, and documentation through the following resources:

• GitHub Repo: Dobb·E GitHub Repository
• Hugging Face Model: HPR Model on Hugging Face
• Paper: On Bringing Robots Home (Arxiv)

Who is Dobb·E for?

Dobb·E is ideal for:

• Robotics Researchers: Who are looking for a fast and efficient framework for teaching robots new tasks.
• AI Developers: Who are interested in contributing to and improving the Dobb·E framework.
• Home Automation Enthusiasts: Who want to integrate robots into their homes and automate household tasks.

Practical Applications

Dobb·E can be used for a wide range of household tasks, including:

• Opening and closing drawers and doors
• Picking up and placing objects
• Pouring liquids
• Operating appliances

Why Choose Dobb·E?

• Cost-Effective: Uses low-cost hardware components.
• Efficient: Allows robots to learn new tasks quickly.
• Versatile: Can be used for a wide range of household tasks.
• Community-Driven: Open-source, with contributions from researchers and developers worldwide.

Homes of New York (HoNY) Dataset Details:

The HoNY dataset includes:

• 22 homes
• 216 environments
• 5620 trajectories
• 13 hours of interactions
• 1.5 million frames

The dataset contains RGB and depth videos at 30 fps, as well as full action annotations for 6D pose of the gripper and the gripper's opening angle normalized between (0, 1).

Using the HPR Model with PyTorch Image Models (TIMM):

You can easily start using the HPR model in a couple of lines of code with TIMM:

import timm
model = timm.create_model("hf-hub:notmahi/dobb-e", pretrained=True)

Conclusion

Dobb·E represents a significant step forward in the field of home robotics, offering a practical, efficient, and accessible framework for teaching robots household tasks. By combining imitation learning with low-cost hardware and a comprehensive dataset, Dobb·E makes it easier than ever to integrate robots into home environments and automate everyday chores. With its open-source nature, Dobb·E encourages collaboration and innovation, paving the way for future advancements in robotics and AI. What is Dobb·E? It's your gateway to bringing robots home.