Large-Scale Robotics Data Collection with UMI-Style Grippers

(Additional contributors, alphabetically: Laz Kopits, Ryan Leong, Sheel Taskar, Daryl Yang, William Zang)

Building robust robotic manipulation policies requires diverse, high-quality demonstration data at scale. The challenge lies not just in collecting data, but in doing so reliably across multiple devices while maintaining temporal synchronization and handling the inevitable network disruptions of real-world deployments. This post details the system architecture behind the SF Fold dataset, a large-scale robotics manipulation dataset collected using UMI-style grippers.

Our Goal: Take the UMI concept from research prototype to production-ready hardware: a device anyone can pick up and use without specialized expertise, seamlessly integrated into cloud-based post-processing pipelines for scalable data collection.

Dataset: huggingface.co/datasets/zeroshotdata/sf_fold

Hardware Setup

The data collection system builds on the UMI (Universal Manipulation Interface) approach, using UMI-style still grippers equipped with OAK-D Wide cameras for 6-DoF pose tracking via ORB-SLAM3 in stereo mode. The setup supports bimanual manipulation tasks through a multi-puppet configuration: Left Puppet, Right Puppet, and Ego Puppet. Throughout this document, we refer to the data collection devices as “puppets”.

Button Controls: Long press creates/closes datasets, short press starts recording episodes. Color-coded LEDs display current system status.

OAK-D Wide Camera

The OAK-D Wide provides stereo vision for 6-DoF pose tracking using ORB-SLAM3. The camera is mounted pointing upward on each gripper to avoid occlusions during manipulation tasks.

Interactive 3D model of the OAK-D Wide camera. Click and drag to rotate, scroll to zoom.

System Architecture

The system follows a distributed architecture where multiple puppets coordinate through a central MQTT broker running on the Ego Puppet. This enables synchronized multi-view data collection while maintaining robustness against network failures.

Architecture showing Right, Ego, and Left Puppets communicating via the central EgoBroker. Components include MQTTCommandListener, ButtonNode, and sensor_app. (Click to enlarge)

Puppet Communication

Puppet interaction follows a state machine: ON → DATASET_ACTIVE → RECORDING → UPLOADING

Each puppet runs three core components:

• MQTTCommandListener: Receives start/stop commands, triggers local recording scripts
• ButtonNode: Handles physical button input on Right Puppet, broadcasts commands
• sensor_app: Publishes sensor data and system status

When a user presses the button on the Right Puppet, the ButtonNode publishes to the Ego Broker, which distributes synchronized commands to all puppets, ensuring simultaneous state transitions and episode alignment.

MQTT Broker Setup

The system implements a dual-broker architecture balancing reliability with discovery:

Key Design: The cloud broker serves only as a discovery mechanism. All data recording and episode management happens through the local broker, allowing fully offline operation after initial setup.

Data Collection Flow

The data flow spans from physical demonstration through cloud storage, with multiple checkpoints for reliability.

Complete data flow between Demonstrator, Puppet, MQTT Brokers, Flask API, Cloud SQL, GCS, and Pub/Sub. (Click to enlarge)

Status Publishing

Each puppet continuously publishes status (IP, health metrics) to the Cloud MQTT Broker, enabling system monitoring and discovery.

Acquisition Flow

When the demonstrator presses the start/stop button:

1. Button press publishes command to local MQTT broker
2. Command triggers acquisition scripts on all connected puppets
3. Each puppet records sensor data with precise timestamps

Upload Mechanism

After recording completes, uploads follow a secure, verifiable sequence:

Reliability and Synchronization

Collecting synchronized multi-view data across distributed devices requires robust handling of network failures.

Reconnection Mechanism

TCP connections can become stale without detection, causing silent disconnections. The system employs two mechanisms:

Failure Behavior: When connections fail, puppets continue recording locally while tracking disconnection events. LED shows CONNECTING_TO_BROKER (pulsing white) during recovery.

Episode Synchronization

When a puppet disconnects during recording, it misses episode transitions, causing numbering to become unsynchronized. The reconnection mechanism operates in three phases:

ego/episode_timestamps.txt:
Episode 1 started at: 1740348759.123456789
Episode 2 started at: 1740348852.987654321
Episode 3 started at: 1740348912.456789123
Episode 4 started at: 1740348975.321654987
Episode 5 started at: 1740349032.789456123
Episode 6 started at: 1740349102.654789321

Episode 1 started at: 1740348759.123456789
# DISCONNECTED FROM BROKER at: 1740348800.123456789
# RECONNECTED TO BROKER at: 1740348920.654321987
Episode 3 started at: 1740348925.789123456
Episode 4 started at: 1740348975.321654987
# DISCONNECTED FROM BROKER at: 1740349000.123456789
# RECONNECTED TO BROKER at: 1740349040.654321987
Episode 5 started at: 1740349045.789123456
Episode 6 started at: 1740349102.654789321

Timing and Clock Synchronization

Precise timing is crucial for multi-device synchronization. When a new episode starts, the system records the exact timestamp to episode_timestamps.txt. Episode boundaries are determined by comparing each data point’s timestamp with recorded start times.

Clock Sync: All devices use Chrony with public NTP servers, providing sub-millisecond accuracy for multi-device coordination.

Sample Data Visualization

The dataset includes interactive 3D visualizations using Rerun, allowing exploration of recorded episodes with synchronized multi-view camera feeds, gripper poses, and sensor data.

Dataset and Usage

The SF Fold dataset is available on Hugging Face:

Download: huggingface.co/datasets/zeroshotdata/sf_fold

The dataset focuses on folding and manipulation tasks, providing synchronized multi-view recordings suitable for:

For more details on the original UMI approach that inspired this work, see the paper and project page.