Six Questions to Ask a LeRobot Dataset Before You Train

The metrics worth computing on a LeRobot dataset before you train a policy or fine-tune a VLA: what each one catches, when to trust it, and where it misleads you.

You recorded 400 teleop episodes last week. You pulled 6,000 more from the Hub because the task looked close to yours. Now you want to fine-tune a VLA.

You can't watch all of it. At 30 seconds per episode, 10,000 episodes is about 83 hours of video. So you scrub a handful, they look fine, and you hit train. Then the policy hesitates at the first grasp.

Somewhere in there is an episode with a frozen wrist camera, one with 11 seconds of dead time before the arm moves, one whose task string is "Hold", and one whose video runs 0.4 seconds behind the actions. None of that is visible in a single episode. It shows up when you compare an episode to its peers.

TL;DR

• Six questions cover most problems: is it broken, is the motion clean, is it wasting frames, did the robot do what was commanded, can you trust the cameras, can a model learn from the instruction. Start with the cheapest.
• Almost everything is signal math on action and observation.state. No model, no GPU. Only camera metrics decode video, at about 0.4 s per camera per episode.
• Compare each episode to its own task group. A pooled score ranks the slowest task as the problem.
• Correlated metrics share one vote. Five time metrics that fire on the same pause are one problem.
• The evidence is mixed. Use scores to order a review queue, tag what you want to drop yourself, and validate with a training run.

See it first

This is the panel for the first 23 episodes of T-Rex (zekaiwang/trex_dataset, MIT), a bimanual robot with 22-joint dexterous hands, scored with a dual-arm layout and three cameras. Every panel screenshot in this post comes from that run.

Overview tab. Top row: score histogram with warn and fail lines, verdict counts, episodes per task. Below: outliers and the worst-first ranking with Score, Flags, and Driver. The amber banner is expected: each of these 23 episodes has its own task string, so all are scored against the whole view.

• The panel follows the view. Filter the grid and every chart, and the ranking recomputes.
• Every bar is a filter. Click one and the grid shows only those episodes.
• The ranking says why. Driver names the group behind the score, Flags counts groups at warn or worse, and an asterisk marks scores normalized against the whole view.
• Nothing is deleted. The footer tag buttons write ordinary sample tags.

Overview tab. Click a red bar in the score histogram to filter the grid, read Score, Flags and Driver in the ranking, and switch normalization groups with the Task dropdown. Footage: HIW-500: Humanoids In-the-Wild (BitRobot/HIW-500-LeRobot), CC BY 4.0.

Pick your metrics

Inside a score, every metric is oriented so higher means worse. The tables show each metric's raw direction.

The format

• A LeRobot v3 dataset stores many episodes per Parquet file and per MP4. Metadata (meta/episodes/) says which rows and which video time window belong to episode N.
• So "episode N" is a pointer into shared files. Pointers drift without anything crashing.
• Timestamps are written from the nominal fps, not a logged clock. Sensor-health checks like jitter and clock drift don't apply. You check structure and alignment instead.
• FiftyOne (1.22+) imports each episode as one sample and reads Parquet rows and video frames on demand.

What you tell the tool

Nothing about your robot is guessed. A 14-dimensional action could be two 7-DoF arms, an arm plus a hand, or end-effector deltas. A silent guess makes every smoothness score wrong in a way you can't see.

Joints are chips, picked the same way as cameras, each offered by name with its observed range beside it (a 0 to 100 range often marks a gripper). Nothing is grouped for you. If the dataset gives no joint names, joints are listed by position (action[0], action[1], ...), and the range tells them apart. A checklist at the bottom of the form shows which metric families will be scored and what each switched-off family still needs. It never blocks the run.

Compute form, Data tab. Joints are chips, and the checklist under them says what will be scored and what is still off (here, gripper metrics, because no gripper joints were picked).

Compute form, Data tab. Confirm the arrays, pick arm and gripper joints (the observed range shows beside each), and finish on the "Will be scored / Off" list.

The metrics

1. Integrity: is it broken?

Integrity has its own verdict, broken or not, and never enters the score. It reads only metadata and Parquet, so run it first.

Figure 1. A v3 episode is a row range in a shared Parquet file plus a time window in a shared MP4 per camera. Each check catches a pointer that stopped matching the data. Illustrative values.

The Hugging Face team's write-up on community datasets lists these as recurring defects: single-frame episodes, Parquet files deleted without reindexing, inconsistent action or state dimensions.

timestamp_dev is reported for information only and never affects the verdict, since timestamps come from fps.

2. Motion: is the motion clean?

Reads action over the joints you marked as Arm (and Hand). On leader-follower rigs (SO-100, Koch, ALOHA), that's the operator's hand, not the robot arm.

How the signal is prepared:

• Each joint is range-normalized before speeds are combined, so units don't matter.
• Gripper and hand dimensions are left out of arm speed. A 22-joint hand is its own signal and never swamps a 7-joint arm.
• Low-pass cutoff is min(10 Hz, 0.4 × fps), so 30 fps data isn't analyzed near Nyquist.
• Fixed-length windows, with a worst-window value kept so one bad stretch isn't averaged away.

SPARC intuition. A clean reach is one speed bump, and its spectrum is compact. Every correction or stop adds frequency content and lengthens the traced path. SPARC is the negative of that length.

Figure 2. The same two reaches, moved three ways. SPARC: smooth -2.57, stop-and-go -5.25, jittery -2.56. LDLJ: smooth -8.87, stop-and-go -14.45, jittery -14.71. SPARC separates stop-and-go. LDLJ penalizes both stop-and-go and noise. Synthetic signals at 30 fps; values computed from them.

Motion & Action tab: one histogram per metric, one colored series per signal (arm_left, arm_right, hand_left, hand_right). The chips isolate a signal, and dashed lines are warning thresholds.

sparc_phase exists because contact is jerky by nature, and scoring the whole episode as one signal counts every grasp as roughness.

3. Time efficiency: is it wasting frames?

The idle threshold is a multiple of the episode's own moving speed, so a slow, careful operator isn't flagged for being slow.

Figure 3. Left: one episode's arm speed with the three idle stretches and the keep window they suggest. Right: length_z places a duration against its task group. Synthetic signals.

Two published pipelines treat dead frames as removable: OpenVLA dropped all-zero no-op actions, and SCIZOR removed samples including "manipulation failures, slow motion, and pauses."

4. Tracking and contact: did the robot follow the command?

On a leader-follower rig, action is the leader arm and observation.state is the follower. When they diverge, something physical happened: a collision, a stall, a slipping joint. These metrics only run when you've set the action to joint positions in the state space.

Figure 4. The follower trails the leader by a few frames (right), then stalls while the command keeps moving. The residual lights up at the stall, and the catch-up produces an acceleration spike. Synthetic signals.

5. Gripper: is the operator fighting the grasp?

Figure 5. Chatter scores 0.88 flips per second, compared with 0.25 for a clean grasp. The regrasp has more transitions than the clean grasp and is tagged as a recovery instead of penalized. Synthetic signals.

Several clean cycles are often an operator correcting a failed grasp, which is behavior a policy may want to learn. That's why recovery is never penalized.

6. Consistency: does it agree with its peers?

Belkhale et al. (NeurIPS 2023, arXiv:2306.02437) call this action divergence and report that "making actions more consistent tends to increase policy success."

Figure 6. Each arrow is the action another episode took from a state near this one's. Variance: 0.00 on the left, 0.59 on the right. Schematic, in a 2D stand-in for the state space.

7. Camera: can you trust the pixels?

Opt-in, because it decodes video. No model, just pixel and signal arithmetic. Every value is stored per camera and compared with the same camera in other episodes, so a wrist camera is never judged against a top camera. An episode's camera score is its worst camera.

Figure 7. The frame metrics on a synthetic scene, computed at the tile's resolution. The plugin measures at a fixed size and compares each camera with itself across episodes. A live camera on a static scene still shows frame-to-frame change; a frozen feed shows none.

• frozen_frac: a still scene with a still robot isn't a frozen camera, so a burst only counts when the robot is moving and the pixels aren't. A live camera on a static scene typically shows 0.3 gray levels of frame-to-frame change or more (the synthetic scene in Figure 7 shows 0.8), so the 0.05 threshold sits well below it.
• video_action_lag_ms: the one integrity can't catch. If video runs 400 ms behind actions, every file is intact, and the durations still match, but the policy learns to predict from a stale observation. The value is relative to peers, so the baseline is the lag every episode shares. Nothing is reported when video doesn't track the action (correlation under 0.3).

Figure 8. Video motion energy trails arm speed by 400 ms here. Across episodes from one camera, the shared lag is the baseline, and the differing episodes stand out. Synthetic signals.

Vision tab: the five camera metrics with one colored series per camera, and a worst-first table over just those metrics.

Vision tab. Compute camera metrics, isolate one camera with its chip, click a histogram bar to filter the grid, and open a flagged episode in the viewer. Footage: HIW-500: Humanoids In-the-Wild (BitRobot/HIW-500-LeRobot), CC BY 4.0.

Cost. The three frame metrics share one decode of 12 frames. The lag pass costs about 0.9 s per episode across all cameras on the development set. Budget about 0.4 s per camera per episode.

8. Language and outliers

Task strings are canonicalized (case, whitespace, punctuation) before grouping. Language flags get their own verdict because a policy trained without language can ignore them, while a VLA user reads them next to the score. We haven't measured whether generic instructions hurt VLA grounding in anything we reviewed. The check exists because the Hugging Face team lists them among the recurring defects in community datasets.

Outliers never score. The length-confound audit (arXiv:2606.10229, more on it in the next section) found isolation-forest curation matched no curation, and a "weird" episode may be your most valuable edge case.

Evidence at a glance

From metrics to a ranking

A ranking needs two decisions: compared to what, and how to combine them.

Compared to what. Episodes are grouped by canonicalized task. Groups with 20 or more episodes are scored against themselves. Smaller groups are pooled over the view and marked low-confidence in the panel. Scores are robust z-scores with a floored scale, and zero-inflated metrics (like idle_lead_s) use a tail-based scale, because a standard deviation over mostly zeros is meaningless.

Figure 9. A slower, smaller task (B) pooled with a larger one (A). Pooled, the warn line sits near A's distribution and flags 26 of B's 30 episodes for being task B. Per task: 2 of 120 in A and 1 of 30 in B. Synthetic data, robust z with a median and MAD scale.

Combined how. Correlated metrics share a group, and a group is one vote.

• Inside a group, the value is the maximum weighted z. Weights are 1.0, except ldlj, idle_trail_s and end_motion_ratio at 0.5.
• The episode score is the maximum group value, so one severe problem can't be averaged away by five fine ones. A frozen camera with perfect motion still ranks near the top.
• n_flags counts groups at or above the warn line (z = 2). driver names the group behind the score.

Figure 10. An example episode with made-up z-scores. Every group is fine except camera, where frozen_frac is at 4.3.

Quick start

1. Get a dataset locally. The plugin reads local LeRobot v3 datasets, so download Hub datasets first:

   pip install "fiftyone>=1.22"
   export VFF_MULTIMODAL=1   # enables the multimodal episode viewer; set it in every process that touches the dataset
   hf download <user>/<dataset> --repo-type dataset --local-dir /data/lerobot/my_dataset
   

   A v2.x dataset needs converting first with lerobot's convert_dataset_v21_to_v30 script; the importer rejects it otherwise.

2. Import it. One sample is one episode.

   import fiftyone as fo
   
   dataset = fo.Dataset.from_dir(
       dataset_dir="/data/lerobot/my_dataset",
       dataset_type=fo.types.LeRobotDataset,
   )
   dataset.add_dir(  # optional: curate across datasets
       dataset_dir="/data/lerobot/other_dataset",
       dataset_type=fo.types.LeRobotDataset,
   )
   session = fo.launch_app(dataset)
   

3. Compute quality. Install the plugin with fiftyone plugins download https://github.com/harpreetsahota204/lerobot_data_curation, open the operator browser, and run LeRobot curation: compute quality. It runs delegated, in the background. Make your picks on the Data tab, optionally pick cameras on the Camera tab, then run. It remembers your picks for the next run.

4. Read the ranking. Open a panel and choose LeRobot Curation. It re-ranks as you filter the grid. Use the driver column to pick a tab, and click any bar to filter the samples.

5. Inspect. Click a row to see every metric with its per-arm or per-camera breakdown, joint traces, speed profile, gripper timeline, and frames. Flagged spans become temporal tags on the episode timeline.

The inspector for the top-ranked episode: score, driver and verdicts, flagged spans as chips, every metric with its per-arm breakdown, and joint traces.

Inspector. Flagged-span chips, per-arm metrics, joint traces, and the gripper timeline, then the same spans as temporal tags on the multimodal viewer timeline. Footage: HIW-500: Humanoids In-the-Wild (BitRobot/HIW-500-LeRobot), CC BY 4.0.

6. Tag, then export. The footer tags review, exclude-candidate and relabel. Nothing is deleted, and marking exclude-candidate is always your call.

   keep = dataset.match_tags("exclude-candidate", bool=False)
   keep.export(
       export_dir="/data/lerobot/my_dataset_curated",
       dataset_type=fo.types.LeRobotDataset,
       export_media=True,
   )
   

   The native exporter writes a self-contained v3 dataset and renumbers episodes. export_media=True is required. Push it to the Hub like any LeRobot dataset.

Tag episodes exclude-candidate in the panel, then export the rest as a LeRobot dataset from a notebook.

What the plugin doesn't do

It's heuristic-only. These model-based methods need a model, a GPU, or a policy, so they aren't included. FiftyOne has the embeddings and similarity and model zoo infrastructure to build them as extra operators.

CUPID's own warning applies to every heuristic here: "human perception of demonstration quality does not necessarily correspond to data that maximizes downstream policy success."

Dataset-level questions aren't per-episode either: task coverage (1,900 episodes of one task and 12 of another isn't balanced), source balance, diversity, and mixture weights (Re-Mix, Hejna et al., CoRL 2024, reports beating uniform weights by 38%). The panel shows coverage as a count on the Integrity & Coverage tab.

Keep it honest

• Scores order a review queue. Use them to decide where to look, never as an automatic accept/reject gate.
• The evidence is mixed. Smoothness filtering helped in one study. Another found most metrics exploited episode length. CUPID found data humans rate highly can underperform data chosen by influence on the policy.
• Thresholds are starting points. The 2% joint-limit margin, the warn line at z ≥ 2, the 20-episode group size, and the camera thresholds were calibrated on a 102-episode development set: the first two episodes from each of the 51 sources in Voxel51/community_v3_10per_embodiment. They aren't adjustable in the form yet. Check them on your data.
• What the checks prove. Each scored metric must catch a corruption (a stalled joint, a frozen video, a blanked task) on real episodes, and every metric is re-tested after truncating episodes to a common length. On 102 episodes, that takes about 5 minutes. It shows a metric detects what it claims to. It doesn't show that detection predicts policy quality.
• Validate. Before you drop data, train on the curated subset and on the full set and compare.

Links

• LeRobot curation plugin: harpreetsahota204/lerobot_data_curation
• Try the LeRobot curation plugin in a Hugging Face Space
• Screenshots: T-Rex, zekaiwang/trex_dataset (MIT), first 23 episodes. GIF footage: HIW-500: Humanoids In-the-Wild, BitRobot/HIW-500-LeRobot (CC BY 4.0).
• Figures from this post, as image files: harpreetsahota/lerobot-curation-metrics-figures. If you reuse them, please credit and cite them. The dataset card has a caption line and a BibTeX entry.
• LeRobot dataset v3 format: https://huggingface.co/docs/lerobot/en/lerobot-dataset-v3
• Hugging Face on community dataset defects: https://huggingface.co/blog/lerobot-datasets
• FiftyOne multimodal and LeRobot docs: https://docs.voxel51.com/user_guide/multimodal.html
• Build your own plugin: https://docs.voxel51.com/plugins/developing_plugins.html