Feeling the Data: A Hands-On Look at 8 Tactile LeRobot Datasets

Try tying your shoes with numb fingers. You can watch your hands the whole time and still fumble it, because the information you need lives at the point of contact. Is the lace slipping? How hard am I pulling? Your eyes can't answer that. Your fingertips answer it constantly.

Most robot policies today are in the numb-fingers position. VLAs have gotten good at the "where": where's the cup, where's the gripper, where should the arm go. They're mostly blind to what's happening at the contact surface. Tactile sensors fill that gap, and a growing number of LeRobot datasets now ship with touch alongside cameras and joint state.

I loaded 8 of them into FiftyOne to see what tactile data looks like in practice: how it's stored, how it differs from one dataset to the next, and what you should check before you train on it. This post walks through what I found, with code you can run on a 240 MB dataset in a few minutes.

TL;DR

• 8 LeRobot v3.0 datasets with touch: 1,679 episodes and 1,877,818 frames, all on the Hub in FiftyOne-ready form.
• Touch comes in three shapes: gel-sensor video (7 of the 8), force/torque vectors (4), and raw taxel grids (1). Each needs a different way of looking at it.
• There's no standard key. observation.tactile is a 60-D force/torque vector in one dataset and the prefix of a taxel grid in another.
• Basic per-episode checks find real issues fast. In the Grabette dataset, one of 48 episodes has no tactile signal at all.

Why touch matters

Cameras fail at the moment of contact. The gripper's fingers block the view of the exact spot you care about. Slip doesn't look like anything until the object is already falling, but a tactile sensor picks up the shear first. A light grip and a firm one look identical in an RGB frame. So do a soft bottle and a rigid one.

The early numbers are encouraging. LeFlexiTac added a 12×32 taxel sensor to the SO-100/SO-101 arm through a single observation.tactile.primary stream. On an in-bag pen task where the camera can't see the target, vision-only ACT succeeded 7 of 30 times, and 23 of 30 with touch. FreeTacMan reports ACT going from 21% average success (vision only) to 71% with tactile input and visuo-tactile pretraining across five contact-rich tasks.

It isn't automatic, though. An open issue on the T-Rex repo trained on the full Sharpa dataset and found the offline benefit of real touch over neutral input stayed within ±0.9%. Touch helps when the task depends on contact and when the data is clean enough to learn from. The second part is a data problem, and it's the part this post is about.

What tactile data looks like in LeRobot

"Tactile" isn't one data type. Depending on the sensor, it lands in a LeRobot dataset in one of three shapes:

1. Video from camera-based gel sensors (GelSight-style). A camera inside the sensor films a soft gel deforming against the object. LeRobot stores these as ordinary video features, so they behave like any other camera stream.
2. Taxel grids from pressure pads, like 6×6 or 12×32 arrays of int16 values. They live in the parquet files, and you only see the contact patch once you render them as heatmaps.
3. Force/torque vectors: Fx Fy Fz Tx Ty Tz per fingertip or per wrist. They only make sense plotted over time next to what the robot was doing.

LeRobot gives all three a home, but it doesn't say what to call them:

Within these 8 datasets, gel video lives under observation.images.* with a different naming pattern in each family. Force/torque is observation.wrench in one family and observation.tactile in another. And observation.tactile.* is also the prefix Grabette uses for its taxel grids. None of these are wrong. They're reasonable choices made by different teams, and they mean every tactile dataset needs a quick look at meta/info.json before you write a loader.

The 8 datasets

All 8 are real-world data, recorded by teleoperation or by hand. They're published on the Hub as FiftyOne-ready LeRobot v3.0 repos under the Voxel51 org, each linked to its original source. Seven are complete copies of their source; the SharpaDex one is a single collection season.

Xense: earbuds, tape, and blocks

Xense publishes these with its own LeRobot fork, lerobot-xense. All three share one layout: 3 RGB cameras, 4 tactile video streams (a left and right pad on each gripper, 400×700), and a 20-D TCP state and action, all at 30 fps. Earbud Case is long-horizon insertion and the largest dataset here (65% of all frames). Strap Tape handles deformable tape and is the smallest tactile-video download at 0.84 GB, so it's a good place to start. Tron2 RT is short pick-and-place. One thing to watch for: the overhead camera is head in two of the sets and top in the third. None of the cards name the tactile sensor model.

Intern: pour, unscrew, insert

Three community uploads by cloudfan on a GELLO-teleoperated 7-DoF arm with a Robotiq gripper: pouring water, unscrewing a cap from a bottle a person holds, and inserting a test tube ("shiguan" is pinyin for test tube). Each episode has two RGB views, two tactile image streams (tactile_*_aug_diff), and a 6-D observation.wrench with per-axis wrench_valid flags. The shipped robot config maps only the RGB views, so out of the box the touch data goes unused. Three things to know:

• Pour episodes 13, 34, and 42 have missing force readings filled with zeros, so filter on wrench_valid, not on the zeros.
• No license is declared.
• The source MJPEG video won't play in browsers, so the FiftyOne copies are lossy H.264 re-encodes.

SharpaDex: dexterous hands, ten fingertip sensors

SharpaDex v1.0 is 28,993 episodes of bimanual dexterous teleoperation with tactile sensors on every fingertip. This copy is one season of assemble_gears_on_base: 410 episodes with raw and deformation tactile video plus a 60-D observation.tactile (left and right thumb through little, each fx fy fz tx ty tz). Computing per-fingertip contact shows the work is concentrated in two fingers. The right index finger registers contact in 61% of frames and the right thumb in 42%, while five fingertips almost never do.

Grabette: exploratory tactile sensing on a handheld gripper

Grabette is Pollen Robotics' open, low-cost handheld gripper for robot-free demonstrations, and the released version has no tactile sensing yet. This dataset is exploratory work on tactile sensing integration. The team added two off-the-shelf DFRobot tactile sensors (6×6 and 4×8 taxel grids, int16) to the right gripper and recorded 48 episodes of putting a textured cube in the correct cup at 50 fps. At the moment, it's the only dataset here with raw pressure grids, and it's small, which makes it a good one to run the code below on.

Hands-on: loading tactile episodes into FiftyOne

FiftyOne reads LeRobot v3.0 directly through fo.types.LeRobotDataset. Each episode becomes one sample with episode_index, task, length, duration, robot_type, and fps fields. The per-frame data stays in the LeRobot parquet and MP4 files, and a media_reference field points into them, so nothing gets duplicated. In the App, you scrub through an episode and the RGB cameras and tactile video streams play side by side on one timeline.

import fiftyone as fo
from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="Voxel51/grabette-tactile-full-synced-48ep",
    repo_type="dataset",
    local_dir="grabette-tactile-48ep",
)

dataset = fo.Dataset.from_dir(
    dataset_dir=local_dir,
    dataset_type=fo.types.LeRobotDataset,
    name="grabette_tactile",
)
session = fo.launch_app(dataset)

Video-shaped touch (Xense, Intern, SharpaDex) plays like any camera. Numeric touch (taxel grids and force/torque vectors) shows up as per-dimension series you can chart in a Plot tile or step through in a Message tile. A 6×6 grid becomes 36 lines, not a heatmap.

What you can't do yet is filter on it. FiftyOne's queries and aggregations run on sample fields in its database, and LeRobot per-frame data deliberately stays in the source parquet behind media_reference. So the next step uses pandas to read the frames and FiftyOne to store, sort, and filter the per-episode results.

Turn taxel grids into per-episode fields

import numpy as np
import pandas as pd

df = pd.read_parquet(f"{local_dir}/data/chunk-000/file-000.parquet")

def taxels(key):
    # (frames, rows, cols)
    return np.stack([np.stack(x) for x in df[key]]).astype(float)

s1 = taxels("observation.tactile.right_sensor_1")  # 6x6
s2 = taxels("observation.tactile.right_sensor_2")  # 4x8
flat = np.concatenate([s1.reshape(len(df), -1), s2.reshape(len(df), -1)], axis=1)

df["peak"] = flat.max(axis=1)
df["in_contact"] = df["peak"] > 50  # taxels read 0 at rest; 50 is a noise margin

stats = df.groupby("episode_index").agg(
    peak_pressure=("peak", "max"),
    contact_fraction=("in_contact", "mean"),
)
stats["idle_taxels"] = [
    int((flat[idx].std(axis=0) == 0).sum())
    for idx in df.groupby("episode_index").indices.values()
]

ep = dataset.values("episode_index")
for field in ["peak_pressure", "contact_fraction", "idle_taxels"]:
    dataset.set_values(field, stats.loc[ep, field].tolist())

session.view = dataset.sort_by("contact_fraction")

The sorted view puts the most suspicious episodes first. Episode 2 sits at the top with a contact_fraction and peak_pressure of 0. All 68 taxels read zero for all 145 frames, even though the task is a grasp. It's also the shortest episode in the dataset, so it may be an aborted take. Open it in the App and watch it before you decide. Episode 0 comes next at 18% contact, then episode 18 at 31%. The median episode is in contact for 40% of its frames.

idle_taxels is a good reminder to check your assumptions. Every episode has between 11 and 68 taxels that never change, which looks alarming. But no taxel is flat in all 48 episodes, so these are cells the object didn't touch, not broken hardware. A taxel that's idle in one episode is normal. A taxel that's idle in every episode is dead.

The same pattern works for the other datasets. For SharpaDex, take the norm of each fingertip's fx fy fz from observation.tactile. For the Intern sets, compute the fraction of frames where observation.wrench_valid is all ones.

What to check before you train on touch

Most of these become one or two fields plus a sort in FiftyOne:

1. Dead or flat sensors. Flag channels with zero variance across the whole dataset, not just one episode.
2. Saturation. Count frames pinned at the sensor's max. Once a sensor saturates, you lose the information about how hard the contact was.
3. Drift. Compare at-rest readings across collection sessions. Gels wear and baselines move, and a model will learn Tuesday-versus-Friday instead of the task.
4. Episodes with no contact. Like Grabette episode 2. They may be fine for vision training and useless for tactile learning.
5. Misalignment. Tactile onset should line up with the gripper closing. If it's a few frames off, the model learns the wrong cause and effect.
6. Rare events. Slips and regrasps are where touch matters most and are usually a small fraction of frames. Find one, then look for more like it.

One more check, and it's the one most likely to fool you: split by episode, not by frame.

Consecutive tactile frames are nearly identical. RCT measured what that does to evaluation. With the encoder held fixed and only the test split changed, tactile-to-text Recall@1 was 80.0% on a frame-random split and 62.3% once whole contact sequences were held out. Since FiftyOne stores one sample per episode, a split over samples is already an episode split.

What would help the community

A few things would make tactile LeRobot datasets much easier to use. These are suggestions, not requirements:

• A shared tactile key convention. For example, observation.tactile.<sensor> for numeric touch and observation.images.tactile_<sensor> for gel video, plus a sensor block in meta/info.json with type, units, layout, and rest baseline. One community SO-arm dataset, ydaichi/tactile_color_pen_v2, already does this: its info.json has a tactile_sensors block with rows, columns, a contact threshold, and a per-taxel baseline.
• Units and sensor models on the card. Most of these 8 cards don't name the tactile sensor model, and none of the numeric tactile streams state units.
• A license. Three of the 8 don't declare one.
• Browser-friendly codecs for tactile video. If you need lossless pixels, keep the lossless copy and add an H.264 preview stream.

Try it yourself

• All 8 datasets: Voxel51 org on the Hub. Each card links to its original LeRobot source.
• FiftyOne: pip install -U fiftyone (docs, GitHub)
• Figures from this post and the script that makes them: harpreetsahota/tactile-lerobot-blog-figures
• Earlier roundup of 11 general LeRobot v3.0 datasets in FiftyOne: voxel51.com/blog

If you're recording tactile data with LeRobot, I'd love to see it. Upload it to the Hub and tag it tactile.