What to Attend, What to Keep

In Brief

Robotic manipulation requires integrating vision, touch, and language. While existing policies fuse modalities uniformly and rely only on the last few steps, we introduce an efficient event memory mechanism. This expands the observation span, allowing the policy to capture salient task history and dynamically prioritize the right modality at the right time.

We address this by breaking long-horizon manipulation down into smaller skills such as reaching, grasping, and alignment. Language serves as an interface that structures the sequence of primitives and guides sensor prioritization at each stage. Our model combines modality-specialized tokenizers, asymmetric temporal horizons for every modality, and a sparse event memory that retains skill-relevant states at their boundaries. We evaluate the learned representation through progress prediction — a structured diagnostic that directly reflects whether the representation captures the right modality at the right time — and compare against conventional multi-modal representations used in manipulation policies.

"Observation formation — not only policy architecture — is a central challenge in multi-modal robotic manipulation."

UMI gripper with integrated tactile sensors

UMI gripper + dual tactile

Dataset

Contact-Rich Manipulation Tasks

We evaluate across three tasks, each composed of distinct subtasks that span the sensor prioritization spectrum. Reach and grasp are tractable from visual proximity alone. Others are not: inspect demands simultaneous vision and touch; slide and twist complete on a slippage event that must be retained in long-term tactile memory; align is detected by seeing a consistent aligned pattern across both touch and vision over a short window.

Lamp Replacement

Replace a light bulb. Subtasks include align bulb and twist bulb. Visual cues are inherently ambiguous during twisting since the repetitive rotational motion looks nearly identical at every step, so completion must be inferred from tactile force patterns.

Bottle Capping

The most tactile-dominant skill. Slippage — a transient force pattern — must be detected from FeelAnyForce tokens. From a visual perspective it's just a repetitive slide motion that looks identical across every frame, making it useless for slippage detection.

Blind Cube Search

Identify a hidden cube among pentagon and square shapes using tactile deformation patterns. The robot must distinguish contact geometry with zero visual information about the cube's location.

↓ Check subtask details below

Skill Decomposition

Primitive Skills

Each task is decomposed into a fixed sequence of language-described skills. Modality importance shifts across phases — reaching skills are vision-dominant, while contact-rich terminal skills depend on tactile sensing. Click any skill to see its description and video clip.

Cont. Continuous progress Disc. Discrete completion Cond. Conditional

Architecture

How does it work?

Short-term vs long-term memory in SkillFormer

Given a language description of the current skill, SkillFormer tokenizes RGB and tactile frames over modality-specific horizons where vision has a short recent window, and tactile has a longer one as it requires longer horizon to , and appends a small set of event tokens sampled from the boundary of each completed skill. A frozen CLIP text encoder encodes the queried skill into a language token. All tokens carry temporal, modality, and event embeddings that tell the transformer when each observation was captured, which sensor it came from, and whether it belongs to the current window or to long-term memory. A GPT-style transformer fuses this mixed context via bidirectional self-attention and decodes a scalar progress estimate — learning, in effect, which modality to attend to and which past states to keep for each skill.

SkillFormer. CLIP ViT and FeelAnyForce encode RGB and tactile frames over modality-specific horizons. A frozen CLIP text model encodes the queried skill. Tokens carry temporal, modality, and event embeddings. A GPT-style transformer fuses them via bidirectional self-attention and decodes scalar progress.

Representation Analysis

t-SNE Visualization of Skill-Conditioned Representations

t-SNE projections of the learned representations. The same observations are queried with different skill descriptions, producing different representations per timestep. Language conditioning meaningfully restructures the representation space: clusters separate by subtask, and coloring by GT progress reveals smooth 0→1 gradients for continuous subtasks and sharp 0/1 boundaries for discrete ones. Switch to Terminal mode to see how well terminal states are separated from the rest of the trajectory, a direct indicator of how distinguishable skill completion is in the learned space.

Hover to zoom

Experiments

Progress Estimation as a Representation Probe

Progress estimation directly measures whether a representation captures the right modality at the right time. We compare SkillFormer against observation formulations common in manipulation policies. Strongest gains appear on skills whose completion depends on contact state, tactile evolution, or transitions between nearby manipulation phases. Smooth pick-and-place primitives like reach and grasp are well-served by simple observation formulations — progress correlates directly with visual proximity, making them tractable even without tactile input or memory. But skills like twist and align require detecting subtle force patterns that only emerge over a longer tactile horizon, and slide depends on retaining a transient slippage event that vanishes within a few frames — demanding an explicit event memory for robust representation.

SkillFormer (top-left) detects the slippage event and predicts skill completion on time. FiLM, no tactile, and events models show delay, only rising toward completion once the robot has already transitioned into the next subtask.

Task

Model

Episode

Speed

Loading...

Fig. 3. Progress estimation with GT progress overlay. Select a model and episode to compare predictions across ablations.

✦

A missed event cannot be recovered. Skills like slide cap depend on a single slippage event for completion. Without memory, the event can fall out of the observation horizon and be forgotten entirely, causing progress to fall back as in FiLM, and the model must infer completion from delayed signals instead. SkillFormer retains the event and predicts completion on time.

✦

SkillFormer excels at skills that require reading tactile patterns over time. Unlike slide cap, which is detected by a single slippage frame, align completion unfolds over a longer horizon of tactile evolution. SkillFormer reliably detects the alignment moment by tracking this gradual pattern. FiLM and w/o events, which simply concatenates observations along the horizon axis, cannot extract this structure and performs only slightly better than Vision Only, which has no tactile data at all.

✦

Event memory resolves ambiguity. The w/o-Events model shows large errors on align cap (0.09 vs 0.01), confirming that skill boundaries are ambiguous without a stored reference to the terminal state of the previous skill.

✦

Blind cube search is fully tactile-dominant. SkillFormer substantially reduces delay and false positives on inspect box compared to FiLM, which cannot distinguish cube geometry from tactile deformation alone.

✦

Tactile pretraining is essential, not just tactile input. The w/o-FAF model, which replaces the FeelAnyForce encoder with an untrained one, performs nearly as poorly as Vision Only. Raw tactile signals without a pretrained encoder contribute little, indicating that the quality of tactile representation matters as much as its presence.