🔥 Can we scale tactile learning without collecting every possible contact in the real world?
Introducing ControlTac — a tactile data generation framework that expands a single reference tactile image into diverse observations with controllable 3D contact forces and contact
Can generated videos help us train more general dexterous controllers?
1/3: GALATEA - a first step toward a general-purpose dexterous tracker.
Generated videos play two roles: they supply diverse motion references during training, and propose motion plans at deployment.
We
@Modern_Gangster Really glad to be involved in this work and to take a small step together toward more general dexterous manipulation. Excited to see where this direction goes!
Thanks for sharing our work! This is an initial exploration of whether a General Dexterous Tracker can enable more generalizable dexterous manipulation across different tasks and objects. We hope this can be a step toward more scalable and versatile robot manipulation.
A generated video showed a robot how to grasp a jar.
The interesting part came after the video.
GALATEA uses Seedance 2.0 to generate manipulation clips, reconstructs the 3D motion of the hand and object, then tries to make that same interaction physically work inside PhysX
A generated video showed a robot how to grasp a jar.
The interesting part came after the video.
GALATEA uses Seedance 2.0 to generate manipulation clips, reconstructs the 3D motion of the hand and object, then tries to make that same interaction physically work inside PhysX before sending it to a real robot.
They generated around 2,500 clips.
About 2,000 produced usable hand-object trajectories, and more than 1,500 could actually be grounded in simulation.
For the real-world tests, they used a Franka Research 3 with a 22-DoF Sharpa Wave Hand.
One detail I really like is that GALATEA keeps the finger motion.
So the reference contains more than "move this object from A to B."
It can preserve how the object is held.
Grab the container around the body. Hold it by the neck. Approach it with a particular finger configuration.
Those choices matter a lot once you leave video generation and start dealing with contact, friction and joint limits.
On 40 unseen real-world plans, they report 27 successful executions.
That makes generative video much more interesting for robotics.
You are no longer looking at a clip only as pixels.
You can start treating it as a rough motion specification that still has to survive physics.
@AnthonyInROS2@YiMaTweets@kaiwynd
@OWW Thanks for sharing our work!
Please feel free to visit our project website ai4ce.github.io/EgoPush/ for more details.
We are currently preparing the code release. Should you have any questions, please do not hesitate to reach out to us via email.
384 Followers 179 FollowingPhD student at ETH; Associated PhD student at ETH AI center.
Working on dexterous manipulation, hand-object motion generation & understanding
384 Followers 179 FollowingPhD student at ETH; Associated PhD student at ETH AI center.
Working on dexterous manipulation, hand-object motion generation & understanding
18K Followers 968 Followinghumanoid robots | Automation | AGI | ASI | Tracking the rise of autonomous systems and the future of intelligence Physical AI embodied
6K Followers 199 FollowingSharpa is an AI robotics company dedicated to developing ultra-high performance robots and core components.
We Manufacture Time by Making Robots Useful.
4K Followers 1K FollowingAssistant Professor @BristolUni, PhD @UCL AI Center, prev. intern @TikTok & @Microsoft. ✨ Reinforcement Learning, Causality, World Models. 🎮 Game Fan!
773 Followers 528 FollowingAssistant Professor @Tsinghua_Uni | Research Scientist @NVIDIA | PhD @NYUniversity | Pushing the frontiers of spatial intelligence
39K Followers 171 FollowingThe Robotic Systems Lab designs machines, creates actuation principles, and builds up control technologies for autonomous operation in challenging environments.
1K Followers 115 FollowingAssistant Prof. @cs_cornell, Researcher @openai. Lead the @PortalCornell group. Building interactive AI agents that self-align
6K Followers 989 FollowingHi, I am the founder of @PokeBot_Global. I like RL, robots, and video games:) I am an Assistant Prof at Tsinghua; Postdoc at Stanford; Ph.D. at Berkeley
6K Followers 1K FollowingAsst. Prof@ Tsinghua University, Co-founder @GalaxeaDynamics. Former Scientist@Waymo, MIT PhD’19. I research on multimodal learning, self-driving and robotics!