Validation: Dextrous manipulation and robot control

Benchmark for bimanual robotic manipulation of semi-deformable objects

Most of our everyday activities, such as the usage of tools to grasp small objects, require coordinated motion of the arms (e.g., reaching) or the hands (e.g., manipulation). Performing these tasks with one arm is often infeasible, mainly because the dexterity and flexibility required for such tasks are beyond a single arm’s capabilities. Similarly, a single robot arm is merely incapable of meeting the requirements of such complex tasks. A dual or multi-arm robotic system, on the other hand, extends the flexibility and capabilities of a single robot arm. It allows, for example, highly complex manipulation of small and deformable objects that would otherwise be infeasible for single arm systems. We are far from having robotic systems that can robustly achieve such dexterous manipulation skills.

One, however, can envision an extensive range of applications in households and factories that can benefit from such strategies. Examples include placing and closing lids, packaging boxes in pallets, inserting the USB cable into sockets. Completing this type of tasks requires to localize both objects relative to one another (lid on top of a box, cable into a tube) and to adapt forces and movements of the two arms in coordination. Insertion of semi-deformable materials is made particularly difficult as one cannot build an explicit model of the deformation and interaction forces. Machine learning provides a structured framework that can allow robots to learn difficult-to-model problems by using their previous experience, without explicit modeling of the task constraints and possibly taking advantage of noisy expert demonstrations. This benchmark proposes an evaluation of machine learning algorithms on a difficult multi-arm insertion task that involves collaboration among the arms to manipulate a small and semi-deformable object.

The watchmaking process highlights these shortcomings of current robotic systems and poses a few challenges to them, including (but not limited to):

• Placing an irregularly shaped object in the correct groove,
• Handling a deformable object,
• Multi-arm coordination:

– One arm stabilizing the system, and the other(s) performing the manipulation,
– Performing object manipulation that requires two tools

Under the SAHR project, we are working to provide the research community with a benchmark that evaluates machine learning algorithms on a difficult multi-arm insertion task that involves collaboration among the arms to manipulate a small and semi-deformable object form the watch assembly. This component is highlighted in the depicted CAD file of the watch assembly.

For the benchmark, and increasing the manipulability of the watch, we work with scaled-up versions of the actual watch to carefully designed scale factors of 3.5 and 5.8. These watches are 3D printed for the robotic setup.

In the benchmark, we will focus on the plate insertion task of the watch assembly process. Thus, the tasks that need to be learned are:

  1. Coordination in the synchrony of the arms to orient plate for ease of control and successful insertion,
  2. Change orientation and pressure in response to force feedback to insert the plate and its leg.

Exploiting hand dexterity for multiple object grasping

Human and robotic hand holding multiple objects

Human hands are highly dexterous systems. Hand dexterity enables humans to grasp everyday objects in distinct poses for different manipulation purposes, and even to grasp multiple objects at the same time for complex manipulation tasks. One main reason for hand dexterity lies in the exploitation of human hand’s abundant degrees of freedom.

Inspired by the human hand dexterity, we are currently developing algorithms to enhance dexterity of robotic hand with redundant degrees of freedom. Our approach largely improves robotic hand’s capability of handling object(s) in dexterous grasping and manipulation tasks.

Dynamic Object Shape Exploration with a Robotic Hand

If our vision is occluded or the environment is dark, we humans use our hands to identify objects around us, e.g., finding an object behind a curtain. Similar situations can frequently occur in robotic applications, where an object cannot be precisely recognized due to occlusion or poor lighting conditions. In this research, we propose an algorithm to imitate the human strategy of adjusting hand pose while exploring an unknown object. During the exploration, an informative path planning approach based on Gaussian Process Implicit Surface is adopted to reconstruct the object based on tactile sensing feedback.

Self-Correcting Model-based Robot Control

In the context of multi-body robotic systems, the coordination of a substantial number of degrees of freedom and interaction forces is a critical task. This coordination presents a considerable challenge, particularly when operating in task space, even for seemingly straightforward tasks, such as those involving a bimanual robot interacting with an object. In contrast, humans and animals possess a remarkable ability to adapt to unforeseen situations and master new skills. They accomplish this feat by learning from their mistakes and engaging in a trial-and-error process.

We envision a similar robotic system that learns through trial-and-error: it tries to achieve the task with a quadratic programming (QP)-based controller, fails (e.g., the robot drops the box), and tries again until it manages to realize the goal. Our approach builds upon existing model learning techniques, such as the ID model, to improve the QP dynamics model over time. Moreover, we utilize adaptive control methods to regulate the feedback term. This is in contrast to prior approaches that focused solely on modifying the QP cost function. By utilizing trial-and-error learning and our proposed methodology, we anticipate that our robotic system will be able to efficiently learn and accomplish complex tasks.


Chatzilygeroudis K., Fichera B., Lauzana I., Bu, F., Yao K., Khadivar F., and Billard A. (2020). “Benchmark for Bimanual Robotic Manipulation of Semi-Deformable Objects.“, IEEE Robotics and Automation Letters, 5(2), pp.2443-2450. DOI:10.1109/LRA.2020.2972837

Yao, K. and Billard, A. “Exploiting Kinematic Redundancy for Robotic Grasping of Multiple Objects.” The IEEE Transactions on Robotics (T-RO), 2023. DOI: 10.1109/TRO.2023.3253249

Farshad Khadivar*, Kunpeng Yao*, Xiao Gao, and Aude Billard. “Online Active and Dynamic Object Shape Exploration with a Multi-fingered Robotic Hand.” *Equal authorship. Submitted to Robotics and Autonomous Systems (RAS), 2023. Under review.

Xiao Gao, Kunpeng Yao, Farshad Khadivar, and Aude Billard. “Real-Time Collision-Free Motion Planning for In-Hand Manipulation with a Multi-Fingered Hand.” Submitted to Robotics: Science and Systems (RSS) 2023. Under review.