Society is rapidly opening its doors to robots in our daily life with autonomous vehicles, rehabilitation devices and autonomous appliances. These robots will face unexpected changes in their environment, to which they will have to react immediately and appropriately. Even though robots exceed largely humans’ precision and speed of computation, they are far from matching humans’ capacity to adapt rapidly to unexpected changes. The SAHR project aims at developing new control methods for robots that will enable them to learn gradually complex motor skills.


To ensure speeding response in the face of unexpected changes, we target controllers that can re-plan at run time and on-the-fly. We take a novel approach to robot learning that follows stages of skill acquisition in humans. To inform modelling, we conduct a longitudinal study of the acquisition of dexterous bimanual skills in watchmaking a unique craftsmanship. We study how humans exploit task uncertainty to overcome their sensory-motor noise, and how humans learn bimanual synergies to reduce the control variables. This study informs the design of novel learning strategies for robots that exploit failures as much as successes. We combine traditional optimal control approach with machine learning (ML) to learn feasible control laws, retrievable at run time with no need for further optimization. We exploit properties of dynamical systems (DS), which have received little attention in robot control, and use ML to identify characteristics of DS, in ways that were not explored to date. The approach is assessed in live demonstrations of coordinated adaptation of a multi-arm/hand robotic system engaged in a fast-paced industrial task, in the presence of humans.

Introduction to Watchmaking Task

We follow a cohort of apprentices in watchmaking over a period of two years. We focus on the acquisition process of dexterous bimanual manipulation skills in watchmaking tasks.

We focus on the assembly and disassembly of three components on the watchface: screw, plate, and spring. Three tools are used during manipulation: screwdriver, tweezers, and stick.

Data Collection

We collect both tactile sensing information (applied force) and kinematic information (hand motion) of human subjects during watchmaking tasks. For this purpose, we have established a data collection system, which consists of two systems: Tactile Sensing System and Motion Capture System.

Experimental Setup for Data Collection

For the tactile sensing system, we use three types of sensors to record contact information during manipulation.
(1) Finger TPS Sensors, to record pressure applied on finger phalanges. For each hand, 6 finger TPS sensors are mounted (12 sensors in total) to cover the distal phalanges (thumb, index finger, middle finger) and intermediate phalanges (index finger, middle finger, ring finger).
(2) ATI Nano17 Force/Torque sensor, mounted on the support of the watchface to record the force and torque signals applied to the watchface.
(3) The DigiTacts sensing systems that comprise sensing elements array are mounted on the surfaces of the tool to record applied pressure during manipulation.

The motion capture system is arranged for the recording of kinematic data of human motion: a camera array consisting of 8 GoPro cameras are mounted surrounding the workspace to capture the human hand movement at high speed and high resolution from multiple perspectives. Twenty-five trackers are pasted on each hand to enable the recording of 3D trajectories of each finger joint, which are used to reconstruct the hand skeleton model as well as to reproduce hand motion afterwards.

The video recording work is undertaken by Pomelo Sàrl.

Data Analysis and Modelling

Breakdown of the Manipulation Time

To analyse the complex human motions during watchmaking task, we segment the recorded human motion data in order to breakdown the entire manipulation process into sequential simple action components. For example, a single complete assembly/disassembly process consists of actions like adjusting base pose, selecting tools, adjusting hand pose, pick and place (of target components), localizing tools, executing assembly/disassembly actions.

Part of the data analysis work is conducted in collaboration with Pomelo Sàrl.

Average Failures per Manipulation

We analyse the collected human motion data from several perspectives. We evaluate and compare the motion of subjects at different skill levels to understand the evolvement of manipulation skills. For example, failures happen during complicated manipulation tasks, such as dropping components, breaking fragile components, failed in localizing tools, and so on. Results from preliminary analysis show that subjects from different skill levels display a significant difference in average failure rate, and failures happen more often in the early stages of learning watchmaking tasks.

Our research also focuses on the failures during executions. We study how human manages to generate new successful motions after the occurrence of failures, in order to develop strategies that enable robots learning from failed demonstrations.

Human Hand Skeleton Model of Manipulating Screwdriver

We reconstruct skeleton models of human hand from the collected kinematic data using OpenSim. Combined with the pressure information recorded on the finger phalanges, we analyse the adjustment of hand pose, synergy of fingers, activities of tendons during different manipulation tasks.
Each human hand has 27 degrees of freedoms (DoFs), some of which are redundant in specified manipulation tasks. We analyse the task constraints to identify the required DoFs for planning the robotic task.
Human changes finger configurations to satisfy different task constraints. As an example, in the manipulation of the screwdriver, three fingers are mainly required: thumb, index finger, and middle finger. The distal phalanx of thumb, along with the distal phalanges of middle and ring fingers, move in collaboration to apply friction to the surface of screwdriver for generating rhythmic rotation movements. In the meantime, the index finger applies constant pressure on the top of the screwdriver to maintain the stability of the screwdriver.

Human Hand Skeleton Model of Manipulating Tweezers for Picking Components

To manipulate tweezers for a pick and place motion, the hand constructs a force closure to realize a stable manipulation as well as to compensate the possible perturbations occur on the tip of tweezers.

Human Hand Skeleton Model of Manipulating Tweezers for Watch Components Disassembly

While using the tweezers for disassembly tasks, subjects often prefer to use fingers to wrap the tweezers, so that the movement of the tweezers along the radial direction is constrained when a force is desired to be applied in the normal direction.

Example: Spring Assembly – A Challenging Bimanual Manipulation Task

The assembly of spring on the watchface is one of the most challenging skills that the apprentices must learn to master during study.

On the one hand, the spring is a very tiny component on the watchface, thus motion of high precision is required during the pick and place movement. On the other hand, the spring must be properly deformed during assembly, in order to save elastic potential energy in it. Therefore, appropriate force must be applied (usually by the dominant hand with tweezers) in the correct direction, while the position of the spring must be maintained (usually by the assistant hand, using the wooden stick) unchanged in the meantime.

The correct way to assemble the spring is demonstrated in the following video.

Either inappropriate manipulation direction or improper application of force may result in failures, even the damage of components. Following videos illustrate the typical failures occur during the assembly: the spring flies away.

Dynamical System-based Skill Modelling

Dynamical system (DS) based control approaches offer a framework for robust control with the capability of fast adaptation from unexpected perturbations. In this project, we use time-invariant DSs to model complex movements, such as combined control of limb motion, from the collected human demonstration data with single or multiple equilibrium states.

Learned Dynamical System from Human Demonstration Data

Dynamical systems offer a robust control of the robot, by allowing to adapt the movement rapidly in case of perturbations. In fact, as we can see in the figure below, we can generalize collected movement data (shown in red) to the whole workspace (blue arrows). The diagram shows a model that can switch between a point-to-point motion towards an attractor (reaching the screw in a. and going back in c.) and a periodic motion (b.), such as that of screwing or unscrewing that we find in watchmaking tasks.

As illustrated in the figure below, a DS model can also be exploited for embedding velocity and force to enable admittance control during a haptic task with an attractor at the origin. This DS modelling may be useful for the control of interactions in bimanual tasks including the interaction of forces.

Watch Assembly as a Benchmark for Robotic Systems

Most of our everyday activities, such as the usage of tools to grasp small objects, require coordinated motion of the arms (e.g., reaching) or the hands (e.g., manipulation). Performing these tasks with one arm is often infeasible, mainly because the dexterity and flexibility required for such tasks are beyond a single arm’s capabilities. Similarly, a single robot arm is merely incapable of meeting the requirements of such complex tasks. A dual or multi-arm robotic system, on the other hand, extends the flexibility and capabilities of a single robot arm. It allows, for example, highly complex manipulation of small and deformable objects that would otherwise be infeasible for single arm systems. We are far from having robotic systems that can robustly achieve such dexterous manipulation skills.

One, however, can envision an extensive range of applications in households and factories that can benefit from such strategies. Examples include placing and closing lids, packaging boxes in pallets, inserting the USB cable into sockets. Completing this type of tasks requires to localize both objects relative to one another (lid on top of a box, cable into a tube) and to adapt forces and movements of the two arms in coordination. Insertion of semi-deformable materials is made particularly difficult as one cannot build an explicit model of the deformation and interaction forces. Machine learning provides a structured framework that can allow robots to learn difficult-to-model problems by using their previous experience, without explicit modeling of the task constraints and possibly taking advantage of noisy expert demonstrations. This benchmark proposes an evaluation of machine learning algorithms on a difficult multi-arm insertion task that involves collaboration among the arms to manipulate a small and semi-deformable object.

The watchmaking process highlights these shortcomings of current robotic systems and poses a few challenges to them, including (but not limited to):

• Placing an irregularly shaped object in the correct groove,
• Handling a deformable object,
• Multi-arm coordination:

– One arm stabilizing the system, and the other(s) performing the manipulation,
– Performing object manipulation that requires two tools

Under the SAHR project, we are working to provide the research community with a benchmark that evaluates machine learning algorithms on a difficult multi-arm insertion task that involves collaboration among the arms to manipulate a small and semi-deformable object form the watch assembly. This component is highlighted in the depicted CAD file of the watch assembly.

For the benchmark, and increasing the manipulability of the watch, we work with scaled-up versions of the actual watch to carefully designed scale factors of 3.5 and 5.8. These watches are 3D printed for the robotic setup.

In the benchmark, we will focus on the plate insertion task of the watch assembly process. Thus, the tasks that need to be learned are:

  1. Coordination in the synchrony of the arms to orient plate for ease of control and successful insertion,
  2. Change orientation and pressure in response to force feedback to insert the plate and its leg.