Society is rapidly opening its doors to robots in our daily life with autonomous vehicles, rehabilitation devices and autonomous appliances. These robots will face unexpected changes in their environment, to which they will have to react immediately and appropriately. Even though robots exceed largely humans’ precision and speed of computation, they are far from matching humans’ capacity to adapt rapidly to unexpected changes. The SAHR project aims at developing new control methods for robots that will enable them to learn gradually complex motor skills.


To ensure speeding response in the face of unexpected changes, we target controllers that can re-plan at run time and on-the-fly. We take a novel approach to robot learning that follows stages of skill acquisition in humans. To inform modelling, we conduct a longitudinal study of the acquisition of dexterous bimanual skills in watchmaking a unique craftsmanship. We study how humans exploit task uncertainty to overcome their sensory-motor noise, and how humans learn bimanual synergies to reduce the control variables. This study informs the design of novel learning strategies for robots that exploit failures as much as successes. We combine traditional optimal control approach with machine learning (ML) to learn feasible control laws, retrievable at run time with no need for further optimization. We exploit properties of dynamical systems (DS), which have received little attention in robot control, and use ML to identify characteristics of DS, in ways that were not explored to date. The approach is assessed in live demonstrations of coordinated adaptation of a multi-arm/hand robotic system engaged in a fast-paced industrial task, in the presence of humans.

Introduction to Watchmaking Task

We follow a cohort of apprentices in watchmaking over a period of two years. We focus on the acquisition process of dexterous bimanual manipulation skills in watchmaking tasks.

We focus on the assembly and disassembly of three components on the watchface: screw, plate, and spring. Three tools are used during manipulation: screwdriver, tweezers, and stick.

Data Collection

We collect both tactile sensing information (applied force) and kinematic information (hand motion) of human subjects during watchmaking tasks. For this purpose, we have established a data collection system, which consists of two systems: Tactile Sensing System and Motion Capture System.

Experimental Setup for Data Collection

For the tactile sensing system, we use three types of sensors to record contact information during manipulation.
(1) Finger TPS Sensors, to record pressure applied on finger phalanges. For each hand, 6 finger TPS sensors are mounted (12 sensors in total) to cover the distal phalanges (thumb, index finger, middle finger) and intermediate phalanges (index finger, middle finger, ring finger).
(2) ATI Nano17 Force/Torque sensor, mounted on the support of the watchface to record the force and torque signals applied to the watchface.
(3) The DigiTacts sensing systems that comprise sensing elements array are mounted on the surfaces of the tool to record applied pressure during manipulation.

The motion capture system is arranged for the recording of kinematic data of human motion: a camera array consisting of 8 GoPro cameras are mounted surrounding the workspace to capture the human hand movement at high speed and high resolution from multiple perspectives. Twenty-five trackers are pasted on each hand to enable the recording of 3D trajectories of each finger joint, which are used to reconstruct the hand skeleton model as well as to reproduce hand motion afterwards.

The video recording work is undertaken by Pomelo Sàrl.

Data Analysis and Modelling

Breakdown of the Manipulation Time

To analyse the complex human motions during watchmaking task, we segment the recorded human motion data in order to breakdown the entire manipulation process into sequential simple action components. For example, a single complete assembly/disassembly process consists of actions like adjusting base pose, selecting tools, adjusting hand pose, pick and place (of target components), localizing tools, executing assembly/disassembly actions.

Part of the data analysis work is conducted in collaboration with Pomelo Sàrl.

Average Failures per Manipulation

We analyse the collected human motion data from several perspectives. We evaluate and compare the motion of subjects at different skill levels to understand the evolvement of manipulation skills. For example, failures happen during complicated manipulation tasks, such as dropping components, breaking fragile components, failed in localizing tools, and so on. Results from preliminary analysis show that subjects from different skill levels display a significant difference in average failure rate, and failures happen more often in the early stages of learning watchmaking tasks.

Our research also focuses on the failures during executions. We study how human manages to generate new successful motions after the occurrence of failures, in order to develop strategies that enable robots learning from failed demonstrations.

Human Hand Skeleton Model of Manipulating Screwdriver

We reconstruct skeleton models of human hand from the collected kinematic data using OpenSim. Combined with the pressure information recorded on the finger phalanges, we analyse the adjustment of hand pose, synergy of fingers, activities of tendons during different manipulation tasks.
Each human hand has 27 degrees of freedoms (DoFs), some of which are redundant in specified manipulation tasks. We analyse the task constraints to identify the required DoFs for planning the robotic task.
Human changes finger configurations to satisfy different task constraints. As an example, in the manipulation of the screwdriver, three fingers are mainly required: thumb, index finger, and middle finger. The distal phalanx of thumb, along with the distal phalanges of middle and ring fingers, move in collaboration to apply friction to the surface of screwdriver for generating rhythmic rotation movements. In the meantime, the index finger applies constant pressure on the top of the screwdriver to maintain the stability of the screwdriver.

Human Hand Skeleton Model of Manipulating Tweezers for Picking Components

To manipulate tweezers for a pick and place motion, the hand constructs a force closure to realize a stable manipulation as well as to compensate the possible perturbations occur on the tip of tweezers.

Human Hand Skeleton Model of Manipulating Tweezers for Watch Components Disassembly

While using the tweezers for disassembly tasks, subjects often prefer to use fingers to wrap the tweezers, so that the movement of the tweezers along the radial direction is constrained when a force is desired to be applied in the normal direction.

Example: Spring Assembly – A Challenging Bimanual Manipulation Task

The assembly of spring on the watchface is one of the most challenging skills that the apprentices must learn to master during study.

On the one hand, the spring is a very tiny component on the watchface, thus motion of high precision is required during the pick and place movement. On the other hand, the spring must be properly deformed during assembly, in order to save elastic potential energy in it. Therefore, appropriate force must be applied (usually by the dominant hand with tweezers) in the correct direction, while the position of the spring must be maintained (usually by the assistant hand, using the wooden stick) unchanged in the meantime.

The correct way to assemble the spring is demonstrated in the following video.

Either inappropriate manipulation direction or improper application of force may result in failures, even the damage of components. Following videos illustrate the typical failures occur during the assembly: the spring flies away.

Dynamical System-based Skill Modelling

Dynamical system (DS) based control approaches offer a framework for robust control with the capability of fast adaptation from unexpected perturbations. In this project, we use time-invariant DSs to model complex movements, such as combined control of limb motion, from the collected human demonstration data with single or multiple equilibrium states.

Learning Dynamical Systems with Bifurcation.

DS for controlling rhythmic patterns are distinct from DS used to control point-to-point motion and current approaches switch at run time across these to enable multiple behaviors. This, however, may lead to jittering and jerky motions when switching across dynamics when these have different equilibrium. The figure below shows a robotic task (polishing) that requires multiple switches from point-to-point motion to cyclic ones.

We here offer an approach that encodes periodic and discrete dynamics in a single time-invariant DS by exploiting the concept of bifurcation, allowing for smooth transitions between the two. The following figure manifests the result of the case where bifurcations are exploited in the desired DS.

To learn a desired periodic motion, we propose a parameterization of the DS to offer an explicit way to control for its behavior, namely speed, location of the target, and shape of the limit cycle. Furthermore, we also explore the use of diffeomorphism and learn a mapping function to modify the initial linear limit cycle into a nonlinear limit cycle. The video below depicts our approach to learn complex-shaped non-linear limit cycles:

Learned Dynamical System from Human Demonstration Data

Dynamical systems offer a robust control of the robot, by allowing to adapt the movement rapidly in case of perturbations. In fact, as we can see in the figure below, we can generalize collected movement data (shown in red) to the whole workspace (blue arrows). The diagram shows a model that can switch between a point-to-point motion towards an attractor (reaching the screw in a. and going back in c.) and a periodic motion (b.), such as that of screwing or unscrewing that we find in watchmaking tasks.

As illustrated in the figure below, a DS model can also be exploited for embedding velocity and force to enable admittance control during a haptic task with an attractor at the origin. This DS modelling may be useful for the control of interactions in bimanual tasks including the interaction of forces.

Learned Dynamical System from Human Demonstration Data

Dynamical systems with multiple equilibrium points are interesting for robot control, as they allow to easily switch across dynamics without incurring discontinuities. While most works assume that the number and type of dynamics are known, we focus on a scenario where unlabeled observations of the dynamical system are given. Determining the number of sub-dynamics and the equilibria’ locations is difficult and can be rarely done analytically. We propose a Graph-based spectral clustering method that takes advantage of a velocity-augmented kernel. We apply the algorithm to identify dynamics of a series of dynamics. We compare our algorithm to Kernel K-Means, Spectral Clustering and Gaussian Mixtures and show that, even when these algorithms are provided with the true number of sub-dynamics, they fail to cluster them correctly.

The proposed method is based on the eigenvalue decomposition of the Laplacian matrix applied to a specific graph structure. Exploiting the spectral properties of the Laplacian matrix, we show that we can cluster and project each sub-dynamics in orthogonal embedding spaces. Moreover, in these embedding spaces, each sub-dynamics is linearized and it is possible to identify the position of the equilibrium point.

Learning Latent dynamical systems

This research aims to learn latent simplistic and low-dimensional dynamical systems from high-dimensional and nonlinear demonstrations which can be obtained during the watchmaking process. The latent dynamical systems are globally exponentially stable and can be used, not only for learning complex dynamical systems, but also for trajectory clustering and classification. We consider exponential stability guarantees which are be obtained from different stability theories such as Lyapunov and Contraction. Furthermore, we investigate the impact of those on the accuracy of the learned model and their robustness to noise and sparse data.

Watch Assembly as a Benchmark for Robotic Systems

Most of our everyday activities, such as the usage of tools to grasp small objects, require coordinated motion of the arms (e.g., reaching) or the hands (e.g., manipulation). Performing these tasks with one arm is often infeasible, mainly because the dexterity and flexibility required for such tasks are beyond a single arm’s capabilities. Similarly, a single robot arm is merely incapable of meeting the requirements of such complex tasks. A dual or multi-arm robotic system, on the other hand, extends the flexibility and capabilities of a single robot arm. It allows, for example, highly complex manipulation of small and deformable objects that would otherwise be infeasible for single arm systems. We are far from having robotic systems that can robustly achieve such dexterous manipulation skills.

One, however, can envision an extensive range of applications in households and factories that can benefit from such strategies. Examples include placing and closing lids, packaging boxes in pallets, inserting the USB cable into sockets. Completing this type of tasks requires to localize both objects relative to one another (lid on top of a box, cable into a tube) and to adapt forces and movements of the two arms in coordination. Insertion of semi-deformable materials is made particularly difficult as one cannot build an explicit model of the deformation and interaction forces. Machine learning provides a structured framework that can allow robots to learn difficult-to-model problems by using their previous experience, without explicit modeling of the task constraints and possibly taking advantage of noisy expert demonstrations. This benchmark proposes an evaluation of machine learning algorithms on a difficult multi-arm insertion task that involves collaboration among the arms to manipulate a small and semi-deformable object.

The watchmaking process highlights these shortcomings of current robotic systems and poses a few challenges to them, including (but not limited to):

• Placing an irregularly shaped object in the correct groove,
• Handling a deformable object,
• Multi-arm coordination:

– One arm stabilizing the system, and the other(s) performing the manipulation,
– Performing object manipulation that requires two tools

Under the SAHR project, we are working to provide the research community with a benchmark that evaluates machine learning algorithms on a difficult multi-arm insertion task that involves collaboration among the arms to manipulate a small and semi-deformable object form the watch assembly. This component is highlighted in the depicted CAD file of the watch assembly.

For the benchmark, and increasing the manipulability of the watch, we work with scaled-up versions of the actual watch to carefully designed scale factors of 3.5 and 5.8. These watches are 3D printed for the robotic setup.

In the benchmark, we will focus on the plate insertion task of the watch assembly process. Thus, the tasks that need to be learned are:

  1. Coordination in the synchrony of the arms to orient plate for ease of control and successful insertion,
  2. Change orientation and pressure in response to force feedback to insert the plate and its leg.


Self-Collision Avoidance for Humanoid Robots

Another issue that we address is related to the self-collisions of redundant robots. Self-collision prevention is necessary to avoid hardware damage. We focus on providing a real-time solution to the self-collision avoidance (SCA) problem for high-dimensional position-controlled humanoid robots, by learning continuously differentiable boundary functions that can be used in real-time to prevent self-collisions while controlling the 29 DoF iCub humanoid. 

Self-collision boundary functions are learned by sampling the robot’s workspace for collided and non-collided joint space configurations. With a sufficiently representative dataset, several function approximation approaches could learn the boundary functions with high prediction accuracy. Since the dataset is collected offline, we are able to use precise triangle mesh representation of the iCub robot body to acquire a highly accurate sampling of the boundary. The challenge we face is that such datasets are in the order of millions. Further, we seek to provide high SCA prediction accuracy and computationally efficiency while ensuring continuity requirements. We offer the former by decomposing the robot’s 29-dimensional joint space into independent submodels of lower dimensionality, each covering a sub-combination of limbs and bodies. To learn the boundary functions, we explore and compare the sparse SVM known as Cutting-Plane Subspace Pursuit (CPSP) algorithm, GPU-accelerated feed-forward neural networks (NN) and GPU-accelerated SVM implementations.

Understanding of Human Kinematic Coordination Patterns in Bimanual Fine Manipulation Tasks

Task demands of assembling watch spring

Skills that require the collaboration of both arms are common in humans’ everyday activities, and the coordination of upper limbs is key to human bimanual dexterity. In this study, we focus on the task of assembling a watch spring, where precise, coordinated, and directionally compliant control of the hands and fingers motions is required. We recorded the motion of subjects and the force/torque applied on the watch face during manipulation tasks.

We compare the performance of 15 subjects participating in our experiments (10 novices and 5 experts). Our analysis shows that experts try to align the direction of force application demanded by the task with the optimal force application direction of the dominant arm. In addition, experts adjust finger placements to improve the dexterity of hands for manipulation. We assume the differences in performance across novices and experts can be explained by the differences in the weights of criteria (manipulability, task compatibility, and discomfort), which are used to evaluate the motion. We develop a computational model to infer the weights. Preliminary results show that novice subjects put higher weight on the task compatibility, while experts have higher weight on motion comfort.

This study has been published in Biological Cybernetics, 2020. Check the publication for more detailed information.

Study of Human Hand Control Strategy

One human subject performs unscrewing task in the free-base condition
One human subject performs unscrewing task in the fixed-base condition

Many manipulations in human everyday activities rely on the collaboration of our two hands. Bimanual fine manipulation tasks are particularly demanding as precise coordination among fingers of both hands is required. Yet, despite the abundant degrees of freedom (DoFs) need
to be controlled, humans can perform such tasks nearly effortlessly. Ample evidence suggests that the human central nervous system (CNS) predetermines hand poses that are compatible with task conditions (e.g. task objective, physical properties of objects, control of force/torque
required by task) prior to manipulation. However, the mechanism by which the CNS encodes task conditions and translates such information into appropriate motor commands is poorly understood.

We designed a comparative experiment to infer task conditions’ effect on human hand control strategies. Twenty right-handed subjects (aged 24.2+/-6.0, 5 women) were tasked to a precise unscrewing task consisting of two steps: placing a screwdriver on a tiny (1.6mm) screw that is tightly mounted on a watchface, and unscrewing it without sliding off. Subjects were equally assigned to two groups and performed the same task in two conditions. In the free-base condition, the watchface was let free to move on the table; whereas in the fixed-base condition, it was fixed on the table. 

Subjects employed similar hand poses across the two successive task steps (placing and unscrewing) within the same conditions, whereas distinct hand poses were observed between these two conditions. This suggests a generic task-oriented motion planning prior to execution.
In the free-base condition, subjects consistently chose similar hand poses across trials. They used each hand to manipulate one piece of equipment, as required to control for more DoFs. In contrast, subjects used both hands to control the screwdriver in the fixed-base condition, and showed diversity in hand pose combinations. Analysis indicates that subjects select hand poses to control simultaneously the task-relevant DoFs (i.e. demanded by the task)
and the task-irrelevant DoFs. The latter, although not crucial to the task goal, may bring in perturbations and impair performance if not properly controlled. Moreover, subjects manifested a tendency to balance virtual fingers (fingers that move together as a functional unity) across the two hands, such that the control of task-demanded force and torque is decoupled. This strategy may be designed to achieve better task performance while reducing the control complexity.

This work has been submitted to  Journal of Neurophysiology, 2020.

Exploiting Hand Dexterity for Multiple Objects Grasping

Human hand grasping multiple objects

Human hands are highly dexterous systems. Hand dexterity enables humans to grasp everyday objects in distinct poses for different manipulation purposes, and even to grasp multiple objects at the same time for complex manipulation tasks. One main reason for hand dexterity lies in the exploitation of human hand’s abundant degrees of freedom.

Inspired by the human hand dexterity, we are currently developing algorithms to enhance dexterity of robotic hand with redundant degrees of freedom. Our approach largely improves robotic hand’s capability of handling object(s) in dexterous grasping and manipulation tasks.