doc. Mgr. Matěj Hoffmann, Ph.D.

Postnatal development in infants involves building the body schema based on integrating information from different modalities. An early phase of this complex process involves coupling proprioceptive inputs with tactile information during self-touch enabled by motor babbling. Such functionality is also desirable in humanoid robots that can serve as embodied instantiation of cognitive learning. We describe a simple connectionist model composed of neural networks that learns the proprioceptive-tactile representations on a simulated iCub humanoid robot. Input signals from both modalities – joint angles and touch stimuli on both upper limbs – are first self-organized in neural maps and then connected using a universal bidirectional associative network (UBAL). The model demonstrates the ability to predict touch and its location from proprioceptive information with relatively high accuracy. We also discuss limitations of the model and the ideas for future work.

Accurately predicting contact between our bodies and environmental objects is paramount to our evolutionary survival. It has been hypothesized that multisensory neurons responding both to touch on the body, and to auditory or visual stimuli occurring near them—thus delineating our peripersonal space (PPS)—may be a critical player in this computation. However, we lack a normative account (i.e., a model specifying how we ought to compute) linking impact prediction and PPS encoding. Here, we leverage Bayesian Decision Theory to develop such a model and show that it recapitulates many of the characteristics of PPS. Namely, a normative model of impact prediction (i) delineates a graded boundary between near and far space, (ii) demonstrates an enlargement of PPS as the speed of incoming stimuli increases, (iii) shows stronger contact prediction for looming than receding stimuli—but critically is still present for receding stimuli when observation uncertainty is non-zero—, (iv) scales with the value we attribute to environmental objects, and finally (v) can account for the differing sizes of PPS for different body parts. Together, these modeling results support the conjecture that PPS reflects the computation of impact prediction, and make a number of testable predictions for future empirical studies.

Recent advancements in object shape completion have enabled impressive object reconstructions using only visual input. However, due to self-occlusion, the reconstructions have high uncertainty in the occluded object parts, which negatively impacts the performance of downstream robotic tasks such as grasping. In this letter, we propose an active visuo-haptic shape completion method called Act-VH that actively computes where to touch the objects based on the reconstruction uncertainty. Act-VH reconstructs objects from point clouds and calculates the reconstruction uncertainty using IGR, a recent state-of-the-art implicit surface deep neural network. We experimentally evaluate the reconstruction accuracy of Act-VH against five baselines in simulation and in the real world. We also propose a new simulation environment for this purpose. The results show that Act-VH outperforms all baselines and that an uncertainty-driven haptic exploration policy leads to higher reconstruction accuracy than a random policy and a policy driven by Gaussian Process Implicit Surfaces. As a final experiment, we evaluate Act-VH and the best reconstruction baseline on grasping 10 novel objects. The results show that Act-VH reaches a significantly higher grasp success rate than the baseline on all objects. Together, this letter opens up the door for using active visuo-haptic shape completion in more complex cluttered scenes.

We present a robot kinematic calibration method that combines complementary calibration approaches: self-contact, planar constraints, and self-observation. We analyze the estimation of the end effector parameters, joint offsets of the manipulators, and calibration of the complete kinematic chain (DH parameters). The results are compared with ground truth measurements provided by a laser tracker. Our main findings are: (1) When applying the complementary calibration approaches in isolation, the self-contact approach yields the best and most stable results. (2) All combinations of more than one approach were always superior to using any single approach in terms of calibration errors and the observability of the estimated parameters. Combining more approaches delivers robot parameters that better generalize to the workspace parts not used for the calibration. (3) Sequential calibration, i.e. calibrating cameras first and then robot kinematics, is more effective than simultaneous calibration of all parameters. In real experiments, we employ two industrial manipulators mounted on a common base. The manipulators are equipped with force/torque sensors at their wrists, with two cameras attached to the robot base, and with special end effectors with fiducial markers. We collect a new comprehensive dataset for robot kinematic calibration and make it publicly available. The dataset and its analysis provide quantitative and qualitative insights that go beyond the specific manipulators used in this work and apply to self-contained robot kinematic calibration in general.

Typically, mechanical design specifications provide the basis for a robot model and kinematic and dynamic mappings are constructed and remain fixed during operation. However, there are many sources of inaccuracies (e.g., assembly process, mechanical elasticity, friction). Furthermore, with the advent of collaborative, social, or soft robots, the stiffness of the materials and the precision of the manufactured parts drops and Computer-aided design (CAD) models provide a less accurate basis for the models. Humans, on the other hand, seamlessly control their complex bodies, adapt to growth or failures, and use tools. Exploiting multimodal sensory information plays a key part in these processes. In this chapter, differences between body representations in the brain and robot body models are established and the possibilities for learning robot models in biologically inspired ways are assessed.

Humans excel in combining information from multiple sensory modalities, controlling their complex bodies, adapting to growth, failures, or using tools. These capabilities are also highly desirable in robots. They are displayed by machines to some extent – yet, as is so often the case, the artificial creatures are lagging behind. The key foundation is an internal representation of the body that the agent – human or robot – has developed. In the biological realm, evidence has been accumulated by diverse disciplines giving rise to the concepts of body image, body schema, and others. In robotics, a model of the robot is an indispensable component that enables control of the machine. In this chapter, we compare the character of body representations in biology with their robotic counterparts and relate that to the differences in performance that we observe. In some sense, robots have a lot in common with Ian Waterman – “the man who lost his body” – in that they rely on an explicit, veridical body model (body image taken to the extreme) and lack any implicit, multimodal representation (like the body schema) of their bodies. The core of this work is a detailed look at the somatoperceptual processing “pipeline” from inputs (tactile and proprioceptive afference, efferent commands), over “body representations” (superficial schema, postural schema, model of body size and shape), to perceptual processes like spatial localization of touch. A direct comparison with solutions to the same task in robots allows us to make important steps in converting this conceptual schematics into a computational model. As an additional aspect, we briefly look at the question of why robots do not experience body illusions. Finally, we discuss how robots can inform the biological sciences dealing with body representations and which of the features of the “body in the brain” should be transferred to robots, giving rise to more adaptive and resilient, self-calibrating machines.

Soft electronic skins are one of the means to turn a classical industrial manipulator into a collaborative robot. For manipulators that are already fit for physical human–robot collaboration, soft skins can make them even safer. In this work, we study the after impact behavior of two collaborative manipulators (UR10e and KUKA LBR iiwa) and one classical industrial manipulator (KUKA Cybertech), in the presence or absence of an industrial protective skin (AIRSKIN). In addition, we isolate the effects of the passive padding and the active contribution of the sensor to robot reaction. We present a total of 2250 collision measurements and study the impact force, contact duration, clamping force, and impulse. This collected dataset is publicly available. We summarize our results as follows. For transient collisions, the passive skin properties lowered the impact forces by about 40 %. During quasi-static contact, the effect of skin covers – active or passive – cannot be isolated from the collision detection and reaction by the collaborative robots. Important effects of the stop categories triggered by the active protective skin were found. We systematically compare the different settings and compare the empirically established safe velocities with prescriptions by the ISO/TS 15066. In some cases, up to the quadruple of the ISO/TS 15066 prescribed velocity can comply with the impact force limits and thus be considered safe. We propose an extension of the formulas relating impact force and permissible velocity that take into account the stiffness and compressible thickness of the protective cover, leading to better predictions of the collision forces. At the same time, this work emphasizes the need for in situ measurements as all the factors we studied – presence of active/passive skin, safety stop settings, robot collision reaction, impact direction, and, of course, velocity – have effects on the force evolution after impact.

Various approaches can ascertain the safety and efficiency of physical human-robot collaboration. This paper presents the concept of robot functional mode switching to efficiently ensure human safety during collaborative tasks based on biomechanical pain and injury data and task information. Besides the robot's reflected inertial properties summarizing its impact dynamics, our concept also integrates safe and smooth velocity shaping that respects human partner motion, interaction type, and task knowledge. We further discuss different approaches to safely shape the robot velocity without sacrificing the overall task execution time and motion smoothness. The experimental results showed that our proposed approaches could decrease jerk level during functional mode switching and limit the impact of safety measures on productivity, especially when guided with additional task knowledge.

We study the performance of state-of-the-art human keypoint detectors in the context of close proximity human-robot interaction. The detection in this scenario is specific in that only a subset of body parts such as hands and torso are in the field of view. In particular, (i) we survey existing datasets with human pose annotation from the perspective of close proximity images and prepare and make publicly available a new Human in Close Proximity (HiCP) dataset; (ii) we quantitatively and qualitatively compare state-of-the-art human whole-body 2D keypoint detection methods (OpenPose, MMPose, AlphaPose, Detectron2) on this dataset; (iii) since accurate detection of hands and fingers is critical in applications with handovers, we evaluate the performance of the MediaPipe hand detector; (iv) we deploy the algorithms on a humanoid robot with an RGB-D camera on its head and evaluate the performance in 3D human keypoint detection. A motion capture system is used as reference. The best performing whole-body keypoint detectors in close proximity were MMPose and AlphaPose, but both had difficulty with finger detection. Thus, we propose a combination of MMPose or AlphaPose for the body and MediaPipe for the hands in a single framework providing the most accurate and robust detection. We also analyse the failure modes of individual detectors---for example, to what extent the absence of the head of the person in the image degrades performance. Finally, we demonstrate the framework in a scenario where a humanoid robot interacting with a person uses the detected 3D keypoints for whole-body avoidance maneuvers.

When leaving the aquatic constrained environment of the womb, newborns are thrown into the world with essentially new laws and regularities that govern their interactions with the environment. Here, we study how spontaneous self-contacts can provide material for learning implicit models of the body and its action possibilities in the environment. Specifically, we investigate the space of only somatosensory (tactile and proprioceptive) activations during self-touch configurations in a simple model agent. Using biologically motivated overlapping receptive fields in these modalities, a variational autoencoder (VAE) in a denoising framework is trained on these inputs. The denoising properties of the VAE can be exploited to fill in the missing information. In particular, if tactile stimulation is provided on a single body part, the model provides a configuration that is closer to a previously experienced self-contact configuration. Iterative passes through the VAE reconstructions create a control loop that brings about reaching for stimuli on the body. Furthermore, due to the generative properties of the model, previously unsampled proprioceptive-tactile configurations can also be achieved. In the future, we will seek a closer comparison with empirical data on the kinematics of spontaneous self-touch in infants and the results of reaching for stimuli on the body.

We investigated the use of impact sounds generated during exploratory behaviors in a robotic manipulation setup as cues for predicting object surface material and for recognizing individual objects. We collected and make available the YCB-impact sounds dataset which includes over 3,500 impact sounds for the YCB set of everyday objects lying on a table. Impact sounds were generated in three modes: (i) human holding a gripper and hitting, scratching, or dropping the object; (ii) gripper attached to a teleoperated robot hitting the object from the top; (iii) autonomously operated robot hitting the objects from the side with two different speeds. A convolutional neural network (ResNet34) is trained from scratch to recognize the object material (steel, aluminium, hard plastic, soft plastic, other plastic, ceramic, wood, paper/cardboard, foam, glass, rubber) from a single impact sound. On the manually collected dataset with more variability in the action, nearly 60\% accuracy for the test set (unseen objects) was achieved. On a robot setup and a stereotypical poking action from top, accuracy of 85% was achieved. This performance drops to 79% if multiple exploratory actions are combined. Individual objects from the set of 75 objects can be recognized with a 79% accuracy. This work demonstrates promising results regarding the possibility of using sound for recognition in tasks like single-stream recycling where objects have to be sorted based on their material composition.

Children are not born tabula rasa. However, interacting with the environment through their body movements in the first months after birth is critical to building the models or representations that are the foundation for everything that follows. We present longitudinal data on spontaneous behavior of three infants observed between about 8 and 25 weeks of age in supine position. We combined manual scoring of video recordings with an automatic extraction of motion data in order to study infants’ behavioral patterns and developmental progression such as: (i) spatial distribution of self-touches on the body, (ii) spatial patterns and regularities of hand movements, (iii) midline crossing, (iv) preferential use of one arm, and (v) dynamic patterns of movements indicative of goal-directedness. From the patterns observed in this pilot data set, we can speculate on the development of first body and peripersonal space representations. Several methods of extracting 3D kinematics from videos have recently been made available by the computer vision community. We applied one of these methods on infant videos and provide guidelines on its possibilities and limitations—a methodological contribution to automating the analysis of infant videos. In the future, we plan to use the patterns we extracted from the recordings as inputs to embodied computational models of learning of body representations in infancy.

The need to guarantee safety of collaborative robots limits their performance, in particular, their speed and hence cycle time. The standard ISO/TS 15066 defines the Power and Force Limiting operation mode and prescribes force thresholds that a moving robot is allowed to exert on human body parts during impact, along with a simple formula to obtain maximum allowed speed of the robot in the whole workspace. In this work, we measure the forces exerted by two collaborative manipulators (UR10e and KUKA LBR iiwa) moving downward against an impact measuring device. First, we empirically show that the impact forces can vary by more than 100 percent within the robot workspace. The forces are negatively correlated with the distance from the robot base and the height in the workspace. Second, we present a data-driven model, 3D Collision-Force-Map, predicting impact forces from distance, height, and velocity and demonstrate that it can be trained on a limited number of data points. Third, we analyze the force evolution upon impact and find that clamping never occurs for the UR10e. We show that formulas relating robot mass, velocity, and impact forces from ISO/TS 15066 are insufficient—leading both to significant underestimation and overestimation and thus to unnecessarily long cycle times or even dangerous applications. We propose an empirical method that can be deployed to quickly determine the optimal speed and position where a task can be safely performed with maximum efficiency.

Humans and animals excel in combining information from multiple sensory modalities, controlling their complex bodies, adapting to growth or failures, or using tools. The key foundation is an internal representation of the body that the agent—human, animal, or robot—has developed. In the biological realm, evidence has been accumulating in diverse disciplines, giving rise to the concepts of body image, body schema, and others. In robotics, a model of the robot is an indispensable component that enables to control the machine. This chapter compares the character of body representations in biology with their robotic counterparts and relates that to the differences in performance observed. Conclusions are drawn about how robots can inform the biological sciences dealing with body representations and which of the features of the ‘body in the brain’ should be transferred to robots, giving rise to more adaptive and resilient self-calibrating machines.

In this paper, we present an integrated system that includes reasoning from visual and natural language inputs, action and motion planning, executing tasks by a robotic arm, manipulating objects, and discovering their properties. A vision to action module recognises the scene with objects and their attributes and analyses enquiries formulated in natural language. It performs multi-modal reasoning and generates a sequence of simple actions that can be executed by a robot. The scene model and action sequence are sent to a planning and execution module that generates a motion plan with collision avoidance, simulates the actions, and executes them. We use synthetic data to train various components of the system and test on a real robot to show the generalization capabilities. We focus on a tabletop scenario with objects that can be grasped by our embodied agent i.e. a 7DoF manipulator with a two-finger gripper. We evaluate the agent on 60 representative queries repeated 3 times (e.g., ’Check what is on the other side of the soda can’) concerning different objects and tasks in the scene. We perform experiments in a simulated and real environment and report the success rate for various components of the system. Our system achieves up to 80.6% success rate on challenging scenes and queries. We also analyse and discuss the challenges that such an intelligent embodied system faces.

The accuracy of robot models critically impacts their performance. With the advent of collaborative, social, or soft robots, the stiffness of the materials and the precision of the manufactured parts drops and CAD models provide a less accurate basis for the models. On the other hand, the machines often come with a rich set of powerful yet inexpensive sensors, which opens up the possibility for self-contained calibration approaches that can be performed autonomously and repeatedly by the robot. In this work, we extend the theory dealing with robot kinematic calibration by incorporating new sensory modalities (e.g., cameras on the robot, whole-body tactile sensors), calibration types, and their combinations. We provide a unified formulation that makes it possible to combine traditional approaches (external laser tracker, constraints from contact with the external environment) with self-contained calibration available to humanoid robots (self-observation, self-contact) in a single framework and single cost function. Second, we present an open source toolbox for Matlab that provides this functionality, along with additional tools for preprocessing (e.g., dataset visualization) and evaluation (e.g., observability/identifiability). We illustrate some of the possibilities of this tool through calibration of two humanoid robots (iCub, Nao) and one industrial manipulator (dual-arm setup with Yaskawa-Motoman MA1400).

Self-recognition or self-awareness is a capacity attributed typically only to humans and few other species. The definitions of these concepts vary and little is known about the mechanisms behind them. However, there is a Turing test-like benchmark: the mirror self-recognition, which consists in covertly putting a mark on the face of the tested subject, placing her in front of a mirror, and observing the reactions. In this work, first, we provide a mechanistic decomposition, or process model, of what components are required to pass this test. Based on these, we provide suggestions for empirical research. In particular, in our view, the way the infants or animals reach for the mark should be studied in detail. Second, we develop a model to enable the humanoid robot Nao to pass the test. The core of our technical contribution is learning the appearance representation and visual novelty detection by means of learning the generative model of the face with deep auto-encoders and exploiting the prediction error. The mark is identified as a salient region on the face and reaching action is triggered, relying on a previously learned mapping to arm joint angles. The architecture is tested on two robots with completely different face.

Robots were largely missing the sense of touch for decades. As artificial sensitive skins covering large areas of robot bodies are starting to appear, to be useful to the machines, sensor positions on the robot body are needed. In this work, a Nao humanoid robot was retrofitted with pressure-sensitive skin on the head, torso, and arms. We experimentally compare the accuracy and effort associated with the following skin spatial calibration approaches and their combinations: (i) combining CAD models and skin layout in 2D, (ii) 3D reconstruction from images, (iii) using robot kinematics to calibrate skin by self-contact. To acquire 3D positions of taxels on individual skin parts, methods (i) and (ii) were similarly laborious but 3D reconstruction was more accurate. To align these 3D point clouds with the robot kinematics, two variants of self-contact were employed: skin-on-skin and utilization of a custom end effector (finger). In combination with the 3D reconstruction data, mean calibration errors below the radius of individual sensors were achieved (2 mm). Significant perturbation of more than 100 torso taxel positions could be corrected using self-contact calibration, reaching approx. 3 mm mean error. This work is not a proof of concept but deployment of the approaches at scale: the outcome is actual spatial calibration of all 970 taxels on the robot body. As the different calibration approaches are evaluated in isolation as well as in different combinations, this work provides a guideline applicable to spatial calibration of different sensor arrays.

The mechanisms of infant development are far from understood. Learning about one's own body is likely a foundation for subsequent development. Here we look specifically at the problem of how spontaneous touches to the body in early infancy may give rise to first body models and bootstrap further development such as reaching competence. Unlike visually elicited reaching, reaching to own body requires connections of the tactile and motor space only, bypassing vision. Still, the problems of high dimensionality and redundancy of the motor system persist. In this work, we present an embodied computational model on a simulated humanoid robot with artificial sensitive skin on large areas of its body. The robot should autonomously develop the capacity to reach for every tactile sensor on its body. To do this efficiently, we employ the computational framework of intrinsic motivations and variants of goal babbling-as opposed to motor babbling-that prove to make the exploration process faster and alleviate the ill-posedness of learning inverse kinematics. Based on our results, we discuss the next steps in relation to infant studies: what information will be necessary to further ground this computational model in behavioral data.

With the advent of different electronic skins sensitive to touch and robots composed of soft materials, tactile or haptic human-robot interaction is gaining importance. We designed a highly realistic artificial hand aiming to reproduce human-to-human physical contact through a special morphology imitating flesh and bones and a heating system imitating human body temperature. The mechanical response properties of different finger designs were analyzed and the most mimetic one came very close to a human finger. We designed three experiments with participants using haptic exploration to evaluate the human-likeness of: (1) finger morphologies; (2) complete hands: real human vs. soft and warm artificial hand vs. rubber hand (3) the hand mounted on a manipulator with fixed vs. passive compliant wrist in a handshake scenario. First, participants find the mimetic finger morphology most humanlike. Second, people can reliably distinguish the real human hand, the artificial one, and a rubber hand. In terms of humanlikeness (Anthropomorphism, Animacy, and Likeability), the human hand scores better than the artificial hand which in turn clearly outperforms the rubber hand. The temperature, or "warmth", was rated as the most human-like feature of the artificial hand.

As robots are leaving dedicated areas on the factory floor and start to share workspaces with humans, safety of such collaboration becomes a major challenge. In this work, we propose new approaches to robot velocity modulation: while the robot is on a path prescribed by the task, it predicts possible collisions with the human and gradually slows down, proportionally to the danger of collision. Two principal approaches are developed—Impulse Orb and Prognosis Window—that dynamically determine the possible robot-induced collisions and apply a novel velocity modulating approach, in which the phase progress of the robot trajectory is modulated while the desired robot path remains intact. The methods guarantee that the robot will halt before contacting the human, but they are less conservative and more flexible than solutions using reduced speed and complete stop only, thereby increasing the effectiveness of human-robot collaboration. This approach is especially useful in constrained setups where the robot path is prescribed. Speed modulation is smooth and does not lead to abrupt motions, making the behavior of the robot also better understandable for the human counterpart. The two principal methods under different parameter settings are experimentally validated in a human-robot interaction scenario with the Franka Emika Panda robot, an external RGB-D camera, and human keypoint detection using OpenPose.

Infant development of reaching to tactile targets on the skin has been studied little, despite its daily use during adaptive behaviors such as removing foreign stimuli or scratching an itch. We longitudinally examined the development of infant reaching strategies (from just under 2 to 11 months) approximately every other week with a vibrotactile stimulus applied to eight different locations on the face (left/right/center temple, left/right ear, left/right mouth corners, and chin). Successful reaching for the stimulus uses tactile input and proprioception to localize the target and move the hand to it. We studied the developmental progression of reaching and grasping strategies. As infants became older the likelihood of using the hand to reach to the target - versus touching the target with another body part or surface such as the upper arm or chair - increased. For trials where infants reached to the target with the hand, infants also refined their hand postures with age. As infants became older, they made fewer contacts with a closed fist or the dorsal part of the hand and more touches/grasps with the fingers or palm. Results suggest that during the first year infants become able to act more precisely on tactile targets on the face.

Průmyslová odvětví bývají doprovázena řadou norem či standardů, které umožňují regulovat dané odvětví. Přestože má norma sloužit jako návod pro pracovníky z oboru, sama je často psaná spíše jazykem právníků než uživatelů. Z tohoto důvodu a také pro svou vysokou cenu mohou normy připomínat spíše bedlivě střežená tajemství než snahu komunikovat obsah srozumitelně. Sami se zabýváme normou pro kolaborativní robotiku, ISO/TS 15066, s kterou jsme se pro naši práci museli podrobně seznámit. V tomto příspěvku představujeme základní poznatky o normách a rady jak je převést do praxe. Pracujeme s normou pro kolaborativní roboty, ale rady jsou platné obecně, nejen normy, které jsou vydávány ISO nebo se týkají robotů (např. pro psychology ISO 10667-1).

We present a neural network learning architecture composed of a Restricted Boltzmann Machine (RBM) and a Conditional RBM (CRBM) that performs multisen- sory integration and prediction, motivated by the problem of learning a representation of defensive peripersonal space. This work follows up on our previous work (Straka and Hoffmann 2017) where we proposed a network composed of a RBM and a feedforward neural network (FFNN). In this work, with a similar 2D simulated scenario, we sought to replace the FFNN with an RBM-like module and opted for the CRBM which is responsible for making a temporal prediction. We demonstrate that the new architecture is capable of learning to map from visual and tactile inputs at a previous time step (without tactile activation) to future activations with the visual stimulus at the “skin” and corresponding tactile activation, including the confidence of the predictions.

The development of reaching in infants has been studied for nearly nine decades. Originally, it was thought that early reaching is visually guided, but more recent evidence is suggestive of 'visually elicited' reaching, i.e. infant is gazing at the object rather than its hand during the reaching movement. The importance of haptic feedback has also been emphasized. Inspired by these findings, in this work we use the simulated iCub humanoid robot to construct a model of reaching development. The robot is presented with different objects, gazes at them, and performs motor babbling with one of its arms. Successful contacts with the object are detected through tactile sensors on hand and forearm. Such events serve as the training set, constituted by images from the robot's two eyes, head joints, tactile activation, and arm joints. A deep neural network is trained with images and head joints as inputs and arm configuration and touch as output. After learning, the network can successfully infer arm configurations that would result in a successful reach, together with prediction of tactile activation (i.e. which body part would make contact). Our main contribution is twofold: (i) our pipeline is end-to-end from stereo images and head joints (6 DoF) to armtorso configurations (10 DoF) and tactile activations, without any preprocessing, explicit coordinate transformations etc.; (ii) unique to this approach, reaches with multiple effectors corresponding to different regions of the sensitive skin are possible.

Reaching to target locations on the body has been studied little despite its importance for adaptive behaviors such as feeding, grooming, and indicating a source of discomfort. This behavior requires multisensory integration given that it involves coordination of touch, proprioception, and sometimes vision as well as action. Here we examined the origins of this skill by investigating how infants begin to localize targets on the body and the motor strategies by which they do so. Infants (7-21 months of age) were prompted to reach to a vibrating target placed at five arm/hand locations (elbow, crook of elbow, forearm, palm, and top of hand) one by one. To manually localize the target, infants needed to reach with one arm to the other. Results suggest that coordination increases with age in the strategies that infants used to localize body targets. Most infants showed bimanual coordination and usually moved the target arm toward the reaching arm to assist reaching. Furthermore, intersensory coordination increased with age. Simultaneous movements of the two arms increased with age, as did coordination between vision and reaching. The results provide new information about the development of multisensory integration during tactile localization and how such integration is linked to action. (C) 2019 Published by Elsevier Inc.

Mechanism calibration is an important and nontrivial task in robotics. Advances in sensor technology make affordable but increasingly accurate devices such as cameras and tactile sensors available, making it possible to perform automated self-contained calibration relying on redundant information in these sensory streams. In this letter, we use a simulated iCub humanoid robot with a stereo camera system and end-effector contact emulation to quantitatively compare the performance of kinematic calibration by employing different combinations of intersecting kinematic chains-either through self-observation or self-touch. The parameters varied were as follows: first, type and number of intersecting kinematic chains used for calibration, second, parameters and chains subject to optimization, third, amount of initial perturbation of kinematic parameters, fourth, number of poses/configurations used for optimization, and fifth, amount of measurement noise in end-effector positions/cameras. The main findings are as follows: 1) calibrating parameters of a single chain (e.g., one arm) by employing multiple kinematic chains ("self-observation" and "self-touch") is superior in terms of optimization results as well as observability; 2) when using multichain calibration, fewer poses suffice to get similar performance compared to when, for example, only observation from a single camera is used; 3) parameters of all chains (here 86 DH parameters) can be subject to calibration simultaneously and with 50 (100) poses, end-effector error of around 2 (1) mm can be achieved; and 4) adding noise to a sensory modality degrades performance of all calibrations employing the chains relying on this information.

So-called collaborative robots are a current trend in industrial robotics. However, they still face many problems in practical application such as reduced speed to ascertain their collaborativeness. The standards prescribe two regimes: (i) speed and separation monitoring and (ii) power and force limiting, where the former requires reliable estimation of distances between the robot and human body parts and the latter imposes constraints on the energy absorbed during collisions prior to robot stopping. Following the standards, we deploy the two collaborative regimes in a single application and study the performance in a mock collaborative task under the individual regimes, including transitions between them. Additionally, we compare the performance under ``safety zone monitoring'' with keypoint pair-wise separation distance assessment relying on an RGB-D sensor and skeleton extraction algorithm to track human body parts in the workspace. Best performance has been achieved in the following setting: robot operates at full speed until a distance threshold between any robot and human body part is crossed; then, reduced robot speed per power and force limiting is triggered. Robot is halted only when the operator's head crosses a predefined distance from selected robot parts. We demonstrate our methodology on a setup combining a KUKA LBR iiwa robot, Intel RealSense RGB-D sensor and OpenPose for human pose estimation.

Humans use signs, e.g., sentences in a spoken language, for communication and thought. Hence, symbol systems like language are crucial for our communication with other agents and adaptation to our real-world environment. The symbol systems we use in our human society adaptively and dynamically change over time. In the context of artificial intelligence (AI) and cognitive systems, the symbol grounding problem has been regarded as one of the central problems related to symbols . However, the symbol grounding problem was originally posed to connect symbolic AI and sensorimotor information and did not consider many interdisciplinary phenomena in human communication and dynamic symbol systems in our society, which semiotics considered. In this paper, we focus on the symbol emergence problem, addressing not only cognitive dynamics but also the dynamics of symbol systems in society, rather than the symbol grounding problem. We first introduce the notion of a symbol in semiotics from the humanities, to leave the very narrow idea of symbols in symbolic AI. Furthermore, over the years, it became more and more clear that symbol emergence has to be regarded as a multifaceted problem. Therefore, second, we review the history of the symbol emergence problem in different fields, including both biological and artificial systems, showing their mutual relations. We summarize the discussion and provide an integrative viewpoint and comprehensive overview of symbol emergence in cognitive systems. Additionally, we describe the challenges facing the creation of cognitive systems that can be part of symbol emergence systems.

With robots leaving factories and entering less controlled domains, possibly sharing the space with humans, safety is paramount and multimodal awareness of the body surface and the surrounding environment is fundamental. Taking inspiration from peripersonal space representations in humans, we present a framework on a humanoid robot that dynamically maintains such a protective safety zone, composed of the following main components: (i) a human 2D keypoints estimation pipeline employing a deep learning based algorithm, extended here into 3D using disparity; (ii) a distributed peripersonal space representation around the robot»s body parts; (iii) a reaching controller that incorporates all obstacles entering the robot»s safety zone on the fly into the task. Pilot experiments demonstrate that an effective safety margin between the robot's and the human's body parts is kept. The proposed solution is flexible and versatile since the safety zone around individual robot and human body parts can be selectively modulated---here we demonstrate stronger avoidance of the human head compared to rest of the body. Our system works in real time and is self-contained, with no external sensory equipment and use of onboard cameras only.

This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both human and robot. The framework, based on a biologically grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor learning algorithms, as well as planning abilities and an autobiographical memory. The architecture as a whole drives the robot behavior to solve the symbol grounding problem, acquire language capabilities, execute goal-oriented behavior, and express a verbal narrative of its own experience in the world. We validate our approach in human-robot interaction experiments with the iCub humanoid robot, showing that the proposed cognitive architecture can be applied in real time within a realistic scenario and that it can be used with naive users.

For robots to share the environment and cooperate with humans without barriers, we need to guarantee safety to the operator and, simultaneously, to maximize the robot’s usability. Safety is typically guaranteed by controlling the robot movements while, possibly, taking into account physical contacts with the operator, objects or tools. If possible, also the safety of the robot must be guaranteed. Not less importantly, as the complexity of the robots and their skills increase, usability becomes a concern. Social interaction technologies can save the day by enabling natural human-robot collaboration. In this paper we show a possible integration of physical and social Human-Robot Interaction methods (pHRI and sHRI respectively). Our reference task is object hand-over. We test both the case of the robot initiating the action and, vice versa, the robot receiving an object from the operator. Finally, we discuss possible extension with higher-level planning systems for added flexibility and reasoning skills.

Using the iCub humanoid robot with an artificial pressure-sensitive skin, we investigate how representations of the whole skin surface resembling those found in primate primary somatosensory cortex can be formed from local tactile stimulations traversing the body of the physical robot. We employ the well-known self-organizing map algorithm and introduce its modification that makes it possible to restrict the maximum receptive field (MRF) size of neuron groups at the output layer. This is motivated by findings from biology where basic somatotopy of the cortical sheet seems to be prescribed genetically and connections are localized to particular regions. We explore different settings of the MRF and the effect of activity-independent (input-output connections constraints implemented by MRF) and activity-dependent (learning from skin stimulations) mechanisms on the formation of the tactile map. The framework conveniently allows one to specify prior knowledge regarding the skin topology and thus to effectively seed a particular representation that training shapes further. Furthermore, we show that the MRF modification facilitates learning in situations when concurrent stimulation at nonadjacent places occurs (“multitouch”). The procedure was sufficiently robust and not intensive on the data collection and can be applied to any robots where representation of their “skin” is desirable.

A large body of compelling evidence has been accumulated demonstrating that embodiment—the agent’s physical setup, including its shape, materials, sensors, and actuators—is constitutive for any form of cognition, and, as a consequence, models of cognition need to be embodied. In contrast to methods from empirical sciences to study cognition, robots can be freely manipulated and virtually all key variables of their embodiment and control programs can be systematically varied. As such, they provide an extremely powerful tool of investigation. We present a robotic, bottom-up, or developmental approach, focusing on three stages: (1) low-level behaviors like walking and reflexes, (2) learning regularities in sensorimotor spaces, and (3) human-like cognition. We also show that robotic-based research is not only a productive path to deepening our understanding of cognition, but that robots can strongly benefit from human-like cognition in order to become more autonomous, robust, resilient, and safe.

Human-robot collaboration including close physical human-robot interaction (pHRI) is a current trend in industry and also science. The safety guidelines prescribe two modes of safety: (i) power and force limitation and (ii) speed and separation monitoring. We examine the potential of robots equipped with artificial sensitive skin and a protective safety zone around it (peripersonal space) to safe pHRI.

The interaction of humans and robots in less constrained environments gains a lot of attention lately and safety of such interaction is of utmost importance. Two ways of risk assessment are prescribed by recent safety standards: (i) power and force limiting and (ii) speed and separation monitoring. Unlike typical solutions in industry that are restricted to mere safety zone monitoring, we present a framework that realizes separation distance monitoring between a robot and a human operator in a detailed, yet versatile, transparent, and tunable fashion. The separation distance is assessed pair-wise for all keypoints on the robot and the human body and as such can be selectively modified to account for specific conditions. The operation of this framework is illustrated on a Nao humanoid robot interacting with a human partner perceived by a RealSense RGB-D sensor and employing the OpenPose human skeleton estimation algorithm.

The safety of collaborative robots’ and human interaction can be guaranteed in two main ways: (i) power and force limiting and (ii) speed and separation monitoring. We present a framework that realises separation distance monitoring between a robot and a human operator based on key point pair-wise evaluation. We show preliminary results using a setup with a Nao humanoid robot and a RealSense RGB-D sensor and employing OpenPose human skeleton estimation algorithm, and work in progress on a KUKA LBR iiwa platform.

This study focuses on how the body schema develops during the first months of life, by investigating infants' motor responses to localized vibrotactile stimulation on their limbs. Vibrotactile stimulation was provided by small buzzers that were attached to the infants' four limbs one at a time. Four age groups were compared cross-sectionally (3-, 4-, 5-, and 6-month-olds). We show that before they actually reach for the buzzer, which, according to previous studies, occurs around 7-8months of age, infants demonstrate emerging knowledge about their body's configuration by producing specific movement patterns associated with the stimulated body area. At 3months, infants responded with an increase in general activity when the buzzer was placed on the body, independently of the vibrator's location. Differentiated topographical awareness of the body seemed to appear around 5months, with specific responses resulting from stimulation of the hands emerging first, followed by the differentiation of movement patterns associated with the stimulation of the feet. Qualitative analyses revealed specific movement types reliably associated with each stimulated location by 6months of age, possibly preparing infants' ability to actually reach for the vibrating target. We discuss this result in relation to newborns' ability to learn specific movement patterns through intersensory contingency.

We have been observing how infants between 3 and 21 months react when a vibrotactile stimulation (a buzzer) is applied to different parts of their bodies. Responses included in particular movement of the stimulated body part and successful reaching for and removal of the buzzer. Overall, there is a pronounced developmental progression from general to specific movement patterns, especially in the first year. In this article we review the series of studies we conducted and then focus on possible mechanisms that might explain what we observed. One possible mechanism might rely on the brain extracting "sensorimotor contingencies" linking motor actions and resulting sensory consequences. This account posits that infants are driven by intrinsic motivation that guides exploratory motor activity, at first generating random motor babbling with self-touch occurring spontaneously. Later goal-oriented motor behavior occurs, with self-touch as a possible effective tool to induce informative contingencies. We connect this sensorimotor view with a second possible account that appeals to the neuroscientific concepts of cortical maps and coordinate transformations. In this second account, the improvement of reaching precision is mediated by refinement of neuronal maps in primary sensory and motor cortices - the homunculi - as well as in frontal and parietal cortical regions dedicated to sensorimotor processing. We complement this theoretical account with modeling on a humanoid robot with artificial skin where we implemented reaching for tactile stimuli as well as learning the "somatosensory homunculi". We suggest that this account can be extended to reflect the driving role of sensorimotor contingencies in human development. In our conclusion we consider possible extensions of our current experiments which take account of predictions derived from both these kinds of models.

The space immediately surrounding our body, or peripersonal space, is crucial for interaction with the environment. In primate brains, specific neural circuitry is responsible for its encoding. An important component is a safety margin around the body that draws on visuo-tactile interactions: approaching stimuli are registered by vision and processed, producing anticipation or prediction of contact in the tactile modality. The mechanisms of this representation and its development are not understood. We propose a computational model that addresses this: a neural network composed of a Restricted Boltzmann Machine and a feedforward neural network. The former learns in an unsupervised manner to represent position and velocity features of the stimulus. The latter is trained in a supervised way to predict the position of touch (contact). Unique to this model, it considers: (i) stimulus position and velocity, (ii) uncertainty of all variables, and (iii) not only multisensory integration but also prediction.

Engineers fine-tune the design of robot bodies for control purposes; however, a methodology or set of tools is largely absent, and optimization of morphology (shape, material properties of robot bodies, etc.) is lagging behind the development of controllers. This has become even more prominent with the advent of compliant, deformable or ‘soft’ bodies. These carry substantial potential regarding their exploitation for control—sometimes referred to as ‘morphological computation’. In this article, we briefly review different notions of computation by physical systems and propose the dynamical systems framework as the most useful in the context of describing and eventually designing the interactions of controllers and bodies. Then, we look at the pros and cons of simple versus complex bodies, critically reviewing the attractive notion of ‘soft’ bodies automatically taking over control tasks. We address another key dimension of the design space—whether model-based control should be used and to what extent it is feasible to develop faithful models for different morphologies.

The contribution of the body to cognition and control in natural and artificial agents is increasingly described as “offloading computation from the brain to the body,” where the body is said to perform “morphological computation.” Our investigation of four characteristic cases of morphological computation in animals and robots shows that the “offloading” perspective is misleading. Actually, the contribution of body morphology to cognition and control is rarely computational, in any useful sense of the word. We thus distinguish (1) morphology that facilitates control, (2) morphology that facilitates perception, and the rare cases of (3) morphological computation proper, such as reservoir computing, where the body is actually used for computation. This result contributes to the understanding of the relation between embodiment and computation: The question for robot design and cognitive science is not whether computation is offloaded to the body, but to what extent the body facilitates cognition and control—how it contributes to the overall orchestration of intelligent behavior.

Humans and animals are constantly exposed to a continuous stream of sensory information from different modalities. At the same time, they form more compressed representations of concepts or symbols. In species that use language, this process is further structured by this interaction, where a mapping between the sensorimotor concepts and linguistic elements needs to be established. There is evidence that children might be learning language by simply disambiguating potential meanings based on multiple exposures to utterances in different contexts (cross-situational learning). In existing models, the mapping between modalities is usually found in a single step by directly using frequencies of referent and meaning co-occurrences. In this paper, we present an extension of this one-step mapping and introduce a newly proposed sequential mapping algorithm together with a publicly available Matlab implementation. For demonstration, we have chosen a less typical scenario: instead of learning to associate objects with their names, we focus on body representations. A humanoid robot is receiving tactile stimulations on its body, while at the same time listening to utterances of the body part names (e.g., hand, forearm, and torso). With the goal of arriving at the correct “body categories”, we demonstrate how a sequential mapping algorithm outperforms one-step mapping. In addition, the effect of data set size and noise in the linguistic input are studied.