Full-Day Workshop at

Humanoids2019 Organized by: Kourosh Darvish, Daniele Pucci, Serena Ivaldi, Eiichi Yoshida

Where: Humanoids2019, Hilton Hotel, Toronto, Canada

When: October 15th, 2019

Call for contribution

Submission deadline: 5th of October, 2019

Notification of acceptance: 6th of October, 2019

Outline

Call for Contribution

Pictures

Objectives

Teleoperation allows humans to integrate their cognitive capabilities with the apparatus to perform given tasks. Teleoperation contributes significantly in a wide range of applications including manipulation in hazardous environments, telepresence, telesurgery, and space exploration. Owing to the anthropomorphic design of humanoid robots, humanoid robot teleoperation facilitate higher capabilities during teleoperation compared to other robots as they have higher manoeuvrability and manipulation capabilities. At the same time, the complexity of humanoid robots offers more challenges for teleoperation, particularly in unstructured dynamic environments. For successful task completion, the level of autonomy, team organization, information exchange between the operator and the robot, as well as the mapping between the human and robot motions play a vital role in teleoperation performance. Moreover, immersive teleoperation systems consider different technologies for perceiving the human motion or commands; and provide feedback to the human, including haptic and visual feedback. The intuitive and natural design of the teleoperation interfaces increases the chance of successful execution of the task.

In that regard, the teleoperation of humanoid robot establishes new questions, that should be answered when designing a teleoperation system. What are the factors which determine the proper autonomy level of the robot for teleoperation scenarios? What is the human role in teleoperation scenario? Does robot need to predict the human motions and intentions in teleoperation scenarios? How we can provide a balance between the robot dexterity, manoeuvrability, and robot stability while it follows the human commands? How we can map the human motions to the robot motions in situations which the human and robot dynamics are largely different, e.g., space exploration scenarios? How we can design intuitive and natural interfaces for human, to increase the performance and effectiveness while providing immersive teleoperation experience to the human?

This workshop aims at gathering researchers from humanoid robotics, teleoperation, and human motions studies together in order to present the latest results in humanoid robot teleoperation and draw the challenges that the research community faces to effectively teleoperate the robot for different scenarios.

Back to the outline

Agenda

Time	Talk	Media

8.45 - 9.00	Kourosh Darvish Welcome & Introduction Teleoperation of Walking Humanoid Robot, iCub	Presentation, Video
9.00 - 9.30	Serena Ivaldi Teleoperation for Human-Humanoid Collaboration	Presentation, Video
9.30 - 9.50	Yuto Nakanishi Teleoperation of humanoid robot, space teleoperation challenges and approaches in Gitai	Video
9.50 - 10.10	ANA Avatar XPRIZE The ANA Avatar XPRIZE: Inspiring creators, inventors and futurists to design and construct the future of robotic avatars	Video
10.10 - 10.20	Allison Bushman Generating Human-Like Motion on Robots Through Teleoperation in Functional Tasks	Paper, Video
10.20 - 10.30	Sanket Gaurav Deep Correspondence Learning for Effective Robotic Teleoperation using Virtual Reality	Paper, Video
10.30 - 11.00	Coffee break
11.00 - 11.30	Enrico Mingo Teleoperation of humanoid robot, in specific WALK-MAN teleoperation challenges, experiences in DARPA robotic challenge	Video
11.30 - 12.00	Yohei Kakiuchi Teleoperation of a humanoid robot with whole-body motion for object manipulation	Video
12.00 - 12.30	Joshua Mehling Challenges and approaches for teleoperation of humanoid robot for space exploration, experiences with Robonaut2
12.30 - 14.00	Lunch Vote this issue to join us for lunch
14.00 - 14.30	Neal Lii Toward multi-modal space teleoperation: A look back at METERON SUPVIS Justin and what lies ahead
14.30 - 15.00	Joao Ramos Dynamic synchronization of human operator and humanoid robot via bilateral feedback teleoperation	Presentation, Video
15.00 - 15.30	Jerry Pratt Humanoids avatar for co-exploration of hazardous environments, lessons learned from DARPA Robotics Challenge	Presentation, Video
15.30 - 16.00	Coffee break
16.00 - 16.30	Rafael Cisneros Limon Teleoperated manipulation and locomotion for humanoid robots in partially unknown real environments by using task sequences	Presentation, Video
16.30 - 17.00	Panel discussion	Video

Back to the outline

Invited speakers

Yuto Nakanishi

Talk Title. Teleoperation of humanoid robot, space teleoperation challenges and approaches in Gitai

ANA Avatar XPRIZE

Talk Title. The ANA Avatar XPRIZE: Inspiring creators, inventors and futurists to design and construct the future of robotic avatars

bio. & abstract.

The ANA Avatar XPRIZE (#AvatarXPRIZE) was launched in March 2018, and is sponsored by All Nippon Airways (ANA). It seeks to incentivize Teams to integrate a range of diverse, cutting-edge technologies to create a physical robotic Avatar System that will transport an operator’s senses, actions and presence to a remote location in real time.

This challenge will extend the capabilities of robots by integrating a broad range of technologies that can increase the application of robotics in the future. The winner of this XPRIZE will demonstrate a functional Avatar System, which consists of a human operator controlling a robotic Avatar at a real and/or Simulated Distance that allows the operator to interact with other humans or the environment, receiving sensory information from the robotic Avatar. The ultimate goal is for a person to feel as if they are truly where the Avatar is, experiencing a sense of Presence through the Avatar.

XPRIZE will present the goals of this competition and invite feedback and discussion from workshop attendees.

Presenters.

David Locke, Prize Director, ANA Avatar XPRIZE
Jacki Morie, Technical Advisor, ANA Avatar XPRIZE

Joshua Mehling

Talk Title. Challenges and approaches for teleoperation of humanoid robot for space exploration, experiences with Robonaut2

Enrico Mingo

bio. Enrico Mingo Hoffman received his Bachelor’s Degree in Electronics Engineering and Master’s Degree in Artificial Intelligence and Robotics (cum laude), both at University of Rome “La Sapienza”, in 2008 and 2011 respectively. In 2012 he was employed in PAL Robotics, Barcelona, on the REEM-H Project. working on algorithms for navigation, Simultaneous Localization and Mapping (SLAM) and localization. In 2016 he got a Ph.D. in Humanoid robotics at the Italian Institute of Technology (IIT). In 2015 he participated in the DARPA Robotics Challenge (DRC) as a member of the WALK-MAN Team. His main interests are Humanoid robots motion control, whole-body manipulation, impedance control, task space whole body control, compliant interaction between robot and environment and software architectures for Humanoid robots. He is Corresponding Guest Editor for the Special Issue on “Humanoid Robot Applications in Real World Scenarios” in the IEEE Robotics and Automation Magazine. At the moment he is a Senior Post-Doc Researcher in the Humanoids & Human Centered Mechatronics (HHCM) group, in IIT, mainly involved in developing motion control algorithms for multi-legged hybrid platforms.

Talk Title. Teleoperation of humanoid robot, in specific WALK-MAN teleoperation challenges, experiences in DARPA robotic challenge

Yohei Kakiuchi

bio. Yohei Kakiuchi is a research associate professor of The University of Tokyo from 2018. He started his academic career as an assistant professor of The University of Tokyo from 2012. He works on Software system(system architecture, control, perception, localization, etc.) for a humanoid robot, Tough humanoid robot working on a real environment.

Talk Title. Teleoperation of a humanoid robot with whole-body motion for object manipulation

abstract. We had the experience to teleoperate the humanoid robot at DRC (Darpa Robotics Challenge). Based on this experience, we develop some methods to teleoperate a humanoid robot with whole-body motion. For humanoid robot executing tasks with appropriate force while balancing, dynamic consistency of whole-body must be considered when performing tasks. I will introduce a method to balance teleoperated humanoid robot, and how to handle force during teleoperation for a humanoid robot.

Neal Lii

bio. Neal Y. Lii received the Bachelor of Science in Aerospace Engineering from Purdue University, USA in 1995, Master of Science in Mechanical Engineering from Stanford University, USA in 1999, and the Ph.D. degree from University of Cambridge, UK, in 2009. he served as the Principal Investigator of the ISS-to-Earth telerobotic experiments, METERON SUPVIS Justin until its conclusion in 2018. He is the founder and head of the Modular Dexterous robots (MODEX) Lab, as well as the domain head of Robotic Assistance, both at the Robotics and Mechatronics Center of the German Aerospace Center (DLR).

Talk Title. Toward multi-modal space teleoperation: A look back at METERON SUPVIS Justin and what lies ahead

abstract. METERON SUPVIS Justin was a space-to-ground telerobotic experiment led by the German Aerospace Center (DLR) together with the European Space Agency (ESA). In 2017-2018, three experiment sessions were conducted between the International Space Station (ISS) and Earth. Using a knowledge driven user interface, astronauts in orbit commanded DLR’s humanoid robot, Rollin’ Justin, to perform increasingly complex tasks in a simulated Martian surface habitat. The teleoperation was realized with the concept of supervised autonomy, which utilized the robot’s local intelligence to function as a coworker on the surface to carry out task-level commands. This approach gave the ISS crew the possibility to command complex tasks without expending high mental and physical workload, as the low-level action planning and execution are delegated to the robot. In total, five ISS crew members from NASA and ESA participated in SUPVIS Justin. Valuable insights were gained through experimental data gathered during the three ISS-to-ground sessions, as well as astronaut feedback in different forms including questionnaires, descriptive observations, and recommendations. In view of our findings from SUPVIS Justin, we aim to help further develop a holistic approach to space telerobotics, where supervised autonomy plays a vital role, together with other forms of teleoperation ranging from haptic feedback driven telepresence to (near) full autonomy. New on-going research and future missions with these various approaches to explore the technical challenges ahead in space exploration are presented. Furthermore, as these technologies can also be helpful on Earth in areas such as assisted living and operation in hazard environment, advances in these applications on Earth are also discussed.

Joao Ramos

bio. Joao Ramos currently works as Assistant Professor at the University of Illinois at Urbana-Champaign. He previously worked as a Postdoctoral Associate working at the Biomimetic Robotics Laboratory, at the Massachusetts Institute of Technology. He received a PhD from the Department of Mechanical Engineering at MIT in 2018. During his doctoral research, he developed teleoperation systems and strategies to dynamically control a humanoid robot utilizing human whole-body motion via bilateral feedback. His research focus on the design and control of robotic systems that experiences large forces and impacts, such as the MIT HERMES humanoid, a prototype platform for disaster response. Additionally, his research interests include human-machine interfaces, legged locomotion dynamics, and actuation systems.

Talk Title. Dynamic synchronization of human operator and humanoid robot via bilateral feedback teleoperation

abstract. Autonomous humanoid robots are still far from matching the sophistication and adaptability of human’s perception and motor control performance. To address this issue, I investigate the utilization of human whole-body motion to command a remote humanoid robot in real-time, while providing the operator with physical feedback from the robot’s actions. In this talk, I will present the challenges of virtually connecting the human operator with a remote machine in a way that allows the operator to utilize innate motor intelligence to control the robot’s interaction with the environment. I present pilot experiments in which an operator controls a humanoid robot to perform power manipulation tasks, such as swinging a firefighter axe to break a wall, and dynamic locomotion behaviors, such as walking and jumping.

Rafael Cisneros Limon

bio. Rafael Cisneros received the B.Eng. degree from the University of the Americas - Puebla (UDLA-P), Mexico in 2006, the M.Sc. degree from the Center of Research and Advanced Studies of the National Polytechnic Institute (CINVESTAV-IPN), Mexico in 2009, and the Ph.D. degree from the University of Tsukuba, Japan in 2015. Since then, he has been working at the Humanoid Research Group of the National Institute of Advanced Industrial Science and Technology (AIST), Japan, from the 2015 to 2018 as a post-doc and from 2018 until now as a tenure-track researcher. He participated as a member of the AIST-NEDO team in the DARPA Robotics Challenge and in the Japan Virtual Robotics Challenge. His research interests include whole-body multi-contact motion control of humanoid robots by using Quadratic Programming, multi-body collision dynamics for performing impulsive manipulation, 3D visualization, and dynamics simulation.

Talk Title. Teleoperated manipulation and locomotion for humanoid robots in partially unknown real environments by using task sequences

abstract. Humanoid robots are expected to be used in areas where other morphologies are not so practical. For example, disaster response, decommissioning or large-scale assembly and manufacturing. However, the corresponding task scenarios are still very complex and it is not yet possible for the robots to be completely autonomous. A human operator must be in the control loop by means of teleoperation, but this operator cannot manually control all the degrees-of-freedom of such complex robotic systems. Also, it is possible that the communication between operator and robot cannot be regarded as reliable, as it is the case of disaster-hit scenarios. This limits the amount of feedback available to the operator. The truly practical solution (currently) must rely in a semi-autonomous behavior. We have implemented this semi-autonomous behavior based on flexible higher-layer procedures that we call task sequences, which reduce the burden on the operator. Furthermore, we have developed a practical strategy to carry out each task by using fast semi-automatic identification aided by force information as well as task-based motion parameterization. In this talk I will introduce this framework and assess its practicality by showing the successful execution of tasks developed since our participation at the DARPA Robotics Challenge. I will also talk about some of our challenges.

Jerry Pratt

Talk Title. Humanoids avatar for co-exploration of hazardous environments, lessons learned from DARPA Robotics Challenge

Back to the outline

Organizers

Kourosh Darvish, Daniele Pucci, Serena Ivaldi, Eiichi Yoshida

Kourosh Darvish

bio. Kourosh Darvish received his B.Sc. and M.Sc. degrees in Aerospace Engineering from K.N. Toosi University of Technology and Sharif University of Technology (Tehran, Iran), in 2012 and 2014, respectively. He finished his PhD in Bioengineering & Robotics from University of Genoa, Italy in 2019. During his PhD, he has developed a hierarchical architecture for flexible Human-Robot Cooperation, in specific for factory and workshop environments. From November 2018, he is working as a post doc at Italian Institute of Technology (IIT) in Dynamic Interaction Control (DIC) Lab, and collaborate on H2020 European project AnDy. His research interests include Human-Robot Collaboration, Telexistence, and Manipulation.

Daniele Pucci

bio. Daniele received the bachelor and master degrees in Control Engineering with highest honors from “Sapienza”, University of Rome, in 2007 and 2009, respectively. In 2013 he received the PhD title in Information and Communication Technologies from University of Nice Sophia Antipolis, France, and in Control Engineering from “Sapienza” University of Rome, Italy. Since then, Daniele has been a post-doctoral fellow at the Italian Institute of Technology at the Dynamic Interaction Control laboratory. Within the CoDyCo project (FP7-ICT-2011.2.1, number: 600716), he has been the principal scientific contributor and he has developed innovative control techniques for whole-body motion control with tactile and force/torque sensing. Daniele’s research interests include control of nonlinear systems and its applications to aerial vehicles and robotics.

Serena Ivaldi

bio. Serena Ivaldi is a tenured research scientist at Inria, leading the humanoid and human-robot interaction activities of the Team Larsen in Inria Nancy, France. She earned her Ph.D. in Humanoid Technologies in 2011 at the Italian Institute of Technology. Prior to joining Inria, she was post-doctoral researcher in UPMC in Paris, France, then at the University of Darmstadt, Germany. She was PI of the EU projects CoDyCo (FP7); she is currently PI of the EU projects AnDy (H2020) and Heap (CHIST-ERA). She is also involved in the French ANR project Flying CoWorker. Her research is focused on humanoid robotics and human-robot collaboration, using machine learning to improve the control, prediction and interaction skills of robots. She strongly believes in user evaluation, i.e., making potential end-users evaluate the robotics technologies to improve usability, trust and acceptance.

Eiichi Yoshida

Back to the outline

Call for Contribution

We will welcome prospective participants of the workshop to submit extended abstracts (up to 4 pages) to be presented at the workshop followed by an interactive poster session. The manuscripts should use IEEE two-column conference format. The accepted abstracts will be posted on the workshop website and will not appear in the official IEEE proceedings. However, the selected contributions may be asked to contribute to a possible review paper addressing the current challenges and trends of the humanoid robot teleoperation. We encourage researchers as well as companies (both hardware and software companies) to contribute to the workshop. The reviewing is single blind, and will be carried out by the workshop organizers. The papers will be selected based on their originality, relevance to the workshop topics, contributions, technical clarity, and presentation. Accepted papers require that at least one of the authors register to the workshop.

In order to submit the manuscript please send an Email to “kourosh.darvish@iit.it” and “serena.ivaldi@inria.fr” with subject “Humanoids 2019 Teleoperation Workshop Contributed Paper”

Important dates: