Audio/visual concepts for human/robot communication in immersive virtual environments

Communication between robots and humans is a challenge in complex built environments. In this project, we are exploring how previous achievements in this area apply to human/robot communication within immersive virtual displays. In our concrete example, we host human-scale robots that interact with humans in our Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab). The CRAIVE-Lab provides a physical/digital environment for collaborative tasks using seamless video from multiple projectors and a 128-channel wave-field system. The system is designed to monitor the whole floor-space area (10 m × 12 m) with a camera network mounted to the ceiling grid and a microphone array combining shotgun, spherical and possibly people-worn microphones. Based on the sensor data, a visual analysis and an auditory scene analysis are performed. The latter includes sound localization, speech and musical-feature recognition. The robots, a Rethink Baxter robot mounted on an electric wheelchair and a Robokind Zeno, are used to perform assistive technology and social communication tasks. Both robots have direct access to these data using a digital feed over a wireless network to augment their own sensor systems of built-in cameras and microphones.