Towards Incorporating 3D Space-Awareness into an Augmented Reality Sign Language Interpreter

Towards Incorporating 3D Space-Awareness into an Augmented Reality Sign Language Interpreter
Fabrizio Nunnari, Eleftherios Avramidis, Vemburaj Yadav, Alain Pagani, Yasser Hamidullah, Sepideh Mollanorozy, Cristina España-Bonet, Emil Woop, Patrick Gebhard
In: Proceedings of the Eighth International Workshop on Sign Language Translation and Avatar Technology. International Workshop on Sign Language Translation and Avatar Technology (SLTAT-2023), located at ICASSP 2023, June 10, Rhodes, Greece, IEEE, 6/2023.

Abstract:

This paper describes the concept and the software architecture of a fully integrated system supporting a dialog between a deaf person and a hearing person through a virtual sign language interpreter (aka avatar) projected in the real space by an Augmented Reality device. In addition, a Visual Simultaneous Localization and Mapping system provides information about the 3D location of the objects recognized in the surrounding environment, allowing the avatar to orient, look and point towards the real location of discourse entities during the translation. The goal being to provide a modular architecture to test single software components in a fully integrated framework and move virtual sign language interpreters beyond the standard ``front-facing'' interaction paradigm.