Voice-driven interaction in XR spaces (VOX Reality)

Summary

• VOXReality aims to facilitate the convergence of Natural Language Processing (NLP) and Computer Vision (CV) technologies in the Extended Reality (XR) field.

• The project aims to develop innovative models that combine language as a core interaction medium with visual understanding. This will result in next-generation applications that comprehensively understand users' goals, surrounding environment, and context.

• The resulting virtual assistants will be deployed in three use cases: a factory setting, a virtual conference (Immersive Tech Week), and a theatre play (Athens Epidaurus Festival).

Keywords

Extended Reality (XR), Natural Language Processing (NLP), Computer Vision (CV), Digital Agents, Virtual Conferencing, Theatre

Date

October 2022 - Ongoing

Budget

€ 4.78M (total)

Role

Design Researcher

---

Role

• In this project, I conducted co-design workshops with experts from various fields to explore innovative applications of XR technology.

  • In Greece, I collaborated with theater experts to envision the integration of AR for real-time translation and enhanced visual effects in live performances.

  • In Germany, I worked on virtual training scenarios, focusing on the development of virtual agents to enhance training environments.

  • In the Netherlands, I facilitated a workshop aimed at envisioning virtual agents and real-time translation for VR conferences.

• These sessions not only laid the core foundation for each project and established a clear development direction, but also deepened the participants' understanding of XR technology and its diverse potential applications.