Enabling Audiovisual User Interfaces for Multisensorial Interaction

Enabling Audiovisual User Interfaces is a research project that investigated how human-computer interactions can be audiovisualized – that is to say, both sonified and visualized – in order to improve user experience and usability. To address this issue, a new UI (User Interface) paradigm was conceived – AVUI (AudioVisual User Interface). AVUI links interaction, sound and image, building upon the concept of Graphical User Interface (GUI) by adding interconnected sound and image. The research hypothesis is: the introduction of AVUI, integrating interrelated sonic and visual feedback, reacting to user interactions, will lead to more usable, accessible, playful and engaging UIs, as compared to a traditional GUI – particularly in use cases where accessibility and/or engagement are determinant. AVUIs were applied to prototypes and case studies, which were developed together with users, following a User-Centered Design approach (UCD). After reaching conclusions from these, an AVUI framework was developed. The 2-year Marie Curie fellowship took place at Goldsmiths, University of London – EAVI (Embodied AudioVisual Interaction) research group, under the supervision of Prof. Atau Tanaka. The project website is: avuis.goldsmithsdigital.com


  • EU Marie Curie fellowship FP7 REA grant 627922


References to the Research

  • Correia, N. N., & Tanaka, A., 2017. AVUI: Designing a Toolkit for Audiovisual Interfaces. In: Proc. CHI ’17. Denver, CO. http://dl.acm.org/citation.cfm?id=3026042
  • Metatla, O., Correia, N.N., Martin, F., Bryan-Kinns, N., Stockman, T., 2016. Tap the ShapeTones: Exploring the Effects of Crossmodal Congruence in an Audio-Visual Interface. In: Proc. CHI ’16. San Jose, CA. http://dl.acm.org/citation.cfm?id=2858456


Sources to Corroborate the Impact

  • Dr. Atau Tanaka, Professor of Media Computing - Goldsmiths, University of London (Supervisor of the research, CHI 2017 paper co-author)
  • Dr. Nick Bryan-Kinns, Reader in Interaction Design - Queen Mary University of London (Partner of the research, CHI 2016 paper co-author)
  • Dr. Oussama Metatla, Research Fellow - University of Bristol (Partner of the research, CHI 2016 paper co-author)