AI See, You See: Human-AI Musical Collaboration in Augmented Reality

Abstract

The integration of artificial intelligence (AI) into creative practices has opened new possibilities for dynamic human-AI collaboration. While prior research has explored structured, process-based co-creative tasks, real-time, open-ended creative contexts such as musical performance remain underexplored. These environments demand tools that facilitate dynamic interaction, mutual understanding, and the explainability of AI contributions. This work introduces a novel augmented reality (AR) co-creative musical system that visualises AI’s musical actions, such as MIDI outputs and control parameters, through real-time AR visualisations. By embedding these visualisations within the performance space, the system facilitates shared creative processes and elevates AI’s presence. Using an auto-ethnographic approach, we evaluated the system in three musical performances across varied configurations, revealing insights into the role of AR in shaping human-AI collaboration and co-creative dynamics. This work offers design and evaluation insights for AR-enabled co-creative systems.