PerformaVis: Real-Time Affective Music Visualization Driven by Pianist's Bodily Expressions

Multimodality is integrated into the majority of our multimedia experiences, most commonly with intent. However, some experience an unintentional, and subjective perceptual response to a stimulus in another sensory domain due to the neurological phenomena synaesthesia. This paper discusses the application of music-colour synaesthetic mappings in live ensemble performance. A succinct discourse on literature related to synaesthesia scientifically and creatively is presented, followed by an overview of the design and development of the system. Exemplification of the mapping strategies is offered through an illustration of one potential output. Finally, the paper concludes with a brief summary alongside ongoing developments.