Material, Embodied, and Spatial Relations in Augmented Reality: An Exploration of AR as a Medium for Musical Composition and Performance
It has been thirty years since the original definition of augmented reality (AR) as a technology used to ‘augment the visual field of a user with information necessary in the performance of tasks’. In this first instance, it was developed with the purpose to ‘improve the efficiency and quality of human workers in their performance of manufacturing activities’ (Caudell and Mizell, 1992). Alongside subsequent decades of funding from the U.S. military-industrial complex (MIC), we have also seen the uptake and reappropriation of AR in creative fields, such as computational art, performance, design, and entertainment - these works often proposing do-it-yourself (DIY) and open-source approaches to their design.
Despite these developments, AR within sound-driven forms of art have been relatively under-explored. If an AR system can be thought of as one that can combine real and virtual multisensory processes, is interactive in real-time, and is registered in three dimensions (Azuma, 1997); why do we, thirty years on, witness the paradigmatic form of AR still being heavily biased (Billinghurst et al., 2015) towards it being a method of visual information overlay?
Standing in stark contrast to the currently unfolding and hyper-commercialised view of AR – as defined by the corporate ‘Metaverse’ – this thesis resituates AR as an artistic medium for the creation of interactive and expressive works by musicians and sound artists. It is guided primarily by the questions: ‘What are AR’s affordances as an artistic medium, what is the resultant experience for participants and audiences (or ‘immersants’) in these experiences, and what might a future of AR digital music instruments look, sound, and feel like?’
To address these questions, this practice-based research takes a DIY Approach to Sound ARt, arguing that, as an medium that combines real and virtual multisensory processes, it must explored with a sensory-process agnostic approach – that is, to approach AR as more than mere visual-information overlay, instead as ‘real-time computationally mediated perception’ (Chevalier and Kiefer, 2020). This has involved making and hacking technology as a necessary aesthetic and political stance against commercial AR technologies in their typical form.
Three sound augmented reality art (ARt) experiences are outlined, and embody the majority of the practical contribution of this thesis: area , polaris , and polygons . In discussing the results of these three study chapters, theoretical propositions are made: ‘augmented materiality’, ‘augmented embodiment’, and ‘augmented space’, that have implications for the use of AR as a sonic medium. Moreover, out of the iterated design of the AR experiences, their study, evaluation, and discussion, three ‘design guidelines’ for those in the field interested in reproducing or devel- oping similar sound ARt have been developed: Designing for Rich AR Experience, Consideration of the AR Instrument, and Role of the Virtual in the AR Environment.
