Sam Bilbow, PhD

It has been thirty years since the original definition of augmented reality (AR) as a technology used to ‘augment the visual field of a user with information necessary in the performance of tasks’. In this first instance, it was developed with the purpose to ‘improve the efficiency and quality of human workers in their performance of manufacturing activities’ (Caudell and Mizell, 1992). Alongside subsequent decades of funding from the U.S. military-industrial complex (MIC), we have also seen the uptake and reappropriation of AR in creative fields, such as computational art, performance, design, and entertainment - these works often proposing do-it-yourself (DIY) and open-source approaches to their design. Despite these developments, AR within sound-driven forms of art have been relatively under-explored. If an AR system can be thought of as one that can combine real and virtual multisensory processes, is interactive in real-time, and is registered in three dimensions (Azuma, 1997); why do we, thirty years on, witness the paradigmatic form of AR still being heavily biased (Billinghurst et al., 2015) towards it being a method of visual information overlay? Standing in stark contrast to the currently unfolding and hyper-commercialised view of AR – as defined by the corporate ‘Metaverse’ – this thesis resituates AR as an artistic medium for the creation of interactive and expressive works by musicians and sound artists. It is guided primarily by the questions: ‘What are AR’s affordances as an artistic medium, what is the resultant experience for participants and audiences (or ‘immersants’) in these experiences, and what might a future of AR digital music instruments look, sound, and feel like?’ To address these questions, this practice-based research takes a DIY Approach to Sound ARt, arguing that, as an medium that combines real and virtual multisensory processes, it must explored with a sensory-process agnostic approach – that is, to approach AR as more than mere visual-information overlay, instead as ‘real-time computationally mediated perception’ (Chevalier and Kiefer, 2020). This has involved making and hacking technology as a necessary aesthetic and political stance against commercial AR technologies in their typical form. Three sound augmented reality art (ARt) experiences are outlined, and embody the majority of the practical contribution of this thesis: area , polaris , and polygons . In discussing the results of these three study chapters, theoretical propositions are made: ‘augmented materiality’, ‘augmented embodiment’, and ‘augmented space’, that have implications for the use of AR as a sonic medium. Moreover, out of the iterated design of the AR experiences, their study, evaluation, and discussion, three ‘design guidelines’ for those in the field interested in reproducing or devel- oping similar sound ARt have been developed: Designing for Rich AR Experience, Consideration of the AR Instrument, and Role of the Virtual in the AR Environment.

The increased use of mixed reality (MR) as a platform for NIME development has appeared out of different disciplines, including the arts, humanities, and computer science, but are potentially restrained by their own fields. With this workshop, we aim to facilitate and co-construct a narrative around the role of MR as NIMEs, and develop a community built on existing concepts of MR through panel talks, demos and theory-generating discussions.

Augmented reality (AR) is increasingly being envisaged as a process of perceptual mediation or modulation, not only as a system that combines aligned and interactive virtual objects with a real environment. Within artistic practice, this reconceptualisation has led to a medium that emphasises this multisensory integration of virtual processes, leading to expressive, narrative-driven, and thought-provoking AR experiences. This paper outlines the development and evaluation of the polaris~ experience. polaris~ is built using a set of open-source hardware and software components that can be used to create privacy-respecting and cost-effective audiovisual AR experiences. Its wearable component is comprised of the open-source Project North Star AR headset and a pair of bone conduction headphones, providing simultaneous real and virtual visual and auditory elements. These elements are spatially aligned using Unity and PureData to the real space that they appear in and can be gesturally interacted with in a way that fosters artistic and musical expression. In order to evaluate the polaris~, 10 participants were recruited, who spent approximately 30 minutes each in the AR scene and were interviewed about their experience. Using grounded theory, the author extracted coded remarks from the transcriptions of these studies, that were then sorted into the categories of Sentiment, Learning, Adoption, Expression, and Immersion. In evaluating polaris~ it was found that the experience engaged participants fruitfully, with many noting their ability to express themselves audiovisually in creative ways. The experience and the framework the author used to create it is available in a Github respository.