Sensing Stories

07.11.2023

The cinema of the future... On the possibility of multi-sensory storytelling in immersive media

In this talk it will be explored, especially with a view to narratological, aesthetic and embodiment-theoretical considerations, if and how a multi-sensory narration can be possible in the cinematographic context, but also within Virtual Reality.

In order to be able to fathom and outline narrative in a future immersive medium, a look into the history of multi-sensory media imagination and intervention will be used to identify and examine lines of development as well as technologies of the myth of total cinema or total immersion - from this, initial ideas for a narrative that goes beyond the sensory modalities of the visual and auditory can already be derived.

Patrick Rupert-Kruse is professor for media theory and immersion studies at the media department of the University of Applied Sciences Kiel; he is also the head of the Institute of Immersive Media and chairman of the Society of Immersive Media Schleswig-Holstein. He is the co-founder of the research group image studies and co-editor of the book series “Bewegtbilder” and the “Yearbook of Moving Image Studies”.
His research areas are theory and conception of immersive media, immersive, multisensory, and transmedia storytelling.

14.11.2023

The Dawn of the Feely Effect – how ancient narrative practices can inform the storytelling of the future

“Take hold of these knobs on the arms of your chair," whispered Lenina,"otherwise, you won't get any of the feely effects." ( Huxley,1932)
Watching is no longer enough. As the crowds who flocked to ‘The Feelies’ in Aldous Huxley’s Brave New World knew, the ultimate narrative experience involves all the senses and the feeling of living viscerally within a story, rather than just experiencing it vicariously. Today immersive performance has become an umbrella concept covering a range of cultural happenings that lean heavily on the promise of a multisensory narrative experience. Spearheading the immersive revolution is new VR technology that offers agency and sensation in fake ‘new worlds’ accessed through digital devices.
This lecture explores how the roots of multisensory storytelling are to be found, however,  not in technology, but in ancient narrative traditions. Much of the evidence of ‘pre-historic’ story-sharing customs has been misrepresented through colonial attempts at appropriation. Consequently, its relevance to emerging artistic practices has been missed. I will explore how contemporary archaeology and anthropology reveals resonances between ancient narrative practices and new immersive arts, which point to new ways of understanding the future of storytelling.

Elizabeth Swift is a writer and director of intermedial performance. She lectures in Drama at the University of Gloucestershire, UK, and  has an MA in Contemporary Performance Practice  from Lancaster University and a Phd in Drama  from University of Exeter. Publications include articles in Critical Stages, International Journal of Performance Arts and Digital Media, Body Space and Technology andin Palgrave’s Framing Immersive Theatre and Performance. Liz represented the UK at the 2019 British Academy International forum on the Future of Storytelling in Australia. She is director of Void Projects performance  company that  has  toured internationally with intermedial  performance work.

21.11.2023

Les cadavres amoureux: Sex and Death in Multisensory (Utopian) Virtual Systems

Starting from a mediarcheological analysis of Guillaume Apollinaire's 1916 short-story Le Roi-Lune (The Moon King), this joint presentation aims to explore the main aesthetic, political and social issues raised by the dream of Virtual Reality (VR) as a machine capable of drastically reconfiguring the ambiguous relationship between presence and absence that is the hallmark of all images. Is it possible to imagine a technology that allows a multisensory experience that is completely immediate but reproducible? Can a man-made device fulfil the obscene fantasy par excellence: to erase the distance between sexually active bodies, between the living and the dead, to momentarily resurrect identities and manipulate them erotically?

By analysing, on the one hand, how sci-fi cinema has imagined this intersection of VR, sexuality and death and, on the other hand, the erotic devices that are currently on the market, this paper will also address the capacity of erotic-technological assemblages to produce forms of narration that are not linked to a classical diegetic development, but rather based on rhythm and resonance between images, bodies, and material technologies.

Pietro Conte is an Associate Professor of Aesthetics at the University of Milan and a team member of the ERC project “An-Icon” (https://an-icon.unimi.it/). After receiving his PhD from the University of Siena in 2007, he spent two years on a postdoc scholarship at the University of Basel (2007-2009). From 2011 to 2015, he worked as a Research Fellow at the University of Milan. He was then awarded a Starting Grant by the Portuguese Foundation for Science and Technology through the scientific programme “Investigador FCT”, thanks to which he was appointed Junior Assistant Professor at the University of Lisbon (2015-2018). In 2018, he got a tenure-track position at Ca’ Foscari University of Venice, where he became an Associate Professor in 2021, before joining the University of Milan in 2022. His scholarly work revolves around illusion, hyperrealism, immersion, and the multifarious practices of un-framing, a thematic cluster that he has addressed in the monographs Unframing Aesthetics and In carne e cera. Estetica e fenomenologia dell’iperrealismo (Flesh and Wax: Aesthetics and Phenomenology of Hyperrealism). Within the “An-Icon” project, Pietro is currently paying special attention to digital immortality and to the media-archaeological reconstruction of the pseudo-scientific attempts to communicate with dead people.

Roberto Paolo Malaspina graduated (BA) in 2016 at the University of Florence in History and Conservation of Cultural Heritages with a thesis on the relationship between museology and New Media. In 2019 he concluded his MA degree in Visual Arts at the University of Bologna, with a dissertation on the theoretical identity of postmodern architecture. During his studies, he worked as curator for Narkissos, an independent gallery based in Bologna, where he further researched on the interrelation between body, technology and feminisms. Between 2019 and 2020 he attended CAMPO, course for curators of the Fondazione Sandretto Re Rebaudengo. He is currently a PhD student at the University of Milan – Department of Philosophy "Piero Martinetti"– and member of the ERC project “An-Icon” where he conducts a research on the relationship between embodied perception and New Media in the fruition of pornographic moving images.

28.11.2023

Other-worldly performances. A virtual reality narrative construction of ‘the other’

Presenting the virtual reality (VR) proto-type Demonised: Bewitched and Possessed, Dr Helen Jackson discusses the promise of an embodied performance of narration, and its ability to leverage within the spatial imagination, a sense of human existence beyond that of what is seen.

Demonised assembles a performance of the trial of the Islandmagee Witches in Co. Antrim in 1711, a young women’s claim that she was bewitched and the ‘witches’ identified as causing her symptoms of demonic possession.

The lecture explains how the Demonised VR experience seeks to intervene in accepted narrative tropes associated with witches and witchcraft by juxtaposing embodied performances of oppression, marginalisation, isolation, and wider concepts of being ‘the other’, with experiences of supernatural power and magical, other-worldly sensations. Exploring these sensorial tensions, where VR is positioned as a means to develop both experiential and informative narrative, the lecture examines the possibilities and the impact of a multisensory narrative operating in the virtual reality space.

Helen Jackson is a Senior Lecturer in Interactive Media at the School of Communication and Media, and researcher at the Centre for Media Research, at Ulster University, Northern Ireland. Her research, for which she has received international awards, involves creative practice-led enquiry into the social and cultural impact of emerging media technologies in contemporary visual culture. This research has led to the successful development and deployment of a range of screen-based digital experiences including interpretive experiences (apps) for smartphones; augmented reality marketing solutions; websites to generate and network communities; digital toolkits for business; and virtual reality interpretive experiences for the museum sector. Recent Publications:Crooke, E., Farrell-Banks, D., Friel, B., Hook, A., Jackson, H., Maguire, T. & McDermott, P. (2022) Museums and Digital Media. A Report of the Museums, Crisis and Covid-19 Project. Ulster University; Jackson, H., 2020. Echoes of the Causeway. [Digital Artefact] Jackson, H., 2021; Over the wall: a drone view of the ‘other’ side. In: The International Journal of Creative Media Research. p. 1-20 20 p.; Jackson, H. Self[ie] reflective practice: revealing student engagement through the photographic performance of the self, Learning, Media and Technology, 2019; Hickey, A. Hook, A. Jackson, H.‘Where is my Googleplex?’ Rethinking vocational learning and teaching spaces for Digital Media curriculums, Journal of Media Practice, and Education, 2018; Jackson, H., 2017. Seeing and knowing Titanic Belfast using augmented reality: an auto-ethnographic view. In: Journal of Media Practice. 18, 2-3, p. 154-170 17 p.

05.12.2023

Zork in the Dark: Worldbuilding and Storytelling Beyond Vision

This talk will outline the scope, methodology and lessons learned from developing The Third Pill, a virtual reality game where half of the action takes place in total darkness. The player completes quests by navigating through hallways, escaping cages, opening doors, and breaking through walls using kinesthetic interaction and responsive soundscapes. There is no visual input of any kind. Our goal was to create an experience totally new to sighted people: the experience of piecing together a spatial reality through embodied interaction and audio input alone. During implementation, we used strategies meant to help the players’ minds develop stable integrations of perceptual information in the absence of visual confirmation.

Andreea Ion CojoCaru is a licensed architect and a software developer. She is the co-founder and CEO of NUMENA, an award-winning German company. Andreea works at the intersection of traditional architecture and immersive technologies to develop projects that require a new approach to cognitive and spatial challenges. She is the recipient of numerous fellowships and is a frequent guest speaker at international events in the creative technologies space, including SXSW, the Architectural Association in London, the UW Reality Lab, Cornell Tech, and several episodes of the Voices of VR, the leading virtual reality podcast.
Formally trained as an architect at MIT and Yale University, she was awarded the AIA Gold Medal for best graduating master student by the American Institute of Architects. Prior to NUMENA, she gained design and project management experience in architecture practices such as Kohn Pedersen Fox and Robert A.M. Stern in New York.
Andreea believes that by expanding the possibilities and affordances of spatial experience, we are expanding and redefining identity, subjectivity and modes of collective being. Currently, she uses virtual and augmented reality to explore new social and economic models for collective agency, and multisensorial approaches to storytelling. You can find a selection of her talk at AndreeaIonCojoCaru.com.

09.01.2024

Speculations on Digital touch futures

Touch matters. It’s fundamental to how we experience and know ourselves, others and the world, and to how we communicate. Emergent digital touch technologies stretch the possibilities of how we feel the world around us and how, what, whom and when we touch.

This lecture speculates on the multi-sensory futures heralded of digital touch through a series of speculative socio-technical imaginaries in the form of research-based fictional multi-modal fragments – short stories, visual adverts, diary entries, and dialogues. These draw on research from the interdisciplinary InTouch project on the social implications of digital touch. These speculations centre on the use of a range of ‘real’ (existing or in development), fictional or imagined digital touch-interfaces or devices.  This imagined landscape offers a space of social and cultural reflection on the contemporary and nascent configuration of digital touch and its social touch values and conventions.  We excavate this landscape to present a digital touch utopian – dystopian continuum, and to highlight key weak signals (themes and directions) for the futures of digital touch.  

Carey Jewitt is a Professor of Technology Interaction, in UCL Knowledge Lab, Culture, Communication and Media department at University College London (UCL). She is Chair of the UCL Collaborative Social Science Domain. Her research centres on digital interaction, multimodal communication, touch, methodological innovation, and interdisciplinary research. Carey is a founding co-editor of two Sage journals Multimodality & Society (2021) and Visual Communication (2000), and author of over 150 publications, including nine books. She has led numerous large research projects (funded by the Economic and Social Research Council, ESRC) and is Director of InTouch, a prestigious European Research Council Consolidator Award (2016-2022).

Sara Price is Professor of Digital Learning in UCL Knowledge Lab, Culture, Communication and Media department, UCL. Her research draws on theories of embodied cognition and interaction, exploring how mobile, tangible, sensor, VR technologies mediate new forms of interaction, cognition, and communication. Her work informs theories of embodied learning, technology design and development, and methodological innovation. She is Associate Editor for Transactions on Computer Human Interaction (ToCHI), author of over 150 publications, and editor of three books. She has led several interdisciplinary research projects (funded by EPSRC, JISC), and most recently Co-PI on Move2Learn (NSF/Wellcome/ESRC), WeDRAW (EU Horizon 2020) and Co-I on InTouch. 

16.01.2024

Touching narratives: Comic and film as interactive media

The talk looks at the underlining sensual trajectories in narrative and interactive (audio-)visual media, employing the logics of the haptic sense in order to analyze aesthetic strategies. It departs from the haptic/tactile opposition used in the physiology of the sense, where it is used to describe different dispositions of the perceiving subject towards stimuli: While 'haptic' refers to an active, investigative attitude towards the environment and requires active movements, 'tactile' stands for a comparatively passive attitude where the perceiving subject does not move in direction of the stimulus. This opposition can be aligned with different attitudes of media spectators/users who have historically often been described as either passive or active. Drawing on comic and film as interactive media, the talk will challenge the active/passive division by exploring the (inter-)active potential of both the haptic and tactile attitude and the affective engagement they require.

Dr. Nina Eckhoff-Heindl studied art history and philosophy at the Ruhr-Universität Bochum. She completed her doctorate in 2021 at the University of Cologne and the University of Zurich (cotutelle de these). Her dissertation on aesthetic experience and visual-haptic narration was awarded the Roland Faelske Prize for the best dissertation in comics and animation studies. She currently holds a position as research coordinator and research assistant of the OER project “Comicforschung.nrw” at the Institute of Media Culture and Theatre at the University of Cologne. Selected publications: Comics begreifen. Ästhetische Erfahrung durch visuell-taktiles Erzählen in Chris Wares Building Stories. Berlin: Reimer 2023; "Aesthetics of Reception. Uncovering the Modes of Interaction in Comics." In Seeing Comics through Art History: Alternative Approaches to the Form, ed. by Maggie Gray and Ian Horton, London: Palgrave Macmillan 2022, 97-119.

Dr. Jana Herwig teaches media studies at the Dept. for Theatre, Film & Media Studies and produces online educational resources (OER) for the Center for Teaching and Learning at the University of Vienna, Austria. Her doctorate explored the history and aesthetics of haptic media; in collaboration with iMooX.at, she coordinates the production of Massive Open Online course, covering societal issues such as Digital Life, Diversity in Higher Education and Natural Disasters. Selected publications: “Dis-/Ability an Hochschulen: Ein Gespräch über Zugänglichkeiten, In-/Kompetenzen und diversitätssensible Lehre” (Springer Nature, forthcoming), (Ed.) Self-Service. Selbstbedienung zwischen Automatisierung und algorithmischen Praktiken, Wien: edition mono / monochrom 2022, with Sonja Buchberger, “Haut, Hand, Schnitt. Zu den affektiven Gefügen des Tastsinns und der Medien”, in: Maske und Kothurn 2019 (65), 1-2, S. 197-212.

23.01.2024

Children’s digital reading enhanced with smell: insights from an empirical study in Norway

In my presentation I will share results from our project Sensory books, funded by the Norwegian Research Council, which is the first project to examine the potential of smell for children’s fictional stories. I will outline a framework for designing and empirically studying children’s engagement with olfactory books, i.e. books enhanced with scent and smell cues. I will set out the connections between key theoretical frameworks that motivated my focus on olfactory storybooks for children, notably Rosenblatt’s transactional theory of reading (Rosenblatt, 1988) and multisensory design (Spence & Gallace, 2011). I argue that a digital olfactory book that purposefully stimulates children’s non-verbal, non-linguistic learning with smell-enhanced text-picture interactions, exemplifies the core ideas of embodied sensory learning with texts. 

Natalia Kucirkova is Professor of Early Childhood Education and Development at the University of Stavanger, Norway and Professor of Reading and Children’s Development at The Open University, UK. Natalia’s work is concerned with social justice in children’s literacy and use of technologies. She is the founder of the International Collective of Children’s Digital Books that connects research and design in children’s e-books and literacy apps and Chair of the Children’s Digital Book Award that is the first award judged entirely by teachers. Her research takes place collaboratively across academia, commercial and third sectors.

30.01.2024

Performative Sensory Environments Unravelled: Listen, Eat, Repeat

My presentation addresses a vision of sensory intertwinement that weaves together the technical and the non-technical through my media art explorations framed as sensory performative environments that encompass sensors, sound, and gustatory infusions. To this end, I will underline the capacity of bio-sensing to measure psychophysiological effects of sensory stimulation as neural responses to stimuli, before drawing on dynamics between bio-sensing and sound as a feedback loop built to set up and measure corporeal sensory responses. Sequentially, I will introduce the interdisciplinary field of gastrophysics with a deeply researched strand being ‘sonic seasoning’ that demonstrates how sound can affect our culinary experience. Building on this concept I will speculate a novel compound that I call ‘culinary sounding’ or having the affectivity of food resonate into sound through the sensor-to-sound coupling. Within this dynamic, while the sound affects the culinary experience, the sensor data taken from the body affected by gustation feeds into sound, modulating its properties. I will conclude the presentation by a lecture-performance intervention that offers a narrative to situate my techno-material sensory experiments.

Mona Hedayati is an artist and a joint PhD researcher at Concordia University, Canada and Antwerp Research Institute for the Arts, Belgium. Her interdisciplinary research-creation draws on sound design, computation arts, and sensory studies.

Select Performances

• Ars Electronica, Linz, Austria.
• Les traversée du Marais Festival, Paris, France.
• Hessian Center for Artificial Intelligence, Darmstadt, Germany.
• Neue Künstlerkolonie Rosenhöhe, Darmstadt, Germany.
• Kunsthal Extra City, Antwerp, Belgium.
• Body Electric Retrospective 2014-20, Toronto, Canada.
• Dyscorpia 2.1: Future Intersections of Body and Technology, Edmonton, Canada.
• Agnes Etherington Art Center, Queen’s University, Kingston, Canada.
• AHVA Gallery, UBC, Vancouver, Canada.

Select Residencies

• Artist-in-science residency, Kultur einer Digitalstadt, Darmstadt, Germany.
• Q-O2 Experimental Sound Residency, Brussels, Belgium.
• Cité International des Arts, Paris, France.

Select Awards

• Social Science & Humanities Research Council of Canada Scholarship (MA & PhD).
• Fonds de recherche du Québec – Société et Culture.
• Excellency Grant, Hexagram réseau de recherche-création en arts, cultures et -technologies.

Select Talks

• 10th Conference on Histories of Art, Science & Technology, Venice, Italy.
• Non-Machines, Bauhaus University Weimar, Germany.
• 11th Conference on Computation, Communication, Aesthetics & X, Weimer, Germany.
• 28th International Symposium on Electronic Arts, Paris, France.
• Exploring Design Beyond the Human, Swiss Design Network, Zurich, Switzerland.