retrospective / prospective logic – creation of meaning – anumana

The following practical account aims to critically examine the significance and impact of various technological ‘interfaces’ — and the data they generate — as instruments for producing knowledge. To develop the concept of ‘interface’, I will expose an argument bound to the biographical method.

I share the view that biographical texts “contribute to a broader cultural production of meaning by presenting exemplary lives and framing individual experiences as miniature models of larger narratives. Biographies create ‘prototypes’ of social experience and historical understanding” (1). The sociologist Pierre Bourdieu describes the logic:

“It is reasonable to assume that behind every autobiographical narrative lies, at least in part, an intention to create meaning—to explain, to impose a logic that is both retrospective and prospective. This involves seeking consistency and continuity, often by establishing intelligible relationships such as cause and effect between successive states, which are thereby elevated to stages in a perceived necessary development.”(2)

However, the title of my text refers to ‘an interface,’ not to the author or her name. In fact, the subject here is not intended as an autobiography in the traditional sense. Rather, I apply the autobiographical method to a specific area of my life practice — namely, the development of an interface, along with its underlying theoretical and practical justification. Ultimately, what unfolds is a description of this interface and its evolution, which might be aligned with one of the types of biographical construction identified by Mikhail Bakhtin: the energetic type. “It is based on the Aristotelian concept of energy. It is not the state, but the action, the active force (“energy”) that constitutes the fullness of being and the essence of man”(3).

The specific triggers, motivations, and fields of energy from my life that shaped my perspective on technological interfaces in relation to knowledge generation – leading to the development of an alternative ‘interface’ – will be presented and discussed in their full scope. In most cases, the accompanying photographs are not intended as mere illustrations, but as documentary evidence that supports the corresponding accounts.

For purely practical reasons, however, I will assign a proper name to the interface discussed here, so that I may refer to it simply as anumana in what follows, rather than repeatedly using the phrase ‘the interface to be described here’.

Anumana is named after N.C. Panda, who, in his chapter “Yoga Psychology”, describes it as a form of knowledge acquisition that arises through inference: “We also obtain knowledge through inference (anumana). Every incidence of fire is accompanied with smoke. Even if we do not see the fire, we infer it by observing the smoke.”(4)

I will begin with a chronological account of the movements that have shaped my personal ‘path of a seeker’ thus far. The term ‘path of a seeker’ refers to an early Platonic model, which Mikhail Bakhtin describes as follows: “The seeker passes through several philosophical schools and subjects them to an examination; the temporal subdivision of the path depends on the seeker’s works“. (5)

EARLY YEARS: EXPERIENCING DIFFERENCE AND GAPS

interface – difference – interdisciplinary – transcultural philosophy

To begin with, it is worth noting that we, as individuals, turn to interfaces because of a perceived or actual difference — whether between systems, organisms, or tools. Because of a gap, a sense of separation that prompts the need to establish connection or reconnection. Without elaborating in detail, I leave it to the reader to consider the relevance of three formative experiences of difference in my early years, and how they may have shaped the forces driving my research. At the very least, these experiences surface as material throughout much of my work. First, I grew up in Germany as the daughter of a German mother and an Indian father. Second, following a family move and change of school, I was one of only two girls in an otherwise all-boys grammar school, until it opened to girls in the sixth form. And third, I chose to study both philosophy and biology at a time when interdisciplinarity was still largely unwelcome within academic departments.

Nancy K. Miller, in Gender and the Subjects of Autobiography, argues that a defining feature of women’s autobiographies is the development of identity ‘through otherness’ (6). She cites Susan Stanford Friedman, who observes that “female writers tend not only to pin the self of their autobiographical project to a single, selected Other, but also – and simultaneously – to the collective experience of women as gendered subjects in a variety of social contexts” (7). This is also linked to Stephen Butterfield’s description of autobiographies from people of color, namely: “The self is conceived as a member of an oppressed social group, with ties and responsibilities to other members”(8).

My need for an interface may well stem from such two “gaps” (related to gender and colour), but I want to affirm, as truthfully as I can, that I have never experienced these constellations as traumatic. Rather, I have come to regard them as sources of incentive and driving energy. In encountering “foreign culture” I felt compelled — as I will show — to find ways of sharing it with those around me. Regarding the “gender gap”, which was clearly evident in German universities in the 1980s, I consistently focused on “taking care of the matter” and simply disregarded gender-based hierarchies, which I considered meaningless (9). Thus, working and living as a freelance artist and private scholar became, in many ways, the natural course for me.

The developmental history of anumana unfolds through various mind-openers, in Filliou’s sense. “Mind-opening goes on every day of your life”, writes the artist Filliou (10), referring to surprising experiences and observations that shift our perceptions in unexpected ways. This element of unpredictability, which calls for unplanned developmental steps, forms a significant part of the evolution of anumana. This way, I describe it in the sense of what Bourdieu calls retrospective logic: only in hindsight do these experiences reveal themselves as meaningful development steps. It is for this reason that I adopt the term mind-opener here.

A first mind-opener on the path toward anumana was my preparation for the final thesis of my first degree, titled Die Wissenschaftstheoretische Begründung der Biologie bei Jakob Friedrich Fries (The Scientific Theoretical Justification of Biology as Proposed by Jakob Friedrich Fries), under the supervision of Prof. Lutz Geldsetzer at the University of Düsseldorf. This was my very first encounter with a fundamental, well-argued critique of the axioms of the natural sciences and with a deeper clarification of the concept of “knowledge”. My specialisation in physiology and genetics during my biology studies stemmed from a desire to understand the world, the universe and human life. The same impulse guided my choice to pursue epistemology and ontology as part of my philosophy degree — and even earlier, my decision to study both. The relevance of my work on Fries, both for the topic at hand and for my life more broadly, has not been exhausted even forty years later. Prof. Geldsetzer captured his position impressively:

“Fries (…) founded the realist ontology of modern natural science and clarified the principles on which it actually rests. And if one looks with an unbiased eye at today’s natural-scientific discourse, one might well admit that the intensity of belief in the reality of the external world among established natural scientists far exceeds earlier beliefs in the reality of ghosts and spirits.”(11)

Interestingly, my connection with Prof. Geldsetzer led to a second mind-opener after I completed my first degree. As a graduation gift, my parents made it possible for me to spend a year studying in Madras, India — something I had wished for deeply. My plan was to explore Indian culture and art while also beginning research for a doctoral thesis. Prof. Geldsetzer accepted me as a doctoral student and tasked me with researching “the historiography of philosophy in India”.

After considerable effort to secure the expert opinions and certified explanatory translations of my academic certificates required by the University of Madras, I was finally admitted as a student and able to present my research topic to a professor of philosophy, hoping to gain his support in sourcing materials (Fig. 1). To my dismay, I discovered that while I had been awarded a university degree in philosophy, it carried no mention of “Western philosophy”. The professor, therefore, not only criticised my lack of familiarity with the subjects taught in his faculty but also pointed out that “historiography of philosophy” was not recognised as a discipline within Indian philosophy, and beyond that, he questioned the value such research could have at all.

Figure 1: The author – Rajele Jain – at Madras University in 1984.

I remember writing a rather vehement letter to Prof. Geldsetzer, accusing him of having told us only ‘half the truth’ when he delivered his lectures on philosophy in Düsseldorf (12). Today, of course, I recognise that this situation could have been turned into an interesting research topic — exploring the meaning of the historiography of philosophy within a tradition of thought that conceives of eternity and infinite cycles. But forty years ago, my respect for academic authority was too great for me to consider developing a topic from such a position of negation.

Thus, my two previous foundations for generating knowledge — natural science and philosophy, that is, academic work — had been shaken, creating a rift through which new ideas and possibilities could flow.

STUDY YEARS: TRYING TO SEE THE OTHER SIDE

resonance – immateriality – performance – emotions – breathing – meditation – masters

In India, I found myself able to question almost all my previous certainties. I say “able to”, rather than “forced to”, because nothing I experienced felt truly foreign. Behind every sensory experience lay a theory unknown to me — the meaning of sound and music; of taste, both in food and in colours and emotions; of smell and breathing; of movement, performance, and communication; of bodily hygiene and rituals; and even of humour, emotions, Vedic logic, knowledge, consciousness, and higher states of awareness. Above all, there was the capacity to think and to coexist with impossibilities.

Today, I can say that I resonated deeply with Indian culture and, in that context, discovered for myself the meaning of immateriality, avatars, mantras, embodiment, frequencies, and oscillations — at a time when the first signs of electronic networks and digital syntheses were beginning to emerge in the West and to extend beyond military research. It was 1984.

I spent a year:

• Immersing myself in the basics of Sanskrit, the primordial language (13);
• Studying South Indian classical dance, Bharatanatyam, with Shyamala, a direct successor of the renowned performer Balasaraswati (1918–1984);
• Learning yoga at the institute of Krishnamacharya (Tirumala Krishnamacharya, 1888–1989);
• Undertaking Vipassana meditation in Igatpuri with S. N. Goenka (Satya Narayana Goenka, 1924–2013).
• I also spent three months on the legendary grounds of the Theosophical Society in Adyar (then Madras, now Chennai);
• And witnessed the premiere of Angika, a dance choreography by Chandralekha Prabhudas Patel (1928–2006), in Bombay.

These experiences helped shape the formation of anumana as explained ahead.

The encounter with Sanskrit and the theory of mantras gave anumana its purpose and central theme: to become an interface that mediates on the fundamental question: “what does the liberated being know, and how does it say?” This question, placed by Anjali Sriram as a guiding key above the mantra ETAT SAMAGAYAN (14), demands reflection on every word: “what” — what truth is at stake; “know” — what does knowledge mean; “the liberated” — what it means to be free; “being” — who can hold knowledge; “how does it say” — what forms of communication are possible; “say” — what is speech, what is sound.

The classical South Indian dance Bharatanatyam taught me that performance can be a medium — a way for the dancer to channel what cannot be expressed in words to the audience. Between 2008 and 2012, I created a film in the genre of “creative documentary”, presenting the theory and practice of this form of communication. The full 98-minute film, titled The Nine Movements of the Eyelid, is available to view on Vimeo.

Understanding and using the power of emotions became essential for shaping anumana. In their book Natya Yoga (15), Anjali and R. Sriram describe the epistemological foundations of Indian performance art and spiritual growth — in essence, the foundations of human life. Their methods show how to create the necessary distance between the observer and the observed, without which unbiased observation is impossible.

Krishnamacharya’s yoga practice emphasised breathing, the individuality of each practitioner (reflected in one-on-one lessons instead of group classes), and the importance of studying yoga philosophy alongside physical practice (16). From this, I learned a lesson that has remained crucial: there is no substitute for regular, long-term practice. As technology increasingly promises shortcuts to “self-optimisation”, this truth seems more relevant than ever. By 2025, I will have practised daily for forty-one years. This deep, embodied experience of the physiological effects of practice on cognitive functions continues to inform how I conceptualise anumana as an interface.

Meditation added another dimension. The technique of occupying the “monkey mind” (17) so that it no longer disrupts deeper concentration is now widely referenced in research in quantum biology and medicine. My first experience at the Vipassana Center in Igatpuri — then still isolated in the mountains, with simple thatched huts for accommodation — was profoundly new for someone raised in the West. Spending ten days without speaking, reading, writing, or communicating in any other way opened up a space for “something different,” for “something new”. The structured daily routine, the special food, and the long hours of concentrated practice left deep imprints, enabling what we now call “embodied experiences”, memories that can be recalled and reactivated. For anumana, even in its early stages, these possibilities cannot be ignored if it is to succeed.

Equally important was my stay at the Theosophical Society in Madras (see. Fig.2). Alongside the knowledge gained from lectures and conversations, I witnessed how the physical and ecological environment, combined with the daily physical and spiritual practices of everyone there, and a shared awareness of the need for silence and peace, deeply shaped research and reflection. At that time — forty years ago — it seemed to me that the so-called ashrams (18) of India were valid alternatives to Western scientific research institutes. Like any faculty, an ashram could have a “dean”, here called a “guru”, who guided the direction and often the methods of research. But in the Indian gurukulam, teaching methods are entirely different, built upon fundamentally different assumptions about how knowledge is transferred. The student, for example, chooses the guru and determines whether there is a resonance strong enough to allow the reception of knowledge. Similarly, the guru’s authority is not institutionally granted but validated by those who come to learn — guided by principles set out in detail in the Guru Gita (19). These guidelines describe an effective teaching–learning dynamic aimed at understanding life itself.

Figure 2: The author Rajele Jain at Theosophical Society in Adyar, Chennai, 1984.

For anumana, this experience has become one of its cornerstones. The ideas of “mastery” and of cultivating a balanced teacher–student relationship are central to how the interface is envisioned as a medium for knowledge creation. I will discuss this further, ahead.

The world premiere of Angika — a choreography by the avant-garde and influential Indian dancer, author, dramaturge, visual artist and activist Chandralekha — in Bombay in 1985, revealed to me the profound power and intelligence of Indian performance art. In the Natyashastra, the foundational theoretical text of Indian performance art (20), every detail is explained — including the dramaturgical tools through which a viewer can be guided toward the intended state: the capacity to acquire knowledge. One could call this interface art, but in relation to the Natyashastra, it might be even more accurate to describe it as interface science. Within its pages, one finds the psychological, aesthetic, physiological and epistemological insights that form the basis of this process.

Long before the term ‘artistic research’ gained attraction in the West — and before the aesthetic research experiments of the 1960s and 1970s (21) were revisited, often without explicit acknowledgment — anumana had already embraced this approach: conducting research through artistic methods. The aforementioned artist book by Robert Filliou, Lehren und Lernen als Aufführungskünste (Teaching and Learning as Performing Arts), published in 1970, can certainly be considered its starting point. While the precise definition of this method cannot be laid out here, I have addressed it in detail in my dissertation (22).

In summary, these are the aspects and findings that can be integrated into the development of anumana to date:

“Knowledge”, as understood or sought in this context, cannot be attained through logical language alone.

Fundamental human emotions must be acknowledged, worked with, and understood in terms of their effects on cognitive processes.

Physiological conditions — both individual and environmental — must be considered.

Regular and long-term practice is essential.

The involvement of masters is helpful if not indispensable.

A teacher–student dynamic that unites both parties in a research process must be established.

Experimentation, performance, and the creation of specific environments function as artistic methods of research.

RESEARCH YEARS: FINDING RELATIONS

art – pioneering – computing – criticism of knowledge – avantgarde – media competence – interactivity – knowledge generation – experimental research

Back in Germany, I developed a deep interest in art and artists. While looking for income, I also encountered the first personal computers, demonstrating ICL machines at a Düsseldorf computer fair—devices that would soon be sold to ordinary households. There was no graphical user interface yet, nor any networking, but these “calculating machines” already exerted a fascination that stayed with me.

For anumana, it matters that I experienced this pioneering arc first-hand: from amber-yellow or green text on dark screens inside beige boxes, to the first Apple computers with black text on light backgrounds; from early vector strokes to colour graphics; teletext as a forerunner of websites and email; modems linking universities (a precursor to the internet); the first scanners; the first digital audio, video, and moving-image files; HyperCard enabling interactivity and non-linear storytelling; hard disks replacing floppy disks; CD burners for home use; the first digital video-editing tools; image-manipulation software; early 3D construction programs (one even named World Construction Set); video projectors; DVD burners; a visit to the Fraunhofer Institute’s CAVE for virtual reality; an early video conference at SONY in Cologne with MIT Boston; text-based internet exchange among a handful of institutions; and, finally, the graphical World Wide Web — for those with a personal computer, otherwise via internet cafés.

I used and experimented with all these technologies (23), both in my artistic practice and to make a living. I taught myself because—apart from the manuals—there were no teachers for tools that were brand new and constantly evolving.

During those years, Fries’ insight into the limitations of our sensory perception was confirmed for me in practice through my collaboration with the experimental filmmaker Werner Nekes — quite literally “brought before my eyes”. Nekes, a pioneer of experimental film art, was also a passionate collector of objects, references, and research materials on the history of optical media. His legendary collection, unmatched worldwide in its scope and diversity, was built over decades of visits to flea markets, auctions, and meetings with inventors and historians.

In his film series Media Magica (1995–1997), Nekes demonstrated many — though not all — of the principles through which our sensory perception can be guided or deceived. His deep knowledge of the predecessors of today’s media, as well as of innovative concepts that had never been realised, was truly unique.

From him, I learned the importance of archives — of understanding the history of the tools and media we work and live with. Only through such understanding can we criticise, improve, or propose alternatives. Nekes applied this principle to film: he exposed the technical illusions of the medium and questioned its production conditions. By co-founding a free filmmakers’ cooperative, he worked to democratise access to what had once been prohibitively expensive production tools. Eventually, one might say, “once it had all been done”, he shifted focus, turning his attention to the prehistory of film through his collection, so that others could learn — and avoid “reinventing the wheel”.

The artist and aesthetician Bazon Brock has made significant observations about the concept of the avant-garde, critically challenging today’s obsession with the pursuit of the completely “new”, whether in art or in science. He writes:

“In truth, the question has always been how the present can relate to its past and how the new can be reconciled with the old. In a sense, the emergence of the new is inevitable; it does not need to be deliberately or intentionally brought into the world. Even in the most radical attempts to keep traditions constant, the new inevitably arises — if only because human language rarely allows for the unambiguous transmission of meaning, even when such clarity is intended.”(24)

I introduced Werner Nekes to the video format and later to digital image processing, whose technical possibilities he was eager to explore in depth. It was also Nekes who recommended me for postgraduate studies at the newly founded Academy for Media Arts Cologne. The academy’s pioneering work lay not only in its focus on new media but also in its quasi-revolutionary, emancipatory stance: that artists, designers, and researchers should have access to the same cutting-edge technologies used by commercial enterprises to shape the media landscape. This approach ensured that the development of these media and their products would not remain the exclusive domain of financially powerful stakeholders and advertising agencies. In other words, it created the conditions for critical engagement and genuine media literacy.

In the first semesters (beginning in 1990), all twenty-five students and roughly twenty professors were swept into an understandable euphoria by the “unimagined possibilities” that these advanced technologies seemed to promise. It felt as if the ultimate interface had arrived — a space where everything could be connected to everything else, where every sensory experience could be expressed and integrated, where the world’s knowledge could be made accessible to all, where alternative realities could be constructed and tested, where digital alchemy could take place. Any image that could be imagined could now be created, just as any sound, film, spatial environment, or “virtual reality” could be brought to life.

Video artists approached the unique qualities of their medium much like experimental filmmakers before them, exploring the simultaneity of recording sound and image, and probing the electronic conditions — feedback, tape wear, and distortion — until, here too, it felt as though “everything had been done.”

During this period, Nekes realised one of his long-held dreams: to make painted images move. For his film Der Tag des Malers (The Painter’s Day, see Fig. 3), I used Photoshop filters within the video animation programme Adobe After Effects, a process so computationally demanding that rendering took several weeks at the time. When the resulting video material was transferred to exposed film and presented to film audiences and curators — many of whom had yet to encounter these emerging image-processing possibilities — it led to the film’s selection for screening at the 54th Mostra Internazionale d’Arte Cinematografica in Venice (Fig. 4).

Figure 3: Poster from Der Tag Des Malers, the film by Werner Nekes.

Figure 4: Text in catalogue of the 54th Mostra Internazionale D’Arte Cinematografica in Venice.

Such was the fascination with digitally generated images — these “new” images. Television audiences, too, were captivated, particularly by the emerging aesthetics of MTV videos. The sensory overload of rapid cuts, garish colours, and virtual “dream” scenarios — as unnatural and exaggerated in their artificiality as they were enticing to viewers accustomed to conventional television imagery — drove rapidly shifting aesthetic trends. These trends were shaped largely by technological developments rather than by the intentions of producers. Form often became more compelling than the content it conveyed.

Following the pioneering years of film and video artists, it fell to media artists to explore, interrogate, and critique their tools through their work. Yet, for some time, there was a noticeable gap between those theorising about technological futures and those already experimenting, almost “virtuosically,” with these new tools. Media philosophers, technologists, and artists (25) often presented their ideas using overhead projectors, printed handouts, or, at best, VHS recordings — an approach that sometimes created striking contrasts with the innovative concepts they were discussing.

Video 1: This short video clip reveals the spirit of optimism that the idea of digital media and its possibilities have sparked in the early 1990s: Peter Weibel at KHM in 1990, recorded and edited by Rajele Jain.

A pioneering spirit fostered bold visions, and the artists, artistic technologists, and media philosophers of the time were deeply invested in the idea of improving the world through greater knowledge — both for all and by all.

In this atmosphere of optimism and exploration, anumana slowly began to take shape. Concepts such as immateriality, network, avatar, interconnectivity, virtual reality and global civilisation resonated strongly with ideas from Indian philosophy. Driven by this connection, I travelled to India once again to support the development of my final installation project at the KHM. This time, I would focus on works and pioneers who were already on their way to bridge Indian and European knowledge, while making comparisons and developing fusions, even. I interviewed Raja Deekshitar, the head priest of the historic Chidambaram temple, exploring perspectives on concepts such as…

Interactivity Three-Dimensionality Legible City Virtual Space Trompe L’oeil Virtual Reality Interface Art Software Video Disk Film Future Vision Interactive Media Network Tree Hypertext Navigation Cluster Strings Museum Network Layering Museum Image Reactor Image Flow Space Cybernetics Television Digital Images High Definition TV Computer Animation Technology Zapping Media Politics Machine Telematics Scanning Spatial Image Immateriality Simulation Cyberspace Visual Language Global Information Society Reality Industry Nonsense Machine Computer Information Aesthetics Visual Communication Information Fractal Mandelbrot Chaos Research Mathematics Networking Informatics Interdisciplinarity Individuality Media Imitation Video Transformation Architecture Computer Graphics Memory Robot Mass Media Time Sign Interactive Art Feedback Science Economy Technology Stereoscopy Holography Laser History Movement Music Video Creativity Kinetic Arts Interference Culture Design Print Artificial life Imagination Intersubjectivity Visualisation

His responses revealed many links between ancient philosophical frameworks and the rapidly emerging language of digital culture.

Figure 5: The author interviewing Raja Deekshitar, head of priests of Chidambaram temple, India 1993.

Due to time constraints, I was unable to visit Dr. Nrusingh Charan Panda, the author of Maya in Physics, whose introduction I had read as a kind of return to Fries’ ideas:

“Does the quantum physicist really work on empirical entities that can be seen, heard, smelt, touched and tasted? If he can be a mystic to explain the empirical phenomena, why is he afraid of metaphysics to explain the totally?”(26)

But before leaving India (in the days before email correspondence, in 1992) I wrote to Dr. Panda, inviting him to participate in my project. Figure 6 shows his reply:

Figure 6: Telefax by Dr. Nrusingh Charan Panda from 18.6.1993.

All Ways – Saraswati and Strange Attractors (Cologne, 1993) became an extensive spatial installation. It encompassed the KHM computer laboratory —housed in a historic building with columns — along with two adjoining rooms and their users, whom I declared to be “temples” (27). By layering a different level of meaning onto an everyday situation, I reframed the scene: the computer workers, sitting motionless before their shimmering monitors, appeared in this context as worshippers of flickering light and immaterial promise – see Fig. 7.

Figure 7: “All Ways – Saraswati and Strange Attractors“, multimedia installation by Rajele Jain, Cologne, 1993.

The experimental juxtapositions of Indian philosophy and the theories of media and quantum physics were staged in a variety of ways – see Fig. 8, for example. Additional explanations can be found on my website (28).

Figure 8: “All Ways – Saraswati and Strange Attractors“, multimedia installation by Rajele Jain, Cologne, 1993.

Just as I presented my approach and preliminary research findings for discussion in various “environments” created with the technological media available to me at the time, I also hoped that relevant institutions would develop multimedia and interactive applications for practical purposes.

Yet, even today, there is still no language or teaching-and-learning application that matches the scope, aesthetic quality, dramaturgical sophistication, adaptability, or conceptual richness of a well-designed computer game. The same shortfall applies to virtual museums, virtual travel, and educational platforms of any kind. This lack of aesthetic appeal, usability and conceptual depth reflects a problem we had already identified during the pioneering years of the KHM: the absence of interdisciplinarity and cooperation among the diverse stakeholders required to craft such multimedia products — projects that fall outside the commercial entertainment industry (29).

While financing these technologies has become progressively less costly, another challenge persists: effective communication among technicians, programmers, philosophers, scientists, curators, producers, artists, authors, performers and designers. Too often, philosophers lack technical familiarity; technicians limit creative ambitions under the pretext of technological constraints; artists struggle to express their concepts to scientists and programmers; and authors clash with producers over budget-driven compromises.

By the end of the 1990s, anumana had aspired to become a fully developed multimedia interface — a tool for generating and shaping knowledge through the integration of emerging technologies. However, the commercial industry’s focus on dazzling technological spectacle, coupled with the tentative and often underdeveloped prototypes produced by some new-media artists, left little fertile ground for such ambitions. Without large-scale, well-funded interdisciplinary collaborations, art remained confined to the studio — or, more recently, the university — while commercial players retained interpretive dominance in the public sphere, where “users” mostly consume rather than truly participate in shaping the collaborative web.

It became increasingly clear that relying on technology alone to guide the search for meaning and truth was misguided. In the context of anumana, this realisation prompted a shift towards seeking alternative methods — ways of discerning relevant information, countering the flood of information, and addressing the deliberate manipulation strategies of today’s “new media”.

This marked a turning point: from actively producing technology-supported interfaces aimed at communicating content to an audience, to creating conditions that empower audiences themselves to transition from passive consumers to active participants in the pursuit of knowledge. What once carried the label “interactivity” in my projects has now evolved into what I call service, a concept that will be explained further below.

NEW STRATEGIES

Ayurveda – analog – expanded concept of art – empowered recipient – authority – nature – quantum biology – service

My shift from the immaterial digital world back to the more analogue world in the early years of the new millennium unfolded through multiple levels. My decades-long daily yoga practice continued, but I also began studying Ayurveda, the Indian “science of life”. Much of the literature available in the West struck me as superficial and poorly grounded, until I discovered the works of Dr. Vinod Verma. As early as the 1980s, Dr. Verma had begun presenting Ayurveda — understood as the science of living — in a manner that was both accessible and scientifically rigorous (see Fig. 9).

Figure 9: Titles of several scientific publication of Dr. Vinod Verma from the years 1977-1984 when she had worked in research departments in Chandigarh/India, Paris/France, Freiburg/Germany and Bethesda/USA.

With her dual doctorate in biomedicine and two decades of experience in the pharmaceutical industry in Germany, she possessed all the expertise needed to lift Ayurveda out of the haze of Western esotericism. She offered a multi-year training programme leading first to the level of “junior assistant” and then “senior assistant,” which I successfully completed. I found her teaching so compelling that it inspired me to produce my second full-length documentary film, Annapakta – One Who Can Digest Food | Yoga and Ayurveda, Science of Life (30).

Ayurveda explains in great detail the importance of lifestyle and its profound influence on thinking, feeling, and cognition. Strikingly, many of these principles align with current findings in quantum biology and medicine. After nearly forty years, this prompted me to revisit the books of N. C. Panda, whose work had anticipated many directions later taken by Western research. Remarkably, after two decades of collaboration on our joint project, I was finally able to meet him in person at his home (see the photo in Fig. 10).

Figure 10: Nrusingh Charan Panda and the author Rajele Jain in Bhubaneshwar, 2013.

The earlier exchange of faxes and letters during my installation project twenty years prior — followed by the eventual visit — reaffirmed the importance of encountering a ‘master’ in one’s own search for knowledge. For anumana, this dimension has since become indispensable.

The experience of engaging with Ayurvedic knowledge was deepened through direct work in and with nature — from cultivating a vegetable and ornamental garden to tending a forest that had remained untouched for more than twenty years. My move from Lisbon to Maria Vinagre, on Portugal’s southwest coast, brought a drastic change of environment. This shift also transformed my working methods, “naturally” — or rather, in line with Ayurvedic principles and recent findings in quantum biology.

In 2005, I undertook a multi-year project to digitise the audio and video archive of artist and aesthetics professor Bazon Brock. Using FileMaker, I systematised the material by cataloguing hundreds of recordings and equipping the database with search functions. Through this work, I was confronted not only with the full scope of artistic developments and ideas from the 1960s and 1970s, but also with Brock’s aesthetics as a method of cognition. He placed research focus squarely on the role of the viewer — the audience — insisting that without a ‘professionalisation of the audience’ there can be no meaningful results from the experience of art (31).

Although it is well known that the “expanded concept of art” introduced by Beuys and others gave far greater importance to the role of the recipient, this legacy — often invoked by so-called “interactive art” aiming to “involve” the audience — is still frequently misunderstood. The mere fact that a visitor to an interactive exhibition can press a few buttons to influence an event, or that their body data are used to manipulate processes, does not make them an ‘active’ co-creator in the original sense. At best, they become unsuspecting extras — a position comparable to that of many internet users today. This imbalance in knowledge between producers and recipients cannot serve as the foundation for a “discourse free of domination”(32) – an ideal to which anumana aspires as a basis for collaborative research and civic participation.

My intensive engagement with the works of Brock, Beuys, Filliou, and the Fluxus community, alongside Ayurvedic principles and practical work in the environment of Maria Vinagre, shaped the further development of anumana. The idea of creating an interface that could guide the recipient toward certain ‘mind-openers’ was set aside in favour of offering what might be called selfless service — a term drawn from the mythology of the Indian deity Hanuman (33). This took form on our so-called Hanuman Farm, a space including a three-hectare forest area, where the aim was to create ideal conditions for knowledge generation. These conditions include:

The opportunity to rest, to sit (or walk) undisturbed.

The possibility of physical preparation, ensuring that “the body does not interfere”, through yoga and meditation techniques.

The possibility of a symptom-free metabolism, supported by preparing food according to Ayurvedic knowledge, along with offering courses on the subject.

The opportunity for research, by designing and making accessible a library with relevant literature on art, science, and philosophy from both Western and Indian sources.

The possibility to experiment with and experience sound, in the sense of mantra theory or in light of recent quantum biological research, using analogue sound generation (34).

Aesthetic and as natural as possible design of rooms and objects.

Good air and natural light.

Furthermore, I defined the factor of “long-term” as a fundamental condition for the optimal development of anumana. This implies valuing processes that unfold over extended periods, since only such duration allows certain results to emerge. It also entails a specific understanding of the notion of practice as applied to scientific and artistic work. As Peter Sloterdijk observes:

“Practice is the oldest and most consequential form of self-referential practice: its results do not flow into external states or objects, as in working and manufacturing, but rather elaborate the practitioner himself and bring him ‘into form’ as a subject-that-can. The result of practice shows in the current ‘condition’, that is, in the practitioner’s state of ability. Depending on the context, this is described as habitus, virtue, virtuosity, competence, excellence or fitness.”(35)

On the other hand, time can also be extended by filling it with less. This leads to my current experimental setup, which follows the tradition of the Indian Forest Academy (36). It involves designating a specific area of forest to concentrate on a single sentence or thought for an entire day (or other chosen duration), either with others or, if no one is available, alone. In solitary practice, the figure of the master inevitably reappears. I would therefore like to explain this principle in more detail.

By ‘master’, I mean a person whose life and work I find exemplary, a source of inspiration that points towards something higher or more true. This can be understood both literally and figuratively. While some figures are widely recognised as masters, it is equally valid for an individual to discern mastery in someone less visible, even absent from history books (37). In Vedic philosophy, mastery is not restricted to persons: a master can also be a question, an experience, or a theme to which one dedicates sustained investigation.

The basic principle remains constant: facing a master means encountering what one cannot yet do, know, or possess — whether in terms of talent, insight, experience, or sensitivity. In the Indian guru-shishya tradition, this constellation represents the ideal condition for knowledge transfer, as described in the Guru Veda. Unlike the Western master-disciple relationship, the Indian guru is not the origin of knowledge but rather a medium, an interface, through which knowledge is transmitted. This becomes possible once the guru has prepared the disciple to be receptive, to recognise “the truth” without falling prey to deception, delusion, or inner obstacles to freedom.

The guru is therefore not the possessor of truth or knowledge, but rather the interface to its realisation. For this reason, a genuine guru will not proclaim “truths” in a dogmatic or authoritative way. Instead, the teachings we find in their recordings or instructions are best understood as preparatory exercises —methods for cultivating our cognitive capacities so that we remain receptive and critical when we encounter something genuinely new, rather than failing to perceive it at all.

This hope for the new is also echoed in descriptions of technological interfaces, which are often said to offer ‘new experiences’. Yet if we consider the narrow bandwidth of our human sensory perception. — for example, the limited frequency ranges within which we can hear or see (Fig. 11) — we must ask ourselves: in what sense, or to what extent, can technological interfaces truly deliver something new? If our perception already recognises so little of the world’s spectrum, what kind of novelty can technology promise within those limits?

Figure 11: The human visible spectrum (same applies to the range of human hearing).

The method of ‘play’ is, of course, a valid means of challenging our perceptual habits — provided it does not claim more than to reveal their influenceability and unreliability. According to yoga psychology, however, physical experiences facilitated by ‘virtual reality’ applications or immersive environments can just as easily occur through pure imagination. N.C. Panda describes this internal mode of constructing thought as follows:

“A ghost may not exist. In spite of the fact of the non-existence of a ghost, we get a verbal knowledge about it by listening to ghost stories and reading books on ghosts. We get a picture of the ghost with all the details of its height, colour, head, face, teeth, tongue, arms, legs, nails, etc.. We also learn the malevolent nature of the ghost. All these informations are printed in our citta. These impressions that are stored in citta come to surface and phenomenalise, at some time opportune for the phenomenalisation. A ghandarva may not exist; a ghost may not exist. But the vikalpa on such objects that are devoid of objectivity (there is no real object corresponding to the concept) causes fluctuations in the citta.”(38)

The mere act of imagining something can affect our physical state. In this way, the kind of physical experience one might have in a Second Life environment —where the body is extended through an avatar — can equally arise by simply standing still, slowly raising both arms, and imagining them to be fifty metres long (39).

Critical questioning of the means by which insights are developed and goals for action defined remains as urgent as ever. Increasingly, it appears that it is not the technologists who possess — or are capable of developing — the means for a truthful understanding of the whole.

However, the alternative — the belief that a human being can never encounter universal truth — is, to me, equally unacceptable and ultimately illogical. As long as we recognise that a person is part of something greater, that they are not alone, then an interface exists—a possibility of access. Discovering this, or at least contributing to its development, is the intention behind anumana. This pursuit should be taken up once more, with a fresh start and expanded possibilities.

WAITING: STARTING ANEW

neuroplasticity – re-wiring – attention manipulation – filter bubbles – truthful speaking – non-domination communication – criticism ability

A well-known method of human cognition appears in Indian meditation practice: neti-neti — a method of exclusion in which, in the search for truth, every arising thought or idea is met with the response: ‘not this, not that.’ Regardless of the meditation technique used, regular practice is known to sharpen one’s critical faculties, as it cultivates the ability not to be swayed by one’s own thoughts and emotions. One explanation for the impact of meditation is the following:

“Meditation is not about quieting the mind. Nor is it about training the mind to only think good or spiritual thoughts. Meditation, properly understood, is about transforming our relationship to the mind. It’s about cultivating the ability to disengage from the mind, to no longer identify with the mind, so that we can discern and discriminate which thoughts are worth listening to and acting on, and which ones aren’t.”(40)

Elaborating on the notion of interface, the author continues:

“As part of an interface, the idea of capturing the ‘evolutionary needs of the moment’ seems to be a desirable capability: we also find an unprecedented ability to be present in a way that most of us have never experienced. And in this powerful presence, we become a finely tuned receptive instrument that can sense into the deeper needs and feelings of those around us, and into what we might call the evolutionary needs of the moment. We can feel it with our being. And this enables us spontaneously discern the path of positive action in a way that transcends mere cognition.”(41)

Some biologists have been researching whether and how the findings of quantum physics can be implemented in the field of biology. Their theories might not yet be considered (scientifically) proven, but they converge with Indian ideas about how the brain works in remarkable ways. Certain key findings about neuroplasticity and the brain’s ability to rewire itself mark a clear departure from conventional neurophysiology — as I studied it — which holds that the human brain remains largely static after reaching adulthood and simply declines with age. In contrast, these new theories offer the promise of healing trauma, enhancing cognitive abilities, transforming limiting habits, and improving overall well-being (42).

As one might expect, there is also a darker side to these developments, with fantasies of ‘social engineering’ and similar ambitions. This makes it all the more important to develop the ability to recognise the potential dangers of new technologies — much like the learning objective of ‘media literacy’ some fifty years ago, when school lessons included comparing how different newspapers reported the same event.

When texts or other outputs generated by so-called AI technologies are used to clarify issues of general interest — yet the underlying resources and strategies (the ‘algorithms’) used to produce them remain opaque, or more crucially, when it is unclear which sources were excluded and why — this violates fundamental principles of truthful communication. Such opacity undermines the very possibility of a just, rational, and socially responsible world, which has long been the declared aim of political decision-makers, regardless of the system of government.

At the same time, the vision of eventually having all prior human knowledge available in digital form — and making it accessible to Artificial Intelligence (A.I.) — naturally gives rise to a previously inconceivable scenario: one in which new combinations or patterns might emerge that are currently beyond the reach of researchers, simply because the existing knowledge base is too vast and fragmented. Until such a comprehensive resource exists, however, A.I. remains of limited value as a tool for anumana, since its outputs cannot be verified in terms of their sources. In his commentary on the Hanuman Chalisa (a Vedic text that describes the principle of Hanuman) Devdutt Pattanaik explains the concept of ‘infinite knowledge’, which differs significantly from its Western interpretation (43).

The same applies to the new possibilities for manipulation that have emerged from the findings of the ‘new sciences’. By understanding these developments, we also gain the ability to recognise and critique them — to free ourselves from their influence or, where appropriate, use them constructively. As great as the early euphoria surrounding immaterial and virtual technologies once was, our capacity for critical reflection does not appear to have kept pace. While media artists have produced powerful critiques of social conditions using advanced technologies, fascination with the ability to mix and connect data across systems still seems to dominate. The technologies themselves — how they function, how they are applied, and how they are designed — are rarely, if ever, subjected to serious questioning.

If the large-scale manipulation of attention — currently attempted and often achieved through social media and other media channels — is combined with insights from brain rewiring and neuroplasticity, both of which are also seen as forms of interface technology, then a well-grounded practice of media criticism becomes essential for the preservation — or realisation — of individual maturity.

It will not only be the “usual” sensational and scandalous news that attracts the most attention, or the manipulation that is already taking place through the so-called ‘filter bubbles’ – which use the enticing concept of user-defined preferences for news topics or similar to trick the recipient into believing that they are “informed”, while all information that would go against these preferences is by definition hidden. According to a study by physicians Michael Reilly, MD, and Amir Hakimi, MD, even the well-known mania for selfies with image filters, in combination with self-beautification through Botox injections, means that people can no longer read each other’s facial expressions correctly, further skewing their perception of reality:

“The embodiment theory states that the processing of emotional information, such as facial expressions, includes reproducing the same emotions on one’s own face. For example, when we observe a smile, our faces tend to smile in an automatic and often imperceptible way as we try to make sense of that expression. Therefore, if one or both members of a two-person interaction are dealing with an altered ability to make facial expressions, the non-verbal communication between those two parties will be impacted and understanding one another may be more difficult.”(44)

For anumana, the obviousness of the possibilities for manipulation by media technologies listed so far provides the justification for moving to the other side by attempting to develop into an interface that is capable of detecting such manipulation strategies. For truthful speech is possible, and as long as someone on this planet upholds this demand, it also exists. The literary scholar Ruth Klüger, a Holocaust surviver, states this very impressively:

“I proceed from the assumption that truthful speech is the glue and cement of all social communication; the possible, if not the usual.. The entire fabric of our interactions, including written communication, begins to collapse the moment we abandon this premise.”(45)

As explained in the introduction to Klüger’s text, truthful speech “has its precondition in the connection of the self of language with the self in the body, which undergoes experiences and is exposed to suffering”(46). Accordingly, the recipient must possess the ability to recognise the truthfulness of such speech — that is, they must have physically experienced what it means to speak truthfully, and to witness another speaking truthfully.

Unlike interfaces designed to enable ‘new experiences’ or to escape human control, anumana — the interface developed here — aims to foster advanced critical skills in engaging with digital data. It does so by offering a form of service that integrates the various aspects discussed throughout this text. As part of my biography, anumana functions as an interface for myself, even as I work to make this service accessible to others. A planned long-term experiment will demonstrate the effectiveness of this approach.

Video 2: Forest Academy #1

REFERENCES AND NOTES

1. Anja Tippner and Christopher F Laferl, eds., Texte zur Theorie der Biographie und Autobiographie (Stuttgart: Reclam, 2016). Own trans. of quote.
BACK

2. Pierre Bourdieu, Praktische Vernunft: Zur Theorie des Handelns, trans. H Beister (Frankfurt am Main: Suhrkamp, 1998) p. 76. First published in 1994, as Raisons Pratiques. Own trans. of quote.
BACK

3. Mikhail Bakhtin, “Die Antike Biographie und Autobiographie”, in Texte zur Theorie der Biographie und Autobiographie (Stuttgart: Reclam, 1998) p.112.
BACK

4. Nrusingh C Panda, Mind & Supermind Vol. I (New Delhi: D. K. Printworld, 1996) p. 140.
BACK

5. Bakhtin (1998) p. 104 [3.]
BACK

6. Nancy K Miller, “Gender and the Subjects of Autobiography”, in Texte zur Theorie der Biographie und Autobiographie, ed. Anja Tippner & Christopher Laferl (Stuttgart: De Gruyter, 1998) p. 281.
BACK

7. Miller (1998) p. 285. [14.]
BACK

8. Ibid.
BACK

9. I was also supported in this by my mother’s research findings in her book Weak-minded Gender or Symbolon? Reflections on the Relationship between Philosophy and Feminism (own trans to English) where she points out that it is not only the “ability to think” that makes an intelligent person, “but also the freedom to apply this thinking. Kant recognised this fact, but he did not apply his findings to women”. Elenor Jain, Schwachsinniges Geschlecht oder Symbolon? Reflexionen zum Verhältnis von Philosophie und Feminismus (Sankt Augustin: Academia‑Verlag Richarz, 1989) p. 88.
BACK

10. Robert Filliou, Teaching and Learning as Performing Arts (Köln [Cologne]: Verlag Gebrüder König, 1970) p. 221.
BACK

11. Lutz Geldsetzer, “Jakob Friedrich Fries’ Stellung in der Philosophiegeschichte” (“Jakob Friedrich Fries’ Position in the History of Philosophy”, own trans to English). Retrieved from <https://wwwalt.phil.hhu.de/philo/geldsetzer/homepag6.html>, accessed December 2025.
BACK

12. I had to correct my opinion, however, because Prof. Geldsetzer actually gave extensive lectures on “Transcultural Philosophy” in later years, see <https://wwwalt.phil.hhu.de/philo/geldsetzer>, accessed December 2025.
BACK

13. The theory and definition of Sanskrit is multifaceted. Etymologically, its name connotes a work that has been “well prepared, pure and perfect, polished, sacred” – Will Durant, Our Oriental Heritage (New York: Simon & Schuster, 1963), p.406. First published in 1935. Of particular interest in our context is the aspect that this language is said to utilise all possibilities of the human voice to generate sound, thereby becoming an universal language. This raises the idea that the meaning of a word or text can be understood by concentrating on its sound. This is also one of the reasons for the meaning of so-called ‘mantras’.
BACK

14. A Sriram and R Sriram, Vedic Chant & Yogasutra of Patanjali (self-published, n.d).
BACK

15. A Sriram and R Sriram, Natya Yoga: Peace Passion Pain: Mapping Emotions (Delhi: Motilal Banarsidass Publishing House, 2023).
BACK

16. The direct student, yoga teacher and researcher R. Sriram has clearly explained how breathing can influence our ability to think, in a book whose title translates as The Secret of Breathing. R. Sriram, Das Geheimnis des Atems (Freiburg: Verlag Herder, 2021).
BACK

17. A term from Buddhist meditation practice that describes a lack of control in one’s thoughts.
BACK

18. An ashram is a place where one strives towards a goal in a disciplined manner. Such a goal could be ascetic, spiritual, yogic or any other.
BACK

19. For example, see how the 352 verses from Guru Gita are translated and commented by Swami Narayanananda. This is verse 270: “The clever, the able, the discriminative, the pure, the knower of the Truths of the spiritual science, the one whose heart is pure, he is really the Guru. His Gurutva (the state of a Guru) shines, not of others. / Notes: Just as it is difficult to find a true disciple, so also it is equally hard to find a really qualified Guru.” Retrieved from Swami Narayanananda, Sri Guru Gita: Translation and Commentary (Rishikesh: Yoga Niketan Trust, n.d.), <https://gurudevsivananda.org/Sri-Guru-Gita-By-Swami-Narayanananda.pdf>, accessed December 2025.
BACK

20. Thus the title of my film: The Nine Movements of the Eyelid (2008). It is available at <https://vimeo.com/1094171780?share=copy#t=0>, accessed December 2025.
BACK

21. See the comprehensive publications by Bazon Brock, e.g. at <https://bazonbrock.de/werke/detail/?id=3965&sectid=3667>,  accessed December 2025.
BACK

22. Rajele Jain, Lehren und Lernen als Aufführungskünste. Künstler als Beispielgeber nach Robert Filliou, doctoral thesis (Wuppertal: 2017). Translation: Teaching and Learning as Performing Arts: Artists as Example Givers after Robert Filliou.
BACK

23. More information at <http://www.rajele.net>, accessed December 2025.
BACK

24. Bazon Brock, “Theorie der Avantgarde”, in Kunst und Gesellschaft, Grenzen der Kunst, ed. Heinrich Klotz (Munich: C. Bertelsmann, 1980) p. 102. Own trans. of quote.
BACK

25. Like Abraham Moles, Achim Lipp, Alfred Biolek, Laurie Anderson, Bernd Girod, Bernd Kracke, Dominik Graf/Peter Brinkmann, Fabrizio Plessi, Gunter Rambow, Hannes Hoff, Hartmut Jürgens, Herbert W. Franke, Ingo Günther, Jeffrey Shaw, Jody Burns, John Sanborn, Jörg Gutjahr, Vilém Flusser, José Encarnação, Jürgen Claus, Ludwig Wilding, Martin Potthof, Matthias Kron/Heiko Idenson, Wolfgang Menge, Horst Königstein, Michael Pfleghar, Michael Rutschky, Patrick Purcell, Peter Kolb, Peter Weibel, Rene Berger, Siegfried Zielinski, Titus Leber, Volker Beckmann, Wolf Donner – I was able to videograph them in the first three months of the KHM. I used the material to produce various works, such as the video installation Erstes Kapitel (Cologne 1992), the interactive DVD Analogischer Alpinismus (Cologne 2000) or the performance Archaeology of New Media (Lisbon 2015) included in the project Posthuman Corporealities, following an invitation by Isabel Valverde.
BACK

26. Nrusingh C Panda, Maya in Physics (Delhi: Motilal Banarsidass, 1991).
BACK

27. “A temple is a characteristic artistic expression of the hinduism and the center of the social and spiritual life of the society whom it is serving.” – a quote from Georg Michell, Der Hindutempel: Baukunst einer Weltreligion (Köln: DuMont, 1991). First published in 1977 as The Hindu Temple: An Introduction to Its Meaning and Forms.
BACK

28. See <http://rajele.net/all_ways_descr.html>, accessed December 2025.
BACK

29. The experience gained from interdisciplinary projects such as Black Mountain College, Monte Verita, Tagore’s Shantiniketan, etc. had, however, made ground-breaking work possible long time ago, without being continued as such.
BACK

30. This film can be viewed at https://vimeo.com/742760179, accessed December 2025.
BACK

31. Cf. Brock’s concept of the “Besucherschule” (“Visitor School”), a unique method to guide visitors through art exhibitions by making the content relevant to them.
BACK

32. “Discourse free of domination” is an important idea in Jürgen Habermas, Theorie des Kommunikativen Handelns (Frankfurt/Main: Suhrkamp, 1981), where the philosopher examines which conditions must prevail so that people can speak “truthfully” among themselves, and thus “optimally rationally”.
BACK

33. In Wikipedia one can read: “Subsequent literature has occasionally depicted him as the patron deity of martial arts, meditation, and scholarly pursuits (…) symbolising a connection with both the physical and the cosmic elements”. Retrieved from <https://en.wikipedia.org/wiki/Hanuman>, accessed December 2025. A very deep and helpful explanation of the principle of Hanuman can be found in Devdutt Pattanaik, My Hanuman Chalisa (New Delhi: Rupa Publications, 2017).
BACK

34. A technique of knowledge generation according to Indian philosophy is Vedadhyayanam or Vedic Chant: “the recitation of the texts is learned in order to come closer to the meaning of the word through the sound” (Sriram, …)  Furthermore, the influence of sound frequencies and waves of Indian ragas on human perception is described . Both methods are to be experimented within the anumana concept by using the sound instrument developed at Hanuman Farm under the name Club of the Knobs, the modular, analogue synthesiser system developed by my husband Kazike, aka Gerd Peun, for the generation of corresponding frequencies and sound events. The current version, resulting from a continuous development over decades, allows for the production of universal sound events, which naturally encompass both the audible and inaudible areas of human perception.
This system was introduced as early as 2005 as an example of an interface in a book by professors at the KHM: Georg Trogemann and Jochen Viehoff, Code@Art: Eine elementare Einführung in die Programmierung als künstlerische Praktik (Vienna: Springer-Verlag, 2005) p. 137.
Also, my first experience in programming, during a course on C++, under Georg Trogemann, introduced me to the basic principles of this programming language. I still remember the feeling of ‘wonder’ that came over me when I was able to make a line candle appear on the screen using the small set of commands that I had laboriously worked out. Today’s assistance in the form of ready-made routines, which can be graphically compiled in applications such as max/msp for specific needs, is also perceived by users as ‘programming’, but the ability to write original software and master the logic of the programming language seems to me, based on my experience, to be different. Perhaps only those who can ‘hack’ are programmers.
BACK

35. Peter Sloterdijk, Scheintod im Denken: Von Philosophie und Wissenschaft als Übung (Frankfurt/Main: Suhrkamp, 2010) p. 16. Own translation of the quote.
BACK

36. Cf. Tapovana (Sanskrit) blends the root words “tapas” – meaning ‘penance’ and by extension ‘religious mortification’ and ‘austerity’, and more generally ‘spiritual practice’ – and “vana” – meaning ‘forest’ or ‘thicket’. Tapovana then translates as ‘forest of austerities or spiritual practice’ <https://en.wikipedia.org/wiki/Tapovana>, accessed December 2025.
BACK

37. I am trying to prove this thesis with my project O Museu Privado (trans. The Private Museum): it is a space in which I introduce the visitor to a master, the one I claim to be one. The possibility of an encounter arises through a sign on the street pointing to the museum. Random visitors come and see the exhibition. The guest book is a testimony to the fact that real encounters (mind-openers) have taken place.
BACK

38. Panda (1996) p. 141. [11.]
BACK

39. I experienced several times the work of Isabel Valverde and her experiments with Second Life applications and motion capturing technologies. Videos from this project, called Senses Places, can be found at <https://www.youtube.com/@isabelvalverde7416>, accessed 2025.
BACK

40. Craig Hamilton, Unlocking the Power of Meditation (Self-published,
2018) p. 12.
BACK

41. Hamilton (2018) p. 43. [40]
BACK

42. See Fleet Maull’s publications and symposia on this topic, for example at <https://www.heartmind.co/home>, accessed December 2025.
BACK

43.  A quote from Devdutt Pattanaik, My Hanuman Chalisa (New Delhi: Rupa Publications, 2024) p. 14f.: “In the information age, as we move towards gathering data about everyone and everything, it is easy to assume we are moving towards infinite knowledge hence God-hood, through computers and databases. However, this data being gathered is material, not psychological. What is being measured is stimulus and its behavioural response. What is being manipulated by technology, is behaviour alone, not thought and emotions. What information is not being gathered is how the mind perceives and processes sensory stimuli. Science today is so focussed on the material, that it assumes measurable input (stimulus) and measurable output (behavioural response) is indicative of thought and emotion, and dismisses the arguments to the contrary. Reality is seen as what we do (measurable), not what we feel (not measurable). At best, doing is seen as an indicator of feeling. At worst, doing is seen as relevant while feeling is considered of no consequence. When the West speaks of an intelligence quotient or an emotional index, it derives all understanding of the mind from measuring behaviour. Measurement limits science. This distinguishes the modern discourse, and disconnects it with traditional Indian wisdom where measurement is seen as establishing a delusion (maya) of certainty.
The obsession with quotients, and indices, hence mathematics, reveals the desire to control, regulate, manipulate human behaviour. Control, in Hinduism, is an indicator of fear. The intelligent seek control: the strong have the resilience to handle the lack of control, and the knowledgeable know the futiliy of control. Hence, we ask Hanuman for strength as well as knowledge, along with intelligence.” Originally published in 2017.
BACK

44. Retrieved from <https://www.psychologytoday.com/us/blog/dissecting-plastic-surgery/201912/botox-and-emotional-expressivity>, accessed December 2025. The authors even go so far as to assume that patients treated with Botox are also changed in their feelings: “Not only does Botox affect the way others perceive one’s emotions but it can also impact the emotional experience of those who have received the injections. In one study, participants either received Botox injections or a wrinkle filler (Restylane), which does not paralyse muscles, and were instructed to watch emotional video clips. Participants who received Botox had a lower self-reported emotional response to the video clips. This finding supports something called the facial feedback hypothesis, which suggests that our facial expressions can affect our emotional experience.”  
BACK

45. Ruth Klüger, “Zum Wahrheitsbegriff in der Autobiographie”, in Texte zur Theorie der Biographie und Autobiographie, ed. Anja Tippner & Christopher F. Laferl (Stuttgart: Reclam, 2016) p. 325. Own trans. of quote.
BACK

46. Klüger (2016) p.321 [45.]
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

Rajele Jain > Vipulamati

Artist and filmmaker whose work spans video, performance, photography and documentary forms. Active since the early 1990s, she co-founded Vipulamati and directed the International Festival for Films on Art. Her practice engages archives, exhibition design and interdisciplinary collaboration to explore cultural memory, visual narrative and embodied media experience

Reviewers

Alan Klima > UC Davis 

Cultural anthropologist and professor at UC Davis whose work explores experimental ethnography, media practices and the politics of affect. His research spans sound, image and sensory experience, advancing innovative methods that bridge anthropological inquiry with interdisciplinary artistic approaches to documentary, performance and technological mediation.

Rui Penha > CESEM & ESMAE 

Composer, researcher and educator working across electroacoustic music, interactive installations and sound spatialisation. As Associate Professor and Vice-President for Artistic Research at ESMAE, he develops projects that integrate composition, technological experimentation and performance, contributing to contemporary discourse on digital creativity and embodied musical practice.

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 3:

I recently came across this blog post, on Hackaday, titled “AI is Only Coming for Fun Jobs” (1). The title says it all – creativity is being superseded by artificial intelligence, drawing upon a massive database to assemble appealing images, text, and sound “assets” in video games, pushing the human creator to the side-lines. So, what can we mere mortal programmers do to provide value to our creative foundation?

From the perspective of a programmer, I call it speculative programming. I do not believe that an A.I. can be ‘speculative’. While it might be able to spit out some code for finding a path through a forest, it cannot create behaviours based on a ‘personality’ encoded into a Non-Player Character (NPC). Can it even understand that as a prompt? Will an A.I. ever have the capacity to just ‘go for it’ and explore a solution, something not built into its training?

In this research, I am using well-established algorithms, implemented in C# and represented in Unity. Vital to the distinction between an A.I. and a human, I’m employing intuition, and perhaps even that complicated word, art, to create a system that enables fungible interpersonal relationships between NPCs and Player/Observers (humans) in real-time – using tiny neural networks clustered in a buckyball configuration, which I am calling a tinyNet.

‘Buckyball’ is the nickname for a molecule named in honor of architect R. Buckminster Fuller, due to its resemblance to his geodesic dome structures. Connecting 60 carbon atoms, its structure is made up of 20 hexagons and 12 pentagons, mirroring the pattern found in geodesic domes – also known as a dodecahedron. See Video 1:

Video 1: Screen capture from the “big” tinyNet in action. John Klima, 2025.

Over the course of this investigation, I kept returning to the notion that a look-up table (LUT) can be encoded as a neural network, such that a neural network can “know” that 3 + 4 = 7 without performing any math. A LUT is essentially a spreadsheet where one looks at the cell that is in row 3, and in column 6, to arrive at a return value. If LUT is replaced with a neural network, it implies that the network can be trained to “re-learn” that 3 + 4 = 8. In other words, the tinyNet can change the rules of the game – training overrides encoded logic.

To investigate this, I started with the most simple game: Rock Paper Scissors. We all know the rules of this game, and the results can be made into a simple 3×3 look-up matrix. But the fact is, every individual will have a proclivity to choose rock, or paper, or scissors each time the game is played. Given two “individuals” on first encounter, the normal rules apply. But if these individuals have several encounters, can they not predict the other’s behaviour? Can they not both change the rules to their favour? This is not a prediction so much as it is simply applying a different set of rules when Fred encounters Jane, and when Jane encounters Julia.

The research brings up another issue. There exists this thing I like to call the “crisis of representation”. How does one demonstrate the underlying concept of the algorithm that is making these decisions? Sadly, video game combat situations seem to be the clearest way to represent cause and effect in a person-to-person encounter. I can easily represent through animations and other means that rock beats scissors. But how does one represent, over time, that one person doesn’t like this other person, and why? Or, indeed, is perhaps in love with that other person, and why? Without having a huge repository of gestures and animations that can clearly demonstrate this cause and effect, we have a crisis, and at best we have a gigantic finite state machine (FSM) that is totally predictable regardless of its depth, and very cumbersome to maintain and expand.

Combat scenarios unfortunately are the clearest form of person-to-person relationships that one can represent in identifiable terms, without using dialogue. There have been great advances in using Large Language Models (LLMs) to create dynamic exchanges in computer-human interactions, primarily around dialogue systems (2). But very little around recognisable physical behaviours, let’s say “body language”, to convey emotion, sympathy, empathy, or hatred.

However, I propose that a hybrid tiny network topology is a step in the right direction, as it has – inherent to its structure – the ability to adapt, and feed fuzzy numbers to a blending of representations in a system designed to accommodate them. A mixture of gestures that can represent a range of emotional responses emergent from the interaction between one character and another, or between the Player (primary agent) and an NPC (a secondary agent); using sound, animation blending, lighting techniques, what have you. A tinyNet is adaptable to every representational challenge that I can imagine. The truly interesting aspect of this is that the same network code, applied to multiple tasks, can easily make for those tasks to inform each other through their inputs and outputs.

I am not sure exactly what I’m going to “get” from this, I’m just working intuitively with the idea in the back of my head that I want to make an NPC “personality” that changes through time, via Player choices. My NPC is not the same as your NPC, even if they are both named “Fred”. Having worked with neural networks for more than 20 years, I simply have this intuitive feeling that I can use them to achieve my goals. That intuitive feeling is not something that an AI possesses, it is fully in the human domain.

THE NON-MATH

I like to use this term, non-math, to describe the purpose of the network.  I use simple math problems to demonstrate that the network is returning “valid” results, but can equally be retrained to produce a different result: 2+3=5 is valid, but 2+3 could also return 23 (the numeric combination).

Though there is a wealth of complex mathematics going on under the hood and I am working numerically, the numbers are symbolic. I use the sigmoid function (a mathematical function that maps any real-valued number to a value between 0 and 1) but I’d be hard pressed to “solve it for x”. I don’t need to, that’s what the computer is for. I’m using numbers but they are simply symbols.

Again, to be clear, input one = 3, input two = 4, output = 7. It did not add the numbers, it simply knew that 3 and 4 is 7. You can almost think of it visually, or in terms of symbols. You have this scratch in the sand, a squiggle that looks like 3, and another that looks like 4. If you combine those two squiggles, don’t they kind of look like 7? Does not 3 combined with 2 kind of look like 5? Is this not what we do in our deep head when asked what 3 plus 2 equals? We know the answer, as long as the equation isn’t too complicated.

Computers count beans. Do we count 3 beans and add 2 beans to them to arrive at 5 beans? No!!! We consider the ‘3’ symbol and combine it with the ‘2’ symbol, to arrive at the ‘5’ symbol. It might as well be 6: the 3 squiggle and the 2 squiggle, when combined, does it not really look like the 6 squiggle? Or maybe, does it look more like the 5 squiggle? I can start with the assumption that 3 + 2 is most likely the 5 squiggle. Subsequently, if I pass the 5 squiggle output and a 3 squiggle output to another node, it will almost certainly arrive at the 8 squiggle. Does 3 plus 5 not kind of look like 8? This is essentially how a neural network functions.

We use beans as a method to confirm our symbolic assumptions. Thankfully, in Principia Mathematica Bertrand Russell proved through logic that 1 + 1 = 2 (3), something mathematicians were struggling with for hundreds of years. Of course, we intuitively know that if I have three beans, then I get two more beans, I have 5 beans; but until Russel, there was no proof of it!

THE NEURAL NETWORK’S “WORKINGS”

I will be making in-game representations of the network using Unity – a well-known digital platform for the creation of video games – thus I will be coding in the C# programming language. I am using my own variation on what is discussed in the section “Coding Neural Network Back-Propagation Using C#” from the Visual Studio Magazine:

“Back-propagation is based on calculus partial derivatives. Each weight and bias value has an associated partial derivative. You can think of a partial derivative as a value that contains information about how much, and in what direction, a weight value must be adjusted to reduce error. The collection of all partial derivatives is called a gradient. However, for simplicity, each partial derivative is commonly just called a gradient.” (4)

I understand more or less what that means, but I don’t really care, as long as it works. I am not trying to reinvent the wheel; I’m modifying that example heavily to suit my needs. For example, none of it is now hard-coded: I have made public variables in a wrapper class so that I can fiddle with it in the Unity inspector, and am saving thousands of small “weight files” – essentially what the tinyNet knows at this given moment, given its training.

THE SIGMOID FUNCTION

At the root of a neural network is a sigmoid function. It is what creates the gradients mentioned above. A sigmoid function refers specifically to a function whose graph follows the logistic function (5). It is defined by the formula in Fig. 1:

Figure 1: The sigmoid function (could not resist an equation).

The best way I can think of it is: a ball rolling down a hill into a valley and up the other side, and back again. It will eventually find a resting place, the “answer”. This also feeds into a programming technique I have used countless times, error correction. Is it too much? Cut it in half. Is it too little, double it, until you have a satisfying result.

START THE BUILD

The number of inputs is arbitrary. Eventually I arrived at 2 inputs, and I found I get great accuracy if each tinyNet has only one output (which of course could go to many other inputs, not just one). There are simply two values that form a question, and a tinyNet will produce one output, the answer.

By taking the code from Visual Studio Magazine (6) and making it “Unity friendly”, I have this rough interface with access to the various parameters I fiddle with to achieve a result (7). Fig. 2 shows the parameters used to train the network:

Figure 2: Initial single node network, encoded in the Unity Inspector.

These parameters are totally intuitive, you do trial and error to then see if the question is answered to your satisfaction. Once it seems like a good set, you leave it. These essentially become constants.

After making the first tinyNet, exposed above, I thought about the interconnection architecture to accommodate many, and chose to implement a scripted buckyball – a dodecahedron – to build and connect the tinyNets. Fig. 3 shows how this looks like:

Figure 3: The buckyball topology.

In this topology there are simply two input nodes and one output node at each intersection; each output becomes an input to the next node, forming the larger buckyball. But it can also regress, such that each node is being informed from yet another buckyball topology, and on, and on.

For a simple example, one could train the tinyNets to “know” math, 2+3=5:

And output = 5 becomes input 1 to another tinyNet, performing another task, perhaps 5 + 3 = 8.

A simple “add” network would be easy to train and test, just to prove that the idea actually works. It is not performing arithmetic; it simply knows the answer is 5. It is not “adding” anything directly, though there is plenty of mathematics happening underneath, within that neural network. I chose a buckyball because they are just cool, but any polyhedron could work. Just so it is as clear as milk, no arithmetic would occur inside the balls, they would simply “know” the answer.

At this point, it is all about the questions I ask. I am converting integers – whole numbers including positive numbers, negative numbers and zero – to floating point decimals because floats are more appropriate to the network architecture, at least from my first glance. After initial training, 0.2 + 0.3 = 0.4999987 as you can see in Fig. 4:

Figure 4: 2 + 3 = 5, more or less.

Probably a floating-point precision rounding error you ask? No: the neural network does not produce the “exact” answer. A double precision floating point (that is a 64-bit format for representing real numbers in computer systems) would just have more 9’s in it, I would expect. Specifically germane to my intended use, the precision shown above is completely unnecessary. Close enough for horseshoes and hand grenades (and government work). Any more accuracy than 0.499 could indeed be undesirable, as unpredictability and organic growth are the core principles I wish to explore.

My neural network uses values between 0.0 and 1.0, or -1.0 to 1.0 to be precise. But we might as well be positive about life, and divide by two. If I want to represent something called “middle”, that would be 0.5. It could not be easier to understand, and it avoids the issue of crossing zero, or indeed somehow resulting in zero, which if used in an equation can wreak havoc. Math forbids I divide by zero, or pass it, or a negative number to a square root function.

I trained it on 0.2, 0.3, 0.5. Thus 0.18 and 0.25 should result in something close to the “truth”. See Fig. 5:

Figure 5: 0.18 and 0.25 results in 0.4999022.

The result is 0.4999022, nearly 0.5. Not as precise as the previous run on “perfect” input data. Not bad at all, though 0.43 would be the exact result.

There is a big difference between 4999987 and 4999022 if one only looks at the divergent decimals, as if they were integers. Subtract them and you get 965 divergent to be exact. That is useful.

I trained the network only upon 0.2 and 0.3 outputs 0.5 and yet it returns a semi-valid prediction for significantly “flawed” input. For example, 1.5 and 2.5 does not equal 4 (in math); it equals 5 because that is the closest pattern match according to the training. I suppose that’s the point, how can it handle flawed input. Which begs the question, why call it flawed?

So now I will add the training for precisely 0.18, 0.25, 0.43 on top of the previous training set. Let’s see what happens (Fig. 6).

Figure 6: Adding the training for precisely 0.18, 0.25, 0.43 on top of the previous training set results in 0.4646704.

Much closer to “truth” – but you see, 0.2, 0.3, 0.5 has an influence on 0.18, 0.25, 0.43. I find this fascinating. Keep in mind that I could train 0.2, 0.3 to return 0.23 – it’s whatever I want it to output, based on input.

Let’s see what happens if I retain 0.18, 0.25, 0.43 and train 0.2, 0.3, 0.23 on top (Fig. 7).

Figure 7: Adding 0.2 and 0.3 trained to equal 0.23 on top of the previous training set results in 0.3312506.

If I want this to be more accurate, I need to train more on 0.2, 0.3, 0.23 – lets see (Fig. 8)…

Figure 8: Adding 0.2 and 0.3 trained to equal 0.23 on top of the previous training set results now in 0.2812851,  converging on the “truth”.

Much closer to 0.23. These are just amazing fun toys to play with. As you see, 2 + 3 no longer equals 5: the resulting value is close to the combination of the individual integers as singular representations, or symbols, rather than a mathematical equation. The same network retrained to do something else, at run-time.

The point now is to solve a game A.I. problem, and my goal is to create character personality that can change over time, without having to build a gigantic finite state machine.

I now have it instancing and running on each node in my buckyball (8). Final code tasks are to connect the output of one node as the input of another. Basically, connect the dots, then I’ll train it to run trivial additions, where each node “knows” the answer to 2+2 etc…, rather than calculate it. Then I’ll test and confirm it can handle a series of addition problems, or “questions.” Then I’ll figure out what to do with it and re-train it accordingly, knowing that it is working well enough based on these simple math tests, I can trust that if I apply a different desire, it will function as long as I can define my desire.

RUBBER STARTS TO MEET THE ROAD

Okay, now is the time to do some “real” network training. When I made the first network I trained it on 0.2, 0.3, 0.5 aka 2 + 3 = 5. Sure enough, I got 0.499999 as a result. So, there happen to be 20 nodes, or vertexes, in a buckyball. And every combination of integers 1 to 5, added together, obtain 20 combinations in each digit order. For example:

Etc… All the way down to 0 + 1 = 1, 1 + 0 = 1.

I have 20 nodes in one buckyball, and each node can start with a “hard training” on each of those 20 combinations! See Fig. 9. What luck, or intuition.

Figure 9: Top: Combinations of integers, hand-written notes on paper by John Klima, 2025. And then, bottom, the same thinking ‘downgraded’ into a more readable format.

If I connect a node that returns 2 as the answer, and a node that returns 3 as an answer, and send those as inputs to a node that “does” 2 + 3 = 5, I should get a perfect answer. I have quantifiable confirmation that the entire structure is valid. Pure magic! I can now create a complete buckyball structure assuming this simple “proof”; one can replace the numbers with symbols, and at the start, have a predictable result. But then the network can “learn” from experience that 2 + 3 = 6.

My whole point with this endeavour is to see if clustered, tiny neural networks can replace massive finite state machines for character behaviour. No more, no less, I am just asking the question. I think they can. I did not find anybody who has been working on this, and that surprises me as I was already looking at this twenty years ago – when machines were slow, by today’s standards, and A.I. was relatively new.

I will start simply, featuring two NPCs battling in Rock Paper Scissors. Each NPC prefers one of those options. So, can one NPC be retrained to predict what the other might select, consider the other’s preference, and modify his own selection to achieve success more often than not? And then, what happens when this trained NPC encounters another with a very different tendency? Through training, it obtains a tendency that can then be exploited by another who trained to defeat it. If I did add a third input node, could the one NPC be trained to learn three different opponents?

The next exciting investigation, now, is to train a node in my neural network for one NPC to detect the tendency of the other in unsupervised learning mode (a type of machine learning that analyses unlabeled data to find hidden patterns and structures without explicit guidance, or labeled data) and win more often than not.

ROCK PAPER SCISSORS

I am intrigued by the idea of developing intelligent NPC behaviours where, instead of using a state machine, a neural network is utilised to train a group of NPCs that all have the same foundational logic, resulting in diverse behaviours. In 2006, for my Master’s thesis, I developed a basic backpropagation neural network – an algorithm for supervised learning. This network was capable of being trained on an input pattern to dictate a character animation. It was unexpectedly effective and could be retrained to obtain a different outcome while the simulation was in progress. Therefore, it appeared clear that a single NPC could improve as an opponent (considering a combat style game), or evolve into either an ally or an enemy, as the game progressed. The same NPC in my game might appear quite different from the same NPC in your game.

I moved forward with training a network to play Scissor, Paper, Rock (apologies for my order, it is how I encoded it). I trained a network to know what it “should have” done, not what it “should do”. For example, if I play Scissors and my opponent played Rock, I should have played Paper. It does not change the fact that I played Scissors. I discovered some very interesting properties of the network.

I configured a net consisting of two input nodes – what I played, what my opponent played, and one output being what I “should” have played. I encoded 0.0 == scissors, 0.1 == paper, 0.2 == rock. A logical place to start if one desires to ascribe meaning to what are essentially integer values in an enumeration. I employed a hidden layer of 64 nodes (hidden layers are the intermediary stages between input and output in a neural network), because that worked well with the simple integer math problem described above (3 + 4 = 7).

It failed miserably! So I went to the cafe and gave it a think. I arrived at two suppositions. Firstly, the numeric difference in my encoding was not broad enough to produce high resolution results. Secondly, for a problem that has only six permutations, maybe I want just six hidden nodes, rather than sixty-four. The approach to configuring a neural network for optimal results is not really a science. It is often simply an intuitive, trial and error process.

After juggling with some numbers here and there, I arrived at 0.25, 0.5, 1.0 to equal Rock Paper Scissors respectively. I came up with this training set of all permutations:

Thus, if I play something around 0.25 and my opponent plays something around 0.5, I should have played something around 0.25, which I did – all good, first line in the set. I played Scissors (0.25), opponent played Paper (0.5), so Scissors is correct (0.25). If I played Scissors (0.25), opponent played Rock (1.0), I should have played Paper (0.5) – second line in the set.

Keep in mind that these are not absolute values. Think of it as being “Scissors Like” vs. “Rock Like”. Meaning that “Paper Like” is what I should have played. That is when I realised I needed threshold values, with cut-offs. To arrive at definitive literal representations, these will be:

However, if say, an animation blending system was passed the actual values, it could directly modulate its “mixing” according to those values. The same would of course apply to audio generation, and indeed any mechanism of representation that is looking for floating point values in this general range. Trust me, there are many of them.

I trained my system for 12000 epochs (in an epoch, we use all of the data exactly once); that is a huge number of epochs but the set is so small it took no time at all. Then I started to ask the questions and got fine results throughout. All answers were well within the correct threshold values (see Fig­­­. 10).

Figure 10: Result output from an animation blending system trained on 12000 epochs.

I love this idea that I am “kinda sorta Rock”, my opponent is “kinda sorta Scissors”, therefore I “kinda sorta Win”.

Assuming my opponent prefers scissors, paper, or rock, and that I have gathered the appropriate responses for this scenario, I can train my tinyNet to indicate that against this opponent I should choose this option. That generates habits in me, leading to situations where, upon facing another rival, they possess an equal chance to learn my habits, which in turn fosters habits of their own. When this adversary confronts my initial opponent, it will modify that opponent’s behaviour so that the next time I compete against that initial opponent, it has adjusted its strategy, and so on.

The point here is not to use this system to play Rock Paper Scissors, but to use it as a clear and understandable indication that the tinyNet functions, and can be trained and re-trained to achieve a variety of results, and “solve” a variety of problems. See Video 2:

Video 2: Demonstration of the tinyNet in its simplest form, by using the principles from Rock Paper Scissors applied to a Ninjas’ combat.

To summarise, I have built a neural network that “knows” the outcome of the game, not from a look-up table, rather it simply spits out the result of a play, just like the previous math example “knows” 2+3=5 without doing math. The interesting thing here is that over time the tinyNet can change the rules of the game such that rock can defeat paper. Changing the rules without changing the code is somewhat mind-blowing, and indeed my goal.

LET’S JOUST!

Let’s now expand the problem to something more complex. I chose the Jousting Matrix from the original Chainmail ruleset (9). See Fig. 11):

Figure 11: My original Chainmail Joust Matrix.

If one has a long look at this, each Player chooses an aim point and a defensive posture, and based on the matrix, a resolution is obtained. However, is this not just a more complex manifestation of Rock Paper Scissors? Yes and no. Read the post “Strategy on the Jousting Matrix” by Stephen Wendell (10).

And there is a SubReddit (meaning a forum dedicated to a specific topic on the website Reddit) discussing “strategy” behind the Joust matrix (11). Believe it or not, there is one. After one joust, in the next joust a Player may be forced to assume a defensive position for that next round, thus the other Player knows how to win that next round because he knows what the other Player must do.

All that having been said, first, can I design and train a neural network to “know” the matrix result (thus it could change the rules)? And second, can I design and train another network to understand the Joust strategy described above? Then, I think, I’m starting to get somewhere (see Video 3).

I hesitate to use combat scenarios, as the aim is to be more general purpose. But the fact is, combat relates to the majority of games in one way or another. It is easy to represent combat with clearly indicative animations, it is easy to comprehend the cause and effect relationship, and it is easily quantifiable. Something more qualitative like dialogue, or personality, is the goal – but it is extremely more challenging to represent.

PERSONALITY

To actually address the notion of “personality” beyond combat, I began to investigate research outside of the computational realm, though some of it has been integrated by other game researchers. At the top of the list is the “5 Factor Model” of personality – here referenced in terms of game research (12), but widely adopted in a variety of fields:

I came across “Maslow’s Hierarchy of Needs”(13), which includes the ‘needs’ listed below:

That is quite an extensive list of traits that must be encoded in a meaningful way such that an individual NPC can demonstrate these traits, and such that the Player/Observer can identify them.

We can also look at the basic set of traits from the original publication of the Dungeons & Dragons rule-set:

How can these lists be integrated and represented on screen? I think to do that one simply starts to assign core values. One can almost think about them as “genes” first to the 5 point factors, then to the rather exhaustive list of basic needs. How much does strength contribute to food? How much does any of them contribute to any other?

That would depend of course on the specifics of an encounter. I need a lot of strength to take down a woolly mammoth, less so for a rat. Though perhaps I need more intelligence for the rat. Then what happens when strength and intelligence are combined against the woolly mammoth? Species extinction, as many investigators believe. Man became so good at killing woolly mammoths in great quantity (drive them over a cliff), that it did not take long before there were no woolly mammoths left.

I think the 5 point factor needs to be added to the classic Dungeons & Dragons (D&D) traits to arrive at a baseline for situational evaluation of the larger list of human “needs”. Again, we are talking about some form of a look-up table, but if any look-up table can be expressed within a neural network, the results become fungible, individual, and hard to predict.

Obvious in D&D if the NPC has low strength and high intelligence, he/she will cast a spell and not swing a sword. However, if the caster is out of mana, and all they have remaining is their dagger, dagger it is, or surrender, or run away. Can we then add emotions, or mental states, such as below:

And then, creating and manipulating that results into representations in the game:

An agent has goals – survive, become rich, kill the murderers of his parents.
To survive, the strategy is to stay away from danger, and drink/ eat regularly.
To drink regularly, the agent’s tactic may be to stay close to a river.

I have the combat working all nicely; when 2 characters enter a volume (box collider), they slug it out. I call it an Encounter. But I wanted to make it OOP (Object Oriented Programming) such that the encounter can marshal a fight, a conversation, kissing, whatever. The volume has a trigger that adds whoever enters the volume to the Encounter, based on an abstract class called “Participant”. The trigger collider “other” has a concrete class called PlayerParticipant. Though Encounter has a list of base “Participants” it is happy to add “PlayerParticipant” to its list. It could be “PuppyDogParticipant” – it doesn’t care.

Once enough participants are part of the encounter, off it goes and resolves it. So, puppy does what it does, and Player does what it does, and the two know everything about each other. So, a Player and a Puppy will resolve what they have in common, and a Puppy and a Puppy will resolve what they have in common, which will certainly be different than Player vs. Puppy.

FEUDAL JAPAN

It was finally time to apply some of what was discussed above, in a more social context. I am no modeler nor animator, and thanks to Synty (14) for creating asset packs for a few bucks, and Mixamo (15) for providing a range of compatible animations for free. I am fond of the Samurai assets from Synty and arrived at this look-up table as the basis of initial reactions based on character type (see Fig. 12). When a “Samurai Grunt” (Sam Grnt) encounters a Geisha, he will flirt. When the Geisha encounters a Grunt, she will ignore him, at least as first encounters go. As with everything thus far discussed, this relationship will change over time.

Figure 12: The reaction look-up table used as the basis of initial reactions, programmed in terms of my tinyNet.

LOOMING DEADLINES

With this journal’s submission deadline around the corner, I needed to wrap up at least one loose end, with a demonstration of how a character can learn. There is still much, much to do, and this research will continue far into the future. For the demonstration, I have used a 20-node network that is trained on two parameters to choose an animation, and started playing with numbers (Video 4). Then I simply made the rule that a kick beats a punch, and that the puncher should kick instead (Video 5). It is a bit more generalised in that if I lose, do what the other guy is doing.

CONCLUSION

I have shown how the tinyNet can replace explicit arithmetic in a narrow mapping task. Given the 2+3=5 “pattern” it knows the answer is 5, without counting any beans. I have also shown that a variety of “combat matrices” can be replaced with a neural network, such that the rules of the game can change, in response to tendencies in a specific opponent. These fuzzy neural networks can then drive representation systems – animation, audio, what have you. I also created a simple fitness algorithm that retrains a character at run-time to essentially mimic what another character is doing. There is still a great deal of research I will continue to “code up” as I feel I am barely half-way to my goal. I will need to build an interface to facilitate a “designer” to easily create the behaviours that fit within the game design. Lastly, I will need a game design that best demonstrates the potentials of the system.

REFERENCES AND NOTES

1. Bryan Cockfield, “A.I. Is Only Coming For Fun Jobs”, Hackaday.com, 5 July 2025, <https://hackaday.com/2025/07/05/ai-is-only-coming-for-fun-jobs>, accessed December 2025.
BACK

2. See, for example, Samuel R Cox and Wei T Ooi, “Conversational Interactions with NPCs in LLM-Driven Gaming: Guidelines from a Content Analysis of Player Feedback”, in Chatbot Research and Design: Conversations 2023 (Cham: Springer, 2024) pp. 167–184.
BACK

3. Alfred N Whitehead and Bertrand Russell, Principia Mathematica, 3 vols. (Cambridge: Cambridge University Press, 1910–1913).
BACK

4. James McCaffrey, “Coding Neural Network Back-Propagation Using C#”, Visual Studio Magazine, 14 April 2015, <https://visualstudiomagazine.com/articles/2015/04/01/back-propagation-using-c.aspx>, accessed December 2025.
BACK

5. A graph that follows the logistic function has an S-shape, called a sigmoid curve. This curve represents a growth pattern that initially increases rapidly, then slows down as it approaches a maximum limit.
BACK

6. See McCaffrey (2015) [4.]
BACK

7. If the reader wants to know more about the various parameters I fiddle with using Unity, refer to McCaffrey (2015) [4.]
BACK

8. An instance is a concrete implementation of an abstract concept. We have the concept of “cat,” and then there exists this actual cat.
BACK

9. Gary Gygax and Jeff Perren, Chainmail: Rules for Medieval Miniatures, 3rd ed. (Lake Geneva, WI: TSR, 1975). Published under the banner of Tactical Studies Rules (TSR), this book served as a foundational rule set for the development of Dungeons & Dragons. 
BACK

10. Stephen Wendell, “Strategy on the Jousting Matrix”, 28 July 2021, <https://www.donjonlands.com/2021/07/strategy-on-the-jousting-matrix> accessed December 2025.
BACK

11. Retrieved from a community platform: “Chainmail Jousting Rules in D&D?”, 2023, <https://www.reddit.com/r/osr/comments/16sa65l/chainmail_jousting_rules_in_dd>, accessed December 2025.
BACK

12. Found in this blog: Saul McLeod, “Maslow’s Hierarchy Of Needs”, Simply Psychology, 3 August 2025, <https://www.simplypsychology.org/maslow.html>, accessed December 2025.
BACK

13. Found in this blog: Annabelle Lim, “Big Five Personality Traits: The 5-Factor Model Of Personality”, Simply Psychology, 20 March 2025, <https://www.simplypsychology.org/big-five-personality.html>, accessed December 2025.
BACK

14. See <https://syntystore.com>, accessed December 2025.
BACK

15. See <https://www.mixamo.com>, accessed December 2025.
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

John Klima > Noroff University

Artist, musician and engineer working with custom electronics, software and electro-mechanical systems that interconnect virtual and physical worlds. His practice spans international exhibitions and collaborations within the game industry, alongside teaching roles in digital media and game design that bridge technical development with creative experimentation.

Reviewers

Rui Penha > CESEM & ESMAE 

Composer, researcher and educator working across electroacoustic music, interactive installations and sound spatialisation. As Associate Professor and Vice-President for Artistic Research at ESMAE, he develops projects that integrate composition, technological experimentation and performance, contributing to contemporary discourse on digital creativity and embodied musical practice.

Rui Antunes > CICANT / Lusófona University 

Artist and researcher whose work spans digital arts, live audiovisual performance and human–machine interaction. He creates installations and performances exploring embodiment, experiential design and computational media, integrating sound, image and interface technologies to examine how artistic practice responds to shifting modes of technological engagement and perception.

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 3:

The drone as a sonic phenomenon has a deep and rich history in various music traditions, religious practices, sound art and performance art. Detailing how its functions and meanings vary across cultures is beyond my purpose here, but a brief historical overview is useful to evidence how these practices share a focus on tonal grounding and spatial resonance (1).

For example, in Scottish music the Highland bagpipe drone works as an acoustic frame that stabilises melodic movement and defines the modal scale. In the music of the Balkans, the ison drone defines the modal colour of the melody and provides a strong base for musical expression. In Indian Raga performance, the tanpura provides a continuous reference tone that articulates time and pitch relationships, anchoring melodic improvisation within an evolving temporal field. In Australian Aboriginal traditions, the cyclical breath patterns from the didjeridoo drone are linked to rhythmic continuity and landscape; they are central to ceremonial, social and ecological contexts. And from the 1950s onward, composers such as La Monte Young, Terry Riley, Pauline Oliveros, Steve Reich, Philip Glass, Hildegard Westerkamp and Phill Niblock (to name a few) reimagined the drone as a sustained field of harmonic stasis and psychoacoustic intensity; a medium for perceptual exploration and intentional listening practices. 

The wealth of existing examples reveals a cross-cultural continuum in which the drone functions not merely as background, but as a structural space.

This idea is fundamental to International Drone Day, which is essentially a celebration of all things that can be construed and understood as drone – including sonic, visual and movement-based practices. Drone Day occurs once a year around the globe, at the end of May, inviting participants from all walks of life to spend time with long tones, with the ephemera, with the inexorable pull of the ambient soundscape around us. It invites curiosity and serious play, offering a resistance to commercialisation, a recognition of the indeterminate and an acknowledgement of the mysterious.

Marie LeBlanc Flanagan (2), who conceived International Drone Day in 2014, announces the date of this special 24-hour period on droneday.org, every year. Beyond showing who has participated in the past, this website includes audio and video archives as well as interactive drone games. One can see how widespread the celebration is, and that it keeps growing. Since 2022, we host our Drone Day celebration at our fixer-upper home in the Hudson Valley, welcoming around ninety musicians and sound artists, and over fifty invited guests from across New York, New Jersey, Connecticut, Pennsylvania, Massachusetts and Canada.

Why is Drone Day important? I will explore this question through multiple angles, starting from a conversation with its progenitor and proceeding with an examination of some of its cultural implications, healing aspects and related sociological phenomena. This overview will further ground a report on the genesis, the evolution and growth of our celebration of International Drone Day in the Hudson Valley – including audio recordings from conversations with several sound artists who have participated regularly, and who have been pivotal forces in drone music as a whole. The photos in Figures 1 through 3 reveal two of our three stage-areas, outdoors and indoors.

Figure 1: Michael Jay’s gong set up and Katie Down’s steel cello on the Drone Stage. Marlboro, NY. May 27, 2023. Photo by Katie Down.

Figure 2: From left – Adrian Di Mateo, David Budd, Peter Blum on the Drone Stage. Marlboro, NY. May 27, 2023. Photo by Katie Down.

Figure 3: From left – Irman Peck, Al Margolis and Craig Chin. Marlboro, NY. May 25, 2024. Photo by Tamalyn Miller.

THE ORIGINS: A TALK WITH MARIE LeBLANC FLANAGAN

To present the origins of Drone Day in a lively manner I transcribed a phone conversation with its progenitor (Fig. 4), whom I met in Montreal two years ago. Flanagan is an artist and programmer who thinks and works within the idea of community, building experimental games and installations. She equally served as the editor-in-chief of Weird Canada, an online platform for Canadian underground music and a launching point for Drone Day – which emerged as a rebellious response to the commodification of music.

Figure 4: Marie LeBlanc Flanagan at Society for Arts and Technology. Montreal, Canada. September 29, 2024. Photo by Maryse Boyce.

Below is an excerpt from our conversation, recorded in 2025.

Marie LeBlanc Flanagan (MF): I grew up next to a creek, and used to crawl into the corrugated metal pipe wearing big rubber boots. I’d go in there and make long, extended sounds — singing in a made-up language, drawn-out tones to the frogs and to my cats waiting outside. I grew up going to music festivals. And Drone Day started about eleven years ago, in 2014, when Record Store Day was coming, and a big Beatles box set was about to be released. I was frustrated, because everyone I knew releasing a record had theirs delayed because of it. I wanted to make a day for my community, for the people around me.

Katie Down (KD): Can you  tell us a little about Weird Canada, where Drone Day started?

MF: Weird Canada began as a tiny music blog. It grew quickly — a few campus radio folks, music lovers. At some point, we were nominated for Best Music Website in Canada. We were not big, but we asked people to vote and were shocked by the response. I think people were voting not just for the blog, but out of frustration with how the internet had promised connection and flat hierarchies — and wasn’t delivering. We incorporated as a nonprofit. I hosted weekly tea parties across the country, where we talked about what we struggled with, what we had to offer, and what we wanted to build. We had six hundred volunteers at one point. We wrote about music, toured, and worked to make music spaces safer — fighting back against transphobia, racism, sexism. It was an incredible, beautiful time.

KD:  And how long did Weird Canada exist? Or does it still?

MF:  When I got tired, I asked: “does anyone want this full-time unpaid job?” No one did. So, Weird Canada is sleeping for maybe six years now… But it is still there. It exists in a dreaming state — a sleeper.

KD:  I would love to hear more about what drone music means for you, specifically. What is it about the drone that resonates? You mentioned non-hierarchical structures — that is something I connect to as well. 

MF: I think it is an easy genre to enter. The door is open. Drone feels like waves — layers of sound and motion. It is a state — being here or not here. Yes, people sell drone records, but it resists commodification.  It is not designed to generate a product.

KD: I would love to hear about how some Drone Day celebrations you’ve experienced, whether in Canada or elsewhere, have shaped or shifted your sense of what drone can be. Even something small or intimate, like a person droning in their closet, can be powerful. What moments have stayed with you?

MF: Some stories really stayed with me. Like the person who created a snowmobile orchestra, or someone in a cage surrounded by insects. Others drone alone in a field every year. One person used to drum with a ball gag in his mouth on the streets of Toronto. There are people curled up on couches droning, and others hosting 24-hour drop-ins where people come and go. I also heard about a sound bath in a pool — that one made me laugh. I am moved by the gatherings where a community clearly loves each other. As well as by just one person droning alone, but connected to everyone else. Drone Day is about drone, yes. And more deeply, it is about finding each other through sound. 

KD: There is something magical about knowing that, on that day, hundreds or even thousands of people are droning. It is a global phenomenon that most media do not cover, but that is part of what makes it so special. It is not about visibility or commerce, it’s open and not transactional. And now, proceeding with our interview, what does listening mean to you?

MF: I remember being at a festival at Grey Area in San Francisco, lying down and feeling the drone through human bones. For me, listening is vibration. Conceptually and physically.

KD: Listening is such a vast concept…

MF: I went to a sound bath once that was all about ‘bad sounds’.  But when you frame it like “I want to hear the most annoying sounds” they stop being annoying. So much depends on the listener, their engagement defines the sound. And magic exists in all spaces. The dusk of existence, the sunrise, the sunset, the edge… Oceans blending. Places where one thing becomes another. I would love to hear about what inspired you to create that kind of space.

My conversation with Flanagan was much longer, but it seems appropriate to interrupt its transcript here: prior to exposing how Drone Day came to exist in the Hudson Valley, in the following sections I will overview several ideas and practices that ground its existence.

WHY DRONE AT ALL?

David Reck wrote: “In India, it is said that the universe hangs on sound. Not ordinary sound, but a cosmic vibration so massive and subtle and all-encompassing that everything seen and unseen (including man) is filled with it” (3). The sonic drone is ubiquitous. Whether in urban or rural settings, crickets, wind and streams create their own continuous drones. Our nervous system, heartbeat, breath and blood flow are all forms of a drone. In consonance with this idea, I adopt a notion of ‘drone’ that expands well beyond any musical genre (see, for example, Video 1).

Video 1: Fanny Peréz Gutiérrez, dance, Michael Jay on gong, Adam Taylor on flutes and pedals, Greg Squared on guitar in the living room, Marlboro, NY. 2025. Footage by Katie Down.

THE DRONE AS CONNECTING FORCE

Examining the idea of a cosmic, all-encompassing vibration as a key concept of the musical drone – understanding its vastness and potential – could be a lifelong pursuit. Beings and objects, both animate and inanimate, are always in motion on a molecular level via translation, rotation and vibration. The vibration of an object causes the surrounding air to vibrate. The eardrum vibrates as a result of the moving air, the brain interprets this vibration as sound, and so on.

In the 1920s the philosopher Martin Buber referred to how a ‘stream of sound’ runs through both humanity and the world – a continuous, underlying presence of the divine. He pointed out succinctly: “We may listen to our inner self – and still not know which ocean we hear roaring” (4). Sufi scholar Hazrat Inayat Khan equally spoke about the connecting and ubiquitous force of sound as the basis of all life:

“The life absolute from which has sprung all that is felt, seen and perceived, and into which all again merges in time, is a silent, motionless and eternal life which among the Sufis is called zat. Every motion that springs from the silent life is a vibration and creator of vibrations. The whole of manifestation has its origin in vibration, in sound. This sound, which is called nada in the Vedanta, was the first manifestation of the universe. Consequently, the human body was made of tone and rhythm.” (5)  

Several decades later, Joachim-Ernst Berendt reactivated the Sanskrit word for ‘sound’ –  nada – in The World Is Sound: Nada Brahma, Music and the Landscape of Consciousness. Nada involves “sounding, droning, roaring, howling, screaming” (6), and related to it is the notion of nadi – a term that means ‘stream of consciousness’.

We can tune ourselves to this stream of consciousness. As the late composer Pauline Oliveros has put it: “walk so silently that the bottoms of your feet become ears” (7). Listening is a whole-body experience; one that goes beyond hearing and demands a holistic, intentional awareness that hearing alone does not require.

Oliveros worked with musical drones on her principal instrument, the accordion, as a way to explore the depth and breadth of listening. In her introduction to Gardner’s book from 1990, she described playing long tones on the French horn as a student to improve her embouchure (8). As she began to notice the physical and psychological effects of the vibrations she developed the practice and philosophy of deep listening.

Deep listening engages all the senses, expanding awareness inward and outward. In Sonic Meditations, Oliveros invites readers — regardless of musical training — to explore sound experientially. She notes: “drones of all kinds, such as motors, fluorescent lighting and freeway noise are ever present. The mantra of the electronic age is hum rather than Om. These constant soundings influence everyone whether consciously or unconsciously” (9). She posits that deep listening is “learning to expand the perception of sounds to include the whole space/time continuum of sound”; it involves “heightening and expanding consciousness in as many directions of awareness and attentional dynamics as possible”.

THE HEALING ASPECTS OF DRONE AND DEEP LISTENING

The musical drone and the act of deep listening became essential to my clinical sound practice through my personal meditation, work in sound design and composition, and study with Pauline Oliveros. I continue to explore its wide-ranging impact across improvisation, community-building and sociocultural events.

The drone resonates with our natural homeostatic state – the body’s ability to maintain a stable, balanced internal environment. It functions as both a personal and interpersonal connecting force. One can clearly sense this in Videos 2 and 3.

Video 2: From left – Marco Dianes, Barbara Gogan, Terry Dame in the barn space. Marlboro, May 24, 2025. Footage by William Loeb.

Video 3: Matt Samolis and Jon Clancy, with bowed metal. Cottage stage. Marlboro. May 24, 2025. Video footage by Bill Loeb.

Stephen Nachmanovitch observes that sound can affect our biorhythms through entrainment (10); the term, which designates a process where one oscillating system influences another to synchronise their rhythms, was adapted to music studies by the music therapist Mark Rider in the late 1980s. We can easily entrain with music and with one another. For example, when we hear music with a beat we often tap our feet. And in a performance context, when entrainment occurs the separateness of audience and performers can disappear. Also, when groups of people are entrained like in a choir, orchestra, theatrical troupe,  or in a ritual in which sound is an integral part, the individual becomes part of a whole. In that process, the dissolution of fragile ego states can take place.

At the basis of healing music is the drone – a long, uninterrupted tone or chord (11). Indeed, the musical drone is integral to many cultural rituals and healing practices, including trance induction, hypnotherapy, Reiki and sound meditation. Many healing practitioners use drone-like instruments such as singing bowls, gongs, monochords, tuning forks, frame drums or vocal toning to enable a process where painful or damaging life stories begin to transform into sources of healing and growth.

The grounding quality of musical drone and the freedom it offers within its steady container can support deep psychological shifts, whether a person feels pain or not. John Cage once described this effect in his own way: he would fall asleep with the TV on, using its background drone as a kind of lullaby (12).

Importantly, drones are not inherently therapeutic. As with any clinical tool, trial and error plays a role, and it is vital to maintain an open dialogue with people before introducing drone-based interventions. Because responses vary, sound-healing practitioners must be trained to recognise when a drone is ineffective or counterproductive, especially in liminal or trance states. Used thoughtfully, it can help the central nervous system regulate and return to homeostasis.

Techniques such as humming can ease pain, aid focus and support nervous system regulation. Sylvia Nakkach speaks of the unstruck sound, which she considers the primordial sound of the universe: “the sound that is not heard except in the heart and in ecstatic consciousness” (13). Notablly, Abdullah Alshami explains that, with around forty-thousand neurons, the heart communicates neurologically, biochemically, biophysically and energetically. The vagus nerve, 80% afferent, sends signals from the heart to the brain — more signals, in fact, than the brain sends back (14).

The unstruck sound, the roar of silence, the liminal space: these terms may help explain why the drone’s healing properties are so accessible. The drone simply resonates with our natural biorhythms — our breath, blood, nervous system, and the quietude within bone and blood. Once we enter the space of deep listening, we can begin to hear what the body-mind is telling us. For the sound-healing practitioner, this informs what sound is needed in the moment. In fact, when in the flow of listening, one might begin to experience the phenomena of being played by music rather than playing music. As Gary Ansdell puts it:

“In music both people play together and respond to each through the sense of the music itself – its rhythmic, melodic and harmonic ‘necessities’ – how, that is, tempo ebbs and flows; how rhythms ‘want’ to repeat and develop; how melodic phrases ‘want’ to complete themselves, and so on. For much of the time, in a way not intentionally directed by either player, the music takes them up and almost, you could say, plays them.” (15)

ONTOLOGY, SPACE AND OVERTONES

One way to understand the ontological aspects of musical drone and its overtones comes from experiments by the ancient Greek mathematician Pythagoras and later philosophers. Using the monochord — a single string stretched over a board with two fixed points (nodes) — they observed that dividing the string in half produced the octave, and other divisions revealed consonant intervals within the harmonic series (16). These experiments demonstrated the open-ended nature of harmonic resonance. The nodes, or zeros, are points of no vibration in a standing wave, causing the wave to resonate between them.

In the simple version of the monochord, a single string was fixed at both ends, with one or more movable bridges to demonstrate musical ratios and harmonic relationships. Alternatively, the monochord was configured with one open drone string and a second string in unison, made tunable by a movable bridge. With two strings, varying intervallic relationships can be heard depending on the bridge’s position.

Peter Hammel observes:

“Under the terms of the ancient techniques of acoustic self-realisation, the knowledge of these interval subdivisions of a string took on the role of a link between nature and the soul. For these basic laws reveal the connection between notes and numbers: the intervals can be physically experienced, and the ratios correspond to particular feelings.” (17)

The word interval connotes space. The therapeutic potential of drone in clinical work lies in how musical intervals affect the body–mind and in our perception of what happens in that space. While a single drone can have an impact, it is often the introduction of a ‘second string’ — a tone, thought, or feeling — that enables movement, growth, self-actualisation and acceptance. Transformation occurs in the space between sounds, between breaths, between ourselves and the others.

By extension, silence is not mere absence, but a material that shapes form, audibility and dramaturgy. Many well-known artists such as Claude Debussy (18) or Sun Ra (19) have elaborated on this idea – which is equally evident in Video 4.

Video 4: From left: Kenji Garland, Tony Cenicola, Matt Motel, Andrea Williams. Living room. Marlboro, May 24, 2025. Footage by Anna Dziechowski.

The continuous vibration of the drone creates a sense of vastness while also holding space, offering containment within expansiveness. Visually, this paradox can be likened to the mandala: a form without clear beginning, middle or end, yet serving as a container for deep meditation and contemplation. Once when travelling in Croatia, I came across a young child selling her artwork on the sidewalk. Each small watercolour painting was carefully placed against the stone wall and held down by tiny rocks to keep them from blowing away. One painting that caught my eye had a series of small and large circles forming a kind of spiral. I was intrigued so I bought it from her and then noticed that there was a title on the back written in Croatian. Later that day, at my friend’s house, I asked her to translate it. It said: space and the place where space is. 

Martin Buber’s I-Thou philosophy, where the flux of life is a stream of sound, addresses a fundamental meeting point between entities and events. In the words of Gary Ansdell,  “that meeting is not in space and time, but space and time in meeting” (20). It pertains to an ontological idea: we are lost as separate entities, until the ‘I’ and the ‘Thou’ eventually become ‘We’. As such, contact and meeting are crucial in a therapeutic context – a space of safety and intimacy, as Donald Winnicott underscores (21). Drone not only supports this space but can also become the space itself, where meeting and meaning unfold.

The term musicking is very significant here: it was coined by musician and educator Christopher Small to describe music not only as a noun, but as a verb (22). Whether we play music with others, alone, or for an audience, we are engaging in the act of musicking, a space where time and self can dissolve. As Small wrote, “the act of musicking establishes, in the place where it is happening, a set of relationships, and it is in those relationships that the meaning of the act lies”. Ontologically, the drone becomes a way of reflecting, relating and entering dialogue. It is both connector and mirror — a source of energy exchange.

EXAMINING THE NOTION OF ALCHEMY

As an archetypal form, the drone can be related to the notion of alchemy – whose basic concept is that all things, animate and inanimate, are part of an evolutionary process, striving to reach their highest potential or most perfect form. The alchemists called this the Great Work or Magnum Opus, meaning ‘uniting one with all’ (23). As Dr. Suzanne Gieser noted in 1996, alchemy is also a psychological process, where the ego dissolves and is reintegrated — our original state being one of chaos and conflict (24). The Great Work, she writes, lies in “bringing these forces together in unified harmony”. In this context, the musical drone acts as both a disintegrating and unifying force, offering sonic space for ego dissolution and reintegration.

Alchemy has been directly referenced in musical compositions and metaphorically applied to musical theory. This is evident in the work of composer Arnold Schoenberg, often labelled atonal — though Schoenberg himself preferred the term atonical, which means ‘a merging together of all keys’ (25).

In The Theory of Harmony he described his Kammersymphonie No. 1, Op. 9. (1906) as “the perfect amalgamation of melody with harmony” where both elements “participate equally in melting down more outlying tonal relationships to form a unity and draw logical conclusions from the problems with which they have landed themselves” (26). The goal of achieving unity could be understood as a return to the fundamental, in which all partials coexist — a universe contained within a single piece of music: 

“What is worth striving for is to discover everything that lies within the natural tone, to attain thereby everything of which the human brain with its powers of association and its ability to systematise is capable. What is attainable with the phenomenon outside ourselves, as far as the tone itself is concerned, theoretically speaking, has no boundaries.” (27).

Because the drone is a constant, anything can be played or sung within or against it, creating sensations of tension and release (such as a minor second resolving to the tonic, or a tritone to the perfect fifth). Our notions of consonance and dissonance continue to evolve, as do our understandings of the fundamental, centre or tonic. That can be sensed in Videos 5 and 6.

Video 5: From left – Dan Lindy, Matthew Talmage and Tamalyn Miller in the Cottage. May 24, 2025. Video by Reneé Bergan.

Video 6: From left – Ben Holmes, Matt Darriau, Hearn Gadbois, Brian Ales and Max Kutner. May 25, 2024. Footage by Katie Down. 

Expanding these definitions is essential in clinical applications of musical drone, where engaging with dissonance — not avoiding it — can deepen awareness and reveal what is needed for healing. There are many possible approaches. For instance, Eileen D McKusick describes how binaural beats – where two slightly different frequencies with intervals only a few cents apart lead the brain to perceive a third, phantom tone – can facilitate an alpha-theta brainwave state (7–13 Hz), promoting calm and relaxation (28). As another example, continuous vocal toning generates bodily vibrations that foster focus and presence; Sylvia Nakkach describes it in depth (29).

The ways in which Pauline Oliveros’ deep listening exercises foster the discernment of consonance within dissonance are especially important to me. As part of an assignment for her students at the University of San Diego, she proposes:

“Listen to the environment for fifteen minutes or a longer, but predetermined time length.
Use a timer, clock, or adequate method to define this time length.
Describe in detail the sounds you hear (heard) and how you feel (felt) about them.
Include internal as well as external sounds.
You are part of the environment.
Explore the limits of audibility: (highest, lowest, loudest, softest, simplest, most complex, nearest, most distant, longest, shortest sound).” (30)

To exemplify the process, she further writes:

“My mind adheres to the sounds of myself and my environment. In the distance, a bulldozer is eating away a hillside while its motor is a cascade of harmonics defining the space between it and the Rock & Roll radio playing in the next room. Sounds of birds, insects, children’s voices and the rustling of trees fleck this space. As I penetrate the deep drone of the bulldozer with my ear, the mind opens and reveals the high-pitched whine of my nervous system. It reaches out and joins the flight of an airplane drone, floats down the curve of Doppler effect. Now, fifteen minutes since the beginning of this writing, the bulldozer has stopped for a while. The freeway one-half mile away, unmasked, sends its ever-shifting drone to join with the train whistle.” (31)

Such perceptual dynamics are foundational to acoustic ecology, which involves the study of relationships between living organisms and their environments, specifically as it pertains to sound. The term  acoustic ecology was coined by R. Murray Schafer, who wrote back in 1977: “The flat lines become curved lines. But they are still without sudden surprises. When flat lines become jerky or dotted or looping lines – the machinery is falling apart” (32). Recently, Hildegard Westerkamp pointed out that “the current wars, the enormity of so many social inequalities and injustices, and most significantly the climate crisis, are signals of volatile times and an uncertain future”, stressing that “listening deeply to all that may stabilise us [and] give us real agency, focus and determination to make changes”(33).

The notion of sharawadji is important here. It was introduced by Jean-François Augoyard and Henri Torgue in 2005, and Schafer discusses it in the foreword of their book:

“The sharawadji effect arises in a situation of rupture, where perceptive confusion gives way to an inexplicable aesthetic pleasure. This element of rupture and the dynamic tension involved in the pleasure makes sharawadji closer to the sublime than the beautiful” (34).

Claude Schryer stresses how this effect is essentially a state of awareness (35). When we are free to listen to whatever we turn our attention to, we can take an active part in the creation of meaning (36). As such, the sharawadji effect helps ground Drone Day – a gathering point for those drawn to the dissolution of structure and the expansiveness of time.

OUR DRONE DAY

I have written a brief account of my personal perspective and relationship to drone music, and will now share an account of our own celebration of Drone Day.

In 2016, my husband Matt Darriau and I bought an old farmhouse on ten acres in the Hudson Valley, about two hours north of New York City. The land included a house, a collapsing barn and a small cottage — all in rough shape — but we saw their potential for art-making, performances, community gatherings and healing arts.

Eight years and countless hours of labour later, that is exactly what the place has become. When our friend and collaborator Craig Chin (37) suggested we host a Drone Day celebration, we had to choose between renovating the kitchen or building an outdoor stage. The choice was easy. Built from mostly salvaged wood donated by our neighbour, and with help from friends and my husband, the stage now hosts much more than Drone Day. While we sometimes call it the yoga deck, it is known best as the Drone Stage.

We also gutted the small, run-down cottage on the land and have been slowly turning it into an ad-hoc venue. It is remarkable how lighting and soft furnishings can transform a rough workspace into a warm, inviting listening space.

Our first Drone Day was in 2022, still in the shadow of COVID, when indoor gatherings carried a sense of risk. We had planned to host the event outdoors on our newly built stage, but rain was forecasted. I asked the performers if they would be comfortable indoors, and most replied “the drone must go on!”. So, musicians and listeners filled the living room, the porch, even the attic (Fig. 5).

Figure 5: From left – Matt Darriau, David Mason, Craig Chin and invited guests in the living room during the first Drone Day celebration at our house. Marlboro, 2022. Photo by Tom Law.

Later, the rain cleared and the music moved outside into the golden evening light. Bansuri master Steve Gorn, overtone singer Timothy Hill, and Peter Blum on Indian tanpura shared a twilight drone of Indian ragas, maple leaves shimmering in the fading sun (Fig. 6).

Figure 6: From left – Steve Gorn, Peter Blum, Timothy Hill on the Drone Stage, year one. Marlboro, 2022. Photo by Katie Down.

The results of our first Drone Day celebration exceeded our expectations by far. The willingness, desire and joy that filled the rooms and outdoor spaces were palpable. There were spontaneous collaborations, deep conversations in corners, ridiculous moments of hilarity and unexpected additions. It was clear: people needed this. To gather, to play, to listen. Because of my long history with drone music, not only as a sound artist, sound designer and composer but also as a music therapist, I was excited about the idea of a festival celebrating musical drone as a defiant act against enforced separation – about a movement back into the power of human connection.

Since that first year, when we welcomed around forty people, Drone Day has grown significantly (see the photos in Fig. 7—12, which are from subsequent years), with hundreds of people attending throughout the day, from noon to midnight. And the effects of Drone Day continue throughout the year. People talk about it, asking when the next date is, who is coming, and can we make it even longer? We have created a monthly Drone jam in the barn, and I’m curating a monthly Drone Night at a local venue.

Figure 7: The singer and vocal alchemist Onome (in red) leading a circle song outside. Marlboro, 2023. Photo by Katie Down.

Figure 8: From left – Michel Marzuli, Jessica Bonds, Tom Law in the cottage. Marlboro, 2024. Photo by Katie Down.

Figure 9: From left – Sean Condron, Andrew Neumann, Rima Fand, Quince Marcum in the cottage. Marlboro, May 24, 2025, Photo by Katie Down.

Figure 10: Adam O’Callahan on theremin in the Barn. May 25, 2024. Photo by Katie Down.

Figure 11: From left – David Freeman, Francesca Genco and Jimmy Garver  in the Barn. May 25, 2024. Photo by Katie Down.

Figure 12: From left – Elana Bell, Eileen Moran, Siewli Stark, Sama Shakti and Stephanie Rooker. May 25, 2024. Photo by Katie Down.

BEHIND THE SCENES

Craig Chin, Tamalyn Miller, my husband Matt Darriau and I form the core production team of our Drone Day (Fig. 13). Planning begins months in advance, often with winter tea by the wood stove as we imagine the warmth and brightness Drone Day brings. We draft a list of musicians and sound artists we think might enjoy playing. Now in our fourth year, that list continues growing.

Figure 13: From left – “Team Drone” aka Katie Down, Tamalyn Miller, Craig Chin, Matt Darriau. Marlboro, 2024. Photo selfie by Katie Down.

Two months before the event, we send out a sign-up sheet – each set is roughly twenty minutes in length, with ten minutes allotted on either side for setup and breakdown of gear. Once the solo and fixed ensemble spots are filled, we weave in improvised musical playdates between artists who have not met or played together before.

The events take place in an outdoor stage, in the back of our barn, and in a small cottage. Performances rotate between these three zones, ensuring continuous sound throughout the day. Guests help however they can: assisting with transitions, setup, food preparation, and cleanup. And there are always listeners, whether gathered at the stages, around the potluck table, or relaxing in hammocks. Families come too, and children often listen with real interest. We have seen ten-year-olds captivated by modular synthesisers, violoncellos, saws and gongs. Some even want to perform, hence we are now considering a fourth stage dedicated to child drone artists. Why not?

THE ARTISTS WHO CORE-PRODUCE OUR DRONE DAY

Our core-production team is continuously experimenting with new ideas; I will briefly introduce who the other sound artists are. To better convey the wealth of their contributions and idiosyncrasies, I will also share audio recordings of my conversations with them.

Craig Chin (Fig. 14) is developing a project called Errant Space, which is equal parts podcast, performance and philosophical inquiry, with a focus on the ambient, abstract and improvisatory nature of drone music. Both his music and his podcast invite contemplation — perhaps not by design, but inevitably by effect. A few years ago, he began hosting his own Drone Day in his backyard in Beacon, NY. It was a casual affair, with about twenty people gathered around the fire pit and one or two musicians at a time playing drone music on the porch, or sometimes just a loop running while everyone ate and drank together in the yard. At his suggestion, in 2022 we began co-hosting the event at our place, which can accommodate more people and multiple drone areas. What began as a simple social gathering around drone music has since grown into a kind of social experiment that we continue to shape as co-curators.

Audio 1 is our conversation about drone, Errant Space, and the evolving celebration we now share. Craig reflects on why he believes Drone Day is important, what he values about its ongoing development, and the moments that continue to surprise and delight him — not only as a co-producer, but as a musical participant as well.

Figure 14: Craig Chin (guitar and synths) with TV monitors installation by the video artist Rooster in the cottage. Marlboro, 2023. Photo by Katie Down.

Audio 1: Conversation with Craig Chin, 2025.

Tamalyn Miller (Fig. 15) and I have been collaborating musically for several years. She is a multi-media artist living in the Hudson Valley, who works in the realms of music, sound, visual curiosities, words, hypnosis and wonder (38). She invents sonic worlds with objects found in nature, drawing out the history of the object to create a story with words and music. Tamalyn recently completed a sound walk recording and album titled Ghostpipe, which features aspects of plants and trees through original text, sound and song. She is also a member of the bands Goddess and Spirit Radio.

Audio 2 is an excerpt from our conversation that touches on how she views the importance of liminal states that the drone can elicit, hypnosis, trance states, and the significance of altered states of consciousness that the drone can provide. She speaks about her personal relationship to Drone Day and what it means for her as an artist and co-curator.

Figure 15: Tamalyn Miller, New Hampshire, X Fest, 2024. Photo by Craig Chin.

Audio 2: Conversation with Tamalyn Miller, 2025.

Matt Darriau (Fig. 16) is an integral part of our Drone Day celebration in that he built the Drone Stage, helped to rehabilitate the back of the barn and is also a regular performer on the day. He is a multi-winds virtuoso whose music spans Jazz, Balkan, Middle Eastern, traditional Irish and free improvisation (39). He plays alto and tenor saxophones, clarinet, Bulgarian kaval and gaida. One of his primary groups, Paradox Trio, was a pioneering force in combining Jazz and Balkan musics in the 1990s; they inspired musicians around the globe to work cross-culturally. Matt is also a long-time member of The Klezmatics, which was awarded a Grammy in 2006. They are set to release their 40th anniversary album soon.

In Audio 3, Matt talks about his role in making our Drone Day happen each year, what he enjoys about it, what stresses him, and what role the drone plays in his music. He speaks about the ‘homebase’ aspect of drone, and about how much of his own original music or arrangement of traditional tunes has a drone aspect, especially in the case of rhythmic ostinati and groove.

Figure 16: Matt Darriau, NYC. 2019. Photo by Katie Down.

Audio 3: Conversation with Matt Darriau, 2025.

Figure 17: David K. Freeman and Oren Neiman. Barn stage. May 25, 2024. Matt Darriau is behind the photo camera.

ADDITIONAL DRONE ARTISTS

Several composers, performers, and producers have become pivotal forces in our Drone Day celebrations through their ongoing, dedicated participation. I will share my conversations with Ken Butler, Sarah van Buren, Melissa Auf der Maur, Terry Dame, Steve Gorn and Michael Jay, along with two sound collages by Skylar Antoine. In these conversations, each artist offers a distinct perspective on drone music — its impact, its place within the broader musical landscape, and its significance in their personal practice.

Ken Butler is an artist, musician, sculptor, inventor, mad scientist and witty raconteur (40). He creates hybrid instruments from chainsaws to tennis rackets as guitars, violins and percussion instruments (Fig. 18). Ken has been an important fixture in the downtown music scene in New York for many years, and there was a recent retrospective on his work at the Culture Lab gallery in New York, which showcased his vast body of both visual and musical works. 

Audio 4 is our phone conversation after Drone Day. It shares some of Ken’s ideas about his personal relationship to drone in his music and his take on Drone Day itself.

Figure 18: Ken Butler, Ann Arbor, Michigan, 2005. Photo by Tobi Hallander.

Audio 4: Conversation with Ken Butler, 2025.

Sarah van Buren (Fig. 19)  is a sound artist, co-curator of the 24-hour Drone festival in Hudson and a DJ – aka SVB – in the Hudson Valley, NY (41).

In Audio 5 she shares how meditating with drone has both personal and social resonance in the wider world. In this interview, she reflects on her role in co-curating that large-scale event, and on how the deep listening philosophy of Pauline Oliveros continues to shape her musical practice. Sarah discusses the power of drone music and communal events like our Drone Day as forms of collective prayer — offering space for peace, connection, and transformation.

Figure 19: Sarah van Buren, NYC, May, 2025. Photo by Nick Schiarizzi.

Audio 5: Conversation with Sarah van Buren, 2025.

Melissa Auf der Maur (Fig. 20) is a sound artist, instrumentalist and former touring musician who has performed with numerous rock luminaries, including Hole (42). She is also a curator of the 24-Hour Drone festival, a widely attended public event held each April at Basilica Hudson, a striking converted warehouse that hosts a range of experimental and community-focused programming.

In Audio 6, Melissa shares her perspective on the importance and urgency of music gatherings and listening-based events, emphasising the transformative exchange between performer and audience. She speaks about her connection to Lisbon and its experimental art scene, and her collaboration with Bob van Er of Le Guess Who – a co-founder of the 24-Hour Drone who now lives in Lisbon. Our discussion took many directions, so I selected a few of the most salient reflections, particularly those that speak to her identity as an artist, dreamer, and facilitator of spaces where artists can grow, connect and flourish. Her insights into drone, listening, and community resonated deeply with me.

Figure 20: Melissa Auf der Maur, Hudson Basilica, March, 2025. Photo by Eve Alpert.

Audio 6: Conversation with Melissa Auf der Maur, 2025.

Terry Dame (Fig. 21) is an inventor and multi-instrumentalist whose creations — ranging from musical instruments and installations to sonic objets d’art — combine found and devised materials (43). As a composer, sound designer and educator at the School of Visual Arts in New York, she builds sculptural instruments that are both sonically rich and visually striking.

Terry has performed at our Drone Day celebration since its inception. This year (2025), she and my husband Matt constructed an interactive gong tree in the yard, featuring mixing bowls, table saw blades, copper tubes, boxing bells, gongs and more. Terry’s inventiveness, humour and generous spirit are part of what makes Drone Day so special.

Audio 7 features an excerpt from our conversation about her relationship to drone music, including her influences from Balinese gamelan, North and South Indian, Persian and Balkan musical traditions — all of which have inspired her original compositions. She also keeps bees (or, as she says, they keep her) and finds their constant drone to be a grounding and sustaining influence on her listening. In our conversation, she reflects on the value of free play, improvisation, and joy in music; qualities that Drone Day fosters and which, she believes, are more vital than ever today.

Figure 21: Terry Dame, Catskill Art Society. August 5, 2025. Photo by Peggy Shaw.

Audio 7: Conversation with Terry Dame, 2025.

Steve Gorn (Fig. 22) is a Grammy Award winner (2011) and five-time nominee who has performed with luminaries across Indian classical music, jazz, world music and contemporary improvisation (44). A virtuoso bansuri flutist and clarinettist, he is widely recognised as one of the foremost Indian bansuri players in the world, and is based in the Hudson Valley, NY. We first met in 2000, during a Pilgrim Theatre production of Jean-Claude van Itallie’s play based on The Tibetan Book of the Dead, for which I had composed a live sound score. Steve had composed a score in 1983 at La MaMa in NYC, for the first iteration of the play. This remount brought us together and we have been friends ever since.

In Audio 8, Steve speaks about his decades-long relationship with drone music, particularly in the context of North Indian classical traditions. He shares his reflections on the psychological, physiological and emotional effects of listening to drone, as well as his thoughts on being a vital part of our Drone Day celebration each year.

Figure 22: Steve Gorn, Blacksmith Studio, Rosendale,  NY May 25, 2025. Photo by Holly Shelowitz

Audio 8: Conversation with Steve Gorn, 2025.

Michael Jay (Fig. 23  and 24) is a sound artist, sound healer, gong master and co-founder of One Tone, a sound healing training programme and instrument distributor (45). He has been a close friend and collaborator for many years and plays a central role in our annual Drone Day celebration. Each year, the day begins with MJ on the Drone Stage, playing a wall of gongs to set the “tone” and welcome everyone into the world of drone. He has long worked with drone instruments — particularly gongs — in performance contexts, including collaborations with jazz musicians such as Melvin Gibbs of the group Harriet Tubman, and Tomchess on oud.

During the COVID lockdown, we recorded an album together in our living room, titled Language of the Alchemists, featuring gongs and steel cello. An excerpt from that recording can be heard in Audio 9. Beyond this conversation through music, Audio 10, then, features a conversation through words.

Audio 9: Excerpt from the album Language of the Alchemists, with Michael Jay and Katie Down on gongs and steel cello.

Figure 23: Michael Jay and Katie Down and their collaboration titled Steel this Gong on the Drone Stage. Marlboro, 2024. Photo by Renee Bergan.

Figure 24: Michael Jay, Sage Academy, Woodstock. August 2017. Photo by Renzo Zanelli.

Audio 10: Conversation with Michael Jay, 2025.

Skylar Antoine (Fig. 25) is a young conceptual artist, sound artist and performer based in Montréal, Canada. She has attended our Drone Day celebrations for the past three years. She created two sound collages drawn from various performances in 2023 and 2024, capturing something of the sonic environment shaped by the many artists passing through (Audio 11 and 12).

Figure 25: Skylar Antoine sitting under photograph by Bill Loeb, May 24, 2025. Barn in Marlboro. Photo by Katie Down.

Audio 11: Drone Day sound collage #1 by Skylar Antoine, 2023.

Audio 12: Drone Day sound collage #2 by Skylar Antoine, 2024.

DISCUSSION

The musical drone is like a sonic mandala: a centre of focus, meditation, connection and presence. It creates a container for thoughts and feelings to flow and take form. Drone Day calls us into this space — to play, to listen, to lie down, to be immersed, curious and connected, in a shared field of sonic play. Many describe it as their favourite day of the year, in part because it is defined by communion, by musicking together, and by the absence of commercial transaction. Of course, its organisation has mandatory production costs, and we do not depend on institutional funding. There is a donation link, but no monetary exchange is required for participation. The potluck is a gesture of collective care, with lovingly prepared food and drink that reflects the same tenderness and thoughtfulness people bring to their sonic offerings. In the months leading up to Drone Day, I notice the excitement — past performers reaching out early to reserve the date, to be sure they can return. 

This kind of devotion feels particularly poignant in our increasingly transactional, polarised world. Drone Day resists that polarisation. It celebrates music and sound that gently awakens a sense of oneness, through subtlety, consistency, and the grounding pull of the drone.

I have attempted to account for all the artists who have performed at our Drone Day – their names are listed in alphabetical order, below. I hope to have included everyone, and apologise if somehow I have missed a few. The list keeps growing –  and we hope to keep expanding…

REFERENCES AND NOTES

1. See, for example: Harry Sword, Monolithic Undertow: In Search of Sonic Oblivion (Nashville: Third Man Books, 2022). And Robin Purves, “Subject of the drone“, Metal Music Studies 6, No.2 (2020), p. 145 – 159. Also Roderick D. Cannon, The Highland Bagpipe and Its Music (Edinburgh: John Donald/Edinburgh UP, 1988). Or Martin Clayton, Time in Indian Music: Rhythm, Metre, and Form in North Indian Rāg Performance (Oxford University Press, 2000) . And The Ashgate Research Companion to Minimalist and Postminimalist Music, edited by Keith Potter & Kyle Gann (Routledge, 2013).
BACK

2. Official website from Marie LeBlanc Flanagan: <https://marieflanagan.com>, accessed December 2025.
BACK

3. David Reck, Music of the Whole Earth (New York: Scribner, 1977) p. 43.
BACK

4. Martin Buber, I and Thou (New York: Charles Scribner’s Sons, 1958) p. 16. First published in 1923.
BACK

5. Hazrat I Khan, The Music of Life (New Lebanon, N.Y.: Omega Publications, 1988) p. 335.
BACK

6. Joachim-Ernst Berendt, The World Is Sound: Nada Brahma, Music and the Landscape of Consciousness (Rochester, VT: Destiny Books, 1991) p. 16.
BACK

7. Pauline Oliveros, Sonic Meditations (Kingston, NY: PopandMoM Publications, 1971) p. 5.
BACK

8. Kay Gardner, Sounding the Inner Landscape: Music as Medicine (Rockport: Element Books, 1997. First published in 1990).
BACK

9. See Oliveros (1971) p.iv [7.]
BACK

10. Stephen Nachmanovitch, Free Play: Improvisation in Life and Art (New York: Penguin Putnam, 1990) p. 51. For a discussion on how the phenomenon of entrainment applies in music therapy, see also Klaus Starcke, Julia Mayr & Rainer von Georgi, “Emotion Modulation through Music after Sadness Induction: The Iso Principle in a Controlled Experimental Study”, International Journal of Environmental Research and Public Health 18, No. 23 (2021).
BACK

11. Gardner (1997) p. 28 [8.]
BACK

12. Richard Kostelanetz, ed., John Cage: An Anthology (New York: Da Capo Press, 1991) p. 7. First published in 1970.
BACK

13. Silvia Nakkach, Free Your Voice: Awaken to Life Through Singing (Boulder, CO: Sounds True, 2012) p. 211.
BACK

14. Abdullah M Alshami, “Pain: Is It All in the Brain or the Heart?”, Current Pain and Headache Reports 23 (2019).
BACK

15. Gary Ansdell, Music for Life: Aspects of Creative Music Therapy with Adult Clients (London: Jessica Kingsley Publishers, 1995).
BACK

16. Peter M Hamel, Through Music to the Self: How to Appreciate and Experience Music Anew (Boulder, CO: Shambhala Publications, 1979).
BACK

17. Hamel (1979) p. 100 [16.]
BACK

18. Julian Johnson, After Debussy: Music, Language, and the Margins of Philosophy (Oxford University Press, 2020). Particularly relevant here is the chapter “Mélisande and the Silence of Music”, pp. 51 to 89.
BACK

19. John F Szwed, Space Is the Place: The Lives and Times of Sun Ra (Duke University Press, 2020. First published in 1997).
BACK

20. Ansdell (1995) [15.] quoting Martin Buber, I and Thou (New York: Charles Scribner’s Sons, 1958).
BACK

21. Donald W Winnicott, Playing and Reality (New York, NY: Routledge Classics, 1971) p. 55.
BACK

22. Christopher Small, Musicking: The Meanings of Performing and Listening (Middletown, CT: Wesleyan University Press, 1998).
BACK

23. Suzanne Gieser, The Innermost Kernel: Depth Psychology and Quantum Physics. Wolfgang Pauli’s Dialogue with C. G. Jung (Stockholm: Springer-Verlag Berlin Heidelberg, 2005).
BACK

24. Gieser (2005) p. 199 [23.]
BACK

25. Jamie James, The Music of the Spheres: Music, Science, and the Natural Order of the Universe (New York: Grove Press, 1993) p. 216.
BACK

26. Arnold Schoenberg, The Theory of Harmony (Berkeley: University of California Press, 1922).
BACK

27. Schoenberg (1922) p. 318 [26.]
BACK

28. Eileen D McKusick, Tuning the Human Biofield: Healing with Vibrational Sound Therapy (Rochester, VT: Healing Arts Press, 2014) p. 78.
BACK

29. Nakkach (2012) p.46 [13.]
BACK

30. Pauline Oliveros, Software for People: Collected Writings 1963–1980 (Kingston, NY. Pauline Oliveros Publications, 2015) p. 28. First published in 1984.
BACK

31. Oliveros (2015) p.17 [30.]
BACK

32. Murray Schafer, The Soundscape: Our Sonic Environment and the Tuning of the World (Rochester, VT: Destiny Books, 1977) p. 80. Reprinted in 1993 and 1994.
BACK

33. See “Speech by Honorary Doctorate Recipient Hildegard Westerkamp at SFU”, 2024 <https://hildegardwesterkamp.ca/resources/PDFs/writings-pdf/Honorary_DR_presentation-2024.pdf>, accessed December 2025.
BACK

34. Murray Schafer, Foreword to Sonic Experience: A Guide to Everyday Sounds ed. by A McCartney and H Torgue (Montreal & Kingston: McGill‑Queen’s University Press, 2005). First published in 1995.
BACK

35.  Claude Schryer, “Searching for the Sharawadji Effect: Electroacoustics and Ecology”, Musicworks No. 70 (1998).
BACK

36. Sal Randolph echoes this idea eloquently in “Sharawadji Mix”, 1 April 2025, <https://www.salrandolph.com/sharawadji-mix>, accessed December 2025.
BACK

37. Official website from Craig Chin: <https://www.errant.space>, accessed December 2025.
BACK

38. Official website from Tamalyn Miller: <https://www.tamalynmiller.com>, accessed December 2025.
BACK

39. Official website from Matt Darriau: <https://mattdarriaumusic.bandcamp.com>, accessed December 2025.
BACK

40. Official website from Ken Butler: <https://kenbutler.squarespace.com>, accessed December 2025.
BACK

41. Sarah van Buren on Sound Cloud: <https://soundcloud.com/sarah-van-buren-1>, accessed December 2025.
BACK

42. Official website from Melissa Auf der Maur: <https://www.chronogram.com/arts/sound-check-melissa-auf-der-mauers-music-recommendations-20168479>, accessed December 2025.
BACK

43. Official website from Terry Dame: <https://www.terrydame.com>, accessed December 2025.
BACK

44. Official website from Steve Gorn: <https://stevegorn.com>, accessed December 2025.
BACK

45. Official website of One Tone and Michael Jay: <https://www.onetone.one>, accessed December 2025.
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

Katie Down > SoundWell Creative Arts Therapy Center

Sound artist, multi-instrumentalist and licensed creative arts therapist whose work integrates trauma-informed practice, mindfulness and deep Listening. She teaches widely, including at SUNY New Paltz, and develops therapeutic and artistic projects that explore sound’s capacity to support healing, embodied awareness and interdisciplinary creative engagement.

Reviewers

Joel Ryan > STEIM & Institute of Sonology 

Composer, performer and pioneer in computer music associated with STEIM and the Institute of Sonology. Renowned for developing real-time digital instruments and interactive systems, he collaborates extensively with experimental musicians and dancers, advancing innovative approaches to improvisation, live electronics and expressive sonic control

Maile Colbert > Faculty of Fine Arts Lisbon 

Intermedia artist, composer and researcher whose work integrates sound, ecology, cinema and immersive media. Grounded in field recording and environmental engagement, her projects include hybrid performances, installations and compositions exploring perception, place, memory and modes of attentive, embodied listening within contemporary artistic practice.

Raquel Castro > CICANT / Lusófona University  

Researcher and curator working across sound studies, acoustic ecology and documentary practices. Founder of the Lisboa Soa festival, she develops projects examining urban soundscapes, environmental awareness and community-based approaches to sonic culture, integrating artistic creation with research into listening, ecology and public space.

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 3:

Somaesthetics is a concept introduced by philosopher Richard Shusterman, which focuses on the body as both a locus of sensory appreciation and a site for cultivating heightened bodily awareness through aesthetic practice (1). This concept is useful to delineate a situated definition of interfaces in digital art. Drawing on David Rokeby’s assertion that “interaction is about encounter rather than control” (2), we propose an understanding of the interface that transcends mere functional control. This perspective challenges widely accepted notions, such as the standardisation of interaction modes and their functional role as intermediaries. It prompts reflection on the complexities of creating an interface that resists subordination.

The present research emerged from artistic practice through the creation of two interactive installations, which explore the interface as an artwork in its own right: ON/CONTACT and EN/GLOBE. In both works, people are invited to interact within a multimodal installation, which engages multiple sensory modalities while allowing the interface to serve as a platform for self-expression and creativity.

AN UNDERSTANDING OF INTERACTION

Reframing the interface and interaction beyond the concept of control creates opportunities for deeper connections and exchanges, fostering the development of diverse dynamic relationships between participants and the system.

Control is defined as “power or authority to guide or manage”, which translates into action as “to have power over”, and refers to “a function or mechanism used to regulate or guide the operation of a machine, apparatus, or system” (3). This notion primarily emphasises function and subordination, making it ideal for interactions that demand precision and efficiency to produce consistent outcomes aligned with the user’s intentions. However, “artists may specifically not want to exercise precise control over their works, preferring to let them gradually transform through the actions of spectators, without actually seeking to know where this transformation will lead” (4). They may explore alternative forms of interaction, to uncover unexpected and emergent outcomes.

In his text “Transforming Mirrors”, David Rokeby examines the implications of interactive media from the perspective of artists. He argues that such art functions as a mirror, reflecting participants back to themselves while situating their sense of self in relation to their environment. The dynamics of control and encounter are expressed through the interplay between the participant’s actions and the system’s responses, revealing how identification processes shape the relationship between them:

“Interaction is about encounter rather than control. The interactive artist must counter the video-game-induced expectations that the interactor often brings to interaction. Obliqueness and irony within the transformations and the coexistence of many different variables of control within the interactive media provide for a richer, though perhaps less ego-gratifying experience. […] The interactive artist must strike a balance between the interactor’s sense of control, which enforces identification, and the richness of the responsive system’s behaviour, which keep the system from becoming closed.” (5)

The idea of encounter proposed by Rokeby redefines the duality of control and entertainment by transforming the relationship between the user and the interface. This shift places emphasis on creative exploration, which lies at the heart of interactive art. It prioritises the process of engaging with the interface and the role of the interface itself, moving beyond a focus on the outcomes or results of that engagement.

An example of encounter emphasising bodily experience in interactive digital art can be found in Rokeby’s Very Nervous System (1986-1990), an interactive sound installation where video cameras and computers translate body movements into real-time audio (6). The installation employs a diffuse, invisible interface that fosters a dynamic, evolving relationship between the system and its participants. This constant feedback loop enables mutual influence, dissolving traditional notions of control into a deeply immersive encounter. As Rokeby describes it, “The interface becomes a zone of experience, of multi-dimensional encounter”.

Another example is Re:MARK by Golan Levin and Zachary Lieberman (2002). This installation uses voice analysis, motion sensing and real-time animation to transform speech into animated letters and shapes that appear to rise from the speaker’s shadow. Participants can interact with these forms by manipulating them with their own body shadows. Recognised phonemes are displayed as letters, while unrecognised speech generates abstract shapes that reflect the vocal timbre. As the artists describe it, “The result is a playful illusion in which visitors become actors in a shadow world of reactive cartoon language” (7). In this way, the artwork functions as a mirror, reflecting a transformed image of the self.

THE NATURE OF THE INTERFACE IN INTERACTIVE ARTWORKS

Interfaces are often associated with computing, yet the term originates in chemistry (8), and broadly refers to liminal spaces. As Jean-Paul Longavesne explains:

“Before being technological, the interface is fundamentally a space, a margin, a zone for articulating communication and spatial and temporal interrelations between two worldviews — a zone of friction and exchange between two spaces […]. It is a place of coexistence […].” (9)

The power of the interface lies in the relationships it establishes, where the result transcends the sum of its parts. Gérard Chazal elaborates:

“The power of the interface lies in the strength of relationships. When two things are connected, the result may sometimes be a mere juxtaposition, in which case the interface is absent. However, more often, something new is created, with properties that are not simply the sum of the attributes of the connected components.” (10)

This phenomenon highlights the interface’s emergent property, described as a characteristic arising from the connection of elements, belonging to the whole but not to its individual components (11). Laurent Diouf notes:

“In the strictest sense, digital arts are the origin of true ‘inter-action’, prompting a movement from the artwork towards the viewer and, conversely, from the viewer towards the artwork. […] Artwork and viewer mutually interact through various modalities.” (12)

This back-and-forth interaction can be seen in Luc Courchesne’s Portrait One (1990), an interactive artwork that facilitates a “framed encounter”: a digitally mediated interaction between the audience and a computer-generated character (13). The work allows participants to engage in an experience resembling a face-to-face meeting, fostering an exchange that is both dynamic and participatory. The interaction is defined by specific boundaries, much like a portrait frames its subject within set constraints; as Courchesne explains, “a portrait of someone is an account of an encounter between the author and the subject” (14). Its design utilises a conversational framework, composed of fragments from the artist’s initial interaction with the digital character. The interface, as a point of contact, also unveils dialogues between the artist and the public. In summary, this piece delves into the complexity of meeting others and examines how such encounters can deepen our understanding of ourselves through the lens of digital interaction.

The artistic creation and exhibition of interactive works foster the generation of a type of knowledge that actively involves the community. In this process, “the artist is no longer the sole creator of a work but often serves as the mediator or facilitator of interactions between the audience and the artwork” (15). These interactions position the artist as a bridge, enabling meaningful engagement between the audience and the piece. Consequently, interactive artworks serve as conduits for discussions about the nature and role of interfaces, creating an environment that encourages participation, exploration, and reflection on the connections between technology, creativity, and human experience.

BEYOND CONTROL AND SUBORDINATION

In digital art, the interface can serve purposes beyond control or subordination, prompting several key considerations. The first consideration examines the body as an interface, the second explores the role of perception in shaping the experience, and the third addresses the artwork’s position as a mediator between the artist and the audience.

Building on the idea that the interface transcends control and subordination, Rokeby asserts:

“Our interface with reality is not only multi-sensory or multimedia but also “multi-modal.” […] We can interpret, analyse or simply enjoy the raw sensation. It’s only a multi-modal approach with multiple simultaneous levels of meaning and communication that can properly express that complex experience of reality.” (16)

Olivier Zattoni further emphasises the essential role of the body in experiencing and understanding the world (17). He argues that the body is not merely a boundary separating the tangible physical world from the intangible realms of imagination or the virtual. Instead, it serves as a continuity — a place where organic materiality merges with the conceptual and emotional interiority of the self. David Le Breton supports this by asserting that “the body is a semantic filter [and] there is nothing in the mind that has not first passed through the senses” (18). This perspective highlights that knowledge is not solely deduced through logic but also shaped by sensory perception, offering rich creative and conceptual potential.

This potential raises critical questions in the context of a hypermediated, technology-driven world. For instance, the 2006 exhibition Sensorium: Embodied Experience, Technology and Contemporary Art presented artworks that exposed discrepancies between the promises of technology and the intentions of users (19). As Anja Bock observes, “when the sensorium is questioned, the operational definition of the self is disrupted” (20). However, multimodality in digital art has the power to evoke states of consciousness that precede thought and resist articulation through language (21). The interface between the body and technology, even when mediated through physical devices, generates its own metaphors and frames of reference, offering a fertile space for exploration and interpretation.

Notably, the dominance of sight — referred to as the hegemony of vision, where one sensory modality overrides others — characterises Western societies, but that is not irreversible (22). Many technological artworks prioritise other senses, particularly touch and gesture, previously absent in the art-viewer relationship (23).

This shift into multimodality informed the development of ON/CONTACT and EN/GLOBE, the artworks discussed in this article. These works emphasise a multi-sensory engagement that moves beyond traditional, vision-dominated experiences. Their creation and public presentation have highlighted the diverse abilities of the audience. To facilitate a wide range of engagements, the design of our interfaces accounts for varying levels of proficiency and accessibility. The interface must be intuitive enough to be understood with minimal or no instructions, enabling users to engage effectively within a short time frame. At the same time, it should be captivating enough to encourage the audience to spend time exploring its possibilities. The gestures involved in interacting with the interface should also be meaningful, enhancing the overall experience.

The role of the body and gesture in somatic experiences with the interface is exemplified in Metamorphy (2013), by Scenocosme (the collaborative name of French artists Grégory Lasserre and Anaïs met den Ancxt) (24). This interactive installation invites participants to engage with a flexible screen, which responds to touch by generating evolving visual and auditory feedback. It transforms the physical act of touch into a dynamic dialogue between the body and the digital world, fostering a heightened awareness of one’s somatic presence within the interactive environment.

Building on their exploration of somatic interaction via touch, Scenocosme’s Lights Contacts extends the concept of encounter through a more social and collaborative approach (25). While Metamorphy focuses on an individual’s interaction with a responsive surface, Lights Contacts involves multiple participants, fostering connections between bodies. The installation features an interactive sphere that responds to electrostatic contact, generating real-time sounds and colours based on the intensity and proximity of participants’ physical interactions. This shift from individual to collective engagement allows for a broader range of social reactions, from intimate exchanges to playful group activities. By encouraging participants to explore how touch influences auditory and visual feedback, Lights Contacts creates a shared sensory experience that transforms physical contact into extraordinary and diverse encounters.

Adopting an approach that emphasises sensory and interactive multimodality has proven fruitful in terms of experience. First, during the creation stage, it serves as a means of finding creative solutions to both technical and conceptual challenges. Second, from the audience’s perspective, they encounter artworks that diverge from the conventions of current audio-visual practices. This allows each individual to engage with the work in their own way, shaped by their abilities, knowledge, and interests. Through this process, the artwork and its spectators develop their own shared rules of interaction, forming feedback loops. Ultimately, the artist’s role is to create a context that facilitates these feedback loops, enabling a dynamic dialogue between the artwork and its audience.

SOMAESTHETICS: THE ROLE OF THE BODY IN AESTHETIC EXPERIENCES

To assess, understand, and integrate the body in the creation and experience of artworks, establishing a structured theoretical framework can be particularly valuable for both artistic practice and analytical insight. Somaesthetics, as a theoretical and practical field, proves well-suited for research-creation, as its outcomes encompass “an artifact or artistic performance accompanied by discursive production” (26).

Somaesthetics, the philosophy developed by Shusterman, grounds on his research about pragmatism, aesthetics and philosophy as a way of life (27). Succinctly, it is defined as “the critical study and meliorative cultivation of our experience and use of the living body (soma) as a site of sensory appreciation (aisthesis) and creative self-fashioning” (28). As an embodied philosophy, it also investigates discourses and structured knowledge of the soma (29), moving away from a mechanistic view of the body and passive receptivity towards recognising it as a “living and sensing body” (30). It emphasises that sensory perception enhances understanding, extending beyond what logic alone can apprehend (31). The body is understood as an intersecting locus of experience; it is both the embodied consciousness of the individual and our primary instrument for grasping the world (32).

Somaesthetics seeks to unify perception and gesture through the lens of aesthetics (33). It provides a means to account for the interactor’s experience, both in relation to the artwork and the broader world it reflects (34).

This suggests a threefold conception of encounter. First, it encompasses an encounter with oneself, involving the refinement of sensory perceptions to foster a deeper understanding of the body through immediate personal experience, as well as self-expression and self-creation (35). Secondly, it promotes an encounter with others, encouraging the “appreciation of the aesthetic qualities of other selves and other things” (36). Finally, it implies an encounter between art and the individual. As Shusterman explains:

“More broadly, our appreciation of the beauties of art has a significant bodily dimension, not so much because we apprehend them through our physical senses (such as proprioception, which aesthetics has traditionally overlooked) but because of the emotional values of art, which, like all emotions, can only be experienced physically. By educating and cultivating the sensitivity of our somatic awareness to enhance our physical perceptions and performances, we not only increase the material means for producing art but also for enjoying it.” (37)

Somaesthetics proves particularly useful as a method by structuring the relationship between sensations and emotions, transforming them into distinct yet interconnected sources of knowledge for art and aesthetic experience. Sensations are understood as immediate, physical perceptions — such as touch, proprioception, or temperature — while emotions are the subjective, embodied responses that arise from these sensations, imbued with personal and cultural meaning. By fostering awareness of these dynamics, somaesthetics enhances our understanding of our own sensory and emotional experiences.

In dance, for instance, the expressive and fluid movements of the body serve as a direct embodiment of somaesthetic concepts. While firmly rooted in art and aesthetic experience, dance uniquely underscores the integration of sensation, emotion and gesture. Contemporary dance, in particular, highlights the body’s role in conveying emotions and narratives, reflecting somaesthetic principles through its physicality and the sensory dimensions of performance.

In theatre, somaesthetics manifests for both performers and audiences by emphasising the sensory and perceptual dimensions of the body in performance. For performers, this is achieved through choreography and staging that foreground bodily sensations, movements and interactions. For audiences, it often involves performances designed to alter or challenge their perceptions of space and movement. This somaesthetic approach fosters a heightened awareness of bodily sensations and spatial relationships, enriching the theatrical experience.

In music, the interplay between bodily rhythms and emotional states creates an immersive and somatic listening experience. Genres such as electronic dance music utilise rhythm and beat to engage listeners on a physical level, encouraging movement and bodily interaction that deepens the connection to the music.

Similarly, in literature, vivid descriptions of physical sensations and movements immerse readers in the embodied experiences of characters, enabling them to sense the physicality of the narrative and fostering a more tangible connection to the story.

Somaesthetics is not a universal solution, but it has proven effective in exploring and articulating the concepts of encounter related to the soma and the aesthetic experience, offering a rich foundation for further inquiry.

AN ARTISTIC RESEARCH PROJECT

This section describes two interactive installations linked through a same research project. They were developed during two research-creation residencies at Hexagram (an interdisciplinary network dedicated to research-creation addressing the relationships between arts, cultures and technologies), which culminated in exhibitions at Ars Electronica 2023 and EuroHaptics 2024. These works approach the interface as an artwork in its own right, enabling participants to engage with the piece while using it as a medium for communication and self-expression. They draw on creative principles inspired by somaesthetics, exploring dynamic relationships between the body, sensory perception and artistic interaction.

ON/CONTACT

ON/CONTACT is an interactive installation in the form of a haptic column that the interactor presses against their body. A closer physical interaction, such as wrapping their arms around the column or leaning into it, activates a custom-made analog tactile sensor. This unfolds a soundscape composed of familiar sounds, renewed with each contact. Simultaneously, it activates a vibratory experience induced by tactile transducers. See Fig. 1 and Video 1:

Figure 1: Different interactions and embraces of the haptic column ON/CONTACT as presented during an art residency at Hexagram 2022. Photos by Flore Boubila @flore_bbl

Video 1: ON/CONTACT during art residency at Hexagram in 2022, presented in DEMO26.

Notably, the terms encounter and contact share semantic connections: encounter refers to the act of coming into intentional contact or relation with others and the world, while contact extends to include both intentional and accidental interactions with an object (38).

ON/CONTACT’s interface acts as a catalyst for encounters, offering multiple interaction modalities through the tactile exploration of its surface. The embrace of the column generates a dynamic sonic and vibratory landscape, while the touch-sensitive surface leaves modulated textures and patterns as traces of interaction (Fig. 2). The soft, inviting material covering the column enhances the tactile experience. During public presentations, it was common to see groups of people interacting with the installation in a collective embrace (Fig. 3). By fostering physical closeness, the artwork can also function as an interface for encounters between individuals.

Figure 2: The surface is modulated when touched, leaving textures and patterns as testimonies of the encounter.

Figure 3: Group embraces during the presentation of ON/CONTACT at Ars Electronica in 2023.

When contact is established, a shared space forms between the human and the haptic column, creating a literal interface. This interface is not merely a mediator but a zone of exchange and connection. It functions as the artwork-in-itself: the audience engages in self-encounter through its feedback, while also connecting to the artist’s intention of fostering interpersonal relationships. These relationships incorporate the diversity of symbolic interpretations expressed by the audience at the point of contact — the haptic column. Furthermore, observing how others interact might be engaging as well, raising intriguing questions about a grey area between encounter and contact. Like any artwork, ON/CONTACT expands beyond existing concepts.

EN/GLOBE

EN/GLOBE is an immersive and interactive installation featuring three spheres that offer a variety of sensory stimuli. It was showcased during an open art residency at Hexagram in 2023, as well as during the symposium Interdisciplinary Encounters — Sensory Environments held at Agora Heart of Science in 2024 (39).

The spheres display video projections that respond to presence and movement, while various mechanisms create physical sensations. A fan emits a gentle breeze scented with peppermint. Meanwhile, sound exciters (which vibrate surfaces to produce sound) transform a sphere into a loudspeaker, generating spatialised audio directly from the sphere. Additionally, a servo motor moves soft filaments that lightly caress the head. See Fig. 4 and Video 2:­

Figure 4: Immersive installation EN/GLOBE, where interaction triggers generative visuals and electronic devices. Hexagram, 2023.

Video 2: EN/GLOBE during art residency at Hexagram in 2023, presented in DEMO26.

These subtle and remarkable experiences, sensed within the spheres without relying on sight, create an ideal context for immediate awareness and interaction between bodily experience (proprioceptive, kinesthetic) and sensory experience (tactile, auditory, visual, olfactory). This approach fosters a holistic integration of sensations within an aesthetic experience.

Additionally, the artwork unfolds dynamically in space and time: the sensory devices generate sounds and movements, while the spheres are enveloped in generative visuals that evolve in response to interactions. The interplay between the audience and the installation, forming an interactive sensory environment, constitutes the universe of the artwork. The ecosystem of the installation and its components acts as an interface, offering different interaction modalities that align with the three modes of encounter: with the artwork, with other audience members, and with oneself.

The stimulation of senses typically underutilised by interfaces, such as smell or passive touch, invites the discovery of subtler sensations across different parts of the body. The imposing size of the spheres encourages full-body movement and gestures on a larger scale than those typically used in manipulation or digital interaction. By combining electronic devices and interaction data to generate evolving visuals and a rich palette of sensory stimulations, the installation merges physical and digital matter, creating a unique environment for sensory experimentation. This space becomes an interface: a zone of contact and exchange, conducive to interactions and encounters.

Through this interplay, the encounter with the artwork and the resulting bodily movements provoke an awareness of personal, immediate experience. Somatic perception surfaces into consciousness, integrating with the flow of thoughts that materialise through a person’s engagement with the work.

The multimodal design of EN/GLOBE accommodates diverse sensory capabilities and modes of expression; various possibilities for perception and interaction.

Firstly, the artwork space acts as an interface that reacts to the audience’s presence, where the meeting between the artist’s intentions and the audience materialises through interaction. The audience completes the “circuit” of the artwork by consciously participating in the event.

Secondly, direct engagement with the artwork involves a combination of sensory perceptions, including movement between the spheres, bodily positioning in space, and triggering sound devices. The artwork becomes an interface that connects audience members as they interact within its shared space.

Finally, each sphere offering a distinct sensory interaction through tactile, olfactory and auditory experiences. In this way, the interface itself transforms into an artwork, enabling encounters with oneself through feedback from the “artwork-interface” (Fig. 5).

Figure 5: EN/GLOBE – Space of the artwork, space of the audience’s encounter, space of each person’s encounter with themselves.

LAYERS OF ENCOUNTER

The concept of encounter applies to the audience’s first experience with an interactive artwork. This encounter begins with the voluntary movement of onlookers toward the interface, followed by contact, interaction and feedback (40). Within this connection space, the whole becomes greater than the sum of its parts (41). The interface transcends its role as a simple intermediary, serving as the point of contact between the audience’s imagination and the universe embodied by the artwork (42). It reveals its potential for encounter, enabling diverse interpretations, experiences, and modes of interaction (43).

These interactions invoke the concepts of agency and multimodality. Multimodality may stem from the artist’s intentions during the creation of the artwork or arise as interactants discover new ways to engage with it. An interface grounded on encounter offers significant agency to the interactant, allowing them to actively shape their relationship with the artwork through participatory experience (44). The encounter also extends beyond the individual, encompassing interactions among audience members, with the artwork acting as a locus for shared contact and exchange. Finally, there is the encounter with oneself, achieved through the feedback loop created by engaging with the artwork (45).

These three modalities — interaction with the artwork, others and oneself — designate the interface as both the subject and object of the encounter, elevating it to the status of a work of art. Ultimately, the interactive artwork becomes a metaphor for encounter.

The concept of encounter is also embedded in the technology used in ON/CONTACT and EN/GLOBE. Both artworks rely on widely accessible electronic devices and basic programming, with their originality stemming from how these tools are utilised. Rather than prioritising precision, accuracy and efficiency to produce consistent results, the focus is on exploring the diverse outcomes that arise from different uses of technology and evaluating their creative potential. What might traditionally be seen as a limitation is reframed as a valuable feature or advantage in specific contexts. This perspective opens up a wide field for investigating interaction and its expression in creative and innovative ways.

In ON/CONTACT, a custom analogue sensor circuit detects contact from the participant’s body, activating a sonic and vibratory landscape through tactile transducers. The circuit features a distinctive, non-linear pattern that challenges conventional applications of such technology. The interface was intentionally designed not to function as a musical instrument, with its custom-made sensor featuring a staggered grid pattern to reflect this intention. The concept of control, often linked to a ‘tool for creative expression’ (46), posed challenges when juxtaposed with the idea of encounter. The installation is designed for the general public, who are responsible for shaping their own aesthetic experience.

In EN/GLOBE, the sensor triggering the generative visuals and devices within each sphere is a standard video camera without infrared capabilities. While the camera’s limitation — interpreting images only on a frontal plane, not in depth — might seem restrictive, it was creatively repurposed to integrate the entire space into the installation, forming a cohesive ecosystem. Approaching technology with a mindset of encounter allows the artist to rethink conventional tools and explore innovative possibilities.

CONCLUDING: OUR SITUATED APPROACH TO INTERFACES

In the context of interactive installations, the concept of multimodal interface implies various levels of interaction, which together form its ecosystem. This perspective offers a specific definition of the interface as an artwork, rooted on the principles of encounter and somaesthetics. Each component of the ecosystem — whether it’s a physical object, sensory stimulus or interactive mechanism — can take on the role of an interface; each can serve as the point of interaction between the participant and the artwork.

This approach is defined by three interaction modalities, each corresponding to a mode of encounter. Firstly, there is the encounter with the artwork, where the entire ecosystem functions as an interface, allowing the artist’s intentions and the audience’s engagement to materialise through interaction. The audience completes the artwork’s circuit by actively participating in the event. Secondly, there is the encounter among audience members, with the artwork serving as a zone of contact and exchange. Here, interaction emerges from movements and interrelations within the shared space. Lastly, there is the encounter with oneself, realised through the feedback generated by the artwork as an interface.

Together, these three modalities establish the interface as an artwork, positioning it as both the subject and the object of the encounter.

Openness to others and introspection are core principles shared by the somaesthetic and encounter models. Somaesthetics encourages us to use our natural sensory and cognitive abilities to appreciate somatic and aesthetic experiences as sources of knowledge and growth. In interactive artworks, the engagement of sensory perceptions and emotions enhances self-awareness. Since each participant interprets the artwork uniquely, aesthetic experiences vary widely. There is no singular “correct” way to interact with an artwork — the right way is the one conceived through the participant’s dialogue with the piece. The interface-artwork invites us to draw on personal, immediate experiences to connect with ourselves and others.

This somatic-aesthetic approach has yielded meaningful experiences for both participants and the artist. For the artist, it is deeply rewarding to work with diverse materials — exploring, listening to, transforming, and drawing inspiration from them to create experiences that transcend language and are accessible to all. The depth of this sensory landscape enables people to develop unique and enriching interactions. Likewise, the concept of the encounter-based interface offers fertile ground for artistic, formal, expressive and technological exploration.

Our creative practice proposes innovative ways to engage with digital art and address the pervasive influence of digital technology in society. By recognising the body as both a medium and the support of sensory knowledge, somaesthetics encourages us to somatically experience and actively shape our relationships with the digital, transforming us from passive spectators into active participants (47).

Building on this foundation, interactive arts contribute to the broader artistic field by integrating ‘original symbolic strategies’ and technological innovations reflective of the present era (48). By offering transformed visions of our identity, human and social nature, interactive arts draw inspiration from technology but are not limited to it. They shift away from the artistic concerns of presenting personal experience, creating interpersonal relationships through their ability to generate ‘specular experience’ (49). This research-creation echoes Jean Dubois’s proposition that interactive art has the ability to “develop a critical poetry of human relationships”, embracing “terms specific to an artifice of encounter” (50).

Through its expressive and self-constructive nature, this perspective adopts a moral and inventive stance. It can be enriched by the concept of encounter, which is essential for the interface to fully embody its status as a work of art. While aesthetically autonomous, interfaces-artworks could also inspire the development of new forms of therapeutic interfaces. This opens opportunities for further experimentation and research into somaesthetic-inspired interfaces, enabling a deeper, broader understanding of encounter-based designs.

Creative exploration in digital arts drives the development of new interfaces, inspiring fresh attitudes and prompting reflection on the future of our relationship with technology. This evolution offers opportunities to engage new audiences and consider the human experience holistically, encouraging a re-examination of concepts like interface and interaction. By centring on the concept of encounter, this project provides a renewed perspective on interfaces, contributing to the creation of a more thoughtful and desirable future (51).

REFERENCES AND NOTES

1. Richard Shusterman, Conscience du Corps: Pour Une Soma-Esthétique (Paris: Éditions de l’Éclat, 2007).
BACK

2. David Rokeby, “Transforming Mirrors: Subjectivity and Control in Interactive Media”, in Critical Issues in Electronic Media (New York: Suny Press, 1995) p. 148.
BACK

3. Definition of “Control” from the Merriam-Webster Dictionary, <https://www.merriam-webster.com/dictionary/control>, accessed 30 November 2024.
BACK

4. Anne-Gaëlle Baboni-Schilingi, “Installations et interactivité numérique”, Les Cahiers du Numérique 1, No.4 (Éditions Lavoisier, 2000) p. 171.
BACK

5. Rokeby (1995) p. 148 [2.]
BACK

6. David Rokeby, Interactive Installations : Very Nervous System (1986-1990) at <http://www.davidrokeby.com/vns.html>, accessed 30 November 2024.
BACK

7. See Golan Levin and Collaborators’s Re:MARK at <http://www.flong.com/archive/projects/remark/index.html>, accessed 30 November 2024.
BACK

8. Definition of Interface from the Centre National de Ressources Textuelles et Lexicales (CNRTL), <https://www.cnrtl.fr/definition/interface>, accessed 30 November 2024.
BACK

9. Original French text: “Avant d’être technologique l’interface est essentiellement un lieu, une marge, une zone d’articulation de communication, d’interrelations spatiales et temporelles entre deux conceptions du monde, une zone de friction et d’échange de deux espaces […]. Elle est le lieu d’une coexistence […].” Quoted from Jean-Paul Longavesne, “Esthétique et Rhétorique des Arts Technologiques. Les Machines Interfaces”, Ligeia 45-48 (2003) p. 167.
BACK

10. Original French text: “La puissance de l’interface réside dans la puissance des relations. Lorsqu’on relie deux choses, on peut parfois n’obtenir qu’une simple juxtaposition, l’interface est alors absente, mais le plus souvent, on crée une chose nouvelle dont les propriétés ne sont pas la simple somme des propriétés des composants ainsi reliés.” Quoted from Gérard Chazal, Interfaces: Enquêtes Sur les Mondes Intermédiaires (Seyssel: Éditions Champ Vallon, 2002) p. 268.
BACK

11. In French, this property of the interface can described as follows: “Propriété [dite] émergente, c’est-à-dire de propriété apparaissant lors de la liaison des éléments, propre au tout ainsi formé et n’appartenant à aucun des éléments.” See Chazal (2002) p. 267 [10.]
BACK

12. Original French text: “Au sens strict, les arts numériques sont à la source d’une véritable ‘inter-action’, en sollicitant un mouvement de l’œuvre vers le spectateur et, inversement, du spectateur vers l’œuvre. […] Œuvre et spectateur interagissent mutuellement, selon des modalités multiples.” Quoted from Laurent Diouf, Anne Vincent and Anne-Cécile Worms, “Les Arts Numériques”, Dossiers du CRISP 81, No 1 (2013) p. 13.
BACK

13. See Jean Gagnon, Blind Date in Cyberspace or the Figure that Speaks, Fondation Daniel Langlois, 1995, <https://www.fondation-langlois.org/html/e/page.php?NumPage=158>, accessed 30 November 2024.
BACK

14. See Jean Gagnon (1995) [13.]
BACK

15. Original French text: “L’artiste n’est plus l’unique créateur d’une œuvre, mais souvent le médiateur ou l’animateur des interactions entre le public et celle-ci.” Quoted from Diouf et al. (2013) p. 7 [12.]
BACK

16. David Rokeby, “The Construction of Experience: Interface as Content”, in Digital Illusion: Entertaining the Future with High Technology (Boston: Addison-Wesley Professional/ © ACM Press, 1998) p. 149.
BACK

17. See Olivier Zattoni, “Du Corps Virtuel au Corps Paysage: Présence à l’Autre et Affectivité à l’Ère Numérique”, Interfaces numériques 7, No. 2 (2019). p. 344.
BACK

18. Original French text: “Le corps est un filtre sémantique [et] qu’il n’est rien dans l’esprit qui ne soit d’abord passé par les sens.” Quoted from David Le Breton, “La Conjugaison des Sens : Essai,” Anthropologie et Sociétés 30, No. 3, pp. 19–28 (2006).
BACK

19. The exhibition Sensorium: Embodied Experience, Technology and Contemporary Art took place at the MIT List Visual Arts Center from October 12th 2006 to April 8th 2007. It was the subject of a book of the same name – see <https://mitpress.mit.edu/9780262101172/sensorium>, accessed 10 December 2024.
BACK

20. Original French text: “Lorsque le sensorium est remis en question, la définition opérationnelle du soi est perturbée.” Quoted from Anja Bock, “Sensorium: Embodied Experience, Technology, and Contemporary Art”, Art Papers 31, 3 (2007) p. 58.
BACK

21.  Bock (2007) [20.]
BACK

22. Le Breton (2006) [18.]
BACK

23. Original French text: “Les œuvres d’art ‘technologique’ privilégient d’autres sens: en particulier le toucher, ainsi que la gestuelle, absente jusqu’alors de la relation art-spectateur.” Quoted from Diouf et al. (2013) p. 13 [12.]
BACK

24. See Scenocosme, Metamorphy, <https://www.scenocosme.com/metamorphy_e.htm>, accessed 30 November 2024.
BACK

25. See Scenocosme, Lights Contacts,<https://www.scenocosme.com/urban_lights_contacts_e.htm>, accessed 30 November 2024.
BACK

26. Louis-Claude Paquin and Cynthia Noury, “Définir la Recherche-Création ou Cartographier Ses Pratiques”, ACFAS Magazine (2018).
BACK

27. See Shusterman (2007) p. 8. [1.]
BACK

28. See Shusterman (2007) p. 11 [1.]
BACK

29. See Shusterman (2007) p. 32-33 [1.]
BACK

30. See Shusterman (2007) p. 11 [1.]
BACK

31. Richard Shusterman, “Conscience Soma-Esthétique, Perception, Proprioceptive et Action”, Communications 86, No. 1, (2010) p. 15.
BACK

32. See Shusterman (2007) p. 13 and p. 34 [1.]
BACK

33. Richard Shusterman, “La Soma-Esthétique de Merleau-Ponty”, Critique d’Art 37, Printemps (2011).
BACK

34. See Barbara Formis, “Richard Shusterman, Conscience du corps. Pour une Soma-Esthétique”, Mouvements 1, No. 57,(2009). p. 156.
BACK

35. Shusterman (2007) [1.]
BACK

36. See Shusterman (2007) p. 12 [1.]
BACK

37. Original French text: “Plus généralement, notre appréciation des beautés de l’art possède une dimension corporelle importante, non pas tant parce que nous les appréhendons à travers nos sens physiques (telle la proprioception que l’esthétique a traditionnellement négligée) mais en raison des valeurs émotionnelles de l’art qui, à l’instar de toutes les émotions, ne peuvent être vécues que physiquement. En éduquant et en cultivant la sensibilité de notre conscience somatique dans le but d’améliorer nos perceptions et performances physiques, nous ne faisons pas qu’accroître les moyens matériels destinées à produire de l’art mais aussi à en jouir.” Quoted from Richard Shusterman, “Le Corps et les Arts : le Besoin de Soma-Esthétique”, Diogène 1, No. 233-234 (2011) p.28.
BACK

38. The French translation of “Encounter” is “Rencontre”. It can be defined as: “fait d’entrer en contact, en relation, de manière intentionnelle, avec autrui et le monde, [le] fait de se trouver en contact, accidentellement ou non avec un objet”. Quoted from Trésor de la Langue Française Informatisé, <https://www.cnrtl.fr/definition/rencontre>, accessed 30 November 2024.
BACK

39. The original name of this symposium, held March 25 to 28 in 2024, is Rencontre Interdisciplinaires — Environnements Sensoriels: Les Agentivités Plurielles en Art Numérique. The talk titled EN/GLOBE occurred March 27.
BACK

40. See Longavesn (2000) [9.]
BACK

41. See Chazal (2002) [10.]
BACK

42. Sandra Gaudenzi and Samuel Gantier, “Vers un Canevas Méthodologique Pour le Design d’Oeuvres Audiovisuelles Interactives”, Interfaces Numériques 7, No. 3 (Université de Limoges, 2008).
BACK

43. Baboni-Schilingi (2000) [4.]
BACK

44. Jean Dubois, “Reflexive Figures: The Encounter in Interactive Arts”, Esse Arts + Opinions 63 (2008). Also available in French : “Figures Réfléchies : la Rencontre en Arts Interactifs”.
BACK

45. Ibid [44.]
BACK

46. See Tim Murray-Browne, Di Mainstone, Nick Bryan-Kinns, and Mark D. Plumbley, “The Medium is the Message: Composing Instruments and Performing Mappings”, International Conference on New Interfaces for Musical Expression (2011).
BACK

47. See Sylvain Brétéché, “L’Incarnation de <<L’Entre>>”, Interface 8 (2015) p. 26.
BACK

48. See Dubois (2008) [44.]
BACK

49. Ibid [44.]
BACK

50. Ibid [44.]
BACK

51. These research-creation projects are associated with a research program on sensory environments in digital art (Fonds de recherche du Québec – Société et culture), led by the Mimesis laboratory at NAD-UQAC.
BACK

ABOUT THE AUTHORS, REVIEWERS AND EDITOR

Authors

Marie-Eve Morissette > Université du Québec à Chicoutimi

Artist and designer creating interactive installations that emphasise touch, encounter and embodied engagement. With a background in environmental, event and digital design, she works with the Mimesis laboratory and contributes to media arts research exploring perceptual experience, spatial interaction and the poetics of technologically mediated contact.

Louis-Philippe Rondeau > Université du Québec à Chicoutimi

Artist and professor whose installations investigate gesture, temporality and image-based interaction. His work has appeared at major international venues including Ars Electronica, ISEA and BIAN, as well as in Phaeno’s permanent collection. He co-founded the Mimesis laboratory, advancing practice-based research into experiential media and embodied interfaces.

Yan Breuleux > Université du Québec à Chicoutimi

Composer and researcher working across electroacoustic music, algorithmic processes and immersive audiovisual environments. A founding member of the collective Artificiel, he creates large-scale works integrating light, sound and experimental technologies, contributing to international discourse on visual music, multisensory experience and technologically mediated artistic practice.

Reviewers

André Sier  > Faculty of Fine Arts Lisbon

Artist and researcher whose work explores computation, simulation and interactive systems across audiovisual installation, performance and programming. Teaching at the Faculty of Fine Arts in Lisbon, he develops generative environments and algorithmic architectures that investigate perception, embodiment and the aesthetic possibilities of technologically mediated experience.

Jo Scott > i2ADS & University of Coimbra 

Artist-researcher working across performance, audiovisual media and improvisation. Her projects explore embodied knowledge, collaborative creation and live interfaces, integrating artistic experimentation with theoretical inquiry. She develops practice-based research that examines mediated performance, multisensory interaction and the dynamic relationships between bodies, technologies and environments.

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 2:

Discussing video editing with AI benefits from first outlining key terminology relevant to a wide range of AI systems. For example, the term cognitive architecture (CA) refers to computational frameworks designed to enable the simulation of human cognition. Drawing from research in psychology, neuroscience and AI, CA’s operate based on theories about human reasoning, problem-solving and learning. They function as blueprints for the construction of digital systems, by specifying fixed structures and interactions among those structures. Examples of CAs include ACT-R, Soar, and CLARION, which have been used to study various cognitive tasks and build intelligent agents – artificial entities that work autonomously (1) (2).

A CA can be integrated with deep learning (DL). DL is a subset of machine learning used in image and speech recognition, natural language processing and autonomous systems, among other computational fields (3). It involves artificial neural networks (NN), often called deep NNs. These are designed to learn hierarchical representations of data; as data passes through the network, it can recognise complex patterns and learn intricate structures.

Artificial neural networks have been applied to a wide range of problems, from speech recognition to ‘prediction of protein secondary structure’, classification of cancers and gene prediction (4). A NN includes a stack of layers with nodes (or “neurons”), as well as input space and output space. The nodes are interconnected, forming a dynamic graph. Throughout the learning period, the NN keeps updating the weight of interconnected nodes, to better fit the training data.

This process enables a mathematical calculation called embedding: to generate embeddings from input data means to derive a numerical representation of a piece of data from a trained NN, given its input space and its previous state. Embeddings are usually produced as a vector map. They can be used in many tasks performed through the graph, including classification, clustering, link prediction and visualisation (5).

Neural network mechanisms have been greatly refined with a deep learning architecture called transformer, developed by Google. This architecture applies an “attention mechanism” drawn from previous models such as recurrent neural networks (RNNs). RNNs can process data sequences but they cannot retain information over long sequences – a temporal problem seen in language translation and speech recognition. A transformer enables the AI system to focus on specific parts of the input data while capturing dependencies over long data sequences. Transformers revolutionised natural language processing, powering applications such as machine translation, text summarisation and language understanding (6).

Yet another important notion in contemporary AI is the one of large language model (LLM). Trained on large amounts of data, LLMs can perform a wide range of tasks, such as understanding and generating natural language. Available systems are accessible to the public through interfaces like Open AI’s Chat GPT-3 and GPT-4 (Generative Pre-trained Transformers), which garnered Microsoft’s support (7). Most LLMs after 2017 employ transformer networks.

LLMs can produce data streams that correlate different modalities – audio signals, video frames or metadata, to name a few (8). Deep learning AI systems have been successfully applied in professional audio-visual production, using neural networks to continuously infer data sequences for direct media generation or search (9). Cognitive architectures proposed prior to transformer architectures and the current DL ecosystem entail characteristics that can equally enhance the design of current generative systems.

In this essay we tackle the development of video sequences through LLMs, and propose a specialised transformer-based system that implements design guidelines from existing cognitive AI architectures. Our research focuses on the modelling of video sequences –  simply called video editing here – and we will present a working implementation that edits arbitrary video datasets.

Our results rely on computing the embeddings of an initial corpus of clips. We use two trained models, connected through natural language: a vision-transformer (ViT) for one-shot image description, and a GPT-J-6B language model for reasoning, contextualisation and text subtitles. Integrating several AI-driven processes advances the current research in automated video sequence editing. To evaluate our approach, we develop a stream of natural language from frame descriptions and we establish mechanics to search and select from the dataset.

The system builds upon existing retrieval paradigms: the current video induces the next one based on semantic constructs (Potapov et al. proposed this (10)) produced by an artificial cognitive architecture system (Choi proposed this (11)). Once we use LLMs, natural language works efficiently in searches by description, matching and context across different data structures (12). Matching modalities is a starting point for the organisation of elements such as samples from a video dataset following a natural language stream, at specific variability rates. Also, the discourse capabilities of LLMs are admirable when set to infer in a loop, and merged with image representations; film professionals and audio-visual practitioners in general might know how that can be advantageous (13).

TYPES OF COGNITIVE ARCHITECTURES: HISTORICAL CONTEXT

CAs specify the structures and functions of cognitive systems as well as the interaction between them (14). They explore a multidisciplinary goal and hypothesis: to effectively define and build a simulation of observable structures of cognition, such as patterns in the human brain, or behaviours in a context environment (15). Researchers in the field attempt to specify a computational structure for intelligent behaviour (16). Using psychological evidence and computer models, they aim to find an infrastructure for general intelligence (17).

CAs implement theories about essential representations and mechanisms (18), including:
– Compatibilities of big data analysis and analysis of brain impulses.
– Prediction and active inference at variable clock rates with temporal changes to weights.
– Long-time continuous learning and reasoning over known possibility sets.
– Value systems and engines for motivation depending on, e.g. valence, arousal and dominance restrictions and differentiations.
– Action-reaction through boundary definers of self-contained systems within controlled environments.

To what extent can a machine simulate human cognition, which is inherently embodied, context-dependent, and changes over time? For at least fifty years, researchers have been modeling AI systems as cognitive architectures (CAs), during which time computing units have evolved dramatically. Both the limitations of computation and scientific understanding of cognition in natural systems have progressed significantly (19). Today, new insights from cognitive science, psychology, and neuroscience continue to reveal complexities and abstractions within the human brain, such as the nuanced role of natural language communication, including the use of embedded vector representations. A key objective has been to determine whether higher-level cognition relies on a unified system, or if distinct systems handle specific functions, such as vision, hearing, or movement, particularly in studies of knowledge representation and production systems (20). Advances in processing power, a deeper understanding of the brain and strategic design choices have led to the development of several domain-general CAs, including those mentioned in the introduction: ACT-R (Adaptive Control of Thought – Rational), Soar, and ICARUS.

Many other cognitive architectures (CAs) will appear throughout this manuscript, showcasing their diverse design choices and task specialisations. These architectures exemplify various approaches to identifiable subsystems of human brain function and intelligence, such as short-term and long-term memory modules. Most literature focuses on digital computations performed in CA simulations, with structures distributed across different physical processing units. Researchers have demonstrated the effectiveness of implementing neural networks (NNs) within CAs, as seen in connectionist approaches like the CA CLARION, which captures significant abstractions based on brain processing through layered structures (21).

Figure 1 displays a simplified taxonomy of CAs, from a comparative study showing their diverging ‘memory‘ and ‘learning‘ modules and processes.

Figure 1: A simplified taxonomy for cognitive architectures, retrieved from Asselman et al., 2015.

Early examples of cognitive architectures (CAs) include the General Problem Solver (GPS), a computer program developed by H. Simon, J. Shaw and A. Newell in 1957. This program explores the psychology of human thinking and proposes a theory to explain why some individuals attempt to solve specific simple formal problems. GPS provides a general theory of cognition that remains independent of particular phenomena, allowing modellers to add representations for specific tasks. Researchers then simulate the architecture on a computer to produce behaviour, which can be compared to human behavior in the same tasks (22).

In their early work, Frixione et al. identified five possible approaches to the problem of Knowledge Representation in Cognitive Science (23):
– Compositional symbolic approaches;
– Local non-compositional approaches;
– Distributed non compositional approaches;
– Cognitive sub-symbolic approaches;
– Neural sub-symbolic approaches.

Soar can be considered a direct successor to GPS, sharing the assumption that mechanisms for problem solving are key to human cognition and intelligence. Soar is a general problem-solving architecture with a rule-based memory developed from the GPS paradigm (24). The authors propose a study on the mechanisms necessary for intelligent behaviour, including search-intensive tasks, knowledge-intensive tasks and algorithmic tasks, to form a general cognitive architecture (25).

At that time, they posited that general learning mechanisms for a performance system would need to exhibit the following properties:
– Task generality
– Knowledge generality
– Aspect generality

To explore the validity of a cognitive architecture, it is worthwhile to examine a particularly successful example of the functional approach: TacAir-Soar, which models a fighter pilot and is built using the Soar architecture (26). The goal is to create multiple simulated pilots that can participate in large-scale warfare simulations, where the models operate alongside human pilots and other personnel. The main criterion for success is whether the models behave like real pilots, thereby contributing to an authentic experience for the humans involved in that specific simulation.

Other architectures do not regard problem solving as a fundamental mechanism but rather as a component of specific models pertaining to particular problem-solving situations (27). For example, the ACT-R architecture, along with its predecessors, draws inspiration from cognitive psychology (28). Just as GPS served as a foundation for Soar, the Human Associative Memory (HAM) model was a precursor to Adaptive Control of Thought (ACT-R) (29). ACT-R is a cognitive architecture featuring a core production system that includes a pattern matcher, which operates on memory and perceptual-motor modules via buffers (30). Building upon the HAM model, John Anderson developed the ACT-R theory through several iterations, culminating in the first major version, ACT* (31).

Later in the post-internet era, the Learning Intelligent Distribution Agent (LIDA) was proposed as an intelligent, autonomous software agent designed to assist personnel in the US Navy (32). The LIDA architecture is an extension of the IDA model, funded by the Office of Naval Research and other Navy sources. LIDA is largely based on Baars’ Global Workspace Theory (GWT) which broadens the focus of cognitive architecture design to encompass a theory of consciousness (33). This theory has, at various points, become the most widely accepted psychological and neurobiological framework for understanding consciousness (34).

Implementations of these cognitive architectures demonstrate functional properties of intelligence through observation and seek to establish general methods for automating task solutions, supported by various specialised procedures or modules.

Figure 2: The Global Workspace Theory metaphor, extracted from Bernard J Baars, 2005.

The rise of neural networks in computing and AI research has developed largely as a distinct field from cognitive architectures. Even prior to advancements like Graphical Processing Units (GPU) and modern machine learning frameworks, CAs successfully implemented NNs and established connectionist paradigms, functioning as networks of diverse processing modules. One example, CLARION, represents a hybrid, dual-process CA that differentiates between explicit and implicit cognitive processes (35). On a similar level, the CA OpenCog implements NNs as well, which can develop to be a system encompassing many different AI models to be used within the same ecosystem, the authors pose CogPrime as a draft architecture for artificial general intelligence (36).

In summary, CAs are certainly a good a candidate for video editing systems, and well-suited to integrate with LLMs or other generative models, as we will show ahead.

SPECIALISED COGNITIVE ARCHITECTURES

In the past two decades, various CAs have been developed with targeted action applications, rather than purely generalised problem-solving, though these applications might still contribute to more general solutions. AI researchers have begun designing CAs to address specific tasks, such as understanding video frames by classifying and detecting anomalies in defined environments.

For example, Doulamis et al. propose a self-configurable CA for monitoring video data in manufacturing, which utilises weakly supervised learning algorithms and self-adaptation strategies to analyse visually observable processes (37). Their advanced architecture, known as the Self-Configurable Cognitive Video Supervision system (SCOVIS), incorporates attention models that focus on regions of interest (ROIs) and objects of interest (OOIs) in static images and video sequences. SCOVIS also supports camera coordination across a network. Principe describes a similar CA developed for object recognition in video, aiming to abstract computational principles from the animal visual system to interpret video images (38).

Li et al. present a novel CA that understands the content of raw videos in terms of objects without using labels (39), achieving three objectives:
– Decomposing raw frames into objects by exploiting foveal vision and memory.
– Describing the world by projecting objects onto an internal canvas.
– Extracting relevant objects from the canvas by analysing the causal relationship between objects and rewards.
Beyond data processing and its patterns, this specialised example (vision and raw image or video frames) clearly indicates a biologically inspired design for the structure of the CA. Further research on this topic can be found in the proceedings of the Annual Meeting of the BICA (Biologically Inspired Cognitive Architectures) (40).

APPLICATION IN GAME ENVIRONMENTS

The application of cognitive architectures has not only been notable in manufacturing environments, but also directly within video game engines. By applying a CA in an ecosystem with its own rules, players, and environments, automation can either encompass all human actions or exist alongside the player, facilitating non-linear dynamics.

Li et al. propose an architecture for frame-oriented reinforcement learning – a type of machine learning that focuses on learning optimal actions based on individual frames or images. This system is designed to understand the content of raw frames and derive meaningful representations (41). The authors demonstrate its capability to recognise and classify characters within Super Mario Bros. The system is modelled to:
– Extract information from the environment by exploiting unsupervised learning (learning from unlabelled data) and reinforcement learning (learning through trial and error, where successful actions receive positive feedback known as rewards).
– Understand the content of a raw frame.
– Exploit foveal vision strategies (focusing on specific details, similar to human vision) analogous to the human visual system.
– Collect new training data subsets (smaller data portions) automatically to learn new objects without forgetting previous ones.

In the same realm, the cognitive architecture OpenCog has also been applied to virtual characters, such as dogs in the OpenCogBot project, which features an interactive question-answering environment. These virtual dogs demonstrate the ability to learn new behaviours through imitation and reinforcement, showcasing the potential of CAs in enhancing the realism and interactivity of virtual characters (42).

Figure 3: Screenshot of OpenCogBot-controlled virtual dog, retrieved from Goertzel et al., 2010.

APPLICATION IN IMAGE AND MUSIC SIGNALS

Cognitive architectures (CAs) have also been developed in the context of computational creativity, demonstrating creative behaviour in specific data patterns such as audio or image frames. Li et al. introduce a system that implements creative behaviour within a CA (43). The system draws from cognitive science and computational creativity, particularly the PSI model and the MicroPSI model (44). It can generate portraits starting from a subject’s snapshot, creating a set of basic styles during a training phase and proposing novel combinations of the input image. The implementation successfully demonstrates motivation in selecting subsequent parts of images based on the flow processes between memory modules.

Focusing on producing content with creative value can be achieved through imitation, which is extensively studied in the context of style transfer and deep neural networks (45). Modelling also extends to audio frames and music sequences, with the generation of music accomplished through the MusiCOG cognitive architecture (46). This CA is integrated with an interactive music composition application called ManuScore. In both examples, the CA systems encode inputs in useful ways for prediction, employing diverging design choices. The flow of information within the CA modules allows for the production of variations. In this case, the CA interoperates with secondary software that includes notation, facilitating an interaction loop between the human and the AI system.

Across various application scenarios and design guidelines, CA architectures consider both symbolic and subsymbolic levels, as well as connectionism through deep neural networks. Chella et al. discuss hybrid architectures where different processing modules coexist and cooperate in a principled manner (47).

Figure 4: Workflow used to place an input image on a Style Map, retrieved from Augello et al. 2013.

TRANSFORMER MODELS WITHIN COGNITIVE ARCHITECTURES

With the advent of neural networks (NNs) and deep learning (DL) models, cognitive architectures (CAs) are undergoing significant re-imagination. Many implementations now leverage specialised modules enabled by NNs for tasks such as one-shot reasoning (drawing accurate conclusions from minimal information), including image classification. In computer vision, convolutional neural networks (CNNs or ConvNets—networks designed to recognise patterns and spatial hierarchies in images) were dominant prior to the emergence of transformers, which have gained prominence recently (48). Transformers, based on an attention mechanism, are typically trained offline on large corpora. Their development facilitated the creation of large language models (LLMs) like GPT (49). It addressed challenges of recurrent neural networks (RNNs), which process data sequences but struggle to retain information over long sequences—a limitation known as the “forgetting problem” (50).

The Transformer architecture, built around a multi-head attention mechanism (allowing the model to focus on different parts of an input sequence simultaneously), is also effective for image data. Dosovitskiy et al. show that dependence on CNNs is not essential for applying attention to images (51). They demonstrate that a pure Transformer, applied directly to sequences of image patches, achieves high performance on image classification tasks when pre-trained on large datasets and tested on benchmarks like ImageNet, CIFAR-100 and VTAB. The vision-transformer (ViT) matches or exceeds state-of-the-art convolutional networks’ performance while requiring significantly fewer computational resources for training (52).

Transformers have also been adopted in cognitive architectures (CAs), retaining some of the structures from early systems. Laird et al. present a proposal for integrating CAs with transformers, featuring a long-term memory system (LTMs store information over extended periods) supported by a neural network itself (53). The authors identify potential capabilities for transformers within a CA, functioning as prediction engines to forecast future working memory states, including:
– Anomaly detection: identifying when a situation deviates from the routine.
– Action modelling: predicting the consequences of an action.
– World modelling: anticipating changes in world dynamics.
– Preparation: taking actions to influence an unexpected future.

Sun recently suggested that incorporating insights from human cognition and psychology, as embodied by a CA, can enhance the capabilities of these systems (54). He emphasises the significance of dual-process architecture (a framework distinguishing between automatic and controlled cognitive processes) and the hybrid neuro-symbolic approach (which combines neural networks with symbolic reasoning) in overcoming the limitations of current large language models.

Sumers et al. further propose a framework called cognitive architectures for language agents (55), aimed at addressing the need for more capable LLMs. They present a unified study of their integration with production systems (computational models that use rules to determine actions). They characterise language agents as a class of LLMs capable of demonstrating common-sense reasoning during conversations or when functioning independently. To enhance LLMs, they propose methods such as feedback (the process of using the output to inform and improve future responses) and recursion (the ability to apply a process repeatedly, allowing for more complex reasoning). This includes techniques like prompt chaining (linking multiple prompts to build context and improve response accuracy) and integrating external resources via search (utilising real-time information from databases or the internet to supplement knowledge).

AI SYSTEMS FOR VIDEO EDITING

In video editing and post-production, automation has extended to both multi-track editing—which layers multiple video and audio tracks in a time-based sequence—and algorithmic approaches, which apply data-driven techniques rather than relying solely on a timeline. With advances in deep learning and artificial intelligence (AI), these automated approaches have introduced new capabilities. Trained models can now restore old footage (e.g., by removing noise or improving resolution), conduct content-based searches (such as locating specific scenes), and predict image content (for example, generating missing frames), tasks that previously required manual adjustments. Additionally, by linking these models with recording data, AI-driven analysis and inference can enhance or even replace traditional methods like segmentation (56), and create new options, such as AI-guided segmentation using large language models (LLMs).

Professional video editing and manipulation involve various tasks, such as adjusting aspect ratios, resizing, and reshaping frames (57). AI systems now add new possibilities, including video summarisation, frame colourisation, and special effects. In film and media, workflows are increasingly enhanced by AI, particularly through the concept of a “virtual camera”—a digital stand-in for a real camera.

AI systems using virtual camera views include:
– Neural Radiance Fields, which create realistic 3D scenes (58).
– Depth estimation, used to gauge distances between objects in a scene.
– 3D point clouds, helpful for constructing and editing 3D environments (59).

Signal processing and understanding of video frames is a process that evolved with the human element at the forefront throughout the entire production chain, incorporating AI systems such as LLMs to assist in specific tasks. Many roles in film production, from camera crew to colour grading, have retained their distinct specialisations, which are often conceptual and based on the expertise of the individuals involved.

In 1994, Tonomura et al. proposed a simplified video computing process, at a time when video was becoming an increasingly important medium, outlining structures still prevalent in today’s professional filmmaking practices (60). The authors envisioned possible visual interaction modes with computers informed by the roles humans play during shooting and editing, designing a system that:
– First analyses a video stream and automatically segments it into shots.
– Each shot is indexed using extracted features, which involves cataloguing and organising the footage based on identified characteristics.
– Detecting camera work information and identifying dominant colours within the shots, which can be used to convey specific moods or themes.

Still in the 1990s, Ueda & Miyatake proposed an approach to integrate image recognition technology into nonlinear (non-sequential) video editing systems. They discussed the results of a prototype with five functions for users:
– Automatic scene separation during disk recording, which segments footage into distinct scenes automatically while recording, simplifying the editing process.
– Tree structure story editing, a method that organises scenes hierarchically, allowing editors to view and manipulate story elements in branching paths for easy narrative structuring.
– Quick simultaneous shot reviews, enabling editors to view multiple shots at once for efficient comparison and selection.
– Film metaphor interface, a user interface that mimics traditional film editing tools, making digital editing more intuitive.
– Integrated multimedia information management, which helps users organise and retrieve various types of media, such as video, audio, and text, within the editing system.

With the development of the desktop computer in that decade, Nack and Parkes presented a specific model of the editing process and outlined the basis for strategies of humour as a case study (62). The authors described AUTEUR – AI utilities for thematic film editing using context understanding and editing rules. – which relied on a model of editing based on a knowledge elicitation exercise that involved studying and interviewing editors in their work environments (63). Figure 5 summarises the system (64).

Figure 5: Architecture diagram of AUTEUR system, retrieved from Frank Nack et al., 1995.

Knowledge can be derived from physical environments by video cameras, recording and editing mechanisms can be developed with real-time incoming footage and real-time processing of that footage. There is virtual representation of the cinema camera to the AI model, and there is also output control for camera movements, these may include editing techniques inscribed in the system. If an AI system would control flight modes for film it would have to at least represent in itself modes of shooting, so conventional procedures in filming can be respected.

Aerts et al. introduce the CAMETRON project that aims to developing an almost fully automated system to produce video recordings of events such as lectures, sports games or musical performances (65). The system includes components such as intelligent Pan-Tilt-Zoom cameras (which can automatically adjust their position and zoom to focus on subjects) and UAVs (Unmanned Aerial Vehicles) that function as camera assistants.

TRANSFORMER MODELS IN VIDEO EDITING

Vasconcelos and Lippman presented an early approach using a Bayesian model architecture — a statistical model that updates probabilities based on evidence — for content characterisation, analysing its potential for accessing and browsing video databases (66). This model demonstrated the ability to browse video databases semantically, which is relevant today with LLMs that use natural language interfaces for search. The video editing process often begins in cinematography, with shooting considered in relation to editing mechanics, making camera control crucial throughout film and audiovisual production.

Within neural networks and cognitive architectures, concepts for image data are deconstructed by models; the graphs compute through their layers, processing data to achieve desired outcomes. Today, transformer models assist in video editing, using natural language guidance and being trained on textual instructions for specific tasks. The video editing software and cloud service Kapwig provides AI-powered tools, such as video trimming using a text-based editor from transcripts (67).

Models of video editing not only consider the roles of the human editor and cinematographer, but also incorporate cultural aspects by focusing on learning from large datasets of images and texts.

In 2024, Wang et al. propose an LLM-powered video editing software and raise a study on a system (68) that allows for:
– Video retrieval
– Footage overviewing
– Idea brainstorming
– Storyboarding
– Clip trimming
– Title generator
In this approach, the editor can engage in a conversation with the LLM, which assists in editing the video by offering interaction through natural language prompts.

When exploring the use of transformers in video editing, one promising approach is applying language-guided transformations. This method enables techniques like style transfer, which allows for the alteration of a video’s visual characteristics by transferring the style of one video to another. A key technique in this process is the diffusion model, which gradually modifies an image or video by adding noise and then removing it, enabling precise control over the transformation. MaskINT, a system built around this model, uses natural language guidance to direct the transformation of videos, facilitating tasks such as style transfer and content modification (69).

OUR SYSTEM: GOALS AND METHODS

Continuously generating embeddings from trained models and measuring distances between these embeddings allows a model to represent diverse input data and find similarities between different pieces. In this context, “distances” refer to measurements of similarity between data points, which are represented as vectors in high-dimensional space. This approach enables the model to identify relationships between video elements based on their similarity. A common technique for handling multimodal data (e.g., video and text) is to establish a shared natural language framework, allowing models to translate specific types of content into text. Training transformers to work with embeddings in this way has proven effective across video and music applications (70).

This research addresses the challenge of automated video sequence editing by using transformers as “one-shot reasoning systems”, which make decisions based on minimal input. This approach relies on embedded features — such as objects, colours and scene types — extracted using a vision-transformer (ViT), pre-trained on large datasets. These features define both the sequence order and duration of video elements, enabling the production of coherent video sequences without human input. To support this, our system generates both image and text embeddings, which include confidence levels indicating how accurately each embedding represents key features of the content.

The ‘confidence’ reflects the semantic similarity between embeddings. After training, the model can detect similar features in new, unlabeled video frames by recognising patterns that match those learned from the labelled data. This labelling process forms the foundation for the model’s ability to automatically detect and define these visual characteristics.

Through the experiences conducted, we reflect on the human editor’s intuitive ability to select the next sequence when creating a montage. This ability often relies on similarity, or, in other words, on memory systems that favour the closest pair to lived experience. We also recognise that this research could benefit from integrating vision encoding and embedding techniques, enabling montage creation by both visual and semantic similarity. This approach shows considerable potential, particularly when working with large video datasets.

To develop this system, we adopted a practice-based methodology (71). We tested our hypothesis by observing the resulting videos, using a small video dataset to accurately identify the undertaken sequencing.

We consider the research successful when the system functions as a tool that disrupts or rearranges the content in professional editing contexts, aiding the editor in generating new ideas or perspectives. The system we implemented can analyse the content of related videos and automatically suggest sequences that offer fresh insights, helping the practitioner reassess the current edit.

Nevertheless, our approach may lead to continuous failure if the original image-text correspondence does not align with the intended meaning or human interpretation. We regard this as acceptable behaviour in experimental contexts. In fact, it could also inspire new creative approaches.

The experiments aim to produce finished video sequences from an arbitrary video dataset using an input prompt, with the following features:
– Present-future correspondence across 10 to 20 elements: This involves aligning video content that connects present actions or scenes with potential future developments, using 10 to 20 key elements to establish this link.
– Clip-based narratives for subtitles: Subtitles are generated from specific video clips, with each clip contributing to a narrative or story that enhances the viewer’s understanding of the video content.
– Varying length based on confidence values inherent to the models: The length of the generated sequences adjusts according to the model’s confidence in its predictions. Sequences generated with higher confidence are longer, while those with lower confidence are shorter.

Our system is a language-based video sequencer. It focuses on three key aspects:
– Production of a sequence of clips sampled from a given video dataset (distribution), where “distribution” refers to the variety and range of video content within the dataset. The clips are ordered according to the semantic meaning derived from a vision-transformer (ViT) for each clip, with the ViT pre-trained on ImageNet21k to extract meaningful features from each video frame (72).
– Definition of the duration of each clip, based on the confidence attributed by the ViT to each label, determining how strongly the model associates each clip with its label.
– Production of a text-based narrative for subtitling, generated by GPT-J-6B, a 6 billion-parameter transformer trained on the Pile dataset, a large text corpus containing diverse publicly available sources like books and websites (73). The narrative is created when the system is prompted with the set of labels derived from the video clips.

Figure 6: Flow diagram from video dataset input to video sequence output.

WORKING OUTCOMES

We conducted a series of experiments using a small open domain video dataset, consisting of 40 clips selected from the Internet. Open domain refers to a dataset that covers a broad range of topics or content types, without being restricted to a specific field or category. The goal was to develop an algorithm that predicts the most likely next video based on an input prompt. After making a couple of random selections (1% of the dataset), it returns to the video sequence that is most probable given the context.

From this algorithm, we generate an embedding space that represents all the videos in the dataset, where each video is encoded by its first and middle frames. This allows the model to capture key visual characteristics of the video at different points in time. The natural language descriptions provided by a vision-transformer (ViT) are used to generate semantic representations of each video, which are grouped into clusters of four frames based on similarity. These clusters are then stored as embeddings in the AI system for further analysis.

Similarity measures are applied to explore the relationships between the videos, where each video is considered a data point in the high-dimensional space. This approach is akin to memory recall, as the system finds connections between the most relatable data points (the ones closest to the input) and those that are more distant, enabling the creation of arcs of reasoning. By searching through these data points, the model can explore the video space from the most relevant to the furthest removed elements.

Figure 7 illustrates a randomiser tool, which initiates video sequences based on the initial prompt “prayer” and randomly selects related videos from the dataset, demonstrating how the system identifies connections between video elements. Thematic similarity appears as a concurrent church drone shot, showcasing the tool’s capability to recognise content alignment.

Figure 7: 20 video sequence thumbnail strip with “prayer” input prompt.

For each keyframe, we generate a 200-character narrative description using GPT-J-6B, a large language model that produces natural language outputs based on ViT labels — semantic tags assigned to each frame by the vision-transformer. These help to capture and describe the main content and themes of the video. These narratives are then aggregated into full sentences and paragraphs, providing contextualised subtitles for each video segment.

To document this process, we allow the system to continue selecting elements after the initial “prayer” prompt, creating a sequence that progressively diverges from the starting theme over 10 selections. In Video 1  the sequence segment is set to approximately 13 seconds, with two subtitle calls per segment, a format that can be quantised to meet standardised time lengths.

Additionally, we developed a time-scaling system (illustrated in Figure 6) to adjust playback duration according to ViT label confidence (a scalar measure that reflects how accurately each label represents the content in each segment). Given the varying lengths of videos in the dataset – which average 32.35 seconds in the 40-video collection – the system applies a scalar to modify playback time according to each segment’s confidence level (see Video 2), even allowing for looping if needed. For consistency in the demonstration, we started by cropping each of the 10 selected videos to the same length.

Video 1: Sequence of 10 videos with same length.

Video 2: Sequence of 10 videos with length from ViT confidence.

The system was developed across two independent computer units with:
– NVIDIA RTX 3070/3090 graphics cards (for GPU computing with CUDA support; intensive computing tasks are significantly faster once parallel processing work is offloaded to the GPU).
– GGML tensor library (for deep learning processing) (74)
– ImGUI (Immediate Mode Graphical User Interface), a library for developing user interfaces).
– All run on Windows 11, with FFMPEG encoders for video processing.

The system deploys trained models, specifically the ViT vision-transformer (75) and GPT-J-6B transformer, which are built to operate on CPUs instead of GPUs, implying lower power consumption. Additionally, we tested the use of prediction lists, where models communicate anticipated outcomes, over HTTP via ethernet. This setup, using TP-Link Archer AC1900 and TL-SG108 (LAN), enables efficient data exchange between devices, supporting research on separate computing instances, such as those used for graphical sequence tasks.

The GUI interface was developed to display key variables for the sequence outputs illustrated in Figure 8. It includes:
– A 10×10 trainable matrix grid visually representing all keyframes from the video dataset, with colour coding to indicate ViT confidence levels, ranging from ‘0’ (blue) to ‘1’ (red).
– A set of callback functions (predefined response triggers) for inspecting and processing system operations.
– A visualisation of the prediction list, showing attributes for each video dataset.

Figure 8: Lightweight GUI for human input and monitoring of the video dataset.

DISCUSSION

Interest in the audiovisual and film industries often centres on using automatic systems for recommendations. Employing automatic sequencing and editing systems as a tool for professional editors or other post-production roles can be valuable for brainstorming. Adjusting such AI systems enables practitioners to explore metacreation, where the algorithm influences the final creative work (76). LLMs and other trained generative models can be used for video editing when a common representation is achieved across separate models with distinct functions (77).

Our research views video editing as a sequence-based process, where we developed an initial but complete AI processing chain from video dataset input to finalised video sequence output, utilising open-source technologies and testing with publicly available videos. It is now common practice to search through embedding representations to recommend semantically relevant content across various video tasks. This research implements a method for achieving semantic consistency across timelines using natural language, enabling the design of trajectories and further computations within embeddings that mirror real editing behaviours.

Prior to this research, other approaches to video editing focused on transforming an input video into something different through AI guidance (video-to-video) rather than sequencing distinct elements (78). We leverage the reasoning capabilities of Transformers, demonstrated in applications like GPT-powered customer service, as the basis for montage and future element selection. This approach provides mechanisms that allow practitioners to engage with sequencing methodologies interactively (79).

Our literature review identifies several valuable use cases where conversational agents (interactive AI systems) are applied in video analysis and image generation, informing our approach to processing video signals. Transformer-based systems can benefit from the memory flows and storage capabilities of conversational agents within a larger system, enabling the modelling of useful representations from video data (80).

The outcomes of this research are framed within the broader context of video editing systems and AI, as explored in the AI conversational agent domain (81). Here, AI systems act as processing nodes, where we define behaviours and working modules that interact based on specific tasks or natural phenomena, such as constructs of long-term and short-term memory, recursive thinking, and pattern recognition (82).

The proposed system can be developed to accept incoming video feeds, write data to disk, and display text on screen without relying on traditional subtitles. Leveraging the performance of contextualising video frames with GPT-J-6B, our results demonstrate the potential to create narrative arcs by defining search modes through keyframes and deriving natural language context from each video clip. The quick graph evaluation (a lightweight computation of node connections) on the CPU enables models to operate on timed triggers, allowing the system to function in complementary real-time applications, such as keyframing new video inputs.

We envision this design as part of a larger, interconnected AI ecosystem capable of producing meaningful representations of video datasets. By coordinating two trained transformers, our methodology also allows for indexing specific sections of larger video datasets through diverse frame selection processes. Image keyframes can be automatically generated from video datasets by using methods such as pixel activity thresholds, colour space analysis, or movement detection. This opens up further research possibilities.

The principles of editing and montage extend to the psychology behind the film experience— how images evoke emotions and sensations in viewers, and how sequences can be composed to enhance that impact (83). Editing also involves reasoning within a framework of similarity, where each subsequent state resembles the current one. This approach is particularly relevant for editing feature films and documentary films, which might span hours.

In this research, we explore reasoning through the sampling of a transformer model over time, reflecting the cognitive processes involved in video editing, with a focus on professional editing system architectures and work environments. The selection of elements from a set of videos can be viewed as the AI system’s current state, analogous to human thought, anchored in known input contexts and described through natural language, which evolves alongside the video frames.

By continuing to develop the system to encompass cognitive architecture processing flows, it becomes possible to formalise and study specific PSI theories — addressing not only cognitive processes but also motivational and emotional ones (84). This interdisciplinary approach can foster a more holistic understanding of montage as a cognitive process when shaped by AI. These representations can help convey specific intended messages within video edits and reveal cultural artefacts — underlying societal values, norms, or biases embedded in trained models.

CONCLUSION

This article details the design and implementation of a language-based transformer video editing system, positioned within cognitive architecture literature. Our proposal leverages ViT and GPT-J-6B models to generate embeddings — mathematical mappings that capture the semantic information of video datasets — which are applied to select video segments that logically follow one another.

The pipeline we developed structures this process by establishing a common natural language foundation for sequencing any chosen video dataset. It involves computing trajectories within an embedding space, meaning it creates paths of semantic relationships between data points. Additionally, it crafts narrative arcs — structured storylines — through subtitles and context for each video clip. Moreover, it determines the duration of each video segment based on AI-defined confidence levels, meaning the segment lengths are adjusted according to the model’s certainty in its interpretations.

Our research aligns with other AI applications in the film and audiovisual industries, focusing on video understanding and automatic sequencing. Through this work, we demonstrate that Transformers can effectively function as semantic similarity engines for sequencing video datasets using nonlinear timelines in natural language. Our approach centres on transition states, designed to support real-time text generation and image description. We document a series of experiments using a small 40-video dataset and prompt inputs, with sets of 10 or 20 videos sequenced based on semantic similarity, image-text alignment during transitions, subtitle generation, and the initial representation of the dataset.

Connecting trained models helps video sequences keep consistent meaning over a scale of seconds. Our AI system allows users to plan sequences by rearranging video clips and previewing the edited sequence based on text prompts. This method also makes it easy to organise collections of different shots using natural language and search.

ACKNOWLEDGEMENTS

Research leading to these results was conducted during the individual research grant 2020.07619.BD and financially supported by the Portuguese Foundation for Science and Technology (FCT) with host institution INESC-TEC, University of Porto and a visiting role at the UAL Creative Computing Institute.

REFERENCES AND NOTES

1. Niels Taatgen and John R Anderson, “The Past, Present and Future of Cognitive Architectures”, Topics in Cognitive Science 2, No. 4, pp. 693-704 (2010).
BACK

2. John E Laird and Paul S Rosenbloom, “The Evolution of the Soar Cognitive Architecture”, in Mind matters (New York: Psychology Press, 2014) pp. 1-50.
BACK

3. Y LeCun, B Boser, JS Denker, D Henderson, RE Howard W Hubbard and LD Jackel, “Backpropagation Applied to Handwritten Zip Code recognition”, Neural Computation 1, No. 4, pp. 541-551 (1989).
BACK

4. Anders Krogh, “What Are Artificial Neural Networks?”, Nature Biotechnology 26, No. 2, pp. 195-197 (2008).
BACK

5. H Chen, B Perozzi, R Al-Rfou and S Skiena, “A Tutorial on Network Embeddings”, arXiv preprint arXiv:1808.02590 (2018).
BACK

6. A Vaswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser and I Polosukhin, “Attention Is All You Need”, NIPS – Advances in Neural Information Processing Systems Conference Proceedings (2017).
BACK

7. Definition extracted from <https://www.ibm.com/topics/large-language-models>, accessed 19 September 2024.
BACK

8. D Li, J Li, H Le, G Wang, S Savarese and S Hoi, “Lavis: A Library for Language-Vision Intelligence”, arXiv preprint arXiv:2209.09019 (2022).
BACK

9. L Engeln, N Long Le, M McGinity and R Groh, “Similarity Analysis of Visual Sketch-Based Search for Sounds” AM – International Audio Mostly Conference Proceedings (2021).
BACK

10. A Potapov, I Zhdanov, O Scherbakov, N Skorobogatko, H Latapie and E Fenoglio, “Semantic Image Retrieval by Uniting Deep Neural Networks and Cognitive Architectures”, Artificial General Intelligence – Lecture Notes in Computer Science() 10999, pp. 196–206 (Springer, 2018).
BACK

11. Dongkyu Choi, “On Using Generative Models in a Cognitive Architecture for Embodied Agents”, AAAI Symposium Proceedings 2, No. 1 (2023).
BACK

12. AS Koepke, AM Oncescu, JF Henriques, Z Akata and  S Albanie, “Audio Retrieval with Natural Language Queries: A Benchmark Study”, Transactions on Multimedia 25, pp. 2675-2685 (IEEE,2022).
BACK

13. T Lin, Y Wang, X Liu and X Qiu, “A Survey of Transformers”, AI Open 3, pp. 111-132 (2022).
BACK

14. Valentin V Klimov and David J Kelley, eds., Biologically Inspired Cognitive Architectures 2021: Proceedings of the 12th Annual Meeting of the BICA Society (Springer Nature, 2022).
BACK

15. A Bubic, D Y Cramon and R Schubotz, “Prediction, Cognition and the Brain”, Frontiers in Human Neuroscience 4 (2010).
BACK

16. Allen Newell, Unified Theories of Cognition (Harvard Univ. Press, 1994).
BACK

17. Lin et al. [13.]
BACK

18. Taatgen and Anderson [1.]
BACK

19. ibid.
BACK

20. John R Anderson, The Architecture of Cognition (New York: Psychology Press, 2013).
BACK

21. S Helie, N Wilson and R Sun, “The Clarion Cognitive Architecture: A Tutorial”, Annual Meeting of the Cognitive Science Society Proceedings 30, No. 30 (2008).
BACK

22. Cecilia Heyes,“New Thinking: the Evolution of Human Cognition”, Philosophical Transactions of the Royal Society B 367, No. 1599, pp. 2091-2096 (2012).
BACK

23. M Frixione, G Spinelli and S Gaglio, “Symbols and Subsymbols for Representing Knowledge: a Catalogue Raisonne”, International Joint Conference on Artificial intelligence Proceedings 1, p. 3-7 (1989).
BACK

24. Laird, Rosenbloom and Newell [2.]
BACK

25. ibid.
BACK

26. R Jones, J Laird, and P Nielsen “Automated Intelligent Pilots for Combat Flight Simulation”, AI Magazine 20, No.1, pp. 27-27 (1999).
BACK

27. Heyes [22.]
BACK

28. Anderson [20.]
BACK

29. John R Anderson and Gordon H Bower, Human Associative Memory (Psychology Press, 2014).
BACK

30. Gerhard Weber, “ACT – Adaptive Control of Thought”, Encyclopedia of the Sciences of Learning (Boston: Springer, 2012).
BACK

31. Anderson [20.]
BACK

32. S Franklin and F G Patterson, “The LIDA Architecture: Adding New Modes of Learning to an Intelligent, Autonomous Software Agent”, IDPT – Integrated Design and Process Technology Conference Proceedings (2006).
BACK

33. S Franklin, T Madl, S D’Mello and J Snaider, “LIDA: A Systems-Level Architecture for Cognition, Emotion and Learning“, Transactions on Autonomous Mental Development 6, No. 1, pp. 19-41 (IEEE, 2013).
BACK

34. Bernard J Baars, “The Conscious Access Hypothesis: Origins and Recent Evidence”, Trends in Cognitive Sciences 6, No. 1, pp. 47-52 (2002).
BACK

35. Helie et al. [21.]
BACK

36. B Goertzel, C Pennachin and N Geisweiller, Engineering General Intelligence, Part 2: The CogPrime Architecture for Integrative, Embodied AGI (Springer, 2014).
BACK

37. A Doulamis, D Kosmopoulos, M Sardis and T Varvarigou, “An Architecture for a Self-Configurable Video Supervision”, ACM – Analysis and Retrieval of Events/Actions and Workflows in Video Streams Workshop Proceedings (2008).
BACK

38. Jose C Principe, “A Cognitive Architecture for Object Recognition in Video”, ICMLA – International Conference on Machine Learning and Applications Proceedings. (IEEE, 2020).
BACK

39. H Li , R Dou, A Keil  and J C Principe, “A Self-Learning Cognitive Architecture Exploiting Causality from Rewards”, Neural Networks 150, pp. 274-292 (2022).
BACK

40. Klimov and Kelley [14.]
BACK

41. H Li, Y Ma, and J Principe, “Cognitive Architecture for Video Games”, IJCNN – International Joint Conference on Neural Network Proceedings (IEEE, 2020).
BACK

42. Z Cai, B Goertzel and N Geisweiller, “Openpsi: Realizing Dörner’s “PSI” Cognitive Model in the Opencog Integrative Agi Architecture”, Artificial General Intelligence – Lecture Notes in Computer Science() 6830, pp 212–221 (Springer, 2011).
BACK

43. A Augello, I Infantino, G Pilato, R Rizzo and F Vella, “Introducing a Creative Process on a Cognitive Architecture”, Biologically Inspired Cognitive Architectures 6, pp. 131-139 (2013).
BACK

44. Joscha Bach, Principles of Synthetic Intelligence PSI: an Architecture of Motivated Cognition (New York: Oxford University Press, 2009).
BACK

45. P K Mital, M Grierson, and T J Smith, “Corpus-Based Visual Synthesis: an Approach for Artistic Stylization”, ACM Applied Perception Symposium Proceedings (2013).
BACK

46. ibid.
BACK

47. A Chella, M Cossentino , S Gaglio and V Seidita, “A General Theoretical Framework for Designing Cognitive Architectures: Hybrid and Meta-Level Architectures for BICA”, Biologically Inspired Cognitive Architectures 2, pp. 100-108 (2012).
BACK

48. LeCun [3.]
BACK

49. Vaswani [6.]
BACK

50. Monika Schak and Alexander Geppert, “A Study on Catastrophic Forgetting in Deep LSTM Networks”, Artificial Neural Networks and Machine Learning – Lecture Notes in Computer Science() 11728, pp 714–728 (Springer, 2019).
BACK

51. A Dosovitskiy, L Beyer, A Kolesnikov, D Weissenborn, X Zhai, T Unterthiner, M Dehghani, M Minderer, G Heigold, S Gelly, J Uszkoreit and N Houlsby, “An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale”, arXiv preprint arXiv:2010.11929 (2020).
BACK

52. Ibid.
BACK

53. Laird, Rosenbloom and Newell [2.]
BACK

54. Ron Sun, “Can A Cognitive Architecture Fundamentally Enhance LLMs? Or Vice Versa?”, arXiv preprint arXiv:2401.10444 (2024)
BACK

55. T Sumers, S Yao, K Narasimhan and T Griffiths, “Cognitive Architectures for Language Agents”, arXiv preprint arXiv:2309.02427 (2023).
BACK

56. YS Xu, TJ Fu, HK Yang and CY Lee, “Dynamic Video Segmentation Network”, IEEE – Computer Vision and Pattern Recognition Conference Proceedings (2018).
BACK

57. X Zhang, Y Li , Y Han and J Wen, “AI video editing: A survey”, Preprints 2022, No. 2022010016 (2022).
BACK

58. K Zhang, G Riegler, N Snavely and V Koltun, “Nerf++: Analyzing and Improving Neural Radiance Fields”, arXiv preprint arXiv:2010.07492 (2020).
BACK

59. For recent findings on human and camera motion reconstruction from in-the-wild videos, see M Kocabas, Y Yuan, P Molchanov, Y Guo, M Black, O Hilliges, J Kautz and U Iqbal, “PACE: Human and Camera Motion Estimation from in-the-wild Videos”,  3DV – 3D Vision Conference Proceedings (IEEE, 2024).
BACK

60. Y Tonomura, A Akutsu, Y Taniguchi and G Suzuki, “Structured Video Computing”, IEEE Multimedia 1, No. 3, pp. 34-43 (1994).
BACK

61. Hirotada Ueda and Takafumi Miyatake, “Tree Structure Operation for Video Editing Utilizing Image Recognition Technology”, in INTERACT. – IFIP Human-Computer Interaction (Boston: Springer, 1997) pp 517–523.
BACK

62. Frank Nack and Alan Parkes, “The Application of Video Semantics and Theme Representation in Automated Video Editing”, Multimedia Tools and Applications 4, 57-83 (1997).
BACK

63. Frank Nack and Alan Parkes, “AUTEUR: The Creation of Humorous Scenes Using Automated Video Editing”, International Joint Conference on Artificial Intelligence – Workshop on AI and Entertainment Proceedings 21 (1995).
BACK

64. Retrieved from Nack and Parkes [63.]
BACK

65. B Aerts, T Goedemé and J Vennekens, “A Probabilistic Logic Programming Approach to Automatic Video Montage”, Frontiers in Artificial Intelligence and Applications 285, pp. 234-242 (2016).
BACK

66. Nuno Vasconcelos and Andrew Lippman, “Bayesian Modeling of Video Editing and Structure: Semantic Features for Video Summarization and Browsing”, ICIP – International Conference on Image Processing Proceedings (IEEE, 1998).
BACK

67. See the Kapwig Software <https://www.kapwing.com>, accessed 22 August 2024.
BACK

68. B Wang, Y Li, Z Lv, H Xia, Y Xu and R Sodhi, “LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing”, Intelligent User Interfaces Conference Proceedings, pp. 699 – 714 (2024).
BACK

69. H Ma, S Mahdizadehaghdam, B Wu et al. “Maskint: Video editing via interpolative non-autoregressive masked transformers”, CVF – Conference on Computer Vision and Pattern Recognition (IEEE, 2024).
BACK

70. P Sarmento, A Kumar, D Xie, CJ Carr, Z Zukowski. and M Barthet, “Shredgp: Guitarist Style-Conditioned Tablature Generation”, arXiv preprint arXiv:2307.05324 (2023).
BACK

71. R Lyle Skains, “Creative Practice as Research: Discourse on Methodology”, Media Practice and Education 19, No. 1, pp. 82-97 (2018).
BACK

72. X Chen, J Liu, Z Liu, L Wan, X Lan and N Zheng, “Knowledge Graph Enhancement for Fine-Grained Zero-Shot Learning on ImageNet21K”, IEEE Transactions on Circuits and Systems for Video Technology (2024).
BACK

73. L Gao, S Biderman, S Black, L Golding, T Hoppe, C Foster, J Phang, H He, A Thite, N Nabeshima, S Presser and C Leahy, “The pile: An 800gb dataset of diverse text for language modeling”, arXiv preprint arXiv:2101.00027 (2020).
BACK

74. A Karapantelakis, M Thakur, A Nikou, F Moradi, C Olrog and F Gaim, “Using Large Language Models to Understand Telecom Standards”, ICMLCN – International Conference on Machine Learning for Communication and Networking Proceedings, pp. 440-446 (IEEE, 2024).
BACK

75. Dosovitskiy et al. [51.]
BACK

76. Pasquier, Philippe, et al. “An introduction to Musical Metacreation”, Computers in Entertainment 14, No. 2, pp. 1-14 (2017).
BACK

77. B Wang, Y Li, Z Lv, H Xia, Y Xu and R Sodhi [68.]
BACK

78. TJ Fu; X Wang; S Grafton; M Eckstein and W Wang, “M3l: Language-Based Video Editing Via Multi-Modal Multi-Level Transformers”, CVF – Computer Vision and Pattern Recognition Conference Proceedings (IEEE, 2022).
BACK

79. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal et al., “GPT-4 Technical Report”, arXiv preprint arXiv:2303.08774 (2023).
BACK

80. Sumers et al. [55.]
BACK

81. Doulamis et al. [37.]
BACK

82. H Li, Y Ma, and J Principe [41.]
BACK

83. Jeffrey Zacks, Flicker: Your Brain on Movies (Oxford Univ. Press, 2015).
BACK

84. Alexei V Samsonovich, “Emotional Biologically Inspired Cognitive Architecture”, Biologically Inspired Cognitive Architectures 6, pp. 109-125 (2013).
BACK

ABOUT THE AUTHORS, REVIEWERS AND EDITOR

Authors

Luís Arandas > University of Porto

Researcher and designer focusing on algorithmic systems, digital tools and experimental media. His work combines creative practice with critical inquiry into computational methods, exploring how code, automation and generative processes influence interactive forms and emergent audiovisual expression in contemporary digital culture.

Pedro Sarmento > Queen Mary University of London

Musician and media artist working across guitar performance, computer-mediated music and creative coding. His projects span sound experimentation, interdisciplinary collaboration and real-time systems, investigating how technology can extend musical practice and shape new modes of performance, composition and interactive artistic production.

Mick Grierson > University of the Arts London

Professor of Computer Science and co-founder of the UAL Creative Computing Institute. His research bridges creative AI, interactive media and machine learning, contributing foundational work in early deep-learning systems for audio, video and performance. He collaborates widely with cultural institutions and industry on innovation in computational creativity.

Miguel Carvalhais > i2ADS / University of Porto

Designer, musician and researcher whose work spans computational art, sound and design. As professor and director of the Faculty of Fine Arts at the University of Porto, he integrates artistic experimentation with analytical inquiry, exploring how digital systems shape aesthetic processes, performance and contemporary creative practice.

Reviewers

Kate Sicchio > Virginia Commonwealth University

Artist, choreographer and researcher exploring how movement, code and embodiment intersect in live performance. As Assistant Professor at Virginia Commonwealth University, she connects creative practice with research in interactivity, dance technologies and algorithmic performance, examining how bodies and systems co-construct expressive and computational forms.

Nuno Correia > Tallinn University

Researcher and media artist whose work spans sound, multisensory experience and embodied interaction. As Associate Professor in Digital Transformation at Tallinn University, he coordinates Creative Europe projects linking dance, technology and AI, investigating how audiovisual systems and interactive media shape contemporary digital culture and artistic practice.

Editor

Adriana Sá > CICANT / Lusófona University Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 2:

Long ago in the pages of Hand’s End: Technology and the Limits of Nature (1), I talked about how artificially tired I would feel after playing a Yamaha WX7 wind controller, a MIDI instrument with the look and feel of a soprano saxophone; MIDI stands for Musical Instrument Digital Interface. With a bunch of buttons you could effortlessly play six octaves up and down. Blow harder on it and the sound gets louder. Easily move your fingers around to create blistering arpeggios! Virtuosity? Power? These things are still around, but as expressive instruments they never really caught on (2).

My apologies if the language may at times seem more colloquial and opinionated than the readers of scientific publications are used to. But the scope of this journal extends beyond usual formats, and this particular contribution reports personal creative experience. I will expose why I enjoy playing a touch screen interface in the making of electronic music, and further explore why I like taking it out in the woods to play with birds and ponds.

So. The artificial tiredness with the MIDI Yamaha WX7 comes with the uncanny feeling that one is playing too much, making more sounds more easily than one should be able to do. It has been more than thirty years since that book of mine, and the world of music is endlessly more effortless than then. Ask your phone to play any piece of music and you got it right away. Why even play anything new? Well, that is another question entirely, a bigger one perhaps, but let’s stick to the playing of electronic instruments. Playing them is one of the toughest things to do with them, since by the time you have mastered one machine it has become replaced with something newer, faster, shinier, hipper, more expensive. Compare that to a cello or piano, an instrument carefully refined over the generations, with a long tradition of how to study, learn, practice and perform it. Compared to that march of centuries’ development, I feel that the swift rise of electronics can only be a passing fad, a rise and fall of the in and the out. Hear an electronic sound on a recording, and nothing dates it more easily. The trendiness of these sounds comes and goes. A wind-controller nearly always brings one back to the 1990s.

Or does it not? Open up a DAW – a Digital Audio Workstation – like Ableton Live, Logic, or Bitwig and the tools you see are a panoply of nostalgia. The realm of possible electronic tones to play is a veritable history of electronic music, and of our puzzling attempts to use technology to produce tones and rhythms that sound musical. All methods of digital sound synthesis are theories put into practice, through possibilities that can be programmed or played in the moment:  emulations of analogue synthesis, which attempt to simulate sound waves generated by rapidly changing voltages; additive synthesis, which generates complex sounds by combining multiple sine waves of different frequencies; FM synthesis, where one audio signal modulates the pitch of another; wavetable synthesis, a form of sample-based synthesis that operates based on digitally recorded sound; sampling; physical modelling, which uses algorithms to simulate a physical source of sound; and granular synthesis, another form of sample-based synthesis, which operates on the microsound time scale. It is not my place to explain these methods here, but there are good summaries online (3).

I sense that today’s electronic music wants to be nostalgic — we smile when a particular sound takes us back to an earlier vision of the future. We no longer try to be ever more realistic in our emulations of acoustic instruments. We recognise that machines offer unique possibilities of expression.

The way we play these sounds might not have changed so much: many people use physical devices with keys resembling a piano or accordion to play all these emblems of history. I would say that keyboards favour too many notes, rather than super-emotional single notes… This is perhaps why the electric guitar, not the piano or organ, became the signature instrument of the amplified age. More emotion, soul, passion, and power! Bend, scream, distort, enlarge, all those amazing things that electricity can do.

Anyway, now we might have reached somewhere else in music, an era of precise refinement in our listening to artificial sound. We may have become connoisseurs of the digital sound, and favour its crispness more than any original warmth. At least the different types of distortion and emulation are now far better known than when people first heard all these effects, say when Les and Mary Paul ran their guitars through the “LesPaulverizer” on prime time TV (4), and people had no idea what was going on. At least we can say that through twenty-four hour immersion in technological media we are all electronic musicians now.

I do not want to talk about the whole transformation of our lives and our art into ‘media’, but about something more specific: how touching a screen, something most of us do hundreds of times a day, can actually become a fine way to make music; an expressive and powerful interface that feels far more direct than those which try to emulate old acoustic instruments. Precisely because it is so far-out, long removed from our expectations as to what constitutes a musical instrument, touch screen interfaces can be surprisingly spontaneous and musically expressive.

I am surprised that little has written about this topic (5), except to point out why the touch screen is such a bad musical interface (6). It is supposed to be bad for the same reasons the wind controller artificially tired me out — there’s no there there. Touch the screen, anything can come out. Yes, this is still the case. But because every touch screen musical interface is designed differently, we can make the abstract concrete. We can actually play it.

Having music that is playable has always been important to me. I don’t want my music to be programmed or produced, but always played. And I have long wanted it to draw from nature.

FROM SOUNDS OF NATURE TO ELECTRONIC MUSIC

Paradoxically, it was the sounds of nature that drew me to electronic music in the first place. I used to prefer acoustic sounds to electronic ones, somewhat agreeing with pianist Colin Farish who once told me “life is too short to mess around with electronics.” The reader might rightfully prefer bibliographic references to random comments from friends, but hey, that one has stuck with me for many years, and he does have a point that many acoustic musicians voice. Thinking of how much time I had spent programming confusing synthesisers like the Yamaha DX7 or the Serge Modular… All the possibilities consumed me. How could one ever settle well enough on the sound to play? I guess that what appealed to me in these machines was that each one seemed to embody a different philosophy of sound. As a philosopher, I often preferred the questions to the answer. Never mind what music I might play — think of the library of all possible musics that could be played at any moment.

That is what the synthesiser offered me, a repository of the unknown. On the other hand, it always seemed incomplete and unknown, empty when compared to the known. It seemed always easy to tell what was artificial from what was natural. A good reason to stop messing with it, for many. But somehow I did not want to give up.

I changed my mind about that thirteen years ago, when I started making music with the sounds of insects. The way scientists talk about their stridulating, describing carrier frequencies and frequency modulation (7), sounds just like manuals explaining how synthesisers work. These bugs are little oscillators, generating oscillatory electric currents out there in the trees, sometimes millions of them vibrating at once.  I will tell more about this later.

Since I play these electronic sounds out in nature, together with the live, actual sounds of the more-than-human environment, it is natural to connect this work with the rapidly expanding field of acoustic ecology. I gladly support that; the investigations and creations of acoustic ecologists are very relevant. However, some people might feel that my activity is quite distinct: my desire to join in and make a music that no one species can make alone does not imply an attempt to understand the soundscape as completely as possible in order to best preserve it.

THE CREATIVE POTENTIALS OF CERTAIN APPS ON THE IPAD

Thirteen years ago, iPads had just been invented and I just wasn’t sure what I wanted to do with such a thing. There were plenty of early apps that made sounds, but it was hard to find one that was good in the touch screen format.

In the following sections I will focus on what makes certain iPad music apps more playable than others, and expose why I still have not found the app I am looking for; I probably never will. Later I will explain why I continue to bring this abstract minimalist machine out into the wilds, to play along with natural environments – the perplexing secret underwater sounds of ponds.

You will see multiple videos shot in the studio, and you might think “aren’t there enough demo videos of all these apps online, why do we need more?”. But actually, I find most of those videos not especially helpful. The people who make them either talk too much or just demonstrate the preset sounds of an app. Here I want to show some of the surprise and joy I feel in playing them, and that is why I include one video for each app. Most of them mix multiple methods of synthesis, except one that is specifically dedicated to additive synthesis. You can decide whether it seems like I play each app in the same way, or differently. I sense that the theory of sound embodied in each app leads me to use each one differently, and express musical ideas that I would not come up with if I were using another tool. That is the sign of a successful musical instrument. Anyway, I do not want to make this essay too much of a review of these instruments, but more an analysis of my interaction with them.

THE ANIMOOG, THE CICADAS AND A MUSICAL DISCOVERY

The first app that grabbed me on the iPad was Animoog, made by the company that made synthesisers a household name. What is special about this instrument is that from its very design it promoted a kind of retro-futurism, allowing you to bend parameters of the sound with your fingers as you played the keyboard. This proved to be a perfect way to play along with millions of cicadas that appear periodically, not every year; they had just emerged in Columbia, Missouri, where I heard them as music for the first time.

Trying to join in with all the natural ‘noise’ I began to understand the connection between electronic music and nature for the first time. Though it was probably years of listening to electronic music that had taught me that the drones, noise, and hissy beats of the various Magicicada species could be heard as music (8).

The Animoog may have first caught my attention because cicadas love it (Fig. 1).

Figure 1: playing Animoog on an iPad with a seventeen year periodical cicada, the Magicicada septendecim. New Jersey 2013. Photo by Asher Jay.

There are so many other reasons for it to remain the single best live musical instrument on the iPad. The design is not the ‘flat aesthetics’ now favoured by IOS and Android, but something slightly steampunky or oscilloscope like, suggesting Forbidden Planet or 1950s visions of the future. And its graphical interface enables several unique forms of modulation. It is just fantastic to play and its sound really cuts through other instruments when you increase the signal level on the filter setting or the master output.

Animoog puts a particular design philosophy into practice, showcasing playability and history more than minimalism. It looks and sounds like what you do not expect on the screen —  old-fashioned looking, with a weird oscilloscope interface, but with a precise sound that is always a bit louder than one expects. The mixer of the graphic pad, the keyboard based on the legendary Moog-Eaton Multi Touch Sensitive keyboard (a precursor of today’s MPE keyboards), and the knobs for parameters such as ‘filter’ and ‘detune’, all easily accessible on one screen – that means that this instrument suggests a way to use the whole touch screen, as shown in Video 1.

Video 1: Introduction and demonstration of the Animoog app on an iPad. Video by Jaanika Peerna.

Note that a few years ago Moog Music basically retired the original Animoog and replaced it with Animoog Z, whose name invokes the “Z” axis from the Cartesian coordinate system, used to define an object’s position in the three-dimensional space – losing the retro-futurism, adding a third “Z” dimension to modulation, and creating separate screens for effects including the filter and detune that I loved having on the one screen (9). My response to this new version has always been “you ruined it”, but to be more charitable I would have to say “I have not really learned how to better play this version”. It does offer some extra features, like arpeggios (notes of a chord played in rapid succession) and that elusive “Z”. Sure, press harder and harder on the screen, you will not get beyond it. This newer version simply does not have the spirit of the original. Newer is not necessarily better, beware the pressure to trade in and get the latest of everything. Remember that in electronic music, like so much else his about history, sometimes you shouldn’t fix what isn’t broken.

ON THE INTERACTION WITH SELF-PLAYABLE MACHINES

Among the tools of electronic music that help an instrument play itself are ‘hold’ keys, arpeggiators for creating an arpeggio when you press down a chord, and sequencers for storing and transmitting sequences of musical notes, chords, or rhythms. These tools let the sound continue while you go on to do something else. There is a long tradition of self-playability, of music going on automatically when the player stops. But what apps (I’d really rather call them instruments) encourage the machine to play itself in new ways?

The simplest solution is often the most brilliant. Though it cannot be too simple. You want something that a five year old or a fifty year old can play. Where you can read and adjust to learn more, or you can just pick it up and make sound.

Waving a pure glass and metal rectangle in the air, unconnected to any wire, is an act of abstraction, a pure postmodern pulse of the Platonic ideal form, a symbol for humanity as a whole, rectangle against flow, culture against nature, the machine in the garden, the silicon dream in the wilderness, minimalism in the midst of maximal mush. I laugh when I play it in the sun and it shuts down in the heat, announcing, “operating conditions are dangerous”.

THE SAMPLR APP AND MY DRIVE TO DANCE WHILE PLAYING THE IPAD

There is something relentless in the pure possibility of the iPad, harsh in its appeal. Especially with another app I worked with, the vowelless eponymous Samplr, which has undoubtedly the best design of any touch screen music app ever invented. So simple that a six year old can figure it out in minutes, so subtle that you can spend years learning to play it. It is a true instrument that looks and feels like no other.

Unique is the fact that there are no words on the screen, just symbols that you can study or intuitively figure out on your own. Six separate samples can be morphed, effected, sliced, remixed, transposed and sequenced, live individually or all on top of each other, in synch or not. Other apps offer a similar functionality, but they do not work as naturally for live on-the-spot playing. I hardly ever play this one with rhythmic beats in sync with each other. I prefer the uncertain, swirling overlap like the patterns of actual life. The included effects are simple, but efficient… They do the job. You can strap it together with more complex effects if you like, but on an iPad this interconnection offers more opportunities for things to go wrong, and technology out in the field must be reliable above everything else.

We are talking about the interface, which includes digital and physical components. The interface must be playful and inspiring beyond being reliable. And it cannot always sound the same. For me it works best when it is not too far away from a vision of playability: when I can play it and move (see Video 2).

Video 2: Samplr app played on the iPad. Video by Jaanika Peerna.

That is also true; I rarely stand still… The machine should speed me up and not slow me down. Inspire me to dance and not to ossify into a screen potato. Watch me and you’ll see…

All right, he moves, but doesn’t he move better playing the clarinet or another wind instrument? Are those acoustic sounds not more authentic, more real? Yes, I think they are… But for me, life is looooooong enough to mess around with electronics.

SHOOM AND A FEW CONSIDERATIONS ABOUT UNPREDICTABILITY

Other apps are simpler than Samplr. One called Shoom is all about placing round dots on a grid, and balancing volume against pitch — each dot can be a tone, noise or rhythm, three separate two-oscillator synths, so three different sounds together at a time, but up to thirty-two possible tones and volumes… The drone possibilities are staggering, and the separation from usual structures of music are very clear. These open-ended dots blow apart my sense of how sounds can be structured, and beautiful experiments come out.

Shoom is far less famous than Animoog, in fact I know few other people who play it.  Basically you touch your finger to the screen, you create an object, a tone that makes a continuing sound.  Could be a drone, could be a rhythm, you can have as many of these screen sound objects as you want, they just stay there continuously. You can move them around, change pitch, volume and other parameters. It feels like having an infinite number of sound creatures to play around in your sound forest — a direct analogy to a screen full of beings, which you play in a whole new way, by moving them around the field of sound.

Forgetting that a sound is playing when you switch to another app is a problem if you have them all open at once, instead of weaving them altogether using a digital connection hub like Aum, or a DAW like Nanostudio, Auria, Logic or Cubasis. You just have to pay attention to the listening and remember what you are doing, while keeping your whole process in your head. Or weave all the apps together. But that sets up more things that can go wrong when playing on the fly, especially outdoors. The simpler the setup the better, and this is where the iPad excels… A purely virtual tabula rasa of an instrument where you wave your hands over a flat rectangle to conjure infinite possibilities of how a musical instrument could be.

I admit it can seem strange, a man waving his flat screen in the air in the wilderness, communing with the music of underwater life. And yet the machine can release any sound, anything can be plugged into it and anything may come out. The flexibility is… daunting. The familiarity is… crushing (see Video 3).

Video 3: Shoom played on an iPad. Video by Jaanika Peerna.

Simplicity and familiarity do not cancel unpredictability. For example, one can easily forget that the Shoom app is running in the background, as it does not like to turn itself off. Additionally, if I am outside and it is slightly rainy, the water drops can play the instrument on their own. Which brings its own charms, of course…

What we are looking for…. Pushing, further, each app deeper into the unknown… After a while, why do they all sound the same? The electronic sound is ethereal, not directly connected to the touching of the screen that creates it… But why, why do we continue? Simply because it surprises us, and we may never be satisfied. As an improvisor when I start to play I am never sure what I’m going to play. The inadvertent machine pulls me ever farther from my oblique intentions or lack thereof. There is the known-unkown and the unknown-known.  I want to remain surprised by what I do or don’t do.

ADDICTIVE PRO AND ITS EFFECT ON BIRDS & BUGS

All right, let’s move on to the fourth synthesiser app, Addictive Pro. The name has a truthful reason-to-be: once you start playing it you cannot stop, it allows for so many discoveries. For example, I love a random roll of the dice button, where anything might happen. One parameter changes while others stay the same, or everything changes at once.

The name of this app also recalls additive synthesis, which means piling frequencies, harmonic (multiples of the fundamental frequency) or enharmonic (tones that sound the same but are notated, i.e. spelled differently). With so many frequencies this sound machine can really get birds and bugs responding. Many species are tuned to come alive only when they hear very particular frequencies (10), and if you layer many on top of each other you are more likely to hit a frequency that makes your responding species excited…. Even if humans may not bond with the sound, other creatures may be unexpectedly roused…

This app is extremely playable; it has basic features that suit well my restless mind. The graphical interface uses XY coordinates to adjust various parameters live, with two fingers. So many randomisation options. The opportunity to draw waveforms and mysterious “spectrums”, some kind of filter, and both volume and drive knobs clearly visible. These are the basic tools to tweak what you play in a live situation, and decide how much it needs to cut through a real-live sound situation in the outside world.

The additive synthesis method is one of the most common and pure-sounding methods in electronic music, probably because it is based on pure sine waves that modulate each other; the more frequencies are combined, the greater is the sonic complexity. Notoriously difficult to program. But with this app we can just draw through the waveforms with an ease the developers of the Fairlight could only have dreamed about! (11) – Video 4.

Video 4: Addictive Synth Pro on an iPad. It really is addictive. Video by Jaanika Peerna.

It never crashes, it is not famous, you should use this one, experiment endlessly. Notice how this video is longer than most of the others…. I just could not stop. It is crisper than Animoog, not as warm. More diverse and more easily interactive, but somehow with less personality and more variation. Every time I explore it I realize I should use it more. What keeps it from being the ultimate iPad performance app? I guess I wish it were based on samples, not only sine waves.

But what is interesting is that the animals seem to especially like it. I think this is because of the many frequencies piled up on top of one another, as I wrote above. One study demonstrated that cardinals preferred synthesised sine wave whoops to their own real ones (12). This suggests a kind of Platonic ideal inside their brains, about how their songs should sound. It is perhaps related with the sense in which some human listeners prefer electronic music, sine waves in particular, to the sounds of flutes and guitars that might ring close to them. We chase the ideal well. So why is it ultimately never satisfying?

We have to keep playing. After a while every sound from any electronic app starts to sound the same. So we jump to another one, where the interface is the thing that most distinguishes one from another, how it feels like to play.

GEOSHRED AND A FEELING ABOUT THE SIMULATION OF ACOUSTIC INSTRUMENTS

GeoShred is one of the most consistently praised iPad performance apps. I mostly use the violoncello tools, which use a technology called SWAM (Synchronous Waves Acoustic Modelling), supposedly to overcome the traditional limitations of sampling – in which each sampled sound can seem accurate but somewhat static, if the starting/ stopping/ connecting does not make it feel natural or expressive. These tools tend to do what is advertised:  make a recording of them and people always ask “who was playing cello?”; on a recording many things can be convincing. Do it live and people sometimes think I am playing a sample of a cellist.

The whole app is guitar-based, hence the ‘shred’ name. So, it often sounds guitaresque, with distortion, reverb, drive. Electrified. It is fun but too easy to play, in a way, like a game. And before playing, the configuration process feels too laborious. The app is deep with all kinds of things to be adjusted, but the layers of effects are exhausting to adjust. That is often a critique with these things: no one wants to endlessly program and adjust, it has to all work on the fly. It should look and sound cool, should impress and make you want to move (see Video 5).

Video 5: GeoShred playing a SWAM cello instrument on an iPad. Video by Jaanika Peerna.

Or maybe that does not matter, it’s music remember, focus on the sound man. The cello and flute SWAMs sound good. I don’t use the flute one so much because I play acoustic wind instruments. Cello is something I always wanted to play, so I like to add it in. Too bad the SWAMs are only in mono, heard as if it were emanating from one single position in the auditory space… (13) Who can be fooled… Expressivity, musicality, does it lie in the ear of the beholder?

I am not sure this app really twists my sense of what music can be, but it does fool the ear into cello-like effects, being produceable with the touch of a screen. Sometimes a touch cello is just a touch cello.

WORLD PIANO: PLAYING THE IPAD I DON’T WANT TO FORGET IT IS AN IPAD

Another twist-tie of an app is World Piano, which used to be called Taq.sim. Most notable here is that you can choose among a series of Middle Eastern Maqams – that is systems of pitches, melodic motives and patterns – to add a quarter-tone detuned but historically musical vibe to your sounds. There are many choices. It is basically a two-oscillator sampler (each patch can combine two distinct sounds), but I stick to piano-related sounds. They are easy to tweak because the app is made with Audiokit, an open-source iOS app-development platform whose results often share a common design language. With this app it is easy to adjust the attack, decay, sustain and release of the sample (i.e. its volume changes), and slightly filter the piano to make it sound more or less like the real thing. Detuning helps to retain a sense of the uncanny valley, a place between real instruments and simulations, so you don’t quite know where you are. I tend to play linearly, short melodies on this thing, though if I plugged in a keyboard I think I would like to experiment more harmonically and judiciously.

It is definitely not monophonic (see video 6).

Video 6: World Piano on an iPad, with a Hijaz descending scale. Video by Jaanika Peerna.

Playing the iPad I don’t want to forget it is an iPad. It is the ultimate in abstract musical controller art… Just a flat screen you touch and cannot explain, this is why the new and surprising can come out. These detuned scales shake up our equally tempered complacency, and we find an enhanced form of harmony and expression.

TC-11 AND THE APPEAL OF OTHERNESS

One of the earliest touch iPad instruments is still the strangest: TC-11. I do not know why it has that name. Visually it is the coolest of all these apps, in fact as you play it is hard to have any idea what action does what. The editing is secretive, hidden, mysterious. The sounds are loose and electronic, difficult to parse. It looks like a communication device from the future, a time when we speak in graphs and figures, and what counts as music becomes something far and beyond.

In a way I would most like to film this, to watch someone playing it, like videos of people playing games popular among six year olds. I do not know what it all means, how it all works. I am not sure I want to know – Video 7.

Video 7: The mysteriously named iPad app TC-11. Video by Jaanika Peerna.

The single greatest lack of this app is that there is no easy volume control slider on the main page. That makes it hard to trust in the field… Especially if several apps are playing at once.

FACTORY AND THE GAME OF SURPRISE 

Some years ago at Superbooth in Berlin I went up to the Sugarbytes booth and I saw an interesting synthesiser plugin being demonstrated, called Factory. It is basically an emulation of an analogue modular synthesiser setup with an undo button, hugely flexible. Seeing how it worked I told them at once, “you really should make this available on the iPad,” and they said, “we’re never going to do that”.

But it came out for iPad a few months later and it is one of the most powerful and unique touch-screen synthesisers ever invented. There are so many choices, so many variations, it can play itself or you can play it, there are so many levels of randomisation and customisation. It takes the number one spot as the most fully realised iPad sound app I use, endlessly surprising, and constantly changing my sense of what live electronic music can be.

Yes it can substitute a vast rack of analogue modules, and yes it has an infinite undo, and constantly sounds fresh and unknown, and like the best of them, it can cut through the mix and be heard.

Playing it is a bit like playing a game. I told the same about GeoShred, but mean it differently here. If you begin by wanting it to do one thing it soon does something else, then if in doubt, jump ship, just try something else, testing, listening, ready to give in to surprise – Video 8.

Video 8: Sugarbytes Factory on the iPad, like a modular synth with an undo button! Video by Jaanika Peerna.

That being said, it can easily just make too much sound, with too many effects and too much drone scratch and noise. Which can be said for a lot of electronic music. The answer: just turn it down, make it variable, sometimes turn it off. Have it play quietly in the background and turn up the live noise: birds, ponds, clarinets, flutes.

If you don’t like it, just pick the X or Y axis on the grid and rewire everything, in a moment. If you don’t like it: undo, go back. The previous state is still there. No other app has that flexibility.

PLAYING WITHIN NATURE

How to make any of these flat screen treasures be useful? Stop playing, take a break, be sure to listen where you are and how what you wish for may or may not fit in.

Do not be afraid to stop, pay close attention, remember that when the machine is in the garden nature will always win in the end, lasting longer than electricity, offering up the best and most eternal sounds.

Still we human are there, endlessly inventing, never satisfied, always looking for the next big playable thing — a tool to extend our desired into the abstract sonic world.

Up until now we have been testing this device in the studio, but I started by warning you I would be taking it out into the wilderness. Sacrilegious you might say, taking such a technology into nature. Why would nature want these sounds? It does not. Why do we want them? If we do, it is because they speak to an abstract purity in our essence, one that we want to test while in the field.

Lately, field recording has been critiqued for being colonialistic, imposing, extracting, taking nature as a soundworld to be mined for its beauty and inspiration (14). Sure, it can be all those things. But listening to nature and taking a piece of it home with you can also be innocent, expansive, beautiful. Most people who do it, from whatever culture, and most who listen to it, feel its value more directly before problematising it.

To me, joining in and morphing this into music seems more honest. As a musician I want to learn more from all possible sounds. As an improvisor I want to delve into new situations where I might engage with the unknown. What is the right or wrong unknown? Sometimes the most right are those sounds that have preceded us on this Earth, and those that will endure long beyond this brief human time. To celebrate this endurance, we must be prepared to dance and explore with the most unexpected of tools. That is why I continue to do this, and why it continues to grab me.

Video 9 shows what it is like to take this very abstract musical machine out into nature, and to play it along with the equally abstract underwater sounds made by real concrete beings: plants, insects, fish.

Video 9: The iPad in itself, out in the world, live along with natural, underwater sounds. Video by Jaanika Peerna.

Here, by the Pirita Jõgi river in Estonia, I demonstrate all that which has been introduced above.

First of all you hear the live sounds of the river, underwater, as picked up by an Aquabeat hydrophone. The regular rhythms come from plants, the grumbles from fish, and the occasional electronic buzzes and whirrs from water bugs who stridulate, making sounds by vibrating their penises against their bodies underwater.

What is exciting to me is that my time spent practicing these different touch-screen digital instruments has changed the way I hear these sounds that come directly from nature. They are buzzy, scratchy, droney and clicky, and because I have gotten more interested in electronic sounds over recent years, I hear these sounds as being more musical than I used to.

This alien, unfamiliar live natural soundworld is the perfect place to add unfamiliar electronic instruments, like those described above. These sounds are playing through a small JBL Charge speaker, pre-amped through a Sony A10 recorder. It is sounds of such strangeness that led me to become more interested in playing electronics live in nature, and the iPad has proven to be a fluid and flexible instrument for this. The sound is coming out through that black and white, camouflaged Ultimate Ears Boom 2 Kool Savas Limited Edition model behind to the right.

Sugarbytes’ Factory is used at first to play a plucked droning sound, which is soon augmented by a sample of a terrestrial insect, a green katydid, pitched up, then down. Then TC-11 comes in with high whirring, abstract sounds, which blend in with the drone and highlight the live underwater clicks and scratches. The human accompaniment stops and then Addictive Pro is brought in for a kinder, additive and filtered drone. At about 3:15 minutes in, the best blend of human and animal appears, as the higher frequencies are attenuated live with the two XY pads on the screen. Then I switch to a GeoShred cello for a solo section. Do the background aquatic plants change their beats in response to all this? It certainly seems so.

I want to enjoy them perhaps more than understanding them, so I treat them as a surrounding soundscape in which to join in. It does not matter to me whether the sounds of the underwater plants and animals change as a result of my engagement with them, though it often feels like they do. I see no need to try to prove it; instead I am just trying to make a music that the human species could not make alone. I need those plants and animals secretly performing underwater for the whole music to be produced.

Some days, what is heard underwater can be truly remarkable. There it was, a strangely warm day in March. The foxes walked slow and silently in the forest. Tried John Allen Pond. Beneath the surface, not a sound. Tried Hidden Lake… What an astonishing volume! Loud as a whale they say. The water boatmen were out. I thought my equipment had gone haywire. Then I realised what was up: this was the exact day their sonic battles began.

The lesser water boatman Micronecta scholzi can make a sound as loud as a whale, minety decibels, by vibrating its penis underwater against its body. Do not, I cannot stress this more strongly, try this at home. In a pool maybe okay, but still it’s risky. When James Windmill and Jerome Sueur discovered this phenomenon, they were sure something was wrong with their equipment. Nothing underwater could be so loud (15).

And yet… here it is as I experienced it. I had the same reaction when I found this sound myself, completely unexpected after the previous pond I tossed my hydrophone in was silent. Here, though, it was all sooooo loud. So I joined in with something exuberant, and had to add this track to my record release Secret Songs of Pond (16).

Audio 1: So Loud, So Still, from David Rothenberg, Secret Songs of Ponds.

All water boatmen, greater, lesser, big and small, sing on, sing on….

You have read this far… Guess the apps I am using. Hint: Addictive Synth Pro, GeoShred SWAM cellos, and at the very end…. You know, I’m not even sure….

In Sweden, in August 2024, I played a forty minute set with the unusually active Länkelven River. Beneath the surface were sounds I had never heard before: truly surprising, very electronic sounding tones, entirely natural, blowing apart my earlier prejudice that I could always tell the natural from the artificial (Fig. 2 & 3 and audio 2). So much for that.

Figure 2: Playing with the Länkelven at the Hyttdreva Festival. Photo by Jan van Boeckel.

Figure 3: Performance introduced by Annika Göran Rodell. Photo by Jan van Boeckel.

Audio 2: Maybe a Monster: Live with the Länkelven River, Hyttdreva Festival, 2024.

Sometimes what we hear makes more sense when we see it. Video 10 is a visualisation of a life iPad performance with a pond full of water boatman bugs, from March 2023 in Fahnestock State Park, upstate New York. This track appears on the album Secret Songs of Ponds, but in the moving image bellow you can see the structure of frequencies against time. It was made with Spectralayers, the graphic sound analysis and editing program.

Video 10: The Many Colours of a Sonic Pond. Made with Spectralayers Pro, by David Rothenberg.

At first it is hard to tell the sampled underwater bugs from the real ones, but after spending time with the moving image, you too will know. Or will you not?

We cannot remove the unknown.

A THIN BLACK RECTANGLE BY THE WATER’S EDGE

So here is what seems a paradox to me: I started out believing that in making music, the acoustic is preferably to the electronic, because it is more natural than artificial. Over time my prejudice began to lessen. Always impatient with new tools, I became enamoured of those that are instantly expressive, and offer quick-changing ways to let me express sounds I am never completely in control of, the hallmark of the possibility for improvisation. I started by taking natural sounds into the machine and twisting and turning them into new directions, always hoping the original naturalness might still shine through. Then I got more excited in bringing this ultimately abstract instrument back onto the land, with its sun, wind, rain, dirt and mosquitoes, and giving the sound back to its source.

Is this any different from playing a guitar or a drum by the water’s edge?

To some people, the spectacle of a clumsy human waving a Malevich-like black square around in the air, supposedly causing sound to come out of a modest, unattached small speaker, is as far from fitting into nature as any cartoon one could dream up. Making music with cicadas looks better with a clarinet or soprano sax, as The New Yorker decided in this caricature of me in 2013 (Fig. 4).

Figure 4: The author in the Talk of the Town, New Yorker (2013). Drawn by Tom Bachtell, used by permission.

Though I was also playing iPad back then, it would have looked even crazier than this. I do not care so much what it looks like, but more how it sounds. People sometimes ask me why bring such a soul-sucking machine into the swamps. Look how pissed off everyone was with the iPad ad that showed the big black flatscreen crushing art studios, music studios, all the mess of human creativity in an advertisement – so offensive that the company pulled it back within a week.

I get that concern, I really do. There is something faceless about a slim glossy rectangle that claims to do everything. Is there any ‘there’ there? We are so afraid of absence and nothingness.

I want to reveal the incredible sounds of these ponds and rivers, including those that previously might have been negatively characterised as ‘noise’. They already sound so much more like music than they once did, given our world’s increased acceptance of all sound as music. Toss a hydrophone into the most modest of ponds, plug in a speaker, and just listen.

If you are a musician, dare to join in. Play as if you are joining a music from another culture, with interest, love, and respect. That is how we should first treat nature: as this enduring world that makes everything we do possible. Beyond an inspiration it can be a collaborator, if you let its wonderful sounds into your sense of what music can be.

I never regret taking my music outside, and blending it with the real sounds that are. Sure, you can be clever and say everything humans do is also natural, because we are natural. But that really depends on how much you want to see, hear, and feel of the world around us. The challenge is to play and explore while also realising where you are — in a resilient world that will no doubt outlast us but is also under siege by our whole civilisation. The human way has a choice: to run counter nature or to listen and fit in.

I want to hear my way in, to a meaningful sound world that just might welcome me. And you too.

Figure 5: Cover of David Rothenberg’s album Secret Songs of Ponds.

REFERENCES AND NOTES

1. David Rothenberg, Hand’s End: Technology and the Limits of Nature (Berkeley: University of California Press, 1995).
BACK

2. Yet, in 2023 Andre 3000, previously a rapper, started playing one in an ambient context. It does sound kind of cool: <https://floodmagazine.com/150671/andre-3000-new-blue-sun-essay>, accessed December 2024.
BACK

3. Among many summaries of different types of synthesis, here is a good one: Adam Douglas, “10 Types of Synthesis, Explained: FM, Vector, and More”, Reverb, 18 June 2019, <https://reverb.com/news/10-types-of-synthesis> accessed December 2024.
BACK

4. Ian Port, The Birth of Loud: Leo Fender, Les Paul, and the Guitar-Pioneering Rivalry That Shaped Rock ‘n’ Roll (New York: Scribner, 2019).
BACK

5. Little has been written about touch screen musical interfaces in practice, but here are two theses: Tuomas Ahva, Padworks: Building a Solo Live Performance with iPad, Masters Thesis (Otaniemi: Aalto University, 2016); and Healy, Daniel, The Influence of Technical Proficiency On Clarinet and iPad Improvisation Achievement, Doctoral thesis (Evanston: Northwestern University, 2021).
BACK

6. See, e.g., Stefano Papeti and Staitis Charalampos, eds., Musical Haptics (Cham: Springer Nature, 2018).
BACK

7. See, e.g. H C Bennet-Clark, Sound Production in Insects. Science Progress 62, No. 246, pp. 263–283 (1975) and Michael Greenfield, “Synchronous and Alternating Choruses in Insects and Anurans”, American Zoologist 34, No.6, pp. 605-615 (1994).
BACK

8. David Rothenberg, Bug Music (New York: St. Martins, 2013).
BACK

9. A review of the Animoog Z: Rob Redman, Music Radar, 8 February 2022 <https://www.musicradar.com/reviews/moog-animoog-z>, accessed December 2024.
BACK

10. See, e.g. Micah R Bregman, Aniruddh D. Patel and Timothy Q. Gentner, “Songbirds use spectral shape, not pitch, for sound pattern recognition”  Proceedings of the National Academy of Sciences of the United States of America 113, No. 6, pp. 1666–1671 (2016).
BACK

11. How does the Fairlight fare today? See Mark Roland, “The Fairlight Era: The Dawn Of Sampling”, Electronic Sound, 8 February 2018, <https://www.electronicsound.co.uk/features/long-reads/the-fairlight-era-the-dawn-of-sampling>, accessed December 2024.
BACK

12. Gabriel J L Beckers , Roderick A Suthers and Carel ten Cate, “Pure-tone birdsong by resonance filtering of harmonic overtones“, Proceedings of the National Academy of Sciences 100, No. 12, pp. 7372-7376 (2003).
BACK

13. See the SWAM instruments, which can be used in various apps: <https://audiomodeling.com>, accessed December 2024.
BACK

14. Lauren Rosati, ed., Sound, Colonialism and Power, MAST – Journal of Media Art Study and Theory 2, No. 2 (2021).
BACK

15. J Sueur, D Mackie and JFC Windmill, “So Small, So Loud: Extremely High Sound Pressure Level from a Pygmy Aquatic Insect (Corixidae, Micronectinae),” PLoS ONE 6, No. 6, e21089 (2011).
BACK

16. See David Rothenberg, Secret Songs of Ponds at <http://secretsoundsofponds.world>, accessed December 2024.
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

David Rothenberg > New Jersey Institute of Technology

Musician, composer and philosopher whose work explores connections between sound, ecology and human–nature relationships. As Distinguished Professor at the New Jersey Institute of Technology, he investigates interspecies music-making and environmental perception, integrating artistic practice with scholarship on acoustic communication, cultural imagination and ecological awareness.

Reviewers

Jo Scott > i2ADS & University of Coimbra 

Artist-researcher working across performance, audiovisual media and improvisation. Her projects explore embodied knowledge, collaborative creation and live interfaces, integrating artistic experimentation with theoretical inquiry. She develops practice-based research that examines mediated performance, multisensory interaction and the dynamic relationships between bodies, technologies and environments.

Rajele Jain > Vipulamati

Artist and filmmaker whose practice spans video, performance, photography and documentary forms. Active since the early 1990s, she co-founded Vipulamati and directed the International Festival for Films on Art. Her work engages archives, exhibition design and cross-disciplinary collaboration, exploring cultural memory, visual narrative and embodied media experience.

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 2:

What do we talk about when we discuss computational art? Do we describe a practice centred around a medium, as when we talk about painting or photography, or do we describe modalities, such as when we talk about visual art or audiovisuals? In reality, both can be accurate answers to this question, although both also fall short of properly defining computational art. Why is that?

Computation is a medium, and like any other medium, it can be used, or misused, to create art (1). Computation can remediate other media forms (2), but its powers extend far beyond that. Computation is a metamedium whose contents are existing media, but also those media forms that have not yet been invented or even that are not physically possible (3). This is possible because of processes of digitisation and imitation that allow systems to effectively resemble the surface features of other media forms, not only looking like them but also, to a large extent, acting similarly. Because of the many affordances of computational systems, and the diversity of ways they can be articulated, there is always a potential for the development of media forms that are new, i.e., without an existing tradition or pattern of usage (4).

For the very same reasons, computation is also a metatool, an everything tool that can be used to create images, sounds, texts, or any other types of signs. In computation, medium and tool, metamedium and metatool, converge in ways where they intersect, merge, and dilute each other’s boundaries. Computational media thus become tools that can create signs but can also keep those signs in memory; they become media that can preserve and transmit signs whilst also being capable of creating information and new signs. They become antinomic systems that are simultaneously able to stabilise information but are compelled to transform it (5).

This is why a modal definition of computational art is problematic. More than being multimodal, computational media tend towards amodality. Computation and computational media exist in a condition across modalities that has the potential to unfold in any or many modalities. Computational media and the computational artworks that use them are computational objects, and, as such, they are in a state of continuous production of signs from data and algorithms. 

In computational objects, signs, as such, do not exist. Signs are produced on demand, when and where necessary, in whatever forms or modalities are required. In this process, information, either in data or generated by a computation, is transcoded to analogue signs so it can be communicated to humans and other analogue systems or to digital formats to be communicated with computers and other digital systems (6).

The processes of transcoding liberate computational objects from modality. While specific objects may be tied to modalities, we may not talk of computational objects as being modal-dependent, but very much to the contrary. Computational objects are in all modalities while being in none. They are in the condition of the metainterface (7) of the omnipresence of computation and its centrality for all other processes.

Computational art is developed from computation and computational objects in ways that make computation its raison d’être (8) as well as its material, to paraphrase Henry Flynt’s definition of conceptual art as an art “of which the material are ‘concepts,’ as the material of for ex. music is sound” (9). In computational art, computation is not only a tool, channel, or resource, but also the thing to be experienced, the centre of aesthetic experience. This can happen in cases where it is used in ways that give rise to other signs — thus producing images, texts, sounds, actions, etc. — but also in those cases where it does not — by producing computation without perceivable outputs.

This is the case with works such as Jaromil’s Forkbomb (2002) or Pall Thayer’s Microcodes (2009-2015). Forkbomb is a code without execution because of the destructive potential of its power to recursively copy itself and crash a host computer (10). Microcodes is a set of small poem-like snippets of code that, although they can be executed, are not meant to be, at least not on a computer, but rather through their reading, interpretation, and mental enaction. Even works with perceivable outputs can foster similar aesthetic phenomena because, very often, the aesthetic experience of computational art extends well beyond those perceivable outputs that may be presented and onto a future of potential realisations that one may never directly interact with but nevertheless feels.

Computation can, therefore, be proposed as the work of art itself, even as the sole aesthetic experience. And in those cases where we find that computation is merely used instrumentally — i.e., to remediate, substitute, or replace other tools and media — or when its presence is not aesthetically relevant per se, even if computers and computational systems may be abundantly used, we may be able to talk about “art involving computers”, but not about computational art (11). It is important to keep this in mind at a time when computation is so ubiquitous and when we may not, therefore, think about artworks as instances of computational art simply because they may in some way resort to computational technologies in their production or deployment. The central question is not whether computational technologies were used or are present, but rather whether the artistic proposal of the work is centred on computation at a fundamental level.

Whenever computation is used in a work, more often than not, it will coexist with one or more modalities and media forms and, consequently, with the traditions that have been developed around them, sometimes for millennia. Because of this, computational art ‌can be visual art  (12), and it can be literature, or audio-visual, or film, or music, or sound art. It can be all of these whilst remaining computational art, existing in a state of superposition where it is many things at once and never collapses into a single one (13).

COMPUTATIONAL AESTHETICS

Computational art promotes aesthetic experiences that are developed throughout a complex network of layers of information, meaning, and significance (14). Florian Cramer points to commonalities between computational and conceptual art. In Concepts, Notations, Software, Art (15), he proposes an idea of “software art”, a set of practices that “pay attention to how and by whom programs [are] written” (16) and of which the materials are formal instruction code whose “instructions and codedness” are exposed in and by works. In this context, Cramer points to how conceptual and software art find a commonality in the usage of language in the works — either in the form of instructions or concepts — and notes that both also collapse concept notation and execution in works. The definition of computational art to which we resort in this text encompasses Cramer’s software art and even expands it. As such, we find that much similarly to conceptual art, computational art is often anchored by objects and signs that allow communication to happen, but ultimately it is not object-oriented but rather systems-oriented (17) as it turns away from objects and towards relationships.

Conceptual art is often reminiscent of specifications for computational machines — or software — because of its usage of notation or plans to construct forms that, before their materialisation, only exist as concepts and afterwards continue to exist as concepts that retain the potential to be expressed more often and are not bound to a single — and scarce — instantiation. This is why “the idea becomes the machine that makes the art” (18), as the art object becomes a non-essential physical mediator for the art experience.

Computational art is more conceptual than perceptual (19), in ways that go beyond the logical and mathematical. It is cognitive, intuitive, fuzzy, and, again, like conceptual art, its practitioners can be seen as being more mystics than rationalists (20), proposing “conclusions that logic cannot reach” (21) in works that, precisely because of their illogical character, promote new experiences (22).

COMPUTATIONAL READING

This is one of the reasons why the experience of reading computational artworks, and computational media in general, is so different from that which is developed in other media.

When we engage with computational artworks, we do so first through their surface layer. The surface is the level of sensorial effusions of a system, and it is primarily through it that we manage to gain access and start to discover the artwork as a network of signs, actions, and causal processes. Through this analogue layer, we can start discovering the digital and the amodal. We can find a way to access the computational core of the artwork that otherwise would be an impenetrable black box. This computational core is what Frieder Nake calls the subface (23). If the surface is sensorially perceivable, the subface is computational, the system of rules that drives the artwork and on which the surface depends and that we are not able to perceive directly. The interface is a bridge between surface and subface, and it exists in a state of duality, in a superposition of computation and of its materialisation. Through the surface, we may attune to the surface, and we are able to couple structurally with the computation (24).

To do this, we need to expand the homolinear and heterolinear reading modes with a hyperlinear reading (25) that is both procedural and ergodic. This is an active reading process that requires readers to invest a “nontrivial effort” (26) and to bring into play far more than the interpretative functions required by most other media. In ergodic forms, readers may explore, configure, influence, or negotiate with texts in ways that go well beyond what is possible in non-ergodic forms. There can be great hermeneutical complexity in non-ergodic forms, but they are usually characterised by a stability of morphology that is quite rare — if at all possible — in ergodic forms.

Our engagement with computational objects is ergodic, as it is situated, iterative and enactive. It is also, as Penny notes, highly subjective and deictic because when we couple with a computational object, we don’t simply become its source of data or an external element to which it responds. Interaction puts us in a position that affects and shapes the computation and that we can, therefore, understand as a position of programming. Through interaction, we become both programmers and programmed. We become computation, and merge — albeit mostly temporally — with the system we couple with. Therefore, we are unable to adopt an objective view of it, of us, and the engagement. With the immersion of structural coupling comes the logical incoherence of assuming “an observing self that exists outside the engagement with an interactive system” (27).

The mesh we engage with when entangling with a computational artwork is not only formal, semantic, and symbolic, but also an interface. It is ergodic, procedural, and computational, pushing us to read and understand it at this level. The way we do that is by developing a gaze focused on function, program, and the recognition of computational patterns. This computational gaze (28) allows us to develop an active interpretation of the works, to simultaneously see them and see-as them, and to act on them while also acting-as them. This is not simply interpretation, but interpretation as epistemic action (29) an effort at deduction of causal relations, induction of mechanics, and abduction of technological intentionality (30) and an exploration of the aesthetic pleasures at the rift between surface and subface.

While reading through the computational gaze, we iteratively build models that are used to explore the artwork, to pose and answer questions, to fill in information, and to anticipate actions, both those from the system as well as our own. We build models that help us predict the future. By engaging with a computational artwork and interacting with it, if possible, we start to recognise some processes, discover others, and identify causal chains. We may, on occasion, duplicate some of the actual processes in a work, but more often than not, the models that we develop will most likely be approximations of the meaningful processes in a work, conjectures that are good enough because of their predictive power (31).

This process starts from the surface and descends towards the subface through a computational reading during which we develop a Theory of the System (ToS) (32). A ToS is not an emulation of the entire system of an artwork but rather a model of the central processes in its subface. A model of the processes that speak to us and that, therefore, become meaningful for the aesthetic and artistic experience of the artwork. It is a model that includes what we perceive as the most significant actors, objects, and behaviours in a system and the rules that organise their relationships. Therefore, it is a model that includes the system, its context, and those other systems — such as ourselves — that may interact with it. And even in those cases when artworks do not allow direct interaction, a ToS allows us to explore the artworks in ways that can be almost as informative as direct interaction, namely through the potential reprisal, repetition, and iteration of the artwork’s processes.

It is important to note that a ToS is not a general-purpose model. Rather, a unique ToS is developed for each artwork we engage with, and if some of its parts may perhaps be repurposed for other systems, their selection and engagement depend on the perceived relevance of the objects they model within the context of the new artwork.

The development of a ToS is a central part of the aesthetic enjoyment of a computational artwork, and it is the one that deeply attunes to the computational layer of the artwork and that is able to relate it to all other layers. This is an understanding of the work through simulation (33) that is, for the most part, developed subconsciously but that can, on occasion, be the focus of conscious attention and be helped by the direct communication of code, pseudo-code, or procedural descriptions within the work itself or in its paratexts.

DEMATERIALISATION

Through the development of a ToS, we model an artwork; we simulate and even emulate parts of it while abstracting less significant aspects. By doing this, we effectively create a new set of computations within our mind. We do not replicate the entirety of an artwork or its computational layer — except perhaps in the simplest of cases — but we do transform and adapt the artwork in the process of transferring it to our minds.

Computational artworks transform the locus of artistic experience because they create reality. This happens, firstly, through the contingent and situated character of their own existence, but it is also a consequence of how computational artworks involve us with their processes and allow us to bring their fields of meaning to existence through the participative process of creating a ToS, directly enmeshing us in the production of new artistic experiences. Through a ToS we absorb and reproduce the artworks, we dematerialise and detach them from their original substrates. We multiply the artworks so we can take them with us in a process that, seen from the perspective of the works, can almost be described as one of mutation and reproduction. It is a process of attunement to computation and to our capacity to discover computations in artworks and enact them.

This is an auratic process. By developing a ToS and transferring the artworks to our mind, we do not diffuse their auras — their here and now (34) — through the creation of mechanical copies that can be fully reproduced. Rather, we create singular, situated, new dimensions and a “dynamic lifetime” (35) that nourishes their auras. Every ToS has its own here and now and, much like an original, it cannot be reproduced or copied because it is developed through processes that are sub-symbolic, intuitive, and impossible to formalise and communicate. Furthermore, a ToS expands the artwork, adding new layers to its complexity and system. The original layers of mechanics, dynamics and aesthetics that are present in any computational system (36) are expanded by a ToS with two layers that we can describe as of simulated dynamics and simulated mechanics.

Here, computational artworks transcend their objecthood and manifest as artworks that are not bound by a finite set of signs or content, that are not bound by story, narrative, form, or other factors but, crucially, by process.

By dematerialising artworks, removing them from the substrate where they are found and embodying them in our minds, we develop something akin to empathy with them, even a very high level of intimacy. We start seeing and feeling artworks not as objects, formal features of physical materials, not as messages, carriers, or encoded meaning; not as surface but as subface, as information and causal processes that are encountered in a particular setting but may potentially manifest in multiple substrates (37). We develop an ecological understanding of the artworks, intuiting how they may develop outputs that are contingent to their mediation, how they may interface with other systems, including ourselves, and how they become in the irreducible phase space of their computations.

Through this, we manage to understand artworks as rifts between surface and subface, feeling them as objects that withdraw from us, from the world, even from themselves  (38). They are objects that need to be interfaced with to be fully realised; objects that exist in a state of suspension — or of superposition — between a past that is captured in code, algorithms, and morphology, and futures that emerge from the potentials of computation. Computational artworks exist in a state between being themselves and not-being themselves (39), with their realisation dependent on their self-destruction through the permanent transformation of their current state. This is not unlike what Graham Harman sees as being at the core of all art, the rift between the real object and its sensual qualities (40), that we explore, attune to and resonate in our aesthetic enjoyment. If this gap exists within all things (41) as an ontological rift between being and appearing  (42), it is hardly ever as clear as it is in computational artworks.

REFERENCES AND NOTES

1. Alva Noë, Strange Tools: Art and Human Nature  (New York, NY: Hill and Wang, 2015).
BACK

2. Jay David Bolter and Richard Grusin, Remediation: Understanding New Media  (Cambridge, MA: MIT Press, 1999).
BACK

3. See Alan Kay and Adele Goldberg, “Personal Dynamic Media”, in The New Media Reader (Cambridge, MA: MIT Press, 2003) pp. 393-404 & Lev Manovich, Software Takes Command: Extending the Language of New Media  (New York, NY: Bloomsbury Academic, 2013).
BACK

4. Janet H Murray, Inventing the Medium: Principles of Interaction Design as a Cultural Practice  (Cambridge, MA: MIT Press, 2012).
BACK

5. Miguel Carvalhais, “Designing (with) Computational Objects: From Metamedia to Metaenvironments”, Joelho – Journal of Architectural Culture 14, pp. 129-140 (2023).
BACK

6. See Trevor Paglen, “Operational Images”, E-flux 59 (2014) & Jussi Parikka, Operational Images: From the Visual to the Invisual  (Minneapolis, MN: University of Minnesota Press, 2023).
BACK

7. Christian U Andersen and Søren B Pold, The Metainterface: The Art of Platforms, Cities, and Clouds  (Cambridge, MA: MIT Press, 2018).
BACK

8. Hiroshi Kawano, “What Is Computer Art?”, in Artist and Computer (Morristown, NJ: Creative Computing Press, 1976) pp. 112-113.
BACK

9. Henry Flynt, “Concept Art”, in An Antology of Chance Operations (New York, NY: 1963).
BACK

10. Josephine Bosma, Nettitudes: Let’s Talk Net Art (Rotterdam: NAi Publishers, 2011).
BACK

11. Rosemary Lee, “Art Involving Computers vs. Computational Art”, in Espaço/ Programa (Vila Nova de Cerveira: FBAC, 2022) pp. 41-47.
BACK

12. For an in-depth reading of the processes of image creation and their influence in visual culture, see Rosemary Lee, Algorithm, Image, Art (New York, NY: Atropos Press, 2024).
BACK

13. We should beware that, as LeWitt notes, “When words such as painting and sculpture are used, they connote a whole tradition and imply a consequent acceptance of this tradition, thus placing limitations on the artist who would be reluctant to make art that goes beyond the limitations.” Sol LeWitt, “Sentences on Conceptual Art”, 0-9 5 (1969).
BACK

14. Rosemary Lee, “The Tangential Realism of @Pepitothecat”, in The Book of X: 10 Years of Computation, Communication, Aesthetics & X (Porto: i2ADS & CITAR, 2022) pp. 79-89.
BACK

15. Florian Cramer, “Concepts, Notations, Software, Art”, Netzliteratur (2002) <http://netzliteratur.net/cramer/concepts_notations_software_art.html>, accessed November 2023.
BACK

16. This perspective can be seen as something like a precursor to the idea of “critical code studies” that has been promoted by Mark C. Marino and others. Mark C Marino, Critical Code Studies (Cambridge, MA: The MIT Press, 2020). I find, however, that Cramer’s focus on art that uses code as its material but also “addresses cultural concepts of software” is more attuned than Marino’s to objects such as those we discuss here, where computation is central. 
BACK

17. Jack Burnham, “Systems Esthetics”, in Art Forum (1968) pp. 30–35.
BACK

18. Sol LeWitt, “Paragraphs on Conceptual Art”, Art-Language 1 (1969)
BACK

19. Miguel Carvalhais, Art and Computation (Rotterdam: V2_Publishing, 2022).
BACK

20. Even when an artwork may seem strongly rational due to its formal features, we can argue that it may not necessarily be the case.
BACK

21. LeWitt, p. 1. [18.]
BACK

22. Ibid., p. 3.
BACK

23. Frieder Nake, “The Disappearing Masterpiece”, xCoAx –  Conference on Computation, Communication, Aesthetics & X, pp. 11-26 (2016).
BACK

24. Simon Penny, Making Sense: Cognition, Computing, Art, and Embodiment  (Cambridge, MA: The MIT Press, 2017).
BACK

25. Espen J Aarseth, Cybertext: Perspectives on Ergodic Literature (Baltimore, MD: Johns Hopkins Univ. Press, 1997).
BACK

26. Ibid., p. 1.
BACK

27. Penny, p. 360. [24];  cf the contrary view of Rodrigo Hernández-Ramírez, “Why Distancing Matters for Aesthetics and Technology”, in The Book of X: 10 Years of Computation, Communication, Aesthetics & X (Porto: i2ADS & CITAR, 2022) pp. 67-77.
BACK

28. Miguel Carvalhais, “Breaking the Black Box: Procedural Reading, Creation of Meaning and Closure in Computational Artworks”, in Artificial Intelligence and the Arts: Computational Creativity, Artistic Behavior and Tools for Creatives (Berlin: Springer, 2021) pp. 347-62.
BACK

29. Carvalhais (2022) p. 64. [19.]
BACK

30. Don Ihde, Technology and the Lifeworld: From Garden to Earth (Bloomington, IN: Indiana University Press, 1990).
BACK

31. It is also important to note that the cognitive economy of these models is a very relevant factor. A model is only good if its predictive power is also possible within a reasonable timeframe and with the limited resources one has within the limits of one’s own organism, i.e., if a model takes longer to predict the work than the time it takes the work to compute itself, or if a model demands mustering computational resources that are too demanding within the context in which the work is experienced, then that model is not useful and will certainly be dropped in favour of any alternatives that are more effective.
BACK

32. For a more in-depth overview of this, see Carvalhais (2022). [19]
BACK

33. Melanie Mitchell, Artificial Intelligence: A Guide for Thinking Humans (London: Pelican Books, 2019).
BACK

34. Boris Groys, In the Flow (London: Verso, 2016).
BACK

35. Aarseth, p. 81. [25]
BACK

36. Robin Hunicke, Marc LeBlanc and Robert Zubek, “MDA: A Formal Approach to Game Design and Game Research”,  Conference of Artificial Intelligence Proceedings (2004).
BACK

37. And that, much like digital computation, can almost be thought of as being independent of these substrates.
BACK

38. Timothy Morton, Realist Magic: Objects, Ontology, Causality  (Ann Arbor, MI: Open Humanities Press, 2013).
BACK

39. Ibid.
BACK

40. Graham Harman, Art and Objects (Cambridge: Polity Press, 2020).
BACK

41. Graham Harman, Object-Oriented Ontology: A New Theory of Everything (London: Pelican Books, 2018).
BACK

42. Timothy Morton, All Art Is Ecological  (London: Penguin Books, 2021).
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

Miguel Carvalhais > i2ADS / University of Porto

Designer, musician and researcher whose work spans computational art, sound and design. As professor and director of the Faculty of Fine Arts at the University of Porto, he integrates artistic experimentation with analytical inquiry, exploring how digital systems shape aesthetic processes, performance and contemporary creative practice.

Reviewers

Kate Sicchio > Virginia Commonwealth University

Artist, choreographer and researcher exploring how movement, code and embodiment intersect in live performance. As Assistant Professor at Virginia Commonwealth University, she connects creative practice with research in interactivity, dance technologies and algorithmic performance, examining how bodies and systems co-construct expressive and computational forms.

Carolina Martins > Film EU University 
Researcher and lecturer working across cinema, expanded media and digital humanities. Her practice investigates spectatorship, augmented textuality and multimodal storytelling, grounding creative experimentation in critical inquiry. She leads interdisciplinary projects that explore how emerging media reshape narrative forms, reading practices and the sensory experience of audiovisual culture.

Silvia Pinto Coelho > NOVA – FCSH & Lusófona University

Researcher and curator whose work engages media arts, experimental methods and contemporary artistic practice. She develops projects spanning sound, installation and hybrid formats, contributing to cultural programming and fostering interdisciplinary enquiry that examines technological mediation, artistic experimentation and evolving modes of creative production within emerging artistic communities.

Editor

Adriana Sá > CICANT / Lusófona University Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience @ seriale online.

Continue with LIVE INTERFACES Volume 1:

Live scoring and artificial intelligence (AI)-based music generation are two fields where a big volume of research has already taken place, providing numerous approaches to the realisation of such projects. AI-generated music is rooted in Computer Aided Composition (CAC), which itself consists of a large collection of works, dating back to the 1950s. More recently, textual AI-based content generation has also seen much development, with generic tools like OpenAI’s ChatGPT and Google’s Bard having drawn a lot of attention. This article presents a new approach to AI-aided composition through textual score writing combined with a character-based Gated Recurrent Unit (GRU) Recurrent Neural Network (RNN). The RNN is a type of AI that can be trained on sequences of data, and then generate similar sequences. One such example is text generators. The two basic types of RNNs are the Long Short-Term Memory (LSTM), and the GRU, which are two different approaches to solving issues with traditional RNNs, that concern vanishing or exploding coefficients of the network during the training process. This means that the values of these coefficients become either too small or too big to make sense or to be handled by a computer.

The approach analysed in this article is based on LiveLily, a bespoke software for live scoring through live coding, and an RNN GRU model created with the TensorFlow library. Live coding is a practice where a computer musician or other artist uses computer code and writes that code live during a performance. LiveLily uses a subset of the Lilypond language. Lilypond is a textual language for score engraving (1). Its language is simple yet versatile and powerful, enabling users to write complex Western notation music scores in an expressive way. The following lines create two bars in a 4/4 time scale with the C major scale in quarter notes starting from middle C, going up one octave, to be played fortissimo:

\time 4/4
c'4\ff d' e' f' \bar "|" g' a' b' c'' \bar "|."

For musicians familiar with the Western notation format, using this language can be very expressive as it uses symbols found in sheet music and terms used in music schools in the Western world. LiveLily is inspired by this language and uses a subset of it to create sequencers and animated scores. It is written with the openFrameworks toolkit and uses a slightly modified version of the Lilypond language with a bit of syntactic sugar – a term used in computer science for syntax that usually enables more concise coding – to enable fast typing for live coding sessions. It can be used either with acoustic instruments as a live scoring system or with electronic music systems that support the MIDI or the Open Sound Control (OSC) communication protocol (2). This is an open-source project, hosted on GitHub.

The idea behind this project is to train a GRU RNN with music phrases in the LiveLily language –  a programming language – and then interact with it during a live coding session. Once the RNN is trained, it can provide suggestions during performance based on what the live coder seeds it with. The live coder can then decide whether he/she will use these phrases intact, edit them, or discard them. Since program synthesis is an AI practice with research dating back to the 1940s and researchers are still active in this field (3), this project has many resources to draw and learn from.

RELATED WORKS

Computer-generated music precedes AI-based music generation. The former is a broad field of practice with many branches. From soundtracks generated through image analysis while looking at a train window (4), to audio mashups (5), or even auto-foley tracks (6), one can find many examples where the computer is solely responsible for the generation of music, once the algorithm has been defined. Besides enabling computer-generated music, LiveLily is a system for live scoring, a field with many research and artistic projects. In this section I will outline related works in all related fields: AI-generated music, computer-generated music, computer-aided composition (CAC), live scoring works, and live coding combined with AI.

COMPUTER-AIDED COMPOSITION AND SOFTWARE

Computer-generated music has its roots in CAC. The first such work dates back to 1956, the Illiac Suite by L. M. Hiller and L. M. Isaacson (7). Another early example is the ST/10-1, 080262 by Iannis Xenakis, a composition for acoustic ensemble realised with calculations made on the IBM-7090 computer (8). According to Curtis Roads, numerous other composers around the time of Hiller or later used computers to generate music (9). Herbert Bruen and John Myhill, James Tenney, Pierre Barbaud, Michel Phillipot, and G. M. Koenig are among these composers. In the context of related works, CAC software should also be mentioned in this section. Such software is nothing new, with the UPIC system where the user could draw various sound parameters being developed and used in the 1970s (10). Other software has been developed more recently, including Open Music (OM) (11). More software can be considered CAC software, but they lean more toward live scoring, so they are mentioned in the respective section.

GENERATIVE SOUNDSCAPES

There are many soundscapes and sound-walks that are also computer-aided or computer-generated. Hazzard et al. created a sound-walk for the Yorkshire Sculpture Park based on GPS data (12). Lin et al. created Music Paste, a system that chooses music clips from a collection of audio that it considers appropriate, concatenating them seamlessly by creating transposition segments to compensate for changes in tempo or other parameters (13). Manzelli et al. create music by combining symbolic with raw audio  to capture mood and emotions (14). Symbolic here refers notes, while raw audio refers to waveforms.

LIVE SCORING WORKS AND SOFTWARE

CAC tends to blend with live scoring works, where usually the composition that is written live is aided by a computer. Comprovisador is a live scoring and CAC work for ensemble and soloist (15). The computer in this work “listens” to the soloist, decodes his/her playing, and creates a score for the ensemble, based on this analysis. A work that is closer to LiveLily is No Clergy by Kevin C. Baird, a live scoring composition using two programming languages, Python and Lilypond (16). This work creates the composition by using audience input on the duration and dynamics of notes, among other data. John Eacott sonifies tidal flow data in his work Flood Tide (17). This composition for acoustic ensemble aims at long performances of slow-changing music, as the tidal flow data are updated every five seconds through a sensor placed in the water.

Live scoring software include the bach external library for Max (18), Maxscore (another Max external object) (19), and INScore (20). OM is also a live scoring software, even though it was initially developed as a CAC software, with its live scoring features added later on (21).

LIVE CODING AND AI

The combination of live coding and AI is not entirely new, although the attempts to combine the two are not numerous. If one refers to the Live Coding book (22) as an exhaustive source on this topic, one can distinguish two notable mentions. These are Cibo v2 (23) and an augmented live coding system by Hickinbotham and Stepney (24). Both projects are based on the Tidal language for live coding (25).

Cibo v2 uses a combination of an autoencoder, a variational autoencoder, and an RNN, to produce Tidal code. The (variational) autoencoder is a type of Neural Network that compresses its training data into what is called a latent space, from which one can produce output by choosing a point in this space. The augmented live coding project uses the Extramuros platform for networked live coding sessions (26) combined with an additional server that both checks whether the Tidal patterns submitted by the participants of a performance conform to the Tidal syntax, and generates its own Tidal snippets, based on mutations of the submitted patterns.

LSTM RNN-BASED MUSIC GENERATION

Music generation with the use of LSTM RNNs has seen a lot of development in recent years. Most of the research done in this field combines RNNs or other types of Neural Networks with MIDI symbolic notation. Agrawal et al. use an LSTM RNN to generate music by using MIDI files to train the network on the relationships between chords and notes (27). This project uses Keras, a user-friendly Python module that builds on top of Google’s TensorFlow ecosystem for machine learning (ML), and combines it with the music21 Python module (28) to get note and chord objects from the MIDI files and feed them to the RNN. Chen et al. use an LSTM RNN to generate Guzheng (Chinese Zither) music, again using MIDI data (29). Mots’oehli et al. compare adversarial with non-adversarial training of LSTM RNNs to create music based on a corpus of MIDI data (30). Adversarial refers to a type of AI where two Neural Networks contest each other, and where one network’s gain is the other network’s loss. Non-adversarial refers to more traditional LSTM-RNNs. Their findings suggest that adversarial training produces more aesthetically pleasing music.

Yu et al. create music from lyrics using an LSTM-Generative Adversarial Network (GAN) RNN (31). They extract alignment relationships between lyrics’ syllables and music attributes to create their training corpus from 12,197 MIDI songs. Ycart et al. create binary piano rolls from MIDI files to train an LSTM RNN and then use it to make polyphonic music sequence predictions (32). Li et al. use an LSTM RNN with MIDI data of piano music to generate such music (33). Huang et al. use a combination of a Convolutional Neural Network with LSTM with matrixes produced by MIDI data to generate music (34). Min et al. use a GAN to generate music, to address the problem of LSTM not being able to produce long music sequences (35). Arya et al. also use music21 combined with an LSTM RNN, based on MIDI files to generate music based on part of an existing composition (36). What we can see in this section is that there is much more research based on MIDI data to train the network and generate new content, than research focusing on text generators. This article suggests a new approach, where AI is trained on text instead of MIDI and it is used to generate text that results in a music score. Even though a solid foundation for this system has already been laid – which is analysed in the next sections – there is still a lot to learn and many resources to draw from. With the advent of Large Language Models (LLM) like OpenAI’s ChatGPT, and Google’s Bard, it is interesting to see how the research that has been conducted and published on text generation through AI can enhance a symbolic music-generating project, where symbolic music refers to notes in the Western music notation system.

THE LIVELILY SYSTEM

LiveLily is an open-source live sequencing and live scoring software based on a subset of the Lilypond notation. The LiveLily language is a slightly modified version of Lilypond with a bit of syntactic sugar to enable fast typing for live coding sessions. It does not produce any sound but sends OSC or MIDI messages to other software or hardware. The idea was to build a system that enables the user to create a music score similar to a Csound score (37) in the context of controlling a synthesiser. The Lilypond language was chosen because of its expressiveness and the similarity it shares with sheet music jargon and Western music terminology. During the development of LiveLily, I started developing the live scoring part of the software. This part led to the idea of using LiveLily to conduct acoustic ensembles through live coding in live scoring sessions. Even though I have used the Lilypond compiler for live scoring in previous work (38), I opted to develop the live scoring part in openFrameworks which was already used for the text editor of LiveLily. Using openFrameworks enabled both the integration of the score in the editor and its instant display, as the Lilypond compiler takes some time to render even a single bar of music. It also enabled the score to be animated by colour-highlighting active notes and showing the beat with a pulsating rectangle that encapsulates the full score. The figure below illustrates a LiveLily session.

Figure 1: A LiveLily session. Alexandros Drymonitis, 2023.

By using the Lilypond language, LiveLily enables numerous expressive gestures to be included in the sequencer and score. As an example, the code chunk below creates two bars of music for four instruments, and then a loop where the first bar is repeated twice, and then the second bar is played once. This code results in the image above, where the first bar is displayed.

\bar 1 {
    \sax c''4\f\>( d''8\mf) fis'' e''4 ees''8 d''
    \cello c,4\ff( cis, c,) d,
    \bass e,,4\f ees,, e,, f,,
    \perc [f'8 c''16*2  a' c'' c''8]*2
}

\bar 2 {
    \sax b'8 bes' c''4-. cis''16*4 d''4--
    \cello c,4 cis, ees d,
    \bass g,,4 fis,, e,, f,,
    \perc [<a' c''>16 c''*3 f'8 <a' c''>]*2
}

\loop loop1 {1*2 2}

The two bars created with the \bar command include dynamics, diminuendi, and articulation symbols. It is also possible to include arbitrary text in the score, like “flutter” or any other text. All this information is communicated with OSC messages, and it is up to the user how these will be interpreted in the audio software. Most of this information can be communicated with MIDI too, excluding the arbitrary text. In the case of live scoring sessions with acoustic instruments, these semantics result in standard Western music notation symbols that are well-known to Western instrument performers. The syntactic sugar LiveLily adds, concerns the multiplication factors shown in the example code and figure above. Multiplying a note or a group of notes encapsulated in square brackets repeats this figure as many times as the value of the multiplier. This is shown in the \perc line of the second bar in the code chunk above. When a bar name is multiplied, as it appears in the \loop command in the code chunk above, then the entire bar is repeated in the loop.

THE GRU RNN

The GRU RNN of this project is created with the TensorFlow library, and it is inspired by a character-level text generator tutorial provided by its developers. Character-level means that the RNN will generate its content by choosing one character at a time, and predicting which character should follow, based on the previous one, instead of choosing entire words. Since LiveLily does not use MIDI information to create its scores, but its own language with a limited set of standard characters, a character-level text generator seemed like an appropriate approach. The code for creating and running this network is hosted on GitHub. This code must be combined with the LiveLily system.

PREPARING THE CORPUS

Preparing a corpus to train the RNN is a challenging task. If one is to use one’s own music only, then one should write a lot of music before a reasonable corpus can be assembled, as a sufficient training corpus must have at least 100,000 characters, as defined by the developer of the Keras library. In the LiveLily examples provided in this article, a single bar of music for a single instrument approximates to thirty characters. For a four-instrument ensemble, to reach 100,000 characters, one would need to write around 800 bars of music, before a corpus that is large enough has been assembled. It is possible though to use music by other composers to train the network, something that is likely to have interesting results, since it will create a hybrid style composition generator.

Since the RNN will be generating text in the LiveLily language, the training corpus must be written in this language. For the purposes of this project, a Python script has been developed to translate MusicXML into LiveLily files. MusicXML was chosen because it is a format used by many score engraving systems, including Sibelius, Finale, MuseScore, and Lilypond. It is also possible to produce MusicXML files from scores in the corpus of the music21 Python module.

The training corpus should consist of bar definitions only, in the LiveLily language. No other information is required, since the network will be asked to generate only bars, and more information from LiveLily sessions would confuse it. Additionally, the corpus should consist of music written for the same ensemble, so that the predictions of the network are aligned with the ensemble for which this system is used.

GENERATING MELODIC LINES

Once the RNN has been trained, the user can start a live coding session with the assistance of the RNN. Before starting to type, a Python script must be launched that will keep track of all keystrokes, even if this script is run in the background. The pynput Python module can do this.

LiveLily, partly being a text editor, creates a new line when the Return character is hit. The Python script with the pynput module keeps track of all keystrokes, and when the Return key is pressed, the script clears the saved contents and starts from scratch. In LiveLily, hitting Ctl+Return executes whatever is written in the line where the cursor is (or a chunk of code, if the cursor is within curly brackets), without proceeding to a new line, and hitting Shift+Return, it both executes a command and creates a new line. In case of Ctl+Return, the Python script should do nothing. In case of Shift+Return, the script tests the saved content from what has been typed. Since LiveLily supports comments, the following line can be used to let the script know that the user is asking for an AI-generated bar.

%generate \bar

A comment is used so that there is no error raised by LiveLily when this line is executed, since the software does not include any such command. When this line is executed with Shift+Return, the Python script seeds the RNN with \bar. As it is possible that the RNN produces content that is not entirely correct in a LiveLily context, a sanity check of the AI-generated content is necessary. In this step, the script checks for false instrument names and rhythmic consistency, where it will be ensured that all lines do not sum up to a greater or lesser duration that expected.

To receive the melodic lines generated by the RNN, but to also not change the source code of LiveLily, the Python script must type the generated content to the LiveLily editor. Even though the pynput module can be used as a keyboard too, besides a keyboard listener, it appears that typing with pynput‘s keyboard controller does not work as expected, either because of feedback that occurs through the module’s keyboard controller and listener being simultaneously active, or due to LiveLily not behaving as expected when controlled with this module. What works well is that the Python script “types” the generated content by sending its characters over OSC. LiveLily supports passive control with OSC, so other software can type in it. Once the Python script types the AI-generated content, it stops and does not execute it within LiveLily. This means that it is up to the discretion of the user whether they will use the AI-content intact, whether they will modify it, or whether they will discard it altogether. This setup protects from faulty content that might be produced, even though there is a sanity check before it is “typed”.

PROOF OF CONCEPT

The video below demonstrates the system in use. The RNN of this video was trained with a corpus of Bach’s four-voice chorals, retrieved from the corpus of music21. The scores were converted to the MusicXML format, and then the necessary information, like meter, instrument name, pitch, and duration, were extracted and converted to the LiveLily format. Once each piece was stored as a LiveLily file, a Python script assembled all these files into one big corpus, consisting of 261,116 characters. The RNN was trained for 30 epochs; its training lasted one hour and fifty-one minutes, and it ended up with a loss of 0.2949. The training was done on a machine without GPU support, which demonstrates that it is feasible to use this system without much infrastructure, as a personal computer should be sufficient. The sound in the video is produced by oscillators in Pure Data, receiving the sequencer data via OSC.

Video 1: The GRU RNN generating melodic lines in the LiveLily format, trained with a corpus of Bach’s chorals. Alexandros Drymonitis, 2023.

It is evident that the resulting music does not imitate Bach very convincingly, but it is also evident that the RNN keeps the register of each voice, without generating melodies that are too high for the bass or too low for the soprano, and this applies to all voices. At some point, the network makes a mistake and generates a half-finished bar, which is discarded, as it is up to the user which bars are used in a session. Sometimes it takes longer for the network to generate a bar, and that is probably because of the test each bar undergoes, where if a generated bar is not of the necessary duration, the network is asked to generate another bar, and this procedure is repeated until a bar that is correct has been produced. During the development of this system though, most of the time, the first bar that was generated was correct. This, together with the integrity of the registers, shows that the network has learnt several aspects of the music quite well.

CONCLUSION

Generating textual content with the aid of AI is a practice that, at the time of writing, is seeing a lot of development. Generic LLMs like OpenAI’s ChatGPT and Google’s Bard have drawn a lot of attention in various academic fields and have initiated many research projects. On the other hand, generating music with RNNs is a field with active research where the problem is approached from many angles. Most such research is based on MIDI data to train the network and generate new musical content, while research on AI-generated music through text generation also exists, although this is limited and focused on electronic music, rather than sheet music.

LiveLily is a software for live sequencing and live scoring through live coding with a subset of the Lilypond language. Being able to create music scores live through textual coding can take advantage of the tools that are being created for textual content generation, in a more domain-specific context. Using an RNN with LiveLily enables a different approach to AI-generated music, that of using a character-level GRU RNN to generate text rather than MIDI data. By generating music scores with a Lilypond-based language, many levels of expressiveness can be included, like dynamics, crescendi/diminuendi, articulation symbols, or even arbitrary text that can be interpreted at will by the performer. Combining LiveLily with a GRU RNN can lead to live coding sessions where the live coder uses the network as an instant composition assistant whose melodic line suggestions he/she is free to use, edit or discard.

From a compositional perspective, this project provides the composer with a tool for creation or experimentation, where AI can be used as a companion. One such case is an RNN trained on a hybrid corpus of music, providing musical patterns that blend compositional techniques. A composer can then draw inspiration for their own creations. From a performative perspective, this AI-enhanced version of LiveLily can enable fast creation of patterns in a live coding session, so the live coder can be partially relieved of the strain of generating new content in the context of the strict time constraints of a live session. As research on AI-generated content in various fields is still very active, there is still a lot to learn and a lot of room for refinement of the system presented in this article. Nevertheless, a solid foundation for experimentation has already been laid, that already works as a functional system.

REFERENCES

1. Han Wen Nienhuys and Jan Nieuwenhuizen, “Lilypond, a System for Automated Music Engraving”,  Musical Informatics Colloquium Proceedings (2003).
BACK

2. Matthew Wright and Adrian Freed, “Open SoundControl: A New Protocol for Communicating with Sound Synthesizers”, ICMC- International Computer Music Conference Proceedings, pp. 101-104 (1997).
BACK

3. J Austin, A Odena, M Nye, M Bosma, H Michalewski, D Dohan, E Jiang, C Cai, M Terry, Q Le, and C Sutton, “Program Synthesis with Large Language Models”, arXiv:2108.07732 [cs.PL] (2021).
BACK

4. P Knees, T Pohle and G Widmer, “Sound/Tracks: Real-Time Synaesthetic Sonification of Train Journeys”,  MM – Multimedia Conference Proceedings,  pp. 1117–1118 (2008).
BACK

5. Marinos Koutsomichalis and Bjorn Gambäck, “Algorithmic Audio Mashups and Synthetic Soundscapes Employing Evolvable Media Repositories”, ICCC -International Conference on Computational Creativity Proceedings (2018).
BACK

6. Sanchita Ghose and John J Prevost, “AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning”,  Transactions on Multimedia 23, pp. 1895-1907 (IEEE, 2021).
BACK

7. Tiffany Funk, “A Musical Suite Composed by an Electronic Brain: Reexamining the Illiac Suite and the Legacy of Lejaren A. Hiller Jr.”, Leonardo Music Journal 28, pp. 19–24 (MIT Press, 2018).
BACK

8. Iannis Xenakis, Formalized Music, Thoughts and Mathematics in Composition 
(Bloomington: Indiana University Press, 1972).
BACK

9. Curtis Roads, The Computer Music Tutorial (Cambridge: MIT Press, 1996).
BACK

10. G Marino, MH Serra and JM Raczinski, “The UPIC System: Origins and Innovations”, 
Perspectives of New Music 31, No. 1, pp. 258–269 (1993).
BACK

11. Jean Bresson, “Reactive Visual Programs for Computer-Aided Music Composition”,  
Visual Languages and Human-Centric Computing Symposium Proceedings, pp. 141–144 (2014).
BACK

12. A Hazzard, S Benford and G Burnett, “You’ll Never Walk Alone: Designing a Location-Based Soundtrack”, NIME- New Interfaces for Musical Expression Conference Proceedings, pp. 411–414 (2014).
BACK

13. HY Lin, YT Lin, MC Tien and JL Wu, “Music Paste: Concatenating Music Clips Based on Chroma and Rhythm Features”,  ISMIR – International Society for Music Information Retrieval Conference Proceedings, pp. 213–218 (2009).
BACK

14. R Manzelli, V Thakkar, A Siahkamari and B Kulis, “An End to End Model for Automatic Music Generation: Combining Deep Raw and Symbolic Audio Networks”, ICCC – Computational Creativity Conference Proceedings (2018).
BACK

15. Pedro Louzeiro, “The Comprovisador’s Real-Time Notation Interface”, CMMR – Computer Music Multidisciplinary Research Symposium Proceedings, pp. 489–508 (2017).
BACK

16. Kevin C Baird, “Real-Time Generation of Music Notation via Audience Interaction Using Python and GNU Lilypond”, NIME – New Interfaces for Musical Expression Conference Proceedings, pp. 240–241 (2005).
BACK

17. John Eacott, “Flood Tide See Further: Sonification as Musical Performance”, ICMC – International Computer Music Conference Proceedings, pp. 69–74 (2011).
BACK

18. Andrea Agostini and Daniele Ghisi, “Real-Time Computer-Aided Composition with Bach”, Contemporary Music Review 32, No. 1, pp.  41-48 (2013).
BACK

19. Georg Hajdu and Nick Didkovsky, “Maxscore – Current State of the Art”, ICMC – Computer Music Conference Proceedings, pp. 156–162 (2012).
BACK

20. D Fober, Y Orlarey and S Letz, “INScore: An Environment for the Design of Live Music Scores”, LAC – Linux Audio Conference Proceedings, pp. 47-54 (2012).
BACK

21. Bresson (2014) [11.]
BACK

22. AF Blackwell, E Cocker, G Cox, A McLean and T Magnusson,  Live Coding: A User’s Manual (Cambridge: MIT Press, 2022).
BACK

23. J Stewart, S Lawson, M Hodnick and B Gold, “Cibo v2: Realtime Livecoding A.I. Agent”, 
ICLC – International Conference on Live Coding Proceedings, pp. 20–31 (2020). BACK

24. Simon Hickinbotham and Susan Stepney, “Augmenting Live Coding with Evolved Patterns”,  in Evolutionary and Biologically Inspired Music, Sound, Art and Design (Springer, Cham, 2016) pp. 31-46.
BACK

25. Alex McLean, “Making Programming Languages to Dance to: Live Coding with Tidal”, 
FARM – Functional Art, Music, Modeling & Design Workshop Proceedings, pp. 63–70 (New York, 2014).
BACK

26. D Ogborn, E Tsabary, I Jarvis, A Cárdenas and A McLean, “Extramuros: Making Music in a Browser-Based, Language-Neutral Collaborative Live Coding Environment”, ICLC – International Conference on Live Coding Proceedings International Conference on Live Coding Proceedings,  pp. 163–169 (2015).
BACK

27. P Agrawal, S Banga, N Pathak, S Goel and S Kaushik, “Automated Music Generation using LSTM”, INDIAcom – Computing for Sustainable Global Development Conference Proceedings,  pp. 1395-1399 (2018).
BACK

28. Michael Cuthbert and Christopher Ariza, “music21: A Toolkit for Computer-Aided Musicology and Symbolic Music Data”, ISMIR –  International Society for Music Information Retrieval Conference Proceedings, pp. 637-642 (2010).
BACK

29. S Chen, Y Zhong and R Du, “Automatic composition of Guzheng (Chinese Zither) Music Using Long Short-Term Memory Network (LSTM) and Reinforcement Learning (RL)”, 
Scientific Reports 12, No. 15829 (Nature Publishing Group, 2022).
BACK

30. M Mots’oehli, AS Bosman and JP De Villiers, “Comparision Of Adversarial And Non-Adversarial LSTM Music Generative Models”, in Intelligent Computing (Springer, Cam, 2023) pp. 428–458.
BACK

31. Y Yu, A Srivastava and S Canales, “Conditional LSTM-GAN for Melody Generation from Lyrics”,  Transactions on Multimedia Computing, Communications and Applications 17, No. 1, pp. 1-20 (IEEE, 2021).
BACK

32. Adrien Ycart and Emmanouil Benetos, “Learning and Evaluation Methodologies for Polyphonic Music Sequence Prediction With LSTMs”,  ACM Transactions on Audio, Speech, and Language Processing 28, pp. 1328–1341 (IEEE, 2020).
BACK

33. G Li, S Ding, Y Li, and K Zhang, “Music Generation and Human Voice Conversion Based on LSTM”, MATEC Web of Conferences 336 (2021).
BACK

34. Y Huang, X Huang and Q Cai, “Music Generation Based on Convolution-LSTM”, 
Computer and Information Science 11, No. 3, pp. 50-56 (2018).
BACK

35. J Min, Z Liu, L Wang, D Li, M Zhang and Y Huang, “Music Generation System for Adversarial Training Based on Deep Learning”, Processes 10, No. 12 (2022).
BACK

36. P Arya, P Kukreti and N Jha, “Music Generation Using LSTM and Its Comparison with Traditional Method”, Advances in Transdisciplinary Engineering 27, pp. 545-550 (2022).
BACK

37. Victor Lazzarini, Steven Yi, John ffitch, Joachim Heintz, Øyvind Brandtsegg and Iain McCurdy, Csound:  A Sound and Music Computing System (Springer, 2016).
BACK

38. Alexandros Drymonitis & Nicoleta Chatzopoulou, “Data Mining / Live Scoring – A Live Performance of a Computer-Aided Composition Based on Twitter”, AIMC – AI Music Creativity Conference Proceedings (2021).
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

Alexandros Drymonitis > Cyprus University of Technology

Sound artist and researcher working with computer music, DIY electronics and robotics. His interdisciplinary practice spans performance, installation and experimental sound, integrating computational processes into embodied artistic contexts. He also lectures in audio-visual arts, contributing to pedagogy and research in technologically mediated creative practice.

Reviewers

Kate Sicchio > Virginia Commonwealth University

Artist, choreographer and researcher exploring how movement, code and embodiment intersect in live performance. As Assistant Professor at Virginia Commonwealth University, she connects creative practice with research in interactivity, dance technologies and algorithmic performance, examining how bodies and systems co-construct expressive and computational forms.

Jonas Runa > CICANT / Lusófona University

Artist-researcher working across sound, digital media and live performance. His work investigates perception, sonic experience and technological mediation, combining experimental practice with research in interactivity and audiovisual expression. As director of Lusófona’s Sound Programmes, he fosters interdisciplinary approaches to creative production and embodied digital arts – veziseriale.

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience

Continue with LIVE INTERFACES Volume 1:

Powerful machine learning (ML) image generators have recently become widely accessible to non-experts including artists, designers, and the general public, which has contributed to contentious discussions on the new aesthetics and creative practices that have emerged as a result. ML enables believable simulations to be produced based on statistical models trained on datasets of multitudes of other images. Generative ML systems have received a great deal of visibility for the affordances they offer, allowing realistic photographic images to be generated based on text inputs without significant technical knowledge. While images may be treated as the product of or interchangeable with data, the theoretical implications of this perspective are connected to longer tendencies in the history of photography and digital graphics, whereby technical modes of imaging have often been framed as observational rather than interpretive. Such perspectives on visual media are in need of critical reassessment as new methods expose a rift between speculative aspects entailed in imaging and the capacity for images to present visual knowledge (1). This research examines interdisciplinary practices working with data-intensive forms of visualisation in relation to existing theories on the epistemic qualities of visual technologies.

The widespread use of generative models to create images has resulted in particular aesthetic tendencies that often emulate the styles of other forms of visual media such as photography or painting while entailing significantly different processes and interpretive frameworks. This results in a discorrelation (2) between the subjective human perceptual interpretation of an image and the highly-automated technical systems that increasingly displace the importance of the latter. Although generative systems may create highly accurate visualisations based on data, the same techniques can also be used to produce images with little basis in reality. Synthetic images, for example, combine aspects of prior, observational approaches to imaging, with new generative methods, resulting in images whose accuracy is difficult to estimate. Not precisely scientifically objective, while also not being entirely fictional, synthetic images highlight the fact that the epistemic value of visual media is highly contextual. Considering these issues in relation to existing notions of scientific objectivity in visual media, this paper proposes a perspective on data-based aesthetics in which the production of images is understood as mediated through, rather than representative of data.

Figure 1: An image sourced online depicting a Russian T72-B3, via Forensic Architecture, 2018. Source: <http://content.forensic-architecture.org/wp-content/uploads/2019/09/spl1.jpg>

Figure 2: A Cinema 4D render of the same perspective of the tank in the previous image, Forensic Architecture, 2018. Source: <http://content.forensic-architecture.org/wp-content/uploads/2019/09/spl2.jpg>

SYNTHETIC IMAGES

Generative ML systems have facilitated the production of synthetic images (3) photorealistic digital renderings that can be used as training data for other ML systems. One way of thinking about this is that they are the product of synthesis rather than apparatus of capture, as in what are referred to in engineering contexts as real or natural images. But while this might seem like a straightforward distinction, it is less so on closer examination. For example, Goodfellow, Bengio, and Courville’s definition of a natural image hinges on the use of a camera:

“A natural image is an image that might be captured by a camera in a reasonably ordinary environment, as opposed to a synthetically rendered image, a screenshot of a webpage, etc.” (4)

While this is a functional definition, it doesn’t touch on a range of nuance that occurs between these categories of natural image and synthetic image, especially since digital photography is now the norm and it is increasingly commonplace for cameras to involve artificial intelligence (AI) features to enhance the images they produce. There is room for a great deal of variability in the qualities of images that have been captured by a camera. Conversely, a screenshot of a webpage can act as an accurate form of documentation of its visual content. The photographic paradigm of imaging, which is currently most dominant, also assumes a particular form of visual referentiality that is not always the case. Images are not necessarily representative of real-world objects and phenomena, and there are many aspects open to variability in the case of representational images.

Looking into this further, we can take insights from the work of Forensic Architecture, which often blurs the boundaries of fact and fiction. In many of their projects, the group aims to generate data that may be useable as a form of legally-recognised evidence, often for the purpose of tangibly proving disputed cases of human rights violations. Discussing their experimentation in the creation of synthetic data, Forensic Architecture explain:

“In order to train machine learning classifiers to detect a given object (the ‘object of interest’) in an image or video frame, a large dataset of images of the object of interest is usually necessary. Images of the objects of interest in the context of human rights violations (for example military vehicles, or chemical weapons) are often rare, however. Forensic Architecture has been experimenting with how to supplement a dataset of real images with additional ‘synthetic’ counterparts.” (5)

From this description, we understand that synthetic images arose in response to the problem of acquiring sufficient training data for a given kind of object. We may also note that there may be a reciprocal relationship between real and synthetic images, with the possibility for synthetic images to be used in place of real images. The figures above demonstrate this interrelationship, Fig. 1 depicting a Russian tank sourced from the internet, while Fig. 2 is a synthetic image of the same perspective of the tank in the previous image that has been generated by Forensic Architecture. Though they are fairly similar in appearance, there are subtle visual cues, such as resolution and detail, that perceptibly allow us to distinguish that one is more likely a real photograph, while the other has been generated. Yet, it’s easy to imagine situations where two such images could be visually indistinguishable. Indeed, Fig. 2 is an example of training data that has been generated, rather than gathered, but this leaves a lot of questions unanswered concerning the defining qualities of these two categories, especially given the fact that images sourced from the internet can have dubious truth value.

While images can, in practice, be used as data, for example, to train a model to produce other images, the relational aspects of synthetic images may render the distinction of real images merely positional within a given context, rather than being rooted in inherent qualities of the images, themselves. In this sense, synthetic images may subvert the forms of objectivity traditionally associated with the visual realism of optical media. For example, highly realistic images can be generated of virtually anything. In a well-known case, researchers demonstrated that it is possible to generate believable photographic images of people who do not exist (6). This is a case of simulacra, which Jean Baudrillard defines as copies without originals, describing simulation as “no longer that of a territory, a referential being, or a substance. It is the generation by models of a real without origin or reality: a hyperreal.” (7)

In generated images, or in this case, synthetic faces, each instance is a copy produced from a model of the human face. But no face is intended as a direct representation of an actual face that exists in the world. But another team of researchers then showed that this was not as straightforward as it initially seemed, in a paper entitled “This Face Does Not Exist … But It Might Be Yours!” (8), where the authors propose that the same system may also reproduce likenesses of real people from its training data. In this sense, it is difficult to say what such a generated image represents, lacking adherence to a specific real-world referent, while entailing the potential to replicate patterns in data with great accuracy.

ML tools can be used in scientific contexts to create highly accurate images, and in some cases this entails processes of visualizing the non-visual or the invisual (9). The relationship between the visual and processual aspects of images has been widely discussed in relation to the concept of the operational image, a term originally coined by Harun Farocki (10). He describes this kind of image as being defined by its performance of a spatial operation as opposed to its visual qualities as traditional perspectives on image-making.

An interesting case of this is the images of black holes produced by the event horizon telescope collaboration (11). In this case, a visualisation is made based on radio frequencies. The original phenomenon is non-visual in nature, meaning that we only have computational measurements to judge against, rather than being able to visually compare the representation against its referent. In another project called Cloud Studies by Forensic Architecture (12), visualisations are produced of cloud phenomena that are extremely ephemeral, and again, often non-visual. Using data visualisation techniques from gathered data about pollutants, or chemical weapons, they were able to track various environmental phenomena and create evidence of alleged criminal attacks that would be otherwise impossible to trace. Another growing area where ML has very promising applications is medical imaging and image analysis, where it can be used to help detect or even predict cancer, or to assist in surgeries.

But while there are cases where ML is able to create very accurate scientific images, a widely discussed aspect of ML is that it can also be subject to serious problems of inaccuracy and bias. This has been notably highlighted by researchers such as Joy Buolamwini, Timinit Gebru, Angelina McMillian-Major and Margaret Mitchell (13), who have pointed to its tendency towards further entrenching already existing inequalities including erring in ways that are deeply gendered, racist, homophobic, and classist. Beyond existing issues in the design, application, and management of ML systems, there is also an interpretive aspect of how we assess their outputs. For example applied to other contexts such as art and design, the resulting images are subject to different frameworks for analysis, interpretation, and evaluation than in technical and scientific research. The interrelationship between art and technoscientific methods, apparatuses, and aesthetics has a habit of destabilising the expectations we have about what the results of these processes mean. For example, in the work of artists who borrow or appropriate techniques from the sciences, the outcomes are then thrown into a new relationship with the world that is not governed by the same principles.

Aspects of the scientific worldview have been normalised to the point that they are taken as given, neutral, even. Artworks may be created using technical and scientific methods, but what they represent has little to do with scientific knowledge. They result, ultimately, in an aesthetic experience that uses the language of scientific imaging to communicate something else. There is therefore a need to develop methods, criteria and conceptual frameworks, such as those in art history and criticism, that allow us to grapple with what is entailed in such instances, where art coincides closely with science and technology.

Dan McQuillan proposes the approach of standpoint theory as a potential way forward from what he calls the necropolitics of AI, saying that “standpoint theory suggests the possibility of alternative ways knowing, rooted in the lived experience of people who are marginalized or minoritized. It is, first and foremost, a challenge, to the aura of absolute objectivity that places the scientific methods above other ways of knowing” (14). Artistic practices explore precisely these alternative ways of knowing that McQuillan describes, and I think critical explorations of synthetic images can help develop new frameworks for grasping them. For example, in the series Material Speculation of Moreshin Allahyari (Fig. 3) (15), the artist created 3D-printed models of cultural heritage destroyed by ISIS. Using what documentation remains of artworks that no longer exist, materially, the cultural memory of these artefacts is revived through the reinterpretation of known facts and concrete data such as YouTube videos of their destruction. Data sticks containing digital files for the 3d models the artist created are then embedded in the sculptures, in a direct sense combining the data with its spatial manifestation. In the process of reconstructing 3d models of these artefacts, Allahyari applies a speculative approach that is grounded in computational methods. The objects that are created through this process are both based in data, while also producing something new through the reinterpretation of the past, undermining the idea of data being self-evident or capable of standing in for its referent.

Figure 3: Material Speculation: ISIS: Marten, Morehshin Allahyari, 2016. © 2022 Morehshin Allahyari.

In a similar sense to the appropriation of technical and scientific methods outside of scientific contexts, synthetic images result in a divergence from prior conceptions of scientific objectivity in visual depictions, such as those outlined by Lorraine Daston and Peter Galison in their book Objectivity. They look specifically at conceptualisations of objectivity in scientific atlas images, from which they outline three main forms, or epistemic virtues (16). The first of these is truth-to-nature, referring to the verisimilitude or quality of being true-to-life in terms of visual referentiality. Botanical and biological illustration are good examples of this. A second form of objectivity is mechanical objectivity, in which direct physical mediation of a phenomenon occurs such as through photography or microscopy. The third form of objectivity Daston and Galison discuss is trained judgement, where the image does not aspire to visual realism, but rather involves a form of visual notation that must be interpreted in a particular, informed way. Importantly, these forms of objectivity are not mutually exclusive: 

“Atlas images — whether rea­soned, mechanical, or interpreted — bear the marks of both episte­mology and ethos. [Objectivity] has traced how epistemology and ethos emerged and merged over time and in context, one epistemic virtue often in point-counterpoint opposition to the others. But although they may some­ times collide, epistemic virtues do not annihilate one another like rival armies. Rather, they accumulate: truth-to-nature, objectivity, and trained judgment are all still available as ways of image making and ways of life in the sciences today.” (17)

Something that Daston and Galison point out that draws an interesting parallel with machine learning is that these methods entail aspects of sample selection. Striving to present visual information in the most accurate way possible also involves choosing a sample that is deemed representative of the phenomenon depicted. This relationship between the many and the one involves the perfecting of nature, selecting an instance that can serve as an exemplary model of a given phenomenon. In that sense, these images strive to capture a generalisation in a specific instance, something that is echoed in the outputs of machine learning models. Lacking a steadfast relationship with direct observation, technical process, or trained judgement, recent visual practices such as creating images using generative models may mark a turn from representational towards what Daston and Galison refer to as presentational strategies of visual depiction:

“At this point, the relationship of science to aesthetics has de­parted from all our earlier models. Art and science are not self-evidently a single enterprise (few today assume that the True and the Beautiful must necessarily converge), nor do they stand in stalwart opposition to each other. Instead, they uneasily but productively reinforce each other in a few borderline areas.” (18)

The synthetic images generated by an ML system can be difficult to differentiate — aesthetically, computationally, and philosophically — from real or natural images. In this sense, they disrupt existing conceptual frameworks for technically-mediated vision, and give rise to new epistemic relationships. While generative image-making systems may be based on the large-scale analysis of data through highly technical methods, they are capable of producing outputs with variable degrees of accuracy.

DATA-BASED AESTHETICS

Discussing these ideas using the term data-based aesthetics invokes both the idea of an image being based on data, as in the synthetic images that are generated computationally from data, and the notion of aesthetics that present a form of visual knowledge. The view of images as the product of, interchangeable with, or as a form of data, in themselves, is connected to longer tendencies in the history of visual technologies, and it is important to reconsider how the epistemic basis of imaging may be reliant on problematic assumptions inherited from older imaging paradigms. Understanding media artifacts as data-based requires new perspectives that are informed by the processes and data behind the images, as opposed to viewing them in isolation.

Anticipating recent developments over fifteen years ago, Victoria Vesna addressed some of these issues in her book Database Aesthetics:

“In an age in which we are increasingly aware of ourselves as databases, identified by Social Security numbers and genetic structures, it is imperative that artists actively participate in how data are shaped, organized, and disseminated.” (19)

This perspective is a poignant reminder that while it may be commonplace to treat data as interchangeable with all things, the relational aspects of data are consequential, and aspects of the things that data stand in for are irreducible. Alva Noë notes, that “nothing is a model by virtue of its intrinsic makeup alone.” (20) Models are ways of exploring the world or accomplishing certain goals. A model is useful, successful, accurate only insofar as it achieves a purpose. So ultimately, it is not the intrinsic qualities of the model that define it as such. In this sense, the measure of a model is its functionality or efficiency at achieving whatever purpose it is applied to. The danger of considering data-based images as separate from the models and worldviews they emerge from would be to neutralise the particular position they derive from.

Like data, models are relational in nature, and this can meaningfully inform our perspective on visual media. One aspect of this is that data don’t tell us much in themselves, and they are always subject to interpretation. This idea is informed by the view of postphenomenolgy (21), which asserts that the mediation of perceptual experience sets up hermeneutic relationships in which what is tangible of the original phenomenon is interpreted and altered in the process. As Daston and Galison describe, recent forms of data-based imaging may constitute a fusing of the artifactual with the natural (22), a form of epistemic practice that produces visual knowledge.

In Johanna Drucker’s work on visual epistemology and interpretive digital tools, she says “The field of visual epistemology draws on an alternative history of images produced primarily to serve as expressions of knowledge.” (23) It holds a base assumption that knowledge may be expressed in visual form, which is something that is often questioned with regard to art. Drucker goes on to explain that the interpretation of visualisations is intimately entangled with the visual interpretation of information, highlighting the contextual and interpretive nature of data. The aesthetic, conceptual and methodological approach applied in image-making may inform the way images are in turn interpreted, and this is further complicated when we add in a basis in data.

Aurora Hoel proposes the operationally real as an alternative to dichotomies typically applied to separate the real from the virtual or the unreal:

“The operationally real is an intermediate reality instituted and sustained by the mediating or resolving action of some technical apparatus (e.g., a living body or a technical machine) that plays the role of adaptive mediator. This implies that the operationally real is a mixed environment of a middle order of magnitude—a complex milieu made up of heterogeneous and conflicting elements and powers that have been brought into communication by technicity.” (24)

In the sense that synthetic images may perform the function of real images, in training an ML system and to be included in their training data, these images may be considered to be operationally real. As Hoel puts it, “The operationally real is a mixed reality, a virtuality instituted and sustained by mediation.” (25) To think of data-based imaging practices from this perspective enables us to acknowledge the positional and relational aspects of data and visualisations based on it, as opposed to attempting to divide these into separate categories. Just as Noë pointed out about models not being defined by their intrinsic makeup, but rather the goals they are applied to, so to can this apply to the data basis of images.

CONCLUSION

In comparison to previous paradigms of image-making, recent technical developments entail complex relationships between the perceptible and its interpretation. The processes behind an image often matter to its interpretation, even if they are not visually accessible within the image. And our understandings of technology inform the expectations we have for interpreting these relationships between the visual and the processual. Consequently, highly technical forms of visual media may require more knowledge external to the image itself, reshaping the way we visually decipher images but also how we manage their contextualisation within larger frameworks beyond the strictly visual.

The ways in which AI influences visual mediation can be difficult to predict, evaluate, remediate, or resist. This underscores the need for perspectives that may challenge the dominant narratives around AI. With the magnitude of visual data that are produced, circulated, and analysed on a daily basis, it is increasingly important to consider alternative perspectives, such as those proposed by standpoint theory, visual epistemology, or the view that there are multiple forms of visual objectivity. Ultimately, the idea of the image as the product of or a form of data itself is one that is incredibly nuanced and it requires us to develop new modes of critical enquiry in order to grasp emerging, data-based aesthetics.

REFERENCES

1. Johanna Drucker, Visualization and Interpretation: Humanistic Approaches to Display (Cambridge: MIT Press, 2020).
BACK

2. Shane Denson, Discorrelated Images (Durham: Duke University Press, 2020).
BACK

3. Forensic Architecture, Experiments in Synthetic Data (2018) <forensic-architecture.org/investigation/experiments-in-synthetic-data>, accessed November 2023.
BACK

4. I Goodfellow, Y Bengio and A Courville, Deep Learning (Cambridge: MIT Press, 2016).
BACK

5. Forensic Architecture (2018) [3.]
BACK

6. T Karras, S Laine and T Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks”, Computer Vision and Pattern Recognition Conference Proceedings, pp. 4401-4410 (2019).
BACK

7. Jean Baudrillard, Simulacra and Simulation (Ann Arbor: Univ. Michigan Press, 1981).
BACK

8. P Tinsey, A Czajka, and P Flynn, “This Face Does Not Exist … But It Might Be Yours! Identity Leakage in Generative Models”, arXiv:2101.05084 [cs.CV] (2020).
BACK

9. Jussi Parikka, Operational Images: From the Visual to the Invisual (Minneapolis: University of Minnesota Press, 2023).
BACK

10. Harun Farocki, “Phantom Images”, Public 29, pp. 12–22 (2004).
BACK

11. Event Horizon Telescope Collaboration, “First M87 Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole”, The Astrophysical Journal Letters 875, No. 1 (2019).
BACK

12. Forensic Architecture, Cloud Studies (2020) <https://forensic-architecture.org/investigation/cloudstudies>, accessed November 2023.
BACK

13. E Bender, T Gebru, A McMillian-Major and Margaret Mitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”, FAccT ’21 – ACM Conference on Fairness, Accountability, and Transparency Proceedings, pp. 610–623 (2021).
BACK

14. Dan MacQuillan, Resisting AI (Bristol: Bristol University Press, 2022) p. 105.
BACK

15. Moreshin Allahyari, Material Speculation: ISIS: Marten (2016) <https://morehshin.com/material-speculation-isis>, accessed November 2023.
BACK

16. Lorraine Daston and Peter Galison, Objectivity (New York, Cambridge: Zone Books, 2007) pp. 39–41.
BACK

17. Daston and Galison, p. 363 [16.]
BACK

18. Ibid., p. 412.
BACK

19. Victoria Vesna, Database Aesthetics: Art in the Age of Information Overflow (Minneapolis: University of Minnesota Press, 2007) p. xiii.
BACK

20. Alva Noë, Strange Tools: Art and Human Nature  (New York: Hill and Wang, 2015) p. 153.
BACK

21. Don Ihde, “Introduction: Postphenomenology”, in Postphenomenology: Essays in the Postmodern Context (Evanston: Northwestern Univ. Press, 1993) pp. 1–8.
BACK

22. Daston and Galison, p. 413 [16.]
BACK

23. Drucker, p. 1 [1.]
BACK

24. AS Aurora Hoel, “Technicity and the Virtual”, Humanities 11, No. 6 (2022) p. 18.
BACK

25. Hoel, p. 20. [24.]
BACK

ABOUT THE AUTHOR, REVIEWERS AND EDITOR

Author

Rosemary Lee  > i2ADS / University of Porto

Artist and media researcher whose work explores the intersections of art, technology and speculative media futures. As an integrated researcher at i2ADS and lecturer at FEUP, she examines machine learning’s cultural implications and develops artistic projects that investigate historical, material and discursive entanglements in digital media practices.

Reviewers

Jonas Runa > CICANT / Lusófona University

Artist-researcher working across sound, digital media and live performance. His work investigates perception, sonic experience and technological mediation, combining experimental @ filme online, practice with research in interactivity and audiovisual expression. As director of Lusófona’s Sound Programmes, he fosters interdisciplinary approaches to creative production and embodied digital arts.

Rui Antunes > CICANT / Lusófona University 

Artist and researcher whose work spans digital arts, live audiovisual performance and human–machine interaction. He develops installations and performances exploring embodiment, experiential design and computational media, integrating sound, image and interface technologies to examine how artistic practice responds to shifting modes of technological engagement

Editor

Adriana Sá > CICANT / Lusófona University

Artist and researcher working at the intersection of sound, performance and interactive media. She investigates perception, embodiment and interface design, integrating intuitive artistic practice with rigorous research. Her work examines how interfaces can function simultaneously as expressive systems and critical tools for exploring technologically mediated experience.

Continue with LIVE INTERFACES Volume 1: