2.1. Non-digital modular music

While computers have unquestionably greatly expanded the means by which music can be created, manipulated and interacted with, composers throughout history have still demonstrated the ability to create structurally plastic musical works through written notation and performance instructions. Perhaps the earliest recorded examples of this are the musical dice games of the eighteenth century such as The Minuet and Polonaise Composer by Johan Philipp Kirnberger or Musikalisches Würfelspiel commonly attributed to Mozart (Hedges Reference Hedges1978). Such works had no inherent structure but instead required audiences to formulate their own structure by cutting and pasting measures of a notated score based on the rolling of dice. In the twentieth century, a number of experimental composers moved towards a level of structural variability by integrating indeterminate elements into how pieces could be performed. For example, via interoperative graphic notation such as in Feldman’s Projections pieces (Boutwell Reference Boutwell2012) or Cage’s Variations I–VIII (1958–78), or by providing only text-based instructions such as in Cardew’s Schooltime Special (1968). Another approach observable in many works is for the various sections of music to be modular, in that the order of musical events can be rearranged in some way. Examples include Stockhausen’s Klavierstück XI (Reference Stockhausen1956) in which 19 brief musical sections may be played in any order, Brown’s Available Forms I (Reference Brown1961) which allows a conductor to freely guide an ensemble through any part of a six-page score, and Cowell’s Mosaic Quartet (Cowell Reference Cowell1935) which contains five movements of no set sequential order. Of a similar disposition are a number of minimalist works such as Riley’s In C (Reference Riley1964) or the original version of Shaker Loops Footnote ¹ (1978) in which music is structured into multiple looping modules that can be variably sequenced and layered with each other. The modular nature of these types of works showcases not only the unique ability for a musical work to vary significantly with each performance but also the capability for a work to be endowed with a level of changeability such that it can be reshaped beyond the direct control of the composer.

In the structurally indeterminate pieces of this period, the performer takes on a unique role; they are no longer just a performer, but also a collaborator invited to creatively contribute to the work itself (Eco Reference Eco and Cancogni1989; Harries Reference Harries2013). Further, the composition itself is no longer strictly delineated but constitutes a type of musical boundary within which varying levels of freedom and expression can be exercised; the composer actively chooses to relinquish control over a work’s structure to forces beyond themselves. In such cases, however, with the exception of the musical dice games, it is difficult to relinquish this power to anyone but an experienced instrumentalist, nor is it readily possible for audiences to participate in this compositional collaboration. These non-digital instances of modularity in composition showcase the creative affordances of allowing a work’s structure to be plastic, but it begs the question of how much further this concept could be taken when utilising digital technology.

2.2. Practices in video game composition

In responding to the interactive aspect of video games, composers have developed a new range of compositional practices heavily integrated with game technology that afford varying levels of structural plasticity depending on creative needs or intentions. The video game framework has been incorporated into a variety of experimental compositional projects from musical puzzle games solved by performing an augmented drumkit (Michalakos Reference Michalakos2021), to virtual environments that encourage competitive composition (Studley et al. Reference Studley, Drummond, Scott and Nesbitt2020), and even to adaptive soundtracks that accompany players within a physical escape room (Délécraz Reference Délécraz2023). However, within more commercial settings, music primarily functions to enhance the players’ emotional experience (Stevens Reference Stevens, Fritsch and Summers2021; Michelmore Reference Michelmore, Fritsch and Summers2021) despite the challenges posed by interactivity. As such, music must, in a sense, narrativise player actions while also satisfying the aesthetic expectations (e.g., instrumentation, style, mood) imposed by a game’s overall design and emotional intentions.

Dynamic music in video games commonly involves the creation of music as separate events that can be rearranged and modified during gameplay. These events, which often exist as rendered digital audio, have been referred to as ‘segments’ (Aska Reference Aska2017), ‘stems’ (Michelmore Reference Michelmore, Fritsch and Summers2021), or ‘modules’ (Medina-Gray Reference Medina-Gray2016; Zdanowicz and Bambrick Reference Zdanowicz and Bambrick2019) as they will be named hereafter. The methods involved in rearranging and modifying these modules can be broken down into two general categories: vertical methods and horizontal methods. Both techniques are discussed in greater detail in section 3.2, but to overview, vertical methods generally refer to the addition, subtraction or substitution of musical layers to make changes to the overall musical arrangement/texture, while horizontal methods refer to changes made to how music progresses over time, that is, the sequencing of musical events. These methods have been discussed in depth by various video game composers (Paul Reference Paul and Moormann2013; Phillips Reference Phillips2014; Sweet Reference Sweet2015; Zdanowicz and Bambrick Reference Zdanowicz and Bambrick2019; Michelmore Reference Michelmore, Fritsch and Summers2021) and many ludomusicologists (Collins Reference Collins2008a; Summers Reference Summers2016; Aska Reference Aska2017; Medina-Gray Reference Medina-Gray2019). The benefit of using digital audio to present sound and music within games is that sonic content can be meticulously crafted according to the stylistic needs of the game while still exhibiting dynamic behaviour by being alterable in how and when constituent audio modules are heard. While the use of audio (a fixed form of media) limits the flexibility of music compared with algorithmic approaches, the favouring of audio-based techniques in modern video games (Plut and Pasquier Reference Plut and Pasquier2020) implies that not only are these techniques suitable for most adaptive music situations, but they also offer certain advantages. These advantages include greater approachability, better stylistic control, and the ability to work within computer processing and memory restrictions, which can outweigh the benefits of highly flexible, algorithmically driven music.

The technological means by which these audio-based methods have been integrated would generally involve programming dynamic behaviour directly within the game engine. However, within the last decade, audio middleware programs such as Wwise (Audiokinetic 2023), FMOD (Firelight Technologies 2023), Elias 4 (Elias Software 2023) and ADX2 (CRI Middleware Co. Ltd 2024) have emerged as more accessible and reliable solutions to integrating sound and music into games. As a result, while these programs provide the tools for creating dynamic music, the video-game-oriented design of these programs arguably inhibits their potential application to other forms of art media, such as enhanced instrument design or sound installations.

A modular approach to game scoring has been integrated into thousands of games; discussions concerning its value as a compositional approach are commonplace within the domain of video game development. However, owing to the close relationship between gaming and modular composition, the latter is rarely discussed on its own terms. The modular compositional approach outlined in this article is highly influenced by dynamic game music practices and seeks to understand its implications on composition detached from considerations defined by the game context.

2.3. Interactive music apps

In a parallel domain to video games, interactive apps that run on smartphones and tablets have hosted an eclectic array of interactive musical works that feature (to varying extent) a level of plasticity in their structures.

In referencing his CEMS system, Joel Chadabe notes that the duality of his system as being both a composition and an instrument also holds true for many music apps (Chadabe Reference Chadabe2015). Apps from Brian Eno such as Bloom (Reference Eno and Chilvers2008), Trope (Reference Eno and Chilvers2009) and Scape (Reference Eno and Chilvers2012) may be considered types of instruments in that they allow audiences to generate their own soundscapes based on how they interact with them. However, the specific ways users can interact with these apps and the range of sounds they can produce give each app a distinct identity, meaning one could also consider them as individual compositions. Other apps that fall into this category include Thicket by Joshue Ott (Reference Ott2010), Bubble Harp by Scott Snibbe (Reference Snibbe2011a), and Borderlands Granular by Chris Carlson (Carlson and Wang Reference Carlson and Wang2012). Composition in this context leans more towards aspects of system programming and interface design through which musical content can be created than the creation of the content itself.

Another category of interactive music apps are those which invite the audience to participate within an auditory world of the composer’s own making. Björk’s Biophilia app (Snibbe Reference Snibbe2011b) and Radiohead’s Polyfauna app (Pyke Reference Pyke2014) both feature musical content created by their respective artists and invite the audience to add to or shape it by interacting within different types of virtual environments. The variPlay system similarly allows artists to consolidate their music into an app format from which listeners are invited to move between alternate mixes and arrangements as the music is playing (Paterson et al. Reference Paterson, Toulson and Hepworth-Sawyer2019). Location based apps such as National Mall (Bluebrain 2011) and the Daoplayer (Hazzard and Greenhalgh Reference Hazzard and Greenhalgh2019) follow similar principles to adaptive video game music and recombine recorded audio events to provide musical experiences based on where the user is and how they move through specific locations. Music in these cases is specifically crafted by the composers/artists and then situated within an interactive system from which audiences are provided with different levels of agency to add to, alter or explore that music. However, with the exception of the Daoplayer, many of these systems rarely enable variability beyond textural alterations. Compared with video games, interactive music apps that incorporate composed music are limited in their musical variability, leaving room for further exploration into how an interactive system can use pre-composed musical material to produce works with a high degree of structural plasticity.

2.4. AI in music: composition and digital music systems

In the context of this discussion, digital music systems are those that generate music via a computer-based algorithmic process. The function and methods behind these systems greatly vary (Herremans et al. Reference Herremans, Chuan and Chew2017), and there is increasing interest in the implementation of deep learning techniques to generate music (Briot et al. Reference Briot, Hadjeres and Pachet2020; Civit et al. Reference Civit, Civit-Masot, Cuadrado and Escalona2022; Ji et al. Reference Ji, Yang and Luo2023), among other forms of machine learning (Roberts et al. Reference Roberts, Engel, Mann, Gillick, Kayacik and Nørly2019; Ens and Pasquier Reference Ens and Pasquier2020; Pachet et al. Reference Pachet, Roy, Carré and Miranda2021). Of particular relevance to this article are the plethora of intelligent music systems designed to engage in compositional activities.

Emerging within the field of Music Metacreation and Human Computer Interaction are numerous digital music systems designed for creative collaboration (D’inverno et al. Reference D’inverno, Gifford, Hutchings, Llano, McCormack and Yee-King2020). From learning and replicating a composer’s style (Lupker Reference Lupker2021) responding to live improvised performance (Erdem et al. Reference Erdem, Wallace and Jensenius2022), to generating multitrack MIDI sequences (Ens and Pasquier Reference Ens and Pasquier2020; Dong et al. Reference Dong, Chen, Dubnov, McAuley and Berg-Kirkpatrick2022), there is a noticeable trend towards using AI as a collaborative creative tool. Notably, plugins such as Google’s Magenta Studio (Roberts et al. Reference Roberts, Engel, Mann, Gillick, Kayacik and Nørly2019) and Sony CSL’s Flow Machines (ibid.) use machine learning to generate new MIDI based musical ideas as a source of creative inspiration for musicians. There is also an emerging trend of AI music tools that are less oriented towards collaboration, but rather involve the full generation of musical works. Systems such as OpenAI’s Jukebox (Dhariwal et al. Reference Dhariwal, Jun, Payne, Kim, Radford and Sutskever2020), Meta’s MusicGen (Copet et al. Reference Copet, Kreuk, Gat, Remez, Kant and Synnaeve2023) and Google’s MusicLM (Agostinelli et al. Reference Agostinelli, Denk, Borsos, Engel, Verzetti and Caillon2023) can be prompted with a type of genre, artist or even a full text description to generate full audio tracks based on that information. For these types of systems, the human user plays a less active role in music creation, instead guiding the creation process by providing descriptive prompts that the system subsequently uses to generate new tracks. However, in the cases outlined thus far, artificially intelligent music systems exist as tools for music creation rather than existing as musical works themselves.

With the growing integration of digital technology and programming in music-making, the term ‘musical work’ may extend to define systems that generate music. This is apparent in the range of programming languages designed for creating algorithmically generated music such as Max/MSP, PureData, SuperCollider or Csound. Within these environments, instead of creating musical content directly, composers create works by modelling composition itself (Collins Reference Collins2008b); they create musical works as systems, and these systems make music. Further examples of this concept can be found within the growing field of affectively driven algorithmic composition in which systems are designed to generate music with specific emotional qualities and affective intentions (Williams Reference Williams, Williams and Lee2018; Williams et al. Reference Williams, Hodge and Wu2020). Additionally, within the field of Music Metacreation there has been extensive experimentation with the creation of quasi autonomous music agents designed to complete creative tasks such as composition, live accompaniment, content generation and arrangement (Tatar and Pasquier Reference Tatar and Pasquier2019; Carnovalini and Rodà Reference Carnovalini and Rodà2020). The act of composing, in these cases, is the act of designing and programming the systems that make music. Structural plasticity is apparent in these systems in that both musical structure (and indeed musical content) is generated in real time and may be guided towards different structures by altering the system itself.

3.1. Music module

A musical module – or simply module – may be defined as an individual pre-composed musical idea, sequence, phrase or progression belonging to a broader collection of modules that constitute a modular musical work. The content of a module is expected to be designed such that spontaneous and variable musical interactions occur as a module is layered and/or sequenced with other modules belonging to a greater collection. This also implies that modules are sonically complementary to their contemporaries. Further, this collection, as well as the ways in which modules rearrange and interact, is what defines the modular composition itself.

Since the variable sequencing of modules is a core aspect of what enables a modular work to be structurally plastic, the duration of a module is generally expected to range from approximately half a second to a minute. This is to ensure that modules can follow one another at a more frequent rate, allowing the musical progression to be more changeable and thus showcasing a greater degree of structural plasticity within the work.

Within our own creative experimentation (presented in section 4), music modules were created in the form of individual audio files. This choice was made for several reasons, the most notable of which being the research focus on structural plasticity as driven by the rearrangement of broader musical components such as musical sequences, phrases and progressions, which can easily be represented by digital audio. Another significant motivation was the fact that recorded audio can be drawn from multiple sources, including human performances on acoustic instruments, sonically versatile virtual instruments, digital or analogue synthesised sounds, and field recordings to name a few. This diversity of potential sound sources provides a broad sonic palette that can be accessed through relatively common music-making tools and is challenging to replicate through synthesis alone. In our case, modules were composed, produced and rendered into audio using Cubase Pro 11, and audio sources included a variety of virtual instrument VSTs as well as recordings of our own acoustic instrument performances.

3.2. Horizontal and vertical methods

Considering the inherent separation of musical material present in this modular framework, the methods by which musical content is combined into new structures carries much compositional significance. The methods outlined here can be broken down into two categories: horizontal and vertical.

Horizontal methods refer to changes made to the succession of musical ideas. This essentially refers to how modules are sequenced and the ways in which broader structures are created via their variable sequential progression. By allowing one module to be followed by a number of potential others, musical progression can branch off in multiple directions.

Vertical methods refer to the addition, subtraction or substitution of musical layers to make changes to the overall musical arrangement/texture. This can be approached in multiple ways, though here we outline two main approaches: verticality within modules (submodules) and verticality through module groups (voices).

Verticality within modules can be achieved by dividing a module into several submodules, with each submodule containing audio for a different musical layer of the main module. For instance, a module containing a single four-bar orchestral phrase might contain separate audio layers (or stems) for each section of the orchestra, or that same module could instead contain several recordings of that phrase played at different levels of intensity from which only one is selected to play.

Alternatively, vertical changes may be made by organising modules into separate layer groups which we refer to as voices. A voice may be defined as a distinct layer within a modular composition, characterised by its specific set of modules that can only be recombined via horizontal methods. So, while the modules within one voice can only vary in their sequence, other voices can also operate concurrently, allowing modules from each voice to play in parallel and combine in variable ways. This brings about considerations regarding the synchronisation of voices to a common metre or tempo but also opens up unique polyrhythmic and polyphonic opportunities.

Both horizontal and vertical methods provide a general means of arranging and moulding the components of a modular composition into alternate structures. Questions regarding the composition of music to suit these methods are highly dependent on how they are intended to be employed and the creative intentions of the composer. The main concern here, however, is how these methods enable musical structure to be fluid, and how they enable the composer to relinquish certain structural decisions to be made within or through the system that the overall composition is situated in.

3.3. System design: relinquishing structural decision-making

Since a modular composition has no inherent structure, all its components must be situated within a system that handles the reconstruction of these components into a single flow of music. The system is just as much a part of the composition as the music itself. The approach to designing such a system fundamentally involves programming behaviours regarding how modules are dynamically rearranged and how the system responds to various inputs such as user interaction or randomisation. This is quite broad and will greatly depend on the musical content, the context of the composition and the creative intentions of the composer. So, while the approach to system design outlined here is not exhaustive, it nonetheless demonstrates an informative process by which modular composition can result in a structurally plastic and living musical work.

Our approach to system design comes down to several key processes: mapping the non-linear structure of a whole work; delineating the means by which modules interact; and defining the boundaries of autonomous behaviour, that is, the freedom by which a work may reshape itself. The latter process also includes considerations of interactivity and the level of control relinquished from the composer to either the system controlling the work or the audience.

4.1. Sum

Sum Footnote ² seeks to showcase variable structural progression in terms of both harmony and textural density. The work comprises three voices each representing alternate musical textures conceptualised as spatial perspectives (e.g., background, midground and foreground; Figure 2). Voices and their respective modules are distinct in their instrumental arrangement, with the overall instrumentation similar to that of a chamber orchestra. Further, modules from both the background and the midground voice contain submodules associated with different instrument groups. These submodules are programmed to have a 50% probability of being heard when the module is played, meaning the instrumental arrangement of a single module can vary slightly. The audio content of modules in this case include a mix of virtual instruments, covering many of the string and wind sounds, as well as some live recorded acoustic guitar and violin.

Figure 2. Structural map of Sum.

The work includes a chordal parameter with five variable states, each representing a different diatonic chord within the key of D minor (Dm, Am, Gm, B♭, C) from which all modules were accordingly composed and categorised. Musical material is not metric, meaning the approach to module rearrangement between voices is asynchronous.

The modules within this work generally range between 15 and 25 seconds in duration. Since there is no tempo or metre, modules tend to slowly swell and ebb in terms of volume and intensity to avoid abrupt transitions in and out of silence. Modules from the foreground voice, however, were exempt as their melodic role suited a more abrupt and sudden beginning.

Both the background and the midground voice contain at least four unique modules for each of the five harmonic states. The foreground voice differs in that it can be set to three different instruments. As such, each instrument in the foreground voice contains four modules per harmonic state. All together this work comprises over one hundred unique modules.

In terms of how overall structural decisions are made, the voices and parameters are the main vehicles by which structure may be actively altered. In this case, the GUI allows the user/listener to make general structural decisions by setting the state of each parameter and by triggering when a voice plays one of its constituent modules (Figure 3). The decision of which respective module plays, however, is random. Additional triggering of any voice will result in modules overlapping with each other until those modules end and no more are triggered. Altering the state of any parameters will dictate which modules may be randomly selected to play whenever a voice is triggered. The parameters in this instance include the chord parameter (associated with all voices), and the instrument parameter (only associated with the foreground voice).

Figure 3. Screenshot of Sum interface.

Sum currently demonstrates a unique ability to vary in its textural and harmonic progression. While there are some limitations in terms of the broadness of its structural plasticity as well as its susceptibility to occasional harmonic and timbral clashing, its overall framework is endlessly expandable and potentially applicable to a variety of musical styles.

4.2. Shifting Patterns

Inspired by the minimalist composers Steve Reich and Terry Riley, Shifting Patterns Footnote ³ is a work whose structure is driven by the recombination of looping polymetric patterns.Footnote ⁴ The work itself contains four voices, each representing a different type of pitched percussion: xylophone, vibraphone, marimba and glockenspiel (Figure 4). Each voice also contains four variable parameters (harmony, metre, pattern type and variation), each of which can be altered independently of other modules. Modules are designed to loop, and they all contain different kinds of two-bar patterns, the sounds of which were recorded in Cubase 11 using various virtual instruments. The length of modules in this case generally range from 3 to 6 seconds (including the reverb tail) and they all have a common tempo of 152bpm.

Figure 4. Screenshot of Shifting Patterns interface.

The selection of which module within a voice is played comes down to the combination of states between all four parameters. Each voice contains the same set of parameters: harmony, metre, pattern and variation. Each combination of parameter states within a voice leads to a unique module which, when allowed, will begin to loop. Any adjustments to the parameters will cause the current module to play out until it is finished, after which the newly selected module will begin in time with the pulse.

The variety of parameter states results in a required 180 unique modules per voice, and 720 modules in total for the entire work. While this is a large number, modules between different voices do follow similar rhythmic accents and pitch contours making their creation more formulaic and efficient. The main differences between modules of each voice, beyond their instrumentation, is that they cover different note ranges and harmonic territory.

Considering the importance of rhythm within this work, voices are all synchronised to a common pulse to ensure temporal homogeny. They are not, however, synchronised to a common meter so that unique pattern combinations can arise as a result of loops of differing pattern lengths metrically phasing with one another.

Since Shifting Patterns currently exists to be controlled via a user interface, the user is afforded a significant amount of structural control. Each module can be individually selected to play by adjusting the parameters between voices and there is no element of randomness to this selection process unlike in Sum. There is, however, a set of controls within the user interface that, when triggered, will set their relative parameters to a random state. There are also macro controls that set the relevant parameter of all voices to a selected value (see Figure 4). This overall format of allowing musical patterns to emerge out of the shifting and alternating of fundamental musical modules enables a unique and versatile approach for a musical work to exist such that it requires active decision-making to structurally coalesce.