1. INTRODUCTION
The role of the composer as an artist who has been bestowed the Atlantean burden of choosing notes and sounds (Murail Reference Murail2005) has led to many formalisations and theories about music. From stochastic and spectral music to neo-Riemannian and transformational theories, different schools of thought have proposed different solutions and techniques, to the benefit of the novice composer who can now choose or unchoose among different paths. Notes can be picked randomly from some scale (a sieve in the sense of Xenakis or a partials reconstruction in the sense of Murail), a series (in the sense of the serial school), a process (in the sense of process music) or by an algorithm (in the sense of algorithmic and generative music). This paper describes the deconstructivist aesthetics that emerges from techniques for generating music that have emerged in new music, notably concatenative synthesis, neural networks, random walks, transformational techniques and variants. It discusses various reconstruction paradox instances in which parts do not reconstruct the whole. We then discuss three case studies: first, random walks in neo-Riemannian spaces learned from a corpus; second, automated orchestration of piano rolls based on distance spaces learned from a corpus; and finally, the use of recursive neural networks (RNN) to generate polyphonic multichannel sketches (i.e., orchestral) based on a learned corpus and steering input data. This article is an exploratory walk in the lands of (this so-called) deconstructivist music, not an exhaustive survey in the field. Other papers have served this purpose (Roads Reference Roads1985; Edwards Reference Edwards2011; Briot, Hadjeres and Pachet Reference Briot, Hadjeres and Pachet2020; Ji, Luo and Yang Reference Ji, Luo and Yang2020).
2. BACKGROUND AND RELATED WORK
2.1. Deconstruction in architecture
Deconstruction in philosophy, as a late structuralist declination by French philosopher Jacques Derrida, has had a long legacy in the arts since as early as the 1980s (Wigley Reference Wigley1993; Vitale Reference Vitale2019). In the fine arts, architecture is perhaps the field in which deconstruction has been the most successfully applied as a technique with palpable and visible aesthetic results, as may be exemplified through the works of architects Daniel Libeskind, Zaha Hadid, Peter Eisenman, Bernard Tschumi, Rem Koolhaas, Frank Gehry, Coop Himmelblau or Morphosis, among many others (Figure 1). About destruction and architecture in a harrowing vision of a lost Europe (in his work Danube), Claudio Magris writes:
It is comforting that travel should have an architecture, and that it is possible to contribute a few stones to it, although the traveller is less like one who constructs landscapes for that is a sedentary task than like one who destroys them … But even destruction is a form of architecture, a deconstruction that follows certain rules and calculations, an art of disassembling and reassembling, or of creating another and different order. (Magris Reference Magris2001)
In architecture and the visual arts, the deconstructivist movement of the 1980s and 1990s found its inspiration partly from the Russian Constructivism of the early twentieth century (Johnson and Wigley Reference Johnson and Wigley1988; Wigley Reference Wigley1993) and partly from challenging new designs coming from computer-generated art (Boden Reference Boden2009). The patterns used for the windows of Libeskind’s Jewish Museum in Berlin are reminiscent of some of Malevitch’s paintings; as are the random geometries early personal computers could generate.
We can describe the deconstructivist works of the 2000s and forward as part of a post-deconstructivist or late deconstructivist era in architecture, where, inspired by the early deconstruction of the 1980s, form has been effectively liberated from the straight line. A myriad of designs has been imagined and built following this initial drive. Computer-aided design and physical models in the design of complex shapes have been instrumental in this artistic movement by allowing greater technical possibilities and greater control over the rendered product. Generative art (e.g., using machine learning) has been a trend in computer graphics and visual arts since at least the 1960s (Boden Reference Boden2009; Audry Reference Audry2021).
2.2. Deconstruction in music
Deconstruction has also been explicitly applied to the theory of music and its meanings (philosophical and sociological). Rose Subotnik seemed to have coined the term in music and used critical theory to provide a new interpretation of classical music (e.g., Mozart, Chopin) (Subotnik Reference Subotnik1995). Deconstuctivism in architecture relates to a set of techniques, effective processes with aesthetic and ‘palpable’ results. In music, however, existing work on deconstructivist approaches is scattered among different niches of contemporary music with diverse labellings.
No deconstructivist school of thought effectively exists in music and association with a heuristic idea of deconstruction may be more of an impression left on an auditor by some work rather than an intentional process.
This being said, the so-called parametric music of serialism and high modernism was an early example of deconstruction (of a piece into its parameteric models). The generative material is represented as a set of tweakable parameters with which a piece can be reconstructed. It was commonplace among composers of the era to believe that once the series was chosen, the rest of the piece would follow and could be reconstructed. Koblyakov’s analysis of Boulez’s Marteau sans maître undeniably demonstrates the parametric and generative aspects of Boulez’s work for instance (Koblyakov Reference Koblyakov1993). Also, composer Iannis Xenakis is known to have parametrised the stochastic generation of some of his most famous pieces, yielding results that still today can surprise and impress (Solomos Reference Solomos2015). Edwards (Reference Edwards2011) lists the following examples as forerunners of the current day algorithmic music: Guillaume Dufay’s (1400–74) isorhythmic motet Nuper rosarum flores; evidence of Fibonacci relationships in the music of Bach, Schubert and Bartók; Mozart’s Musikalisches Würfelspiel (1792); and the Quadrille Melodist sold by Professor J. Clinton of the Royal Conservatory of Music, London (1865) (a set of cards that allowed a pianist to generate quadrille music).
The music of Brian Ferneyhough and the visual scores of John Cage or Sylvano Bussotti are also proto-deconstructions (Ferneyhough Reference Ferneyhough1981; Attinello Reference Attinello1992; Bogue Reference Bogue2014; Hidalgo and Ipinza Reference Hidalgo and Ipinza2016). Figure 2 shows several such examples of deconstruction in music. In Ferneyhough’s work, the serial use of rhythm trees (i.e., in the sense of the Patchwork or OpenMusic softwares) disjuncts the time dimension; for John Cage, the deconstruction of the time domain comes from the use of indeterminacy and chance composition; for Bussotti, the deconstruction is visual (as it relates to deconstruction in the visual arts) – the straight line, like in architectural deconstruction, is literally perturbated. The work Piano Pieces for David Tudor: 1. Rhizome by Bussotti was incidentally inspired by the ‘rhizome’ concept of post-structuralist philosopher Gilles Deleuze (Reference Deleuze2013): a ‘rhizome’ (a concept borrowed from botany and dendrology) is a structure of constant splitting, such that synthesis (in a Hegelian sense) of the parts is no longer possible. Corbussen (Reference Corbussen2002) describes a work by Gerd Zacher, Die Kunst seiner Fuge (1968) – a set of ten variations of Contrapunctus I by Bach where the composer interprets the piece in various ways – as deconstructivist. Variations are a musical form that adequate well with the idea of deconstruction or parametric music in general. Generated content and reconstructions can be obtained by slightly varying some parameters, yielding significantly different results.
Granular synthesis – deconstructing (segmenting into ‘grains’) and reordering audio signals (often with additional parametric signal processing) – is a technique notably pioneered by Xenakis (Solomos Reference Solomos2015). The same process applied to music notation (via MIDI) is at the source of techniques such as concatenative synthesis – which could be performed on audio or MIDI (Zils and Pachet Reference Zils and Pachet2001; Schwarz Reference Schwarz2004, Reference Schwarz2005; Maestre, Ramírez, Kersten and Serra Reference Maestre, Ramírez, Kersten and Serra2009).
The work on real-time generative accompaniment and concatenative synthesis (Dannenberg Reference Dannenberg1993; Lewis Reference Lewis2000; Thom Reference Thom2000; Young Reference Young, Kronland-Martinet, Ystad and Jensen2007; François Pachet, Roy and d’Inverno Reference François Pachet, Roy and d’Inverno2013; Carsault Reference Carsault2017; Nika, Déguernel, Chemla-Romeu-Santos and Vincent Reference Nika, Déguernel, Chemla-Romeu-Santos and Vincent2017) could also be described as a form of musical deconstruction. In the OMax project (Dubnov, Assayag and El-Yaniv Reference Dubnov, Assayag and El-Yaniv1998; Assayag, Bloch, Chemilliera, Cont and Dubnov Reference Assayag, Bloch, Chemilliera, Cont and Dubnov2006; Assayag and Bloch Reference Assayag and Bloch2007), segments are classified in a suffix tree using the Oracle Factor algorithm. These research systems segment musical sources and reconstruct them out of place based on different heuristics. In the DYCI2 project (Nika et al. Reference Nika, Déguernel, Chemla-Romeu-Santos and Vincent2017), the musical material is broken into memories that are triggered in real time by an instrumentalist. The work Ex Machina by saxophonist Steve Lehman and artistic director Frédéric Maurin is a hallmark of the use of DYCI2. Interestingly, most of these systems have parameters designed to improve the continuity of the reconstructed pieces, or their contextual/genre readability. These parameters can, however, be tweaked to their extremities and yield completed deconstructed, disjuncted results. The work of Umberto Eco (Reference Eco1965) on open works can be noted, but also indeterminacy as pioneered by John Cage and the New York School (Iddon and Thomas Reference Iddon and Thomas2020).
3. RECONSTRUCTION PARADOXES
A fundamental concept in the construction of tonal music, which is often overlooked when considering other such concepts such as tonality, harmony, counterpoint and timbre, is the enchaînement concept. Simplistically speaking, enchaînement is to music what the straight line is to architecture. As a related concept, voice leading and its perception was extensively studied in the cognitive sciences (Huron Reference Huron2016). Enchaînement is thus the composition of the rules defining what comes before and what comes after a given musical event (e.g., a certain chord, a dissonance, an incomplete melodic pattern), the actualisation of harmony and tonality in a temporal structure. In neo-Riemannian theory, or transformational theory, some chords occur with higher probability after some other chords based on their position in some neo-Riemannian embedding space. In Hollywoodian music, this syntax has been made evident (Lehman Reference Lehman2018). It is our opinion that enchaînement conventions define musical order in at least as pervasive a manner as tonality or timbre.
In the piece Morphogenesis (2022) (Figure 3), we used as source material Beethoven’s Symphony no. 9 (4th movement), or Bee9, and a recursive neural network to reconstruct the material using Boulez’s Notations for orchestra as a steering material. The reconstructed pieces were composed of reordered (tonal) segments from Bee9, and they sound atonal.
In a sense, this relates to the Banach–Tarski paradox where reconstruction from the parts does not yield the whole. Sets in complex spaces (a 3D ball or a musical piece) may not always follow expected behaviours. The following quote gives a description of the Banach–Tarski paradox (de Rauglaudre Reference de Rauglaudre2017: 37):
[The] Banach-Tarski Paradox states that a ball in 3D space is equidecomposable with twice itself, i.e. we can break a ball into an infinite number of pieces, and with these pieces, build two balls having the same size as the initial ball. This strange result is actually a Theorem which was proven in 1924 by Stefan Banach and Alfred Tarski using the Axiom of Choice.
In music, we could call this a reconstruction paradox. In the same way, the reverse transformation or rotations in post-romantic and serial music could be seen as generative techniques (e.g., Rachmaninov’s Variations on a Theme by Paganini). Breaking the enchaînement conditioning of musical sources effectively generates new sonorities and aesthetics. We will discuss the details of these compositional processes in later sections.
Figure 4 demonstrates a reconstruction paradox using an example from Rachmaninov’s Piano Concerto no. 2. In the study sketch for the piece Morphogenesis (2022) (Figure 3), a neural network was used to learn associations between segments of Rach2 and their MIDI representations. The neural network was then used to transform segments from Boulez’s Notations for orchestra into multitrack polyphonic MIDI signals (i.e., orchestral segments). Judging from the atonality of the resulting sequences, the perceived sense of tonality is not the consequence of choosing notes from a tonal scale but rather an artefact of tonal enchaînement patterns (i.e., tonally syntactic chord progressions, dissonance resolution conventions and so on).
4. PIECES, SLABS, WEDGES, CASES AND STUDIES
4.1. Learning generative transformations in neo-Riemannian spaces
At least two trends in algorithmic music and composition can be distinguished in the massive grove of new music techniques and technologies: on one hand, generative techniques, such as OMax, DYCI2, Somax (Assayag et al. Reference Assayag, Bloch, Chemilliera, Cont and Dubnov2006); and on the other hand, a perpetual search for understanding inner structures of musical composition through neo-Riemannian and transformational theories (some examples include Lewin Reference Lewin1987; Lerdahl Reference Lerdahl1996; Tymoczko Reference Tymoczko2011; Cohn Reference Cohn2012). Concepts from transformational theories, such as chordal distance and spatial embeddings, have been used in generative techniques to steer the generation of music from pure randomness to some sense of coherent aesthetics. The ability to choose notes among the infinitude of possible assemblies of frequencies has been at the core of contemporary music research since Xenakis’s exploration into stochasticity and his development of sieve theory (Xenakis Reference Xenakis1992; Solomos Reference Solomos2015), Messiaen’s quest for modes of limited transposition (Messiaen Reference Messiaen2000), and Cage’s chance music, to the usage of partials and microtonality generated from sound analyses in the spectral school (Murail Reference Murail2004, Reference Murail2005). The quest for theories of compositional choice is foundational to the design of algorithms and processes of generative music.
Figure 5 shows the embeddings-into-nearest-neighbour graphs of segments obtained from distance matrices using the Spiral Array distance (Chew Reference Chew2014) and computed for different works (i.e., Rach2 and Schoenberg’s Klavierstücke, op.11, no. 3). Different pieces yield different structures (which is reminiscent of the work on generative meshing in the new architectural geometry field). These graphs and structures can be used as terrain for a random walk steered by the concavities and convexities of the terrain. We start from an input chord and the algorithm chooses some optimal path based on the data it has learned from the different works. Reproducing the genre or style of the piece is not the primary goal of such methods. Following the concept of reconstruction paradoxes, we can say that the reconstituted wholes rarely display all the properties of the original works. Figure 6 shows examples of such reconstructions. Figure 7 shows a piece generated from a random walk in a space learned from ‘O Fortuna’ from Carmina Burana by Carl Orff. The random walks generated on a single channel were then orchestrated using Orchidea, a software developed at IRCAM (Maresz Reference Maresz2013).
The generated random walks sometimes preserve some characteristics of the original corpus (in ‘O Fortuna’, the grandiloquence could still be perceived), however, reconstruction paradoxes (in the perception of tonality and metric rhythm especially) still occur in the generated artefacts. The random walk algorithms are described in Algorithms 1 and 2.
4.2. Distance-based automated orchestration
Automated orchestration designs have offered a smaller number of candidates for review than algorithmic composition. Orchidea, the latest creation of the Orch* series developed at IRCAM by Carmine-Emanuele Cella, uses a customisable database of sounds to output an optimal instrumentation using genetic algorithms (Maresz Reference Maresz2013). IRCAM has also developed another piece of software, Orchestral Piano by Leopold Crestel (Crestel and Esling Reference Crestel and Esling2017), where orchestration rules are learned from real orchestration of piano reductions. The machine-learning algorithms then learn association predictions from the piano reduction and orchestration data using variants of Restricted Boltzmann Machines (RBM, cRBM and FGcRBM). Finally, Handelman and Sigler (Reference Handelman and Sigler2012) proposed an automated orchestration method based on the concept of z-chains.
We demonstrate a technique that is based on some chordal distance measure (e.g., chromagrams, Estrada distance, Costère distance and Chew distance) that has a lightweight learning phase (i.e., updating a distance matrix) and that can therefore be used in real time with minimal footprint (storing a transformation matrix, which in the case of subsets of 12 notes is 4096×4096). A first learning phase where an existing MIDI orchestration of some work A is segmented and tagged by its pitch-class content (i.e., the multichannel MIDI signal is collapsed, and duplicate pcs across all octaves are deleted) is performed. A second-generation phase is performed with MIDI from some work B that is used for generating sketches: each segment of work B is matched to some segment in Work A and a reconstruction of A is generated, steered by B.
This process is explained in Algorithms 1 and 3 (in Figure 8). Algorithm 1 is shared between the random walk algorithm and the distance-based automated orchestration algorithm; as the generation of a transformation matrix uses the same techniques. This process can be used to handle timbre as well in the distance-based automated orchestration algorithm. Timbre, here, is extracted from the information on the channels/instruments provided in the MIDI file. Creating a distinct transformation matrix for the channel information in addition to the note information (pitch-class sets, or harmony) is possible. Joining both transformation matrices using a weighing strategy is also possible. During the past decade, the recent interest in artificial intelligence and embodied knowledge has provided novel important insights to rethink mediation technology, thereby paving the way for a transdisciplinary approach to artificial intelligence and other technologies such as wearables or CAC. Stemming from a phenomenological-based approach and considering current trends in sonic interaction design and interactive composition, our paper also proposes experiments on artificial intelligence and embodied approaches to mediation technology and underlines the increasing importance of somatic and generated knowledge within the computing field. The understanding of the phenomenal body and mind and the interpretation of mediation technology as an environment capable of affording transformative feedback, both allow, in fact, for an enrichment of the phenomenological experience. The paper also presents an autoethnographic analysis of our own performances, which provide an original contribution to the artistic application of artificial intelligence and other technological applications. Stemming from an ongoing research-creation on musical improvisation and generation with artificial intelligence, the case study suggests a new way of understanding viscerality as a key feature to design interactive systems and interactive composition and empower the performer’s agentivity. This contribution also tries to emphasise how an embodied and visceral approach to interaction can transform artificial intelligence into an active sensory-perceptual mode of experiencing, which is capable of stimulating the performer’s sensorimotor metaplasticity. The reconfiguration of generativity through the use of sound feedback is a process that unfolds with a high degree of sensitivity in which the body can be poetically understood as an emergent territoriality, inhabited and transfigured by the sound.
4.3. Recursive neural networks for steered orchestral concatenative synthesis
Neural networks in music synthesis have been studied extensively since the idea was first put forth (Lewis and Todd Reference Lewis, Todd, Touretzky, Hinton and Sejnowski1988; Lewis Reference Lewis1998). Also, more recent RNN, ConvNets, GAN, VAE and concatenative synthesis-based techniques in music synthesis have notably been experimented with in IRCAM’s automated accompaniment libraries (e.g., OMax, DYCI2, Somax).
Here we demonstrate the use of a robust and well-tested technique used in speech recognition/classification (the so-called spoken digit classification problem) using RNNs and apply it to music synthesis. As usual, neural networks are sensible to input data and training parameters.
We generated three sets of input data using Beethoven’s Symphony no. 9 (4th movement) or Bee9. One with an average segment length of μ = 0.137405s (a short segment length corresponding to a segment generated at each onset of the MIDI file modulo a precision parameter handling notes not hitting the same exact onset), one with an average μ = 1.04207s (a coarse segment length where segments could contain many onsets – motifs) and a coarser segmentation with average μ = 2.05122s.
Different datasets used to train the RNN yielded different properties in the generated music. Shorter segments yielded less diversity in the generated segments. Coarser segments yielded more diversity in the segments. Coarser segments also increased the recognisability of the input corpus. Reconstruction paradoxes are clear in the generated material, even in longer segments in the Bee9 example. This is illustrated in Figure 9.
5. ALGORITHMS
Algorithm 1 is identical for the random walk and distance-based orchestration algorithms and one can overload the other. When dealing with orchestration, the MIDI files used are multichannel: the feature extraction and embedding algorithm (Algorithm 1) learns the pitch-class space and collapses the channel information to keep only the pc-set data (note that this technique can be extended by learning the timbral embedding space and storing the MIDI channel information on another segment).
Algorithms 2 and 3 assume that a transformation matrix was learned using Algorithm 1. In the case of Algorithm 2, a single channel segment is returned. In the case of Algorithm 3, multichannel information is returned.
Algorithm 4 assumes a trained RNN of the type described in Figure 10. The RNN effectively classifies orchestral audio segments to their MIDI polyphonic multichannel counterparts (additional technicalities are necessary to handle MIDI files with more than 16 channels).
6. CONCLUSION
The study of deconstruction in music promises to be a testbed for many new works. Reconstruction paradoxes are real consequences of many generative and algorithmic music techniques that are being studied in new music (e.g., neural networks, random walks, concatenative synthesis, variational autoencoders, machine learning). When comparing Figures 4a (Rach2 non-deconstructed) and 4b (Rach2 deconstructed), we can see that the excerpt generated by AI has an embedded deconstructivist ‘logic’ to it: the traditional patterns of enchaînement do not seem to hold. The AI deconstructivist creations may share some traits and parameters with their component parts and the original pieces these parts were extracted from (mostly at the microtemporal domain); but they demonstrate unique aesthetical features, distinguishable by the absence of traditional enchaînement conventions – resulting from the dismantling of their original syntactic tonal/metric contexts. As we continue to research and create in the area of deconstructivist AI music, other paradoxes, phenomena and epiphanies are bound to be discovered, leading to a growing body of works and theoretical base.
Acknowledgements
We wish to thank the following for providing permission to reproduce artworks and scores in our paper: Edition Peters for the work Fontana Mix (1958) by John Cage; the John Cage Trust, Edition Peters and MOMA/Scala/Art Resource for the works II from Mushroom Book (1972) by John Cage; Casa Ricordi for the work Introduction: Rhizome – From Five Piano Pieces for David Tudor (1959); Edition Peters for the work Lemma-Icon-Epigram (1981) by Brian Ferneyhough; MOMA/Scala/Art Resource for the picture of the Project for a Monument to the Third International (1919) by Vladimir Tatlin; and Studio Libeskind for the drawing Micromegas (1979).