4.1. Cumulative orderings from Task 1 and Task 2/3 are distinct

We expected correlation between orderings in the two tasks as performed by any individual participant, since once used to conceive relationships a sonic feature is likely to remain in memory. But did participants nevertheless present different orderings in the two tasks, given the different demands, and as assessed by the transposition distances? Forty-five participants used all four boxes in Task 2/3, while one used only three (22, Figure 3). This latter was accommodated within the analysis code (using only six permutations of the box order for this participant).

Figure 3. Sound grouping: the distribution of the sounds by participant and box in Task 2.

Note. Abbreviations: The sound design abbreviations are as before. Boxes are numbered 1–4 from left to right; anonymised participants are indicated above each individual display as the first two numbers. ‘Sound’ refers to items 1–30.

Every participant’s Task 2/3 orderings (i.e., all the permutation orders) were all different from their ordering of Task 1. The mean, median and standard deviation of the minimum transposition distances, participant by participant, were 155.4, 146.9 and 58.7 with the range 48–276. Even the lowest value corresponds to more than 10% of the maximum possible. For the maximum distances, mean, median, standard deviation and range were, respectively, 398, 410, 33.3 and 334–450 (from 74.2 to 100% of the maximum possible). This shows that every participant behaved differently in Task 2/3 than in Task 1, as we hoped.

A first-order adjacency matrix was constructed for each of the tasks, based on the frequencies with which each sound was placed next to another. Figure 4a shows the matrix for Task 1: the diagonal is occupied by 0, since no item can appear twice. The matrix is symmetrical with respect to the diagonal (because these are undirected frequencies, AB equivalent to BA, providing one adjacency shared by A and B). In Task 1, all sounds are aligned in a single sequence, left to right, so this construction is straightforward. However, in Task 2/3 we provided no suggestion that the four boxes were themselves ordered. Figure 3 shows the disposition of the design groups by box for all participants: several members of a design group were often co-boxed, with no regard for the position of that box. So DSP-manipulated sounds (magenta) commonly appear in the same box, but that box may be any of the four. The fact that the box ordering giving the lowest transposition distance between Task 1 and 3, was blue–green–orange–turquoise, the screen order, in only 9/46 cases, confirms that that the boxes were not treated as part of an ordering process. Consequently, for the adjacency analysis we assume no relation between the four suborders in Task 2/3, and each suborder is treated as a partial realisation of the overall ordering. This means that there are three less adjacency values obtained per participant from Task 2/3 than Task 1 (because no adjacencies between the boxes are assumed). Thus the adjacency matrix for Task 2/3 (Figure 4b) is obtained simply by adding together those from the occupied boxes from each participant.

Figure 4 (a) Undirected adjacency matrix for Task 1. (b) Undirected adjacency matrix for Task 2/3

Note. Numbers 1–30 are the test sounds. The sum of the adjacencies in Task 1 is 2,668 (corresponding correctly to 29 adjacent pairings × 46 participants × 2 since the matrix is symmetric, undirected). There are 2,394 adjacencies in Task 2/3 because of the omission of counts between boxes, and the fact that one participant only used three of the four possible boxes. The easiest way to interrogate these data is probably to choose a column, and read downwards from the diagonal of zeros, bearing in mind that the matrices are symmetrical about the diagonal.

Given this evidence on the interpretation of Task 2/3 ordering, there still remain two ways of interpreting orderings cumulated across participants: as being directed (a left to right adjacency of x and y is distinct from its reversal), or undirected (where the two cases are not differentiated).

The proliferation of 0 values outside the Figure 4 diagonal shows that some adjacency pairings are unexploited: 172 (6.4% of counts) for Task 1, and 314 (13.1%) for Task 2/3. These two adjacency matrices are distinct, in accord with the observations comparing the individual participant sequences from Task 1 and Task 2/3. However, as commented already, one would expect correlation between the matrices. The Mantel test (a form of permutation analysis involving repeated correlations) confirms this, showing a strong correlation of 0.86 (p = 0.001, two-sided). This means that about 26% of the variance in the adjacency data is not shared in the two orderings, revealing the flexibility we are focused on. The distinction between the matrices remains even when a random 10% of participants’ data are removed from the matrix for Task 1 (since there are approximately 10% less counts in the Task 2/3 matrix). The Mantel test does not take account of intra- or inter-personal differences: it is rather like taking two global averages. So the strong correlation is unsurprising, and it remains clear that each individual performed differently in the two tasks.

4.2. Restricted Markov chain predictions of orderings

Next we consider the adjacency matrices as directed (i.e., we distinguish L to R sequences 1–2–3 and 3–2–1). This is treating the sequences as first-order Markov chains restricted by the fact that no item can recur. Put simply, in such a Markov chain, the probability of an item appearing in position n+1 is entirely predicted by the occupancy of position n. This is termed the transition probability between the two positions. This allows us to determine the directed adjacency matrices using the R ‘Markov’ package, and then to construct the corresponding Markov chain models. The values in these directed adjacency matrices (not shown) are now transition probabilities, but they again allow the Mantel test. This confirms the difference between the two matrices, which remains true when 10% of the Task 1 participants are removed at random. We can again conclude that the adjacency matrices of Task 1 and Task 2/3 are distinct. Figure 5 illustrates this by comparing the Markov ordering predictions for the two tasks.

Figure 5. Markov model predicted orders for Tasks 1 and 2/3.

Note. The Markov models of Tasks 1 and 2/3 are used to predict the overall most likely ordering patterns across all participants, in each case given a starting sound of 29, and sequentially predicting the next sound using the transition probabilities. Predictions at each point are restricted to sounds which have not already been allocated.

The all-participants’ Markov chain model also allows prediction of individual participants’ orderings in Task 1 based on their choice of the extreme left sound. These predictions are much better than chance; for example, showing 4–10 correct predictions (for which the chance probability is only approximately 10–⁶⁾. Various tests indicate that the data are legitimately described as Markovian, and there is no indication that Markov orders higher than one are required for their best modelling.

Thus whether directed or undirected, the Task 1 and Task 2/3 orderings are statistically distinct, and unlikely to arise by chance.

4.3. Relationships between sounds’ design group and their orderings: participant strategies

If our five design groups were perceptually influential, then members of a group would be adjacent more often than they are to members of another group, even though our purpose was to partially disrupt these relationships. A simple analysis of this issue used the data from Task 2, where categorisation is explicitly required. Table 2 shows a co-occurrence matrix representing these results. Because one group only contained four members, Table 2 shows the frequency that >25% of the sounds of one group were co-boxed with >25% of another. The diagonal values show that members of each design group commonly co-located with other members of that group, and the main ‘confusion’ (off-diagonal entries) was between convolved and instrument sounds, which is unsurprising since convolutions were done on the instruments.

Table 2. Co-occurrence matrix from Task 2 (categorisation)

Note. Each number shows the frequency that >25% of the sounds of the design group specified on the vertical axis were assigned to the same box as were >25% of the group specified on the horizontal axis. Design group abbreviations are as before. Note that the column and row sums are heterogeneous.

In response to enquiries on ordering strategy(ies) used in any task, participants often mentioned using pitch, and quite often loudness (even though the sounds were RMS-equalised). We did not ask about their interpretation of our instructions on ‘cohesively’ and ‘grouping … by affinity’, so as to minimise directional influences of our questions. A few people identified sonic categories (e.g., ‘music’, referring primarily to instruments), and a few mentioned their degree of liking. A very few explicitly used ‘categorisation’ in the box Task 2 more than in ordering Task 1, while a few considered ordering as a series of nearest neighbours in features. Most participants had no interest in attaching labels to the boxes after they had occupied them (suggesting they might not have explicitly identified design features). Twenty-four respondents wanted additional boxes in Task 2/3, 11 wanted fewer (some having difficulty recollecting numerous sonic features), while the remaining 11 were indifferent to this. Participants were thus not invested in shared ordering strategies, rather, they were diverse, as we anticipated. A slight majority, preferring additional boxes, implicitly recognised the categorisation pressure of Task 2.

4.4. Network analysis of orderings

The co-occurrence matrix of Table 2 emphasises that sounds from different design groups co-relate in diverse ways. This suggests that a desired compositional usage of timbral items (partially recreated by our two tasks) is to form relationship networks that can be used diversely. Thus it is interesting to consider the network features (Betzel and Bassett Reference Betzel and Bassett2017) implied by our data. Network models identify the intensity of the relationship between pairs of sounds, rather than providing a more abstract ‘distance’ measure. For example, some social network analyses might use numerous features that link any individual person (in this case, the ‘agents’ that are the subjects of any generic network model are individual people), such as frequency of contact, topics of contact, shared activities or disciplines, age, gender and financial status. In the present case, the information we are using that links our agents (being individual sounds) is solely their frequency of contact (adjacency in the orderings determined by the participants), but as we point out later, other features could be relevant for different purposes (such as acoustic features, as distinct from perceptual relationships). We present both descriptive networks, which display the observed structures, and then generative network models, which suggest the processes by which the networks arise and allow extrapolation to generate the full set of networks consistent with a particular model.

Given the task natures, we only consider undirected, but in this section, valued networks. So an L to R link (adjacency) between sounds 1 and 2 is taken as equivalent to R to L, but its intensity is considered: the frequency of occurrence in either direction. Because of the sound group size unevenness and the networks observed, we sometimes subdivide the Environmental into Natural and Machine/Human categories (defined in Table 1). Figure 6 shows the descriptive networks of the two tasks. Note that each link intensity is represented by the thickness of its line, but 2D layout is algorithmically chosen to optimise clarity, and has little bearing on the interpretation of the network.

Figure 6. Descriptive networks for Tasks 1 and 2/3.

Note. Each node represents a numbered sound. The configurations use the Fruchterman–Reingold force-directed algorithm to maximise legibility. The graph shows the interspersing of the design groups, and the frequencies of node links are represented by edge widths.

In Figure 6 the Impulse response and DSP groups are relatively discrete in the networks, while Instruments and Convolution are interspersed, as are Instrument, Machine and Natural Environment in Task 2/3. This is partly due to sound 28, a ‘waterphone’ instrument that produces complex rapidly changing timbres like the sound of a stream of water. This sound could be classified as ‘Machine’ (but there is a sense in which all musical instruments are machines). As emphasised, the most objective feature of the representation is the edge width, proportional to the link strength. For example, note that some of the strong edges within the DSP in Task 1 are weakened in Task 2/3, as are some within the Impulse response group. There are also some stronger edges for Task 2/3 than 1, such as 13–25 and 5–30. Overall, these network representations confirm and display the distinctiveness of Tasks 1 and 2/3.

The fundamental features of each node in a network include not only the intensity of its connection to other nodes, but also its ‘degree’ of interconnectedness (the number of distinct connections it makes). Many important network features depend on further consideration of its linkage structure, such as detecting ‘communities’: do user perceptions indicate sound communities corresponding to or different from the design groups? A community is a group of nodes that are more connected internally than to the rest of the network. These further features suggest mechanisms that may be responsible for network generation. For example, if items A and B are strongly linked, and so are B and C, then this is quite likely to result in the formation of a link between A and C, as we know from our own experiences of social networking (this is termed the triangular relation in models). Such analyses identify community distinctions between the tasks, and are not simply congruent with the design groups. Figure 7 shows communities detected by the so-called label propagation algorithm (note the different colouring system from Figure 6). The label propagation algorithm (see Betzel and Bassett Reference Betzel and Bassett2017) assigns randomised labels to all nodes to commence, and then by progressively (empirically) changing the labels of connected nodes (e.g., applying the same label to the members of a linkage triangle) and assessing the outcome, it maximises the number of nodes which are directly connected to nodes with the same label, and in this case gave stable results.

Figure 7. Communities based on label propagation for Tasks 1 and 2/3.

Note. Sounds are numbered as before, but the colouring systems of both background and vertices solely reflect the identified communities. Within community edges are black, between are red.

Figure 7 suggests that in both tasks communities are formed by the impulse responses (sounds 10–13), DSP (sounds 1, 7, 21, 23, 26), and the mixture of convolutions (3, 4, 5, 6) with all but one of the instruments (14–17). The remaining (mainly Natural and Machine environmental) sounds form two communities in Task 1, and are interspersed as one in Task 2/3. Figure 7 is based on label propagation, but other aspects of the connections and weights can be used for community detection (Betzel and Bassett Reference Betzel and Bassett2017). Among these, detection by ‘greedy optimization of modularity’ gives the same patterns as Figure 7. ‘Community betweenness’ is an alternative criterion for community discrimination, maximising the difference between intra-community and inter-community edge frequencies. For Task 1, it indicates 10 communities, nine of which contain a single item, while for Task 2/3 it indicates nine, seven of which have single members (four shared with Task 1), while one contains four, and the final large community contains the remaining 19. Thus diverse community analyses again confirm the distinctness of networks from the two tasks (for a detailed comparative review of community detection methods see Yang, Algesheimer and Tessone Reference Yang, Algesheimer and Tessone2016).

Table 3 compares some important network statistics, supporting their distinctiveness.

Table 3. Network statistics for Tasks 1 and 2/3

Note. Edge density describes the proportion of possible edges that are occupied, and transitivity is closely related (there are 435 possible undirected edges in these networks). Other measures are also closely interrelated. Diameter represents the maximum distance (number of edges) required to get from any node to any other node. Mean degree summarises the number of nodes connected to any one other node. Centrality is a global betweenness measure, while mean distance represents that between pairs of nodes on average. Note that in principle, part of these differences between Tasks 1 and 2/3 might flow from the fact that 10% less edge counts are provided by the latter task (as discussed earlier): again, comparisons based on randomly removing 10% of participants from Task 1 suggest any such effect is minimal.

Besides differentiating the networks, the mean degree parameters (the mean number of other nodes to which a node is connected) indicate both networks are profuse (19.5 and 24.3 out of a maximum attainable of 29): whereas the evolutionary popular music networks (Serrà et al. Reference Serrà, Corral, Boguñá, Haro and Arcos2012) only have comparatively sparse degree medians of four for pitch classes, and eight for timbre. While timbre contrasts may be ‘rarely the basis for musical discourse’ (Serrà et al. Reference Serrà, Corral, Boguñá, Haro and Arcos2012) in popular music, this is clearly not true of our sounds and our experiment, nor likely to be true generally for EAM.

4.5. Generative network models: modelling the two different networks

Such models seek to explain network generation, rather than simply describe networks; for example, as mentioned earlier, the formation of ‘triangular’ node sets can be a generative path. Exponential random graph models (ERGM) are a widely used class of generative network models (Betzel and Bassett Reference Betzel and Bassett2017). Given clear criteria of connection, describing a network is unambiguous. In contrast, generative models have to be assessed for relative ‘goodness of fit’ (GOF: the degree to which they reflect the properties of the data); and GOF may assess a selection of features, not just link frequencies. Ideally, a generative model would be assessed not only by comparing its modelled predictions (i.e., the features that it explicitly sought to explain) with those observed, but also by comparing unmodelled predictions with those observed. In these ways models can be compared such that the best available can be selected, and some of its limitations understood.

For valued nets, such as those studied here, the R ERGM-count package (from within the statnet package) does not yet provide a GOF test which goes beyond the predicted variables. In contrast, with simpler non-valued nets one can also test whether the generative model produces nets whose statistics coincide with the unmodelled parameters, such as some of those in Table 3. Thus, the models of Table 4 (valued nets) achieved good fit solely to the modelled variables, and were optimised by means of residual deviance and information criteria (representing precision and efficiency of the model, where efficiency is a measure of predictive power per predictor variable used, with a preference for simpler models with less predictors given equal precision). Some further technical details are given in the Table 4 notes.

Table 4. Exponential random graph models for the valued networks of Tasks 1 and 2/3 Formula for both models: net(1 or 2/3) ∼ sum + nodesqrtcovar(center = TRUE) + transitiveweights (‘min’, ‘max’, ‘min’) + nodematch (‘type’, diff = TRUE, form = ‘sum’). Both models show Monte Carlo Maximum Likelihood Estimate results.

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Null Deviance: 0 on 435 degrees of freedom; Residual Deviance: -9036 on 426 degrees of freedom. AIC: -9018 BIC: -8981

Null Deviance: 0 on 435 degrees of freedom; Residual Deviance: -10081 on 426 degrees of freedom. AIC: -10063 BIC: -10026

Note. The formula indicates that the model predicts the basic network edge weights by means of the specified variables. Estimate: the determined coefficient of the specified predictor in the model formula; SE, standard error (of estimate): SE that are low in relation to the Estimate indicate a reliable estimate, and the final Pr(probability) value indicates how likely it is the result could occur by chance rather than as modelled; AIC, Aikake’s Information Criterion; BIC, Bayesian Information Criterion – in all cases lower values are better, and these values are used in model selection, but the two task models cannot be compared in this respect as explained below. All parameters are highly significant as indicated by the triple asterisks.

All ‘sum’ parameters refer to the sum of the relevant edge weights (the modeled response parameter for these valued networks, while nodematch… refers to the design group of the sound, with the same abbreviations at the end of the name as before, in this case considering six groups. Here, nodesqrtcovar.centered is an index of the distinctiveness of the behaviour of the individual nodes; while transitive weights is a triadic closure measure for valued nets, analogous to the concept of edge ‘triangles’ in non-valued nets.

Table 4 models the distinctness of the two networks, particularly in terms of the coefficient on nodesqrtcovar, the index of the individuality of the behaviour of individual sounds. The quantitative interpretation of the estimated values is as follows: the odds determining the likely weight between any pair of sounds in Net 1 model is exp(-2.13), which is 0.12. That determining the weights between any pair of Impulse responses is exp(2.41), that is, 11.1 times higher. The corresponding figures for Net 2/3 are 0.10 and 18.5. While the general pattern of coefficients is similar for the two nets, nodesqrtcovar confirms the substantially lesser diversity of the weights of edges in Net 2/3, in accord with our hypothesis of transforming conceived relationships between sounds in the two task conditions.

4.6. Modelling the temporal transformation of the timbral networks

The R statnet tergm package allows modelling a temporally changing network on the basis of two (or more) distinct successive samples of it, which is essentially what we can take Net 1 and Net 2/3 to be. Tergm for valued networks is not readily feasible presently, but we determined models of the edge structure of unvalued Nets 1 and 2/3 to be of the form net ∼ edges + triangles (as noted earlier ‘triangles’ is a term representing a simple form of triadic closure). Nodematch terms (concerning whether or not an edge links members of a single sound design group) were not effective in these unvalued models (cf. Table 4 for the valued version). A tergm model includes both an edge formation and a dissolution model (since with time, both processes can occur), and Table 5 summarises our optimised model. Note that the deviance and information criteria values of this table cannot be compared with those of Table 3. This model met GOF criteria both for modelled and unmodelled (e.g., degree) variables.

Table 5. A separable temporal exponential random graph model of unvalued Nets 1 and 2/3 Formation model. Formula: net ∼ edges + triangles Monte Carlo MLE Results

Null Deviance: 98.43 on 71 degrees of freedom. Residual Deviance: 98.13 on 69 degrees of freedom. AIC: 102.1; BIC: 106.7

Null Deviance: 504.6 on 364 degrees of freedom. Residual Deviance: 435.3 on 362 degrees of freedom. AIC: 439.3; BIC: 447.1. The probabilities show that this model was less strong than those of Table 5.

As noted, Net 1 has 364 edges, Net 2/3 293 (in each case out of a possible maximum of 435), and these numbers give rise to the degrees of freedom of the Formation and Dissolution models. Taking these two nets as one step apart (with Net 1 first, as it was produced by the participants) in the possible transformation of the temporal network, one can simulate a series of steps from the model of Table 5. Starting with Net 1, and applying the generative model of Table 5, each and every single transformation step creates and dissolves multiple edges. The product of one step can be then used sequentially in a further transformation. Over multiple such simulations, one can assess the cumulative network adoption of the possible edges: this showed that all 435 possible edges always occurred at least once in the first nine networks, often earlier. When the two networks were treated as if in the reverse order (which could readily occur), with the categorisation/ordering task (2/3) first so that the number of edges rises between the two time slices, simulating with the different resultant tergm model always showed full coverage of all 435 edges even within six time steps. Thus tergm confirms the potential of the modelled generative dynamic network process representing participants’ possible developing concepts of inter-sound linkages to rapidly survey all the possible linkages. This again supports our proposal that listeners (including composers) can readily conceive diverse and changeable relationships between timbres. We wished to avoid categorisation as a first step in our experiment (since it would likely drastically influence a subsequent unilinear ordering task). On the other hand, doing the categorisation step first, contrary to our experiment, may well be a composerly approach (Dean Reference Dean2017).