3.1 File selection

Stage 1 of the process requires the specification of the files to be analysed. This depends on the engineering project and dependencies between the files one wishes to investigate. Examples include the use of CAD files to determine product dependencies and emails to determine process dependencies.

With the files selected, a further thresholding of the files is performed based on file activity. At this initial stage, a file activity of greater than four updates is often chosen. This represents the creation followed by three more edits to the file.

The aim is to remove the files that show little to no activity and, thus, may represent areas of the product/project that cannot be altered by the team (e.g. a bought-in part such as the engine block and/or gearbox). Imposing a low threshold for the number of files helps prevent an overreduction in the size of dataset; however, too low and a probabilistic model cannot be achieved as there is a lack of edits to build up confidence in the candidate dependencies.

3.2 Generation

Figure 2. Detecting co-occurrences of file edits.

To generate the co-occurrence DSM, one needs to identify when a co-occurrence of file activity has occurred. To achieve this, the process iterates through each file edit timestamp and identifies the files that have changed within a specified time period. The matrix is then updated to reflect the frequency of the co-occurrence of edits between the files.

Figure 2 illustrates the process of identifying the co-occurrence of file edits. Taking File A as the file of interest, the process identifies all the timestamps where the file has changed and generates the time periods where the occurrence of edits to other files are retrieved. $t_{p}$ defines the maximum time period post a file edit and can often be smaller due to A being edited again within this period. This is demonstrated in Figure 2 where $t_{p}$ is set to 12 h, but 4, 3 and 2 h are present due to additional edits being made to A. Dynamically adjusting the time period ensures no double counting of file edit co-occurrences.

With the time periods for a co-occurrence with respect to A defined, one can now refer to the edit timestamps for the other files and check whether a co-occurrence has occurred. Figure 2 shows the three possible scenarios. Scenario (i) occurs where File A has changed and File B changes within the post-edit time period. This leads to a co-occurrence of file activity being recorded. Scenario (ii) occurs where File B changes again within the same post-edit time period. Since a co-occurrence has already been recorded, no further co-occurrences are recorded. Scenario (iii) occurs where File B changes within a period of inactivity of File A. In this case, there is no co-occurrence of file activity.

This results in a matrix consisting of the number of file edit co-occurrences. This is taken to Stage 3, which normalises the matrix with respect to the number of edits to the files.

3.3 Normalisation

At this stage, the process has generated a DSM consisting of the number of file edit co-occurrences. If one would partition the current DSM, the resulting partitions would be highly influenced by the files with a high number of edits.

Thus, normalisation against the number of edits is performed to adjust for the unequal distribution of edits across the files. To do this, the process applies conditional probabilities. Referring back to the theoretical model, this is the assessment of the consistency of file edit co-occurrences and is achieved by taking the number of co-occurrences between two files $u$ and $v(C_{u,v})$ and dividing it by the total number of edits of $u(E_{u})$ to give the probability of an edit in $u$ leading to a co-occurrence edit in $v(P_{u,v})$ . This is shown in the following equation:

(1) $$\begin{eqnarray}P_{u,v}=\frac{C_{u,v}}{E_{u}},\end{eqnarray}$$

where $P_{u,v}$ is the conditional probability of an edit in $u$ post $v$ , $C_{u,v}$ is the number of co-occurrences between $u$ and $v$ and $E_{u}$ is the number of edits to file $u$ .

The result of this normalisation is a directed DSM that represents the conditional probabilities of a dependency between the files. To continue, the process evaluates the ‘directedness’ of the DSM to determine whether a directed or undirected DSM should be used and the appropriate partitioning technique is applied.

3.4 Evaluating the ‘directedness’

Up until this point, the DSM has been inherently directed where a change in the file in the row vector $(u)$ has a likelihood of leading to a change in the file in the column vector $(v)$ . This ‘directedness’ ( $D$ ) is subsequently analysed to assess its significance in order to ensure that the appropriate directed or undirected partitioning techniques are applied (Equation (2)). This is evaluated by taking the average of the absolute differences in the conditional probabilities between the files within the DSM. If $D$ is close to ${<}0.1$ , then it is argued that no significant directionality exists within the DSM.

(2) $$\begin{eqnarray}D=\frac{\sum \unicode[STIX]{x1D6E5}P_{u,v}}{\sum E}=\frac{\sum |(P_{u,v}-P_{v,u})|}{\sum E}.\end{eqnarray}$$

As the focus of this paper is not to discuss the process entirely, the paper will focus on the analysis of undirected matrices as this was found to be the case for the systems design projects to be compared. Equation (3) details the transformation from a directed to undirected matrix $(U_{u,v})$ , which takes the average of the two directed conditional probabilities $(P_{u,v},P_{v,u})$ .

(3) $$\begin{eqnarray}U_{u,v}=U_{v,u}=\frac{P_{u,v}+P_{v,u}}{2}.\end{eqnarray}$$

With the DSM normalised and directionality determined, the process moves to the pruning of the DSM where further noise is removed from the DSM ahead of performing the partitioning.

3.5 Pruning

The objective of pruning is to omit candidate dependencies that are a likely result of randomness and concurrency of engineering work. In addition, this reduces the computational time taken to partition and analyse the DSM by increasing the spareness of the matrix. This enables the DSM to be maintained and updated in real time as the project evolves. Pruning is the act of running through all the candidate dependencies and removing any that are below a defined pruning value $(p)$ . The optimum value for $p$ is generated during Stage 7 (Section 5.8). Pruning the matrix also has the ability to disconnect files and groups of files from the rest of the matrix. In the first case, the process removes this file from the rest of the analysis, while in the second case, the group of files remain within the matrix, thereby forming components. With the pruning complete, the co-occurrence DSM is ready to be partitioned to uncover the underlying and evolving architecture of the engineering project.

3.6 Partitioning

Partitioning reveals the DSM’s underlying structure. While there are many algorithms that could be applied to reveal the underlying structure of the DSM, this paper applies the Louvain community partitioning algorithm (Blondel et al. Reference Blondel, Guillaume, Lambiotte and Lefebvre2008). This algorithm is typically applied to communication transactions, such as email and social media, to reveal communities within social networks. As the co-occurrence measurements used to form the DSMs are also of a continuous and transactional form, it is argued that the partitioning algorithm will also be suitable for this case.

The algorithm works by allocating files to partitions with the objective of returning a partition set that produces the highest modularity score for the DSM. Modularity ( $Q$ ) is an assessment of the quality of the matrix partition and is defined as (Newman Reference Newman2004)

(4) $$\begin{eqnarray}Q=\frac{1}{2m}\mathop{\sum }_{u,v}\left[A_{u,v}-\frac{k_{u}k_{v}}{2m}\right]\unicode[STIX]{x1D6FF}(c_{u},c_{v}),\end{eqnarray}$$

where $m=\frac{1}{2}\sum _{u,v}A_{u,v}$ and is the sum of the co-occurrences’ conditional probabilities within the matrix. $\unicode[STIX]{x1D6FF}$ is the Kronecker delta function and is 1 if a co-occurrence exists between two files and 0 otherwise. $k_{u}k_{v}/2m$ is the probability that a co-occurrence may exist between two files, where $k_{u}$ is the number of files that have co-occurrences with file $u$ and $k_{v}$ is the number of files that have co-occurrences with file $v$ . And $A_{u,v}$ is the weighted co-occurrence between two files in the matrix.

To start, the algorithm assigns each file to its own partition. The algorithm then sequentially moves one file to a different partition and calculates the change in modularity (Equation (5)).

(5) $$\begin{eqnarray}\unicode[STIX]{x1D6E5}Q=\left[\frac{P_{\text{in}}+k_{i,\text{in}}}{2m}-\left(\frac{P_{\text{tot}}+k_{i}}{2m}\right)^{2}\right]-\left[\frac{P_{\text{in}}}{2m}-\left(\frac{P_{\text{tot}}}{2m}\right)^{2}-\left(\frac{k_{i}}{2m}\right)^{2}\right],\end{eqnarray}$$

where $P_{\text{in}}$ is the sum of all the co-occurrence probabilities within the partition that file $i$ is being included in, $P_{\text{tot}}$ is the sum of all the co-occurrence probabilities to files within the partition that $i$ is being included in, $k_{i}$ is the co-occurrence degree of $i$ and $k_{i,\text{in}}$ is the sum of the co-occurrence probabilities $i$ and other files within the partition that $i$ is merging with.

From this, the maximum modularity change can be identified. The associated files are then assigned to the same partition and the algorithm repeats the previous step of identifying the next file movement that will result in a further increase in modularity. If no further movement of files achieves an increase in modularity, the algorithm terminates, resulting in the partition set that gives the highest modularity score. This iterative process results in a partition set that are highly connected internally and weakly connected to one another.

The process of generating the partition from the bottom-up makes the approach inherently hierarchical and maps well to the hierarchical nature of product architectures within CAD systems. Hence, it is suitable for analysing DSMs generated from the co-occurrence of edits to files.

3.7 Optimisation

With the generation process defined, Stage 7 optimises and evaluates the stability of the DSM partitioning as a function of time period $(t_{p})$ and pruning threshold $(p)$ . The stage seeks to optimise for the following objective $O$ function:

(6) $$\begin{eqnarray}O=f(t_{p},p)=\frac{QN_{p}R}{N_{c}},\end{eqnarray}$$

where $Q$ is the modularity of the matrix partition, $N_{p}$ is the number of matrix partitions, $R$ is the ratio of the matrix that remains after pruning and $N_{c}$ is the number of matrix components.

To find the optimum values for $t_{p}$ and $p$ , a grid search using a range of time periods (0 to 24 h in increments of 1 h) and pruning values (0–1 in increments of 0.05) is used to generate the co-occurrence DSM and perform the partitioning. From this, the structural characteristics of each DSM are computed and $O$ is obtained. The values that produce $O_{\max }$ are then selected as the optimum time period and pruning values. Further analysis of this grid search can also reveal the sensitivity of the identified structures and provide confidence in their significance to the project.

3.8 Summary

With the dynamic DSM method introduced, the paper now continues by describing the set of near-identical systems design projects, the dataset that has been generate and example process of applying the dynamic DSM generation method.

4.1 Design intent

Table 2. FS team’s design intent as presented to the FS judges

The design intent statements given to the FS judges by the three teams are presented in Table 2. From these statements, it can be seen that Team A’s design intent was centred on generating a reduced design cycle in order to allow for more time to be associated with the build and testing of the product. In contrast, Team B’s focus is on the transient performance and user-centred design of the vehicle alongside a rapid design cycle in order to increase time for building and testing. Team C further emphasises the importance of developing a car with good transient performance, and to achieve this, their aim is to have a highly detailed build plan that allows for significant time for testing.

Table 3. Most common terms used across the team reports (in descending order)

In addition to the competition design and business reports, each engineer is required to produce a report on their work as part of a university assessment. Thus, the work discussed in these reports can provide an insight into the specific areas of focus for each team member and provides an indication of the allocation of resources across the design and manufacture of the car.

Natural language processing was applied to extract the commonly used terms across the reports. Stop-word lists were used to remove context independent terms from the analysis (e.g., the, a, and). Table 3 presents the top 12 terms for each team.

A consistent theme across all three teams is the importance of the engine in the development of the car. This is a logical outcome as the engine is integral to the power train and forms a central part of the chassis. Therefore, this is a crucial area for the team to understand in order for the rest of the car to be built.

Continuing down the list of terms, one can see the teams diverge due to their respective design intent. Team A focuses on wheel, weight, suspension, fuel, torque, chassis and tyres, which aligns with their intent of an ‘elegant’ design where they are wishing to create a simple robust design. These terms all relate to the fundamental principles of vehicle design and features of the car that have to exist in order for a car to function. The focus on simplicity is further highlighted by the lack of any terms relating to the aerodynamics and/or sensor data capture that one might expect on a racing car. Therefore, one would expect their DSM to have partitions centred around the chassis, suspension and wheel and tyres.

In contrast, Team B focuses on aerodynamics, data capture and testing with terms test, data, pressure and flow being used. In addition, model appears in the list, which highlights the greater use of digital modelling and simulation technologies used by Team B. Again, the set of terms aligns well to their design intent that focuses on being ‘lightweight’ and ‘responsive’ as it is argued that a considerable amount of modelling is required in order to understand how design concessions affect the transient performance of the car. Given the focus on testing, one would hypothesise that Team B’s DSM will be more modular to reflect the many tests that they wish to perform.

Team C appears to be an amalgamation of the previous two teams with a mix of fundamental features of the car such as weight, monocoque and power, whilst also considering the modelling and aerodynamics side of the car with the use of terms data, pressure and model. As the design intent of Team C is to be ‘dynamically excellent’, it can be argued that to achieve this, a team would need to be able to harmonise the fundamental chassis and suspension design with the additional loads generated by the supporting aerostructures. In addition, test is rated particularly high in Team C’s results, which further confirms the team’s focus on their design intent of ‘Testing’. Thus, it is hypothesised that Team C’s DSM would also contain more partitions to reflect testing and that the partitions will also centre around chassis and aerodynamic components.

This analysis of the teams’ design intent highlights that although each team is creating the same product, the approach each team has taken has had a considerable impact on the allocation of resources across the product architecture. Thus, there is potential in that similarities and differences can be observed in the comparison of DSMs for the FS teams and that structures within the DSMs may correspond to the design intent of the teams.

4.2 Product performance

Table 4. FS competition results

Table 4 presents the results from the FS competition and the hypothesis is that the comparison of DSMs may provide structures that indicate the potential performance of the final product.

In terms of overall position and top UK team, Teams B and C both outperform Team A with both teams placing 4th in their respective competition years. However, there is little difference in the UK team performance. Moving to the vehicle performance events, Team C outperformed Team A and B in the endurance event, which suggest that they were able to meet their design intent of ‘testing’ as reliability is the main concern for this event. Team B performed the best in efficiency and further confirms that they were able to translate their design intent of ‘efficient’ into the real-world product. However, in the sprint event, it is Team C that is the top performing team with a time of 52.232 s. This clearly aligns with their ‘dynamically excellent’ design intent.

Overall, Team C outperformed the other two teams at the competition; however, there were events that Team B did excel at. In contrast, whilst Team A did complete all the events of FS, their competition performance was not at the same level as Teams B and C. Therefore, there is potential for the study to uncover relationships in the DSM structures and overall competition and/or individual event performance.

4.3 Summary

From the discussion of both the teams’ design intent and product performance, the ES one hypothesises will exist within the DSMs are the following:

1. (i) Team A’s DSM partitions should be centred around the chassis, suspension and wheel and tyres.

2. (ii) Team A’s aim for an elegant and simple design will be reflected in a DSM with few disconnected modules.

3. (iii) Team B’s DSM is expected to be more modular to reflect the priority on testing the various systems of the car.

4. (iv) Team C’s DSM will contain more partitions to reflect the testing and that the partitions will also centre around chassis and aerodynamic components.

5. (v) Team C’s as-designed architecture revealed by the DSM will be more closely aligned to their as-planned architecture, given their focus on planning and organisation.

6. (vi) The joint results of Teams B and C reveal that their car is easier to modify and tune for each event, which would be reflected in lower change propagation statistics of their DSMs.

5.1 Systems design projects dataset

The data capture focused on the CAD files of the three teams. This is due to the maturity of the CAD file management system that the teams had in place, which managed the naming conventions, relationships, organisation and check-in/out status of these product models. The teams also associated their models to a consistent product architecture featuring 10 systems:

In comparison, other common files found in engineering projects, such as computational fluid dynamic and finite element analysis files, were stored in a more ad hoc manner and across many storage devices, thus making it more challenging and erroneous to capture.

A Raspberry PiFootnote ² connected to the network was used to monitor the CAD file management system. The Pi recorded the status of the file management system using a custom-built folder monitoring package written in Node.js.Footnote ³ The tool monitors the entire folder/file structure containing the CAD files and detects any accesses or modifications to them. These are then recorded in a NoSQL database.

Figure 4. Distribution of product models with respect to the team defined product sub-systems.

Table 5. FS CAD file statistics

Table 5 presents a summary of the number of files and edits across the three teams. The summary reveals that Teams B and C produced almost twice the number of models compared to that of Team A; however, the total number of edits recorded was consistent between Teams A and B, whilst Team C contained fewer. To further investigate this, Figure 4 shows the distribution of product models by sub-system. This reveals that the additional models present in Teams B and C come from the Frame and Body and Suspension sub-systems, respectively. This aligns with the design intent of the two teams where they have focused on the transient performance and user-centred design aspects of the vehicle and indicates the potential of mapping intent and product performance to structures within the DSM. In all other sub-systems, the number of product models remains fairly consistent across the three teams. The final aspect to note is the reduction in the Standard Part counts across the teams, and following a discussion with the team members, it was revealed that these were allocated to more specific areas of the sub-system architecture. This provides an indication that Teams B and C were more confident in their allocation of the product models during the early design phases than Team A.

5.2 File selection

Figure 5. Frequency of edits per product model.

To select the files of interest, Figure 5 shows the frequency distribution of file edits. It is interesting to note that all the teams follow a similar profile with the additional files present within Teams B and C’s datasets appearing to extend the tail of the frequency distribution. This suggests that these files represent standard/bought-in components that are not edited further by the team.

The files are filtered based on the number of updates made. For this study, this has been set to four as this includes the creation of the product model followed by three further updates. This removes models that show little to no activity and may represent constraints in the team’s design space that they are unable to alter (e.g., a bought-in part such as the engine block and/or gearbox) as well as standard parts/fixings (such as nuts and bolts). Following this filtering, the files of interest totalled 353, 379 and 311 for Teams A, B and C, respectively.

5.3 Generation

A time period $(t_{p})$ of 2 h has been selected following the optimisation strategy being performed across the datasets (see Section 5.8). Figure 6 presents the output of this stage, which is an $N\times N$ matrix containing the frequency of co-occurrences between the files.

Figure 6. Co-occurrence DSMs.

5.4 Normalisation

Following the generation of the co-occurrence DSM, the three teams’ DSMs are normalised. Figure 7 shows the impact of normalisation on the DSMs of the three teams. It is immediately apparent that the normalisation of the co-occurrence of values and distribution of the ratios is consistent across the three teams. This provides an initial indicator that a similar structure is present across the teams.

Figure 7. Conditional probability of file edit co-occurrences.

5.5 Evaluating the ‘directedness’

Table 6 reveals the ‘directedness’ of the teams DSMs with respect to the measure  $D$ . In all cases, the measure is significantly low enough to permit the application of an undirected DSM for further partitioning. This was performed on all DSMs for consistency in the analysis.

Table 6. ‘Directedness’ of the DSMs

5.6 Pruning

The pruning level $(p)$ for the three teams’ datasets was set to 0.3 (see Section 5.8) and the results are presented in Table 7. The pruning has had a considerable effect on reducing the candidate dependencies within the DSMs. The results from the pruning also give an indication to the level of co-occurrence activity with Team A having much greater co-occurrence of activity (40,018) compared to that of Teams B and C. The impact of this is discussed in Sections 6 and 7.

Table 7. Reducing the effect of false positive dependencies

5.7 Partitioning

The DSMs are then partitioned and Figure 9 demonstrates the ability for the partitioning technique to reveal the underlying structure in the DSMs. The red boundaries indicate the architecture as planned by the team, whilst the blue boundaries indicate the architecture as designed and produced from this analysis. This visual indication highlights the potential of the technique to uncover the ES within the teams’ architectures. These results, along with the mapping to the hypotheses, form the ES that are presented in the following section.

5.8 Optimisation

To determine $t_{p}$ and $p$ for the DSM generation, the optimisation strategy was performed across all three datasets and the average for each set of $t_{p}$ and $p$ values was taken. Figure 8 shows that a $t_{p}=2$ and $p=0.3$ provides the optimum values the three datasets. Also, the variance across the three datasets was very low for the entire range of $t_{p}$ and $p$ values, which further highlights the consistency in working practices across the three teams.

Figure 8. Aggregated optimisation results for the three FS teams.

Figure 9. Partitioning of a DSM.

6.1 End-of-Project DSMs

Figure 9 and Table 8 present the results from the analysis of the end-of-project DSM. Team A’s DSM has the fewest partitions (9) and components (1) and the lowest modularity (0.56), which reveals that Team A has the most interdependent product model out of the three teams. This is surprising as the team were focusing on an ‘elegant’ design with the aim of keeping it simple. The DSM analysis appears to counter their argument for a simplified design and is a potential reason for them under-performing when compared to Teams B and C.

Table 8. End-of-project DSM summary

Team B’s design is the most modular with their DSM containing 38 partitions as opposed to 9 and 28 of Teams A and C, respectively. In addition, 29 of these partitions were components. This highlights that Team B had a better understanding of the dependencies between components and were therefore able to generate a more modular design. This is further confirmed by their as-planned partitions scoring a modularity of 0.41, the highest of all the teams. In addition, the team have the smallest $\unicode[STIX]{x1D6E5}Q$ ( $+0.23$ ) between their as-planned partitioning and the as-designed partitioning, revealing that they had a good understanding of their intended architecture and maintained it through the development and build phases. This is in line with Team B’s design intent of an efficient design process to ensure more testing can be performed.

In contrast, Team C’s DSM has 28 partitions with 21 being components. This shows that Team C’s design consists of fewer, larger modules that are disconnected from one another (Figure 9f). In addition, Team C’s design achieved the highest modularity (0.81) score by partitioning through co-occurrence. This reveals that their as-designed product model is the most structured and well-defined when compared to the other teams, and although you would not expect a modularity score of 1 as interfaces must exist between product models, it highlights that Team C’s interfaces are more defined. In addition, their modularity score increases the most from their as-planned architecture ( $+0.44$ ), revealing that they were better at further modularising the architecture during the design phase. This aligns with Team C’s design intent of ‘organisation and planning’ in order to create a well-defined product model. This clarity of interfaces between components is also reflected in their competition performance and indicates a greater knowledge and understanding of their product model. Thus, $\mathbf{E}\mathbf{S}_{\mathbf{1}}$ is that a focus on planning and organisation leads to a more defined product architecture with $\mathbf{E}\mathbf{S}_{\mathbf{2}}$ showing that a more modular architecture positively affects competition performance.

Figure 10. Composition of DSM partitions.

Figure 10 reveals the compositions of the as-designed partitions. Through this, it can be seen that Team A’s partition composition is the most diffuse with each partition featuring a range of product models from the pre-defined sub-systems (Figure 10a). This further highlights that Team A were unable to identify a clear sub-system structure during the early design phases, and this may have led to the issues in the product’s performance and the appropriate allocation of resources during the project.

Figure 10(b) further highlights that Team B’s more modular design can be clearly seen through the greater number of smaller partitions that are focused on single pre-defined sub-systems. In addition, the larger partitions for Team B are noticeably more in line with the initial sub-system classification outlined by the team, which further highlights that Team B were able to better classify the models during the early design phases.

These features are also present in Team C’s composition of clusters (Figure 10b) where there is a number of large inter-related partitions as well as a number of smaller partitions relating to specific sub-systems. These smaller partitions may also relate to specific work packages that were assigned during the generation of the project build. The fact that they exist in the final DSM highlights that Team B and C were both able to identify independent elements of work within the build of the vehicle during the early design phases. This aligns to their design intent of ‘organisation and planning’. Thus, $\mathbf{E}\mathbf{S}_{\mathbf{3}}$ is a clear identification, and adherence to the pre-defined sub-system structure leads to improved competition performance as well as further confirming $\mathbf{E}\mathbf{S}_{\mathbf{1}}$ .

6.2 Change propagation

A typical application of DSMs is to use them as a predictor/indicator for the capability of a system to accommodate changes along with an evaluation of the level of propagation a change will likely generate (Clarkson, Simons & Eckert Reference Clarkson, Simons and Eckert2004). To investigate this, the study processed the cumulative conditional probabilities of change for each file within the DSM.

This was achieved by taking the file of interest and selecting the files that exist within one and two branch paths. One and two branch paths were selected as the inference of a likely change to a file beyond the third branch has been shown to be unreliable (Pasqual & de Weck Reference Pasqual and de Weck2012). With this list of files and associated one and two branch paths, it is possible to calculate the conditional probability of a change propagating to the neighbouring files.

For example, take File A as the file of interest and File B as a neighbouring file with one and two branch paths between the files. The conditional probability of File B changing is determined by taking the product of probabilities along each of the paths, which gives the probability of each of the paths being traversed, and calculating the probability of at least one of these being traversed if a change occurs. A threshold value for the conditional probability (0.5 in this case) is taken to indicate that a change would occur in B as a result of a change to A.

Table 9. Change propagation statistics

Performing this analysis across all the files gives the distribution of the likely level of change that is to occur across the DSM and gives an insight into the DSMs’ resilience to handle changes. Figure 11 highlights this characteristic through a histogram depicting the proportion of the system that is likely to change by file changes. Table 9 shows the characteristics of each team’s histogram.

Figure 11. Change propagation distribution.

Comparing the distributions, it can be seen that Team A has a much wider distribution over the proportion of system to change with some models requiring checks across almost the entire product architecture. This is in stark contrast to Teams B and C, which exhibit much lower maximum proportions of change values. Inspecting the mean, it can be observed that Team A has a much higher average than that of Team B or C. This indicates that a change within Team A’s product architecture would require significantly more resource in order to handle the propagation of change, thus making it potentially more challenging for the team to accommodate changes. Hence, $\mathbf{E}\mathbf{S}_{\mathbf{4}}$ is that smaller change propagation trees correspond with improved competition performance.

Figure 12 shows the distribution of the top 30 product models that are most likely to change as a consequence of a change in another product model. In Team A’s DSM, the top 30 product models are distributed across a range of sub-systems, whilst the most likely models to change from Team B are exclusively from the Frame and Body and Engine and Drivetrain. Team C is similar to that of Team B with the addition of models assigned to the suspension sub-system. As the Frame and Body models form the external structure of the design, this may indicate that Team B’s design was such that any internal changes within the design could be accommodated by the Frame and Body alone. This can also be said for Team C with the suspension sub-system as this is an external element of the vehicle. In contrast, Team A’s design required more extensive cross sub-system alterations to accommodate changes. Therefore, $\mathbf{E}\mathbf{S}_{\mathbf{5}}$ is that competition performance improves when the engineering changes can be accommodated by external sub-systems.

Figure 12. Model most likely to change as a consequence to a change in another model.

6.3 Evolution of DSMs

A particular affordance of automatically generated DSMs is that they can be produced in real time. To investigate this, a pseudo real-time generation of the DSMs for the three projects has been generated. This has been achieved by generating and analysing the DSM as it accumulates product model updates throughout the project. At every product model update, the following characteristics of the DSM have been calculated:

1. (i) the number of product models that form the DSM;

2. (ii) the number of dependencies between product models;

3. (iii) the number of partitions within the DSM;

4. (iv) the modularity of the DSM;

5. (v) the number of components within the DSM.

Figure 13. Evolving DSM statistics.

The results are presented in Figure 13 where the cumulation of updates has been normalised in order for comparisons to be made between the three teams. In addition, it can be seen that the lines representing each teams’ DSM do not start at 0 but further along the cumulative update scale. This is due to the fact that during this period, the number of co-occurrences is insufficient to form a DSM.

Looking at the number of product models within the DSM (Figure 13a), it is interesting to note the similarity in the generation of models over the course of each project. This indicates that there is a relationship between the number of updates and maturity of the product architecture. This result provides evidence for $\mathbf{E}\mathbf{S}_{\mathbf{6}}$ , which states that normal and predictable behaviours exist in the design process of near-identical projects.

However, differences start to appear when one focuses on the number of co-occurrences weighted ${>}0.25$ (Figure 13b). Co-occurrences appear early and see considerable growth for Teams A and B, whilst Team C’s co-occurrences ramp up more gradually and plateaus. Both Teams A and B see a continuing rise with an abrupt change in the number of co-occurrences near the end of the project, which indicates a significant change in product architecture and potential rush to finish the product model before the deadline. With Team C performing the best out of the three teams, $\mathbf{E}\mathbf{S}_{\mathbf{7}}$ is that gradual and steady growth of co-occurrences between product models has a positive relation to competition performance.

Figure 13(c) reveals the number of partitions identified with an almost linear relationship between partitions and cumulative updates being exhibited. This suggests that if the design begins with a high number of partitions, then the product architecture will continue to be highly partitioned. In addition, the lines appear rather jagged, showing the continual addition, merging and parting of partitions within the DSMs as they evolve. Thus, $\mathbf{E}\mathbf{S}_{\mathbf{8}}$ is that a modular architecture at the start of the design process will promote further modularisation of the product architecture.

Figure 13(d) reveals how the modularity of the DSM evolves throughout the cumulation of updates. Modularity indicates the level of structure that resides within the matrix with a score of ${>}0.25$ indicating a structure that is beyond random chance. In the case of the three teams, it can be seen that the values calculated are consistently higher than pure chance and, thus, provides further evidence that the method is capturing and providing design process insights. Overall, all teams’ modularity scores increase as the updates accumulate with Team C achieving the highest modularity score. Thus, $\mathbf{E}\mathbf{S}_{\mathbf{9}}$ is that the design process can be described as a process of continually increasing the level of structure between a set of product models.

In addition, Team C provides an interesting feature within Figure 13(d) where there is a significant decrease in modularity before returning to its previous value. Upon reviewing the design reports of the team, it was discovered that each sub-team developed their assemblies away from one another before integrating them to form the complete assembly model. The feature within Figure 13(d) aligns with this event.

The final aspect monitors the number of components throughout the evolution of the DSM (Figure 13e). This follows a similar trend to that of the partitions within the DSM where Teams B and C see a continual growth in the number of components, whilst Team A contains very few. This highlights the potential increase in the modular architecture of Teams B and C compared with Team A. In addition, Teams B and C both have a sudden increase in the number of components as they move beyond 0.8 cumulative updates. As these are components and not partitions, this suggests that these product features are separate to the main product architecture. Interrogation of the product models within these components highlighted that these were finishing touches to the product such as the decals on the bodywork and ancillary elements to the aero package (e.g. gurney flap). It is interesting to see that Team A were unable to reach this part of the design process and may relate to the relative lack of performance in the competition as their product had not reached the maturity of Teams B and C. Therefore, $\mathbf{E}\mathbf{S}_{\mathbf{10}}$ is that the final stages of an FS design process are indicated by a sudden increase in the number of components to the DSM.