2.1 Creativity

Creativity is defined as ‘the ability to imagine or invent something new of value’ (Childs et al. Reference Childs, Hamilton, Morris and Johnston2006), or ‘the production of novel, useful products’ (Mumford Reference Mumford2003). It is considered as a vital element in design, which initiates innovations, assists problem solving, and increases a firm’s market share (Sarkar & Chakrabarti Reference Sarkar and Chakrabarti2011). Stewart & Simmons (Reference Stewart and Simmons2010) indicated that it is creativity which will drive business in the future.

Creativity is a fundamental feature of human intelligence (Cross Reference Cross2011). The outputs of creativity can be objects, actions, or ideas, and these outputs are conceived to be novel, high in quality, and useful (Carruthers Reference Carruthers2011). The creativity output arises from the combination of some essential mental capabilities, which is often the result of long periods of work with a series of mini-breakthroughs (Childs Reference Childs2014). Most creative ideas can be characterised by the diverse thoughts from previous experience stored in memory elements (Liikkanen & Perttula Reference Liikkanen and Perttula2010; Benedek et al. Reference Benedek, Jauk, Fink, Koschutnig, Reishofer, Ebner and Neubauer2014).

2.2 Combinational creativity

Boden (Reference Boden2004) has proposed three approaches to achieve creative ideas in our human brain, which are combinational, exploratory and transformational creativity. Exploratory creativity includes the generation of creative ideas through exploring structured conceptual space (structured styles of thought), for instance, the different styles of coffee mugs. Transformational creativity involves transforming the conceptual space to produce transformative ideas in new structures, for example, the origination of aromatic chemistry by transforming string molecules to ring molecules. Combinational creativity, also known as conceptual combination, involves generating new ideas through exploring unfamiliar combinations of familiar ideas (Ward & Kolomyts Reference Ward, Kolomyts, Kaufman and Sternberg2010).

Combinational creativity can be achieved through forming associations between ideas that were previously only indirectly linked (Boden Reference Boden2004). The ‘ideas’ to be combined can be concepts, words, images, sounds, and even more abstract ones such as, music styles, artistic genres, and so on (Ward & Kolomyts Reference Ward, Kolomyts, Kaufman and Sternberg2010). Combinational creativity is the easiest approach to creativity for human beings, as it is a natural feature of the human associative memory system (Boden Reference Boden2009). Combining ideas could be the best way to fully use nowadays abundant information for producing creativity (Yang & Zhang Reference Yang and Zhang2016). Many people have used the term ‘combinational creativity’ to describe creativity, for example, Steve Jobs defined that ‘creativity is just connecting things’, Frigotto & Riccaboni (Reference Frigotto and Riccaboni2011) indicated the nature of creativity is to combine, and Henriksen et al. (Reference Henriksen and Mishra2014) described creativity as ‘the process of making alterations to, and new combinations with, pre-existing ideas and artifacts, to create something new’.

Combinational creativity has been used widely in creating various products. For instance, the well-known chocolate bar ‘Kit-Kat’ can be regarded as just a simple combination of chocolate and wafers; the toy ‘LEGO’ is a series of combinations of numerous different bricks; the ‘Dyson’ vacuum cleaner combines cyclone and ball with vacuum cleaner which provided a strong suction power and a steering mechanism, respectively; the ‘Apple Watch’ is a combination of watch and data device, albeit with a very sophisticated operating system.

Although combinational creativity is the easiest type of creativity for humans, it is challenging for computer-based implementation. Boden (Reference Boden2004) noted that combinational creativity requires a very rich store of knowledge, and the ability to form links of many different types. However, combinations of ideas might lead to a ‘combinational explosion’, of which it could take years to produce and evaluate all the possibilities (Simonton Reference Simonton, Kaufman, Glaveanu and Baer2017). These are also some of the main reasons why acclaimed designers are usually rich in knowledge and experience. It is not difficult for computers to form combinations or even novel combinations of stored ideas, but it is extremely difficult for computers to generate valuable or interesting combinations and to recognise their values (Boden Reference Boden2009). Compared with the human brain, it is difficult for computers to achieve the cornucopia of human associative memory. Moreover, a computer cannot naturally form links between ideas and evaluate them, while a human can naturally associate ideas of many different kinds during everyday thinking. Therefore, modelling combinational creativity on computational systems requires a rich knowledge database and methods of generating links comparable to a human. Some successful computer programs that involve combinational creativity are reported by Gastronaut (Butnariu & Veale Reference Butnariu and Veale2006) and Portrait Robot (Augello et al. Reference Augello, Infantino, Pilato, Rizzo and Vella2014).

In order to simulate aspects of the human brain in producing combinational creativity, understanding how the human mind and human memory work is essential. The following sections briefly illustrate the basic working principles of the human mind and memory in producing combinational ideas. In this study, human memory is considered as a knowledge database containing various ideas, while the human mind is regarded as an associative system that can form links among stored ideas.

2.3 Human memory – the knowledge database

Memory can be described as the ability to store, retain, and recall information (Shergill Reference Shergill2012). It is significant to designers, as an idea generation process involves controlled retrieval of existing information from memory (Liikkanen & Perttula Reference Liikkanen and Perttula2010; Benedek et al. Reference Benedek, Jauk, Fink, Koschutnig, Reishofer, Ebner and Neubauer2014). The multi-store memory model proposed by Atkinson & Shiffrin (Reference Atkinson and Shiffrin1968) comprising long-term memory and short-term memory is often cited, and it is used to explain how human memory works in this study.

Short-term memory is a system that underpins the capacity to retain small amounts of materials over periods of a few seconds (Baddeley, Eysenck & Anderson Reference Baddeley, Eysenck and Anderson2009). Short-term memory is also known as working memory, and it is the place where information is manipulated and processed (Cowan Reference Cowan2008). Long-term memory stores information over long periods of time (Baddeley et al. Reference Baddeley, Eysenck and Anderson2009). Information from the outside world flows into short-term memory through sensory perceptions (such as visual and auditory), and then the information can flow into long-term memory through encoding. The information can be passed back into short-term memory by retrieval while cues from the outside world or ourselves are provided. Through refreshing (Vergauwe & Cowan Reference Vergauwe and Cowan2015), the key information in short-term memory can be maintained for a relatively long time. An information flow diagram of the human memory system using this embodiment is shown in Figure 1.

Figure 1. Information flow diagram of a human memory system model.

Although human memory has extensive store capacity and duration, the performance on tasks involving long-term memory declines steadily through the adult years. It seems that human memory is less reliable than augmented memory (non-single human mental memory) (Baddeley et al. Reference Baddeley, Eysenck and Anderson2009). A number of augmented memories, for example, books, Internet, and professional memory augmentation systems such as Ubiquitous Memories (Kawamura et al. Reference Kawamura, Fukuhara, Takeda, Kono and Kidode2007), are available to help us store and recollect memory elements.

2.4 The human mind – the associative system

The human mind is widely considered as a product of natural selection and evolution (Bolhuis & Wynne Reference Bolhuis and Wynne2009). Intuition is a fundamental facet of our thinking and used by us to solve all kinds of problems (Pinker Reference Pinker1999). Connectionism is a widely accepted approach for explaining how the human mind works, which is based upon highly interconnected processing units (Thomas & McClelland Reference Thomas, McClelland and Sun2008). The human associative mind is a connectionist system, with information elements associated together in the mind through experience (Anderson & Bower Reference Anderson and Bower2014). Associative memory is an attribute of the mind’s associative power, which is the ability to learn and remember the relationships between different information. For instance, we recall a person’s face and other characteristics immediately when the known person’s name is mentioned (Anderson Reference Anderson1976; Suzuki Reference Suzuki2005). Due to this ability, an enormous associative network involving information and relationships has been formed in the human long-term memory through learning and experiencing. Many cases of creativity depend on the mind’s associative power, especially combinational creativity (Boden Reference Boden2004).

A semantic net is an artificial associative network in a graph structure that represents knowledge in relation patterns of interconnected nodes and links (Sowa Reference Sowa and Shapiro1992). It is a depiction of human associative memory as well as an associative model of cognition, in which each idea can be linked to some other relevant ideas and sometimes ‘irrelevant’ ideas. Combinational creativity, as defined, involves generating new ideas through associating unrelated ideas. Therefore, achieving combinational creativity can be regarded as forming new combinational links between unrelated or indirectly related ideas in a semantic net.

3.1 The basic algorithm of the Combinator

This section and the following two sections illustrate the novel approach used for developing the Combinator. This type of approach can be regarded as a new method for developing design tools, especially idea generation tools. In this study, the Combinator is aimed at ideation in the product design field. However, it is believed that the Combinator can be used in various fields, such as engineering, fashion, music, and art. Understanding how the human mind and memory work is crucial for simulating combinational creativity on computers. Memory is the place where structured information is stored, and the mind is related to how structured information and new links between unrelated information are formed. Therefore, in order to simulate combinational creativity, modelling a rich structured database and an idea linking system are necessary. The basic algorithm of the Combinator, is shown in Figure 2.

Figure 2. The basic algorithm of the Combinator. © Ji Han.

The algorithm starts with trawling and retrieving relevant user-defined information from the Internet using web crawlers, which is similar to a human in gaining knowledge from augmented memory. The information is then stored in the core database and linked with a semantic network after being analysed and processed by a natural language processing tool. This is a process simulating how knowledge is processed and related to each other in human memory. When a user input, which can be a design keyword or a desire, has been delivered to the Combinator module, a cue is produced for instructing the retrieval process from the Combinator database to the Combinator module. This is similar to how information in human long-term memory is retrieved and transferred into short-term memory. The ideas in the Combinator are linked together, thereby generating combinational ideas in text forms as well as visual forms. This imitates aspects of how the human mind forms links between ideas to generate combinational ideas.

As illustrated above, the basic algorithm of the Combinator conforms to natural features of the human mind and memory system in generating combinational creative ideas. Thereby, the natural feature of the algorithm has provided a method accommodating a human’s natural thinking pattern, which indicated simple-to-learn and easy-to-use characteristics.

3.2 Creating the Combinator database

Creating a human-like structured knowledge database is the initial stage, but not every piece of information is relevant to product design. Therefore, it is necessary to narrow down data elements and discard irrelevant elements in order to produce creative ideas (Childs Reference Childs2014). The database can be simply generated via manual input, but this is time-consuming and capacity limited. This study takes advantage of natural language processing tools, information retrieval technology, and existing knowledge bases to produce the Combinator database.

A web crawler is a program that browses and retrieves specific resources and information from the Internet (Patil, Bhattacharjee & Ghosh Reference Patil, Bhattacharjee and Ghosh2014). This technology has been used for various purposes, such as data mining, information searching, web monitoring, and web archiving. It has been demonstrated that web crawlers can be used to mine design-related information for idea generation (Wang, Childs & Jiang Reference Wang, Childs and Jiang2013).

An open-source web crawler is used in this study for developing a text-based knowledge database. This allows users to define which domain or group of domains to explore, for example, what kind of data to extract and which data format to use. In order to narrow down the database in a product design field, the web crawler was programmed to retrieve only main body texts describing design-related products or ideas from a design website that covers the best in product design. Simultaneously with the crawling, the crawled texts were processed by a natural language processing tool, to extract requisite words. Texts such as proper nouns, prepositions, and plurals were discarded using the natural language processing tool to improve the quality of the database. The words were then stored into corresponding databases in the Combinator according to their part of speech. These databases are ‘the core database’ of the Combinator, as shown in Figure 2.

In order to construct a human-like associative knowledge database, the core database is required to be linked with a semantic net. ConceptNet is used in this project, which is a knowledge base that provides a large semantic net representing general human knowledge and the commonsense relationships between them (Liu & Singh Reference Liu and Singh2004; Speer & Havasi Reference Speer and Havasi2012). The knowledge contained in ConceptNet is mined from crowd-sourced resources (for instance, Wikipedia, Wiktionary, and Open Mind Common Sense), expert-created resources (such as, WordNet (Fellbaum Reference Fellbaum1998) and JMdict (Breen Reference Breen2004)), and other resources (such as DBPedia (Lehmann et al. Reference Lehmann, Isele, Jakob, Jentzsch, Kontokostas, Mendes and Hellmann2015)). Besides, ConcepNet allows users to add data and even to create their own versions.

The information elements in the core database are unorganised and unrelated, for example, eight individual elements from the core database are shown in Figure 3. After connection with the ConceptNet, the core knowledge elements are connected to each other as well as linked with information that is correlated but not included in the core database. This associated network formed the Combinator database. For instance, the eight core items of information are linked with each other as well as associated with correlated information via commonsense relationships such as ‘Is A’, ‘Used For’, ‘At Location’, and ‘Related To’, as shown in Figure 4. The associative network example shown in Figure 4 is an epitome of the Combinator database, which demonstrates how information is structured in the Combinator database.

Figure 3. Information elements in the core database.

Figure 4. Information elements in the Combinator database.

3.3 Establishing the Combinator module

As shown in Figure 2, the Combinator module provides an imitation of aspects of human short-term memory while the human mind is processing combinational creativity. It retrieves information from the Combinator database and combines it with the user input to produce combinational ideas. The user input can be a desire or a design keyword, and the output idea is a text combination along with a correlated image combination. The Combinator module can provide a cue, provided by the user input, to retrieve a word or a phrase that is associated with the input from the Combinator database to replace the input for the idea combination. For example, if the user input is ‘chair’, the Combinator module will retrieve an associated word, such as ‘sofa’ or ‘wood’, and combine it with another core information word retrieved from the core database which was crawled from the Internet. This replicates the human mind’s associative power that people often think of other information associated with the one in the mind. For instance, when someone comes up with ‘car’, he or she might also think of ‘window’, ‘seat’, ‘park lot’, and so forth.

The Combinator module can produce a noun–noun combinational phrase, as well as adjective–noun compound phrase and verb–adjective–noun combinational phrase. The use of language or words as stimuli to increase creativity has been previously explored (e.g. Chiu & Shu (Reference Chiu and Shu2012)). However, producing only text-based combinational ideas can have limitations, as sometimes it is difficult for a user to capture the combinational meaning of a compound phrase. This might be caused by the limitation of the user’s vocabulary or the lack of corresponding images in the user’s long-term memory. Combinational images provide an opportunity to enhance the understanding and interpretation of a generated compound phrase, as the human brain activities are mainly triggered by the input gathered visually (Luis-Ferreira & Jardim-Goncalves Reference Luis-Ferreira and Jardim-Goncalves2013). Superimposed and merged images, which are regarded as combinational images in this study, can lead to more creative outcomes (Ward & Kolomyts Reference Ward, Kolomyts, Kaufman and Sternberg2010). Combined images have been identified as effective stimuli for creativity (Ward & Kolomyts Reference Ward, Kolomyts, Kaufman and Sternberg2010).

Images need to be gathered from a source such as the Internet or an image database in order to generate combinational image-based ideas. The open-source crawler used can crawl images according to the existing text-based database and store them in an image database, but creating an image database via the web crawler can take weeks or even months as it is usually enormous. Therefore, the authors have developed a live feed image crawler using Python software to be more time-efficient. The live feed image crawler, which is integrated into the Combinator module, can retrieve corresponding images in real time from the Internet according to the generated combinational phrases.

The Combinator is capable of producing combinational image-based ideas with the assistance of a vision algorithm library. The Combinator provides two main methods for image combination: direct combine and blending. In direct combine, images are cropped first and then merged next to one another. In blending, images become transparent first and then superimposed on one another. For example, Figure 5 shows the original images of an ice cream scoop and a shell, and their direct combination image as well as the blending combination image. Ideas can be derived from the combinational images, such as an ice cream scoop based on a nautilus shell for serving (Millar, Goltan & Cheng Reference Millar, Goltan and Cheng2010). The Combinator implemented the alternative image combination methods to avoid the Einstellung effect that might block the mind of sticking to just one type of image-based idea combination.

Figure 5. An example of image combinations.

Ward & Kolomyts (Reference Ward, Kolomyts, Kaufman and Sternberg2010) point out that mentally combined visual-based ideas are essential for developing inventions and exploring discoveries. Therefore, the combinational ideas produced by the Combinator, especially the image-based ideas, are potentially beneficial for designers in idea generation. Users of the Combinator can be further stimulated by the generated combinational ideas, and thereby producing or deriving more ideas.

3.4 Examples of using the Combinator

The Combinator software, in the formulation described here, provides a simply designed user interface, as shown in Figure 6. There is an input box at the top that allows users to input their design keyword or desire. The Combinator provides three option menus respectively for selecting a semantic relation, choosing the number and the part of speech of words used for combination, and selecting an image combination method. The semantic relation option menu can be switched on or off by the switch button next to it, which allows users to choose a relationship from over twenty types, for instance, ‘Related To’, ‘Is A’, ‘Part Of’, ‘Member Of’, ‘Has a’, ‘Used For’, ‘Capable Of’, ‘At Location’, ‘Causes’, and ‘Synonym’. Thereby, the Combinator can retrieve information associated with the input keyword in a selected relation. However, the retrieved word or information might not belong to the same part of speech category as the input word. There is also a ’Random’ option which can choose a relationship randomly from the list. The ‘Select what to combine with’ menu is used for defining the combination output type, for example, to combine the input with ‘one noun’, ‘two nouns’, or ‘verb $+$ adjective $+$ noun’. Users can select the output combinational image style, ‘Direct Combine’, ‘Blending’, and ‘Random’ (Randomly select a method), via an image combination method menu.

Three function buttons, ‘Generate’, ‘Show Image/Change Direction’, and ‘Play’, are located at the bottom of the interface. A text-based combinational idea is produced after clicking the ‘Generate’ button. For example, the combinational idea ‘Origami Handbag’ is generated according to the input ‘Handbag’, as shown in Figure 6 (while the semantic relation function has been switched off). A correlated image-based combinational idea pops out while ‘Show Image/Change Direction’ is clicked. For instance, a combinational image of ‘Origami Handbag’ is generated, as shown in Figure 7. The combinational image produced can be saved on the user’s computer. By clicking the button again, the image combination method or direction is changed to gain more stimulus, for example, blending can transform into direct combine, and left and right merging direct combine can change into up and down merging. The ‘Play’ button yields a slideshow of combinational ideas, including both image-based and text-based ideas, correlated with the input keyword. New information is retrieved from the database for producing a series of new combinational ideas according to the input. This can help users improve divergent thinking and avoid design or cognitive fixation.

Figure 6. The Combinator user interface.

Figure 7. A generated combinational image.

In the current version of the Combinator, the combinational ideas are produced in a random manner to provoke the users’ brain. In other words, ideas are randomly selected from the database to be combined with the input to produce the combinational creative idea. Randomness has been identified as a vital element of creativity (Carruthers Reference Carruthers2011), and random stimuli have been indicated to be beneficial in ideation (Howard et al. Reference Howard, Culley and Dekoninck2011).

As shown in Figures 6 and 7, a handbag idea, an origami handbag, has been prompted by the Combinator. A number of ideas can be interpreted or derived from the generated compound phrase and the combinational image, for example, a handbag that can be folded similar to origami (Kawamoto Reference Kawamoto2013). As a result, it is expected that numerous ideas related with a handbag can be generated by the Combinator.

Another example of generating ideas about a ‘violin’ is shown in Figure 8, while the semantic relation option is switched on and ‘Has A’ relation is selected. The Combinator has retrieved a semantically related idea ‘String’, as ‘Violin’ has ‘String’. This is similar to the human while a person has ‘Violin’ in the mind, he or she might think of ‘String’. According to the generated outcome ‘Spider-Silk String’, ideas, such as a violin made using spider silk for string or for influencing the properties of the construction, can be produced or implied (Alesandrini Reference Alesandrini2016). The simple examples demonstrated in this section have shown how the Combinator works and indicated its capability of assisting users in generating creative ideas.

Figure 8. An example when the semantic relation is switched on.

4.1 Outcome-based evaluation

An outcome-based approach is the most common method for evaluation, for example the study conducted by Istre et al. (Reference Istre, McCoy, Moore, Roper, Stephens-Stidham, Barnard, Carlin, Stowe and Anderson2013). The outcome approach is easier than the process approach, especially for engineers, as there are fewer difficulties for experts to evaluate a set of ideas in their domains for a given problem (Shah et al. Reference Shah, Smith and Vargas-Hernandez2003). In an outcome-based evaluation of creativity, psychometric measurements have been used extensively for decades. A good measure of the outcomes can objectively reflect the idea generation method, as specific metrics can be used to relate the creativity of ideas to the performance of the ideation method.

Researchers have proposed several sets of psychometrics to evaluate the creativity level of outcomes generated by an ideation method, for example, Plucker & Makel (Reference Plucker, Makel, Kaufman and Sternberg2010) used fluency, flexibility, originality, and elaboration of ideas to measure creativity. Dean et al. (Reference Dean, Hender, Rodgers and Santanen2006) indicated a set of psychometric composed of novelty, workability, relevance, and thoroughness for creativity measurement. Diedrich et al. (Reference Diedrich, Benedek, Jauk and Neubauer2015) only used novelty and usefulness for creativity evaluation. In the study conducted by Chulvi et al. (Reference Chulvi, Sonseca, Mulet and Chakrabarti2012), novelty and utility were applied for creativity assessment through using a questionnaire based on the CPSS (Creative Product Semantic Scale) (Besemer & O’Quin Reference Besemer and O’Quin1989). The taxonomical form of this approach allows to select which items to use in each assessment. Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011) employed novelty and usefulness to assess creativity of products. In this method, novelty is determined by the FBS (Function–Behaviour–Structure) model and the degree of novelty is assessed by the SAPPhIRE model (Chakrabarti et al. Reference Chakrabarti, Sarkar, Leelavathamma and Nataraju2005), while usefulness is evaluated by level of importance, rate of popularity of use, frequency of usage, and duration of use. It has been indicated that this method can reflect the intuitive notion of novelty and usefulness of experienced designers.

Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003) proposed quantity, quality, novelty, and variety as the four creativity measures. In this method, quantity is the total number of ideas produced, which is known as fluency in idea generation. It is commonly considered that the more ideas generated, the higher chance of better ideas occurring. Quality, which is commonly described by usefulness (Kudrowitz & Wallace Reference Kudrowitz and Wallace2013), is the feasibility of an idea and how close the idea comes to meet the design specifications. High-quality ideas generally have higher design success rates. Novelty is the unexpectedness or unusualness of an idea comparing with the others, which implies originality. A novel idea is usually the result of expanding the design space. Variety or flexibility reflects the exploration of the solution space during an ideation process. Generating similar ideas indicates low variety, which shows a limited exploration of ideas in other areas of the solution space. It has been indicated that this approach is focused on measuring the creativity of solving design problems and the effectiveness of an idea generation method, while other approaches, such as Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011), are focused more on measuring the creativity of products or outcomes. Therefore, Shah et al.’s (Reference Shah, Smith and Vargas-Hernandez2003) psychometric evaluation method was chosen in this study to evaluate the Combinator.

4.2 Process-based evaluation

Compared with an outcome-based approach, a process-based approach is time-consuming and difficult, because observations of the occurrence of creative cognitive processes are required while designers are producing ideas. In addition, there are no generally agreed protocol studies that can be used to collect data for idea generation process evaluation (Shah et al. Reference Shah, Smith and Vargas-Hernandez2003). Simple cognitive models exist, such as the Geneplore Model (Finke, Ward & Smith Reference Finke, Ward and Smith1996), and the Roadmap Theory (Smith Reference Smith, Hedley, Antonacci and Rabinowitz1995), which can be used for analysing data in simple tasks. These models could not be applied to ‘complex’ problems, for example, challenges involving engineering knowledge.

However, a process-based evaluation can demonstrate how an outcome is achieved, for example, the study conducted by Fiedler et al. (Reference Fiedler, Knippertz, Woodward, Martin, Bellouin, Ross and Heinold2015). Observation is used as a process-based evaluation method in this study, as it can assist in understanding the actual uses of a new technology and detecting potential problems (Yin Reference Yin2013). Video recording and screen recording are used in the case study to assist the observation. To some extent, the observation result can demonstrate how and why designers produce ideas through using the Combinator.

Table 1. Basic participant information.

4.3 Design challenge and participants

A practical design challenge was introduced to evaluate the Combinator. Waste separation for recycling at homes and offices is necessary. Two or more dustbins are often used to achieve the goal, but this is often space consuming and messy. The design challenge was to design a new solution that can efficiently use space as well as provide waste disposal and recycling attributes. Customer needs, such as waste separation, space saving, easy of use, no unpleasant odours, and stylish appearance, were provided, which were regarded as design specifications in this design challenge.

Thirty-six individuals, who were familiar with ‘dustbins’, participated in this case study and the following interview. The participants were highly interested and intrinsically motivated to participate in this idea generation case study voluntarily. They have signed up with standard case-study protocols concerning use of data and giving permission for the use of HD video recording. They were also rewarded with two pieces of high-quality stationery after completing the challenge and the interview. The basic information of the participants is shown in Table 1. Six of the participants are considered as experienced designers having over three years of design experience. The other thirty participants are regarded as novice designers, as none of them has sufficient experience in either engineering design or industrial design. Twelve of the participants conducted the study by using the Combinator, another twelve participants were not using any tools, and the remaining twelve used Google Image. The participants were regarded as the Combinator participants, the non-tool participants, and the Google Image participants, respectively. In order to have a fair competition, the division of participants was based on the participants’ experience and background, and thereby constituted three categories possessing similar capabilities.

4.4 Evaluation of the Combinator

The thirty-six participants conducted the challenge separately, as a person’s performance may be influenced by the action and ideas of others while producing ideas in groups (Perttula & Sipilä Reference Perttula and Sipilä2007) in a quiet room without any interruptions. Each of them was provided with A3 paper and a pen for sketching and writing ideas. The participants were given the same amount of time to generate as many ideas as possible. The Combinator participants generated ideas under the assistance of using the Combinator and was trained to use the tool before starting the challenge. The non-tool participants came up with ideas based on their intuition and experience, while the Google Image participants produced ideas by using Google Image. During the task, all the participants were observed by a recorder and recorded with high-definition cameras. The observations were conducted silently in order to minimise the impacts on the participants. Interviews were conducted after each participant had accomplished the design challenge. The generated ideas from each participant were collected and mixed together before the evaluation in order to eliminate bias.

Sarkar & Chakrabarti (Reference Sarkar and Chakrabarti2011) indicated that identifying creativity of a product is challenging, as one cannot be knowledgeable enough about all products. Experienced designers are often used to evaluate creativity of ideas, concepts, and solutions in design companies. It was also pointed out that the experienced designers who are selected to judge creativity should come from similar domains. Therefore, two experienced design engineers having over three years’ experience in design engineering were selected to evaluate the ideas respectively. The ideas were evaluated under the same guidance and the same inter-rater agreement of scoring 8 to 10 for excellent, 5 to 8 for good, 3 to 5 for fair, 1 to 3 for poor. The final scores are the means of the scores graded by the two raters.

The four metrics, quantity, novelty, quality, and variety, were measured according to Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003). Quantity was measured by counting the total number of ideas generated by an individual. Variety was applied to the entire group of ideas produced by an individual, which was measured by counting the total number of idea categories. The ideas generated were grouped based on the different physical principles used to satisfy waste separation which made the ideas very different from one another. For instance, using a chest of drawers to store waste and using barrels to store waste for satisfying waste separation were grouped into two categories, and thereby indicate two idea varieties.

Novelty of an idea was assessed by scoring each key function 1 to 10 from ‘poor’ novelty to ‘excellent’ novelty while comparing with the existing conventional dustbins on the market. An image including a collection of these conventional bins for waste separation (a swing bin for general waste, a caddy bin for food waste, and two rectangular recycle bin for paper, plastic, and glass) was provided to assist comparison. These bins are commonly used in people’s daily lives, and were considered as not novel in this study. Waste separation and space saving were considered as the two key functions of a new dustbin idea. These two functions were applied with the same weight 0.5 (total weight 1), as they are equally significant to a new dustbin design. The coefficients of the weights of novelty were decided through consulting a professional design expert who has over 10 years’ practical experience in consumer product design. A higher novelty score would be given to an idea’s ‘waste separation’ or ‘space saving’ function, if the function was more unusual or unexpected in comparison with non-novel ideas. In other words, ideas with higher novelty scores have less overlapping elements with non-novel ideas. The overall novelty of each idea was calculated by using Eq. (1) (adapted from Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003)). $M_{1}$ is the overall novelty score for an idea with $n$ functions. In (1), $f_{i}$ is the weight assigned to the function $i$ , while $S_{i}$ is the score of function  $i$ . The novelty score of an individual participant was the mean novelty score of all the ideas generated.

(1) $$\begin{eqnarray}M_{1}=\mathop{\sum }_{i=1}^{n}f_{i}S_{i}.\end{eqnarray}$$

Quality of an idea was evaluated by scoring each key attribute 1 to 10 from ‘poor’ quality to ‘excellent’ quality considering the idea’s feasibility and to what extent each attribute meets the design specifications. A total weight of 1 was assigned to 6 key attributes according to their importance to the idea as follows, feasibility (0.25), waste separation (0.25), save space (0.25), easy to use (0.1), no odours (0.1), and stylish appearance (0.05). The coefficients of the weights of quality were decided through consulting a professional design expert considering that feasibility, waste separation, and save space were more important than the other design specifications for an initial idea. A higher quality score would be given to an idea, if the idea’s attributes could meet the design specifications in better degrees respectively. In other words, an idea with a higher quality score has a higher feasibility and usability. The overall quality of each idea was computed with Eq. (2) which was adapted from Shah et al. (Reference Shah, Smith and Vargas-Hernandez2003). $M_{2}$ is the overall quality score for an idea including $m$ attributes. $S_{\!j}$ is the score of attribute  $j$ , while weights ( $f_{\!j}$ ) were assigned to each key attribute. The quality score of an individual participant was the mean quality score of all the ideas generated.

(2) $$\begin{eqnarray}M_{2}=\mathop{\sum }_{j=1}^{m}f_{\!j}S_{\!j}.\end{eqnarray}$$

The individual level mean scores of quantity, novelty, quality, and variety of each participant categories were calculated respectively for comparison. It is more effective to compare the scores at the individual level than the totals scores of a category, due to the different amount of ideas produced by each participant.

However, in this evaluation, new functions or attributes, other than the ones stated for evaluation, which might benefit the product, were not considered. This is due to only key attributes or functions are used for evaluation according to Shah et al.’s (Reference Shah, Smith and Vargas-Hernandez2003) psychometric method. Moreover, in initial design phases, it is significant to assess whether a product’s key attributes can satisfy the design requirements. In this study, subjectivity and potential biases might originate during the evaluation, due to issues such as subjective judgements caused by an evaluator’s understanding and perception of the attributes, inconsistent evaluation standards caused by insufficient time as well as inconsecutive evaluation processes, and misunderstanding of ideas caused by vague idea demonstrations and descriptions. Therefore, two experts were introduced to evaluate the Combinator, in order to minimise evaluation biases.

4.5 Results of evaluation

4.5.2 Statistical analysis

Based on the psychometric evaluation method and the equations illustrated above, the mean scores of quantity, quality, and novelty at the individual level of each category were calculated and presented in Figure 9. The Combinator participants generated 4.42 ideas at the individual level, while the non-tool participants generated 2.17 ideas and the Google Image participants produced 1.75 ideas at the individual level. The mean novelty score of the Combinator participants was 6.78 which is 0.43 and 0.82 higher than that of the non-tool participants (6.35) and the Google Image participants (5.96) at the individual level, respectively. The non-tool participants and the Google Image participants achieved 5.87 and 5.83 respectively on quality at the individual level, while the Combinator participants scored a 6.67. 3.42 idea categories were demonstrated at the individual level among the Combinator participants, whilst 1.67 varieties were shown by the non-tool participants and 1.50 varieties were presented by the Google Image participants at the individual level.

Figure 9. Psychometric evaluation results.

The standard deviation was introduced to quantify the dispersion of the data values. The standard deviation of the number of ideas (quantity) generated by each participant from the Combinator participants was 1.97, while the standard deviations of the non-tool participants and the Google Image participants were 1.34 and 0.97, respectively. The standard deviation of novelty scores and quality scores at the individual level of the Combinator participants, the non-tool participants, and the Google Image participants were 0.74, 0.71, 0.96, and 0.52, 0.92, 0.76, respectively. The individual level variety scores of the Combinator participants, the non-tool participants, and the Google Image participants were 1.38, 0.67, and 0.98, respectively.

Although the means are different, this might occur by chance. Therefore, statistical analysis is required to determine whether there are statistically significant differences between the means. The statistical analysis is conducted by using IBM SPSS Statistics software and the significance levels of the statistical tests are set as 5% ( $\unicode[STIX]{x1D6FC}=0.05$ ) as a convention. A Shapiro–Wilk test is conducted to analyse whether the data of each metric of the three participant groups are normally distributed. In the test result, if a $p$ -value is greater than 0.05, it indicates that the corresponding data obey normal distribution. As shown in Table 2, the Shapiro–Wilk test result shows that all the data of the Combinator participants are normally distributed, while only the novelty and quality values of the non-tool participants and the Google Image participants are normally distributed, respectively.

Table 2. Shapiro–Wilk test result of data normal distribution.

Table 3. Independent sample T-test result of ‘Novelty’ and ‘Quality’.

An independent sample T-test is conducted to determine whether there are significant differences between the means of novelty and quality, which requires the assumption of normal distributions. During the T-test, a Levene’s test is conducted to assess homogeneity of variances for selecting the proper $p$ -values produced by SPSS. In the test result, if a $p$ -value is less than or equal to 0.05, then there is a significant difference between the means of two conditions. The results of the T-test are shown in Table 3. Comparing the Combinator participants with the non-tool participants, the $p$ -value of the mean quality value is less than 0.05, while the $p$ -value of the mean novelty score is 0.162 which is greater than 0.05. Therefore, there is a significant difference between the mean quality scores and a non-significant difference between the means of the novelty scores. With regards to the Combinator participants and the Google Image participants, the $p$ -values of the two metrics are all less than 0.05. This determines that there are significant differences between the means of novelty and quality, respectively.

A Mann–Whitney U test is conducted to determine whether there are significant differences between the means of quantity and variety. The Mann–Whitney U test is a nonparametric test, which does not require the assumption of normal distributions, used for assessing significant differences. Similar to the independent T-test, there is a significant difference between the means of two conditions, if the $p$ -value is less than or equal to 0.05. As shown in Table 4, all the $p$ -values are less than 0.05. This indicates that there are significant differences between the means of quantity and variety respectively, while comparing the Combinator participants with the non-tool participants and with the Google Image participants.

Table 4. Mann–Whitney U test result of ‘Quantity’ and ‘Variety’.

The effect sizes (Cohen’s d) are calculated to define the differences between the means to support the statistical analysis. The independent T-test has indicated a non-significant difference between the mean novelty scores of the Combinator participants and the non-tool participants. As shown in Table 5, comparing the Combinator participants and the non-tool participants, there is a medium effect of the non-significant difference between the mean novelty scores. However, there are large effect sizes of significant differences between the mean scores of the other three metrics. In terms of the Combinator participants and the Google Image participants, there are large differences between the means of all four metrics.

Table 5. Effect sizes (Cohen’s d) between the metric of different participants.

We have also calculated the robust (bootstrap) confidence intervals, as presented in Table 6, by using 95% confidence interval of means and 1000 samples. It provides robust confidence intervals for estimates based on one thousand samples. The result shows that 95% of the Combinator participants could score 3.21 to 5.50 in quantity, 6.34 to 7.15 in novelty, 6.39 to 6.95 in quality, and 2.58 to 4.12 in variety. The performances of the Combinator participants are better than the other two types of participants based on an estimated 1000 samples data size, according to the calculation result.

Table 6. Robust confidence intervals by using 95% confidence interval of means.

According to the statistical analysis above, in terms of comparing the twelve participants using the Combinator with the twelve participants without using tools concerned in the case study, there are significant increases in quantity, quality, and variety at individual levels. However, there is a non-significant improvement in novelty. Comparing with the twelve participants using Google Image at the individual level, there are large improvements in all four aspects. This indicates that, concerning the conducted design challenge, the Combinator had improved the users’ fluency in idea generation as well as enhanced the originality, usefulness, and flexibility of the ideas produced in comparison with the non-Combinator users.

4.5.3 High-novelty and high-quality ideas analysis

In addition, the number of high-novelty and high-quality ideas are also significant criteria, as generally only high-novelty and high-quality ideas will be accepted and developed into final products. In this case study, an idea is considered as a high-novelty idea or a high-quality idea while its novelty score or quality score is greater or equal to 7, respectively. As shown in Figure 10, the twelve Combinator participants generated 28 high-novelty ideas out of 53 ideas in total, while the twelve non-tool participants only produced 9 out of 26 and the twelve Google Image participants generated 6 out of 21. In terms of high-quality ideas, the twelve non-tool participants and the twelve Google Image participants came up with 3 and 1, respectively, while the twelve Combinator participants generated 19 in total. Thus, 53% of the ideas generated by the Combinator participants are high-novelty ideas, while only 35% and 29% of the ideas generated by the non-tool participants and the Google Image participants are high in novelty, respectively. The Combinator participants generated 36% high-quality ideas, which is about three times of that of the non-tool participants and seven times of that of the Google Image participants.

Figure 10. High-novelty and high-quality ideas.

4.6 Observations and interviews

Observations and video recordings were introduced as a process-based evaluation method. Although this method cannot provide very convincing results, to some extent, it could be used to understand how ideas were produced by different groups of participants during the idea generation sessions. According to observations and video recordings, the recorder would need to record the time a participant spends on sketching and writing ideas, as well as the time a participant spends on thinking (not producing ideas). The recorder would also need to record the reactions of the Combinator participants and the Google Image participants, such as starting to produce ideas, while corresponding types of images are shown to the participants. The results have shown that the participants who were using the Combinator spent less time than the other participants on the thinking process. This could be that non-Combinator users had to retrieve relevant knowledge to produce a creative idea, which could be challenging as well as time-consuming, especially for novice designers. In addition, the Combinator users could always come up with new ideas after being stimulated by the ideas produced by the tool. This implies that ideas or stimuli generated by the Combinator could be used for further idea derivation. According to Howard et al. (Reference Howard, Culley and Dekoninck2011), a stimulus does not directly inspire new ideas, but divert designers into new design spaces to enable new ideas. Thus, the process-based evaluation has revealed that it was the ideas generated by the Combinator that assisted users in generating creative ideas. This is how the Combinator functioned during the idea generation challenge and the reason why the Combinator users generated more creative ideas than the others.

Interviews were conducted after each participant had accomplished the design challenge. The participants were asked to evaluate their ideas, user experience, and creativity levels during the study, which was called participant evaluation. Scatter charts were employed to demonstrate the individual evaluation results. Please note that the lines in the charts do not represent data variations or trends, the lines are only used for better presentations. In the charts, the vertical axis represents the evaluation scores, while the horizontal axis represents the participant ID from 1 to 12 of the Combinator participants, the Google Image participants, and the non-tool participants, respectively.

Figure 11. Participants evaluation of outcomes: the Combinator VS. Google Image.

Figure 12. Participants evaluation of user experience: the Combinator VS. Google Image.

The Combinator users as well as the Google Image users were asked to score the ideas produced by themselves, from 1 to 10 (from ‘poor’ to ‘good’) in terms of quality, novelty, and variety. As shown in Figure 11, in general, the ideas produced by the using Combinator have higher scores in all three aspects comparing with the ideas produced by using the Google Image. This indicates that the users were satisfied with the outcomes of the Combinator. These participants were also asked to evaluate their user experience of the tools in four aspects: usefulness, easiness, comfort, and enjoyment. According to the user experience evaluation, as shown in Figure 12, the Combinator had provided a superior user experience and been considered as a useful, easy, comfortable, and enjoyable tool. In terms of Google Image, it had been shown as an easy and comfortable tool, but unenjoyable and less useful. All the participants were asked to grade themselves from 1 to 10 to describe how creative they feel during the idea generation sessions. As shown in Figure 13, overall, the Combinator participants had graded themselves with higher creativity level scores comparing with the others. This implied that the Combinator had positively influenced the creative thinking of the participants during idea generation.

Figure 13. Participants evaluation of creativity level: the Combinator VS. Google Image VS. Non-Tool.

All the twelve Combinator participants provided highly positive feedback considering the Combinator as a very useful tool for idea generation. However, some participants using the Combinator pointed out that the qualities of several images produced from the Combinator were poor, which interfered with their thinking. Several non-design background participants indicated that the Combinator generated many high-quality and useful ideas, but it was difficult for them to translate these ideas into creative outcomes. Most of the Google Image users considered Google Image as a useless tool, as the images provided by the tool were monotonous. In terms of the non-tool participants, some thought that the design task had a certain degree of difficulty, and thus it was challenging for them to come up with a number of creative ideas. Most of the non-tool participants acknowledged that they need some stimuli, especially visual stimuli, to help them in idea generation.