Introduction
Optimizing the way information is presented to engineering designers can substantially enhance their ability and their capacity to use that information in their design activities. Designers frequently need to acquire information from nonhuman sources (Robinson, Reference Robinson2010), such as drawings, written reports, physical models, and presentations (Wodehouse and Ion, Reference Wodehouse and Ion2010). To utilize information in new scenarios, the designer must first comprehend the information and integrate it with their prior knowledge. This can be challenging because sources of engineering information can be incomplete or inconsistently formatted (Filieri and Alguezaui, Reference Filieri and Alguezaui2015), and the volume of information that engineers need to process can lead to information overload (Anette Wulff et al., Reference Anette Wulff, Rasmussen and Westgaard2000). Moreover, as they use these sources for a wide range of different design activities, it cannot be determined in advance which parts of an information source will be relevant for a given task. Fortunately, representing the same underlying content in different ways can enhance the performance of design activities. This has been demonstrated in inspiration-aided ideation (Vasconcelos and Crilly, Reference Vasconcelos and Crilly2016), idea evaluation (Kwon and Kudrowitz, Reference Kwon and Kudrowitz2023), creativity assessment (Kudrowitz et al., Reference Kudrowitz, Te and Wallace2012), design reviews (Lukačević et al., Reference Lukačević, Škec, Perišić, Horvat and Štorga2020), and learning (Hegarty, Reference Hegarty2014). This phenomenon could be leveraged by using simplification tools (Siddharthan, Reference Siddharthan2014; Al-Thanyyan and Azmi, Reference Al-Thanyyan and Azmi2021; El-Kassas et al., Reference El-Kassas, Salama, Rafea and Mohamed2021) to present information to designers in the optimal format at the right time. However, to evaluate the effectiveness of such tools, there is a need to (i) establish how a designer processes and understands information, (ii) determine what makes for “good” information presentation, and (iii) measure the designer’s performance. The motivation of the present research was to address these needs.
One particularly valuable source of information is the worldwide patent database. Comprising over 130 million documents (EPO.org, n.d.), the patent database is one of the largest, freely available repositories of technical knowledge in the world, containing information that is often not disclosed elsewhere (Asche, Reference Asche2017; Golzio, Reference Golzio2012). By reading patent documents, designers can gain detailed knowledge about the problems being addressed by patents (Souili et al., Reference Souili, Cavallucci and Rousselot2015), functions, behaviors, and structures of artifacts (Russo, Reference Russo2012; Fantoni et al., Reference Fantoni, Apreda, Orletta and Monge2013) and affordances (Chiarello et al., Reference Chiarello, Cirri, Melluso, Fantoni, Bonaccorsi and Pavanello2019). Such information can enhance design activities throughout the design process, including market research, problem formulation, detailed design, and testing (Vasantha et al., Reference Vasantha, Corney, Maclachlan and Wodehouse2017). Yet, despite their potential utility, patent documents are regarded as difficult to read compared to other documents (Verberne et al., Reference Verberne, D’hondt, Oostdijk and Koster2010; Brügmann et al., Reference Brügmann, Bouayad-Agha, Burga, Carrascosa, Ciaramella, Ciaramella, Codina-Filba, Escorsa, Judea, Mille, Müller, Saggion, Ziering, Schütze and Wanner2015; Suominen et al., Reference Suominen, Ferraro, Nualart Vilaplana and Hanlen2018; Casola and Lavelli, Reference Casola and Lavelli2022). This has been attributed to the linguistic characteristics of patents, such as long sentences and the use of novel terms (Mille and Wanner, Reference Mille and Wanner2007; Verberne et al., Reference Verberne, D’hondt, Oostdijk and Koster2010), complex syntactic structure (Verberne et al., Reference Verberne, D’hondt, Oostdijk and Koster2010), and complex linguistic style (Brügmann et al., Reference Brügmann, Bouayad-Agha, Burga, Carrascosa, Ciaramella, Ciaramella, Codina-Filba, Escorsa, Judea, Mille, Müller, Saggion, Ziering, Schütze and Wanner2015).
Computational methods can be used to make sources such as patent documents easier to read, without removing or altering the semantic content of the information (Casola and Lavelli, Reference Casola and Lavelli2022). Unlike summarization tools, simplification tools reduce the complexity of information without altering the content (Casola and Lavelli, Reference Casola and Lavelli2022). This is particularly valuable in the context of patent documents because the wording is legally significant. Notably, these methods attempt to reduce complexity not by focusing on the challenges stemming from the linguistic properties of patents but by manipulating the visual properties of the information. These are presenting images and text side-by-side (e.g., Google’s patent viewer (Google, 2023)), highlighting key words (Okamoto et al., Reference Okamoto, Shan and Orihara2017), annotating a claim with additional text from a description (Shinmori and Okumura, Reference Shinmori and Okumura2004), and restructuring patent claims to highlight different aspects of their hierarchical structure (Andersson et al., Reference Andersson, Lupu and Hanbury2013; Ferraro et al., Reference Ferraro, Suominen and Nualart2014; Sheremetyeva, Reference Sheremetyeva2015). This implies that there is a lack of correspondence between the claimed problems with patent documents and the methods that can be used to simplify patent document complexity. Unfortunately, all of these tools are limited in two ways. First, none appear to be guided or informed by any theoretical or empirical evidence about how to effectively present information to improve comprehension performance. Only one survey-based user study was guided by theories of information design (Suominen et al., Reference Suominen, Ferraro, Nualart Vilaplana and Hanlen2018), and it is not clear how the theories led to the selection of interventions. Second, these tools are evaluated by using subjective user ratings (Mille and Wanner, Reference Mille and Wanner2008; Codina-Filbà et al., Reference Codina-Filbà, Bouayad-Agha, Burga, Casamayor, Mille, Müller, Saggion and Wanner2017; Suominen et al., Reference Suominen, Ferraro, Nualart Vilaplana and Hanlen2018), or quantitative lexical methods (see Casola and Lavelli (Reference Casola and Lavelli2022) for an overview), but none can demonstrate whether a given intervention has improved the users’ understanding of the information.
In this work, we present an approach for improving the comprehensibility of technical information in engineering design, without changing the semantic content of the content (“An Approach for Enhancing and Measuring Information Comprehensibility in Engineering Design” section). It involves remodeling information sources with theoretically and empirically validated information design principles and measuring the change in comprehension of the original and modified sources. The approach distills a substantial body of research on document search, question answering, and learning. It is uniquely tailored for engineering design applications in that it is suitable for long, complex documents, evaluated in experiments that simulate real-life information-sharing scenarios, where the goal is to measure comprehension of domain-specific engineering design knowledge. To evaluate the approach, we test the hypothesis that information design principles could be used to remodel patent documents to make them easier to comprehend. This is done via an experiment with 28 professional engineering designers reading and answering questions about two patent documents (“Applying the Approach to Enhance and Measure the Comprehensibility of Patent Documents" section). Our assumption was that the approach would substantively enhance the comprehensibility of any patent documents used as stimuli. However, the results of the experiment were not statistically significant (“Results” section). We discuss the reasons for this, attributing it to a combination of sample size limitations with unexpected moderators (“Discussion” section). Nonetheless, the approach provides a valuable resource for interdisciplinary knowledge transfer from cognitive psychology to engineering design and provides the first experimental method for evaluating the comprehensibility of patent documents or complex visual and textual technical information. Our approach is further applicable in the evaluation of computational simplification tools that have thus far relied on more subjective judgments or lexical measures of usability.
An approach for enhancing and measuring information comprehensibility in engineering design
We present a generic approach for enhancing and measuring the comprehensibility of information sources for engineering designers. It is applicable for long sources of multimedia information comprising text and images, where the user has to read through multiple pages of information to find and comprehend information. Figure 1 shows the conceptual elements of the approach, which have been distilled from a substantial body of research in the cognitive and learning sciences (for relevant compilations of theoretical and empirical research, see Mayer Reference Mayer2014b; Jean-Francois Rouet Reference Rouet2006a; Van Meter et al. Reference Van Meter, List, Lombardi and Kendeou2020).
-
I. Establish the constitution of the information source
-
II. Understand the cognitive processes of document search (searching through a document to find information or answer questions, “Document Search” section) and comprehension (forming a mental model of the information and integrating it with prior knowledge, “Comprehension” section).
-
III. Identify relevant information design principles to improve information source – information design principles are prescriptive guidelines for how to present information to influence cognitive processes.
-
IV. Remodel the information source – this can be conducted manually or using computational methods.
-
V. Conduct an experiment to measure performance – document search and comprehension can be elicited with specific questions and performance can be scored with a mixture of scoring rubrics and coding schemes
Figure 1. The cognitive approach used to remodel patent documents.
In this section, we elaborate on steps II–V that represent the major theoretical and methodological aspects of the approach.
The cognitive processes involved in reading technical documents
The use of technical documents by designers can be understood in terms of two cognitive processes: search and comprehension. Search refers to document search, that is, looking through a document to explore its contents or find relevant information (Jean-François Rouet, Reference Rouet2006b; Rouet and Britt, Reference Rouet and Britt2011). Comprehension refers to the understanding of the content of that document and the building of a mental model of the content of the information (Schnotz, Reference Schnotz2005; Mayer, Reference Mayer2014b). Document search is relevant because search and comprehension processes are not independent (Jean-François Rouet, Reference Rouet2006b; Cataldo and Oakhill, Reference Cataldo and Oakhill2000; McTeague and Chatzimichali, Reference McTeague and Chatzimichali2022). A person’s comprehension performance can be influenced by their search strategy (Rouet et al., Reference Rouet, Vidal-Abarca, Erboul and Millogo2001, Exp. 1).
Document search
Document search refers to the activity of reading through text- and image-based media to find relevant information that fits a preexisting knowledge need (Jean-François Rouet, Reference Rouet2006b; Rouet and Britt, Reference Rouet and Britt2011). According to the Task-Based Relevance Assessment and Context Extraction model, this knowledge need is then satisfied by setting a search goal, searching through memory and external sources for information that satisfies that goal, and iteratively evaluating the relevance of the material being processed (Jean-François Rouet, Reference Rouet2006b). Reading to find specific information may not be linear; readers can adopt various search strategies to navigate through the document (Cerdán et al., Reference Cerdán, Vidal-Abarca, Martínez, Gilabert and Gil2009).
Performance during document search is influenced by a variety of factors, relating both to the type of information that is required (the search goal) and the properties of the text in which the answer is located (Mosenthal, Reference Mosenthal1996). These properties include the correctness of the required information, the degree of integration and inference required to satisfy the search goal, the complexity of the document, and the presence of information that is similar to, but not relevant to the search goal.
Comprehension
Comprehension is the formation of a mental representation of the referent (the thing being represented) in the mind of the reader, and the integration of that referent with the reader’s prior knowledge (Kintsch, Reference Kintsch1991; Schnotz, Reference Schnotz2005; van den Broek and Helder, Reference van den Broek and Helder2017). Multimedia comprehension refers to the processing of multiple information modalities, such as the texts and images found in patent documents. Cognitively, multimedia comprehension can be understood in terms of models of text comprehension (Kintsch, Reference Kintsch1998), multimedia comprehension (Schnotz, Reference Schnotz2005; Hegarty, Reference Hegarty2014), and learning (Mayer, Reference Mayer2014b). The three facets of comprehension are key for understanding document use by engineering designers. Document processing places demands on working memory that can lead to effective and ineffective performance, people can comprehend information at multiple levels of depth, and the rhetorical structure of documents can be understood in terms of scope (micro- and macro-structure).
Multimedia comprehension can be improved by presenting information in a way that guides the reader’s cognitive processing without overloading their working memory (Mayer, Reference Mayer2014b p. 59). Comprehension involves active processing through multiple processing streams in a system with limited working memory capacity. The demands on this cognitive system can be described in terms of cognitive load theory (Paas and Sweller, Reference Paas and Sweller2005; Sweller, Reference Sweller2011; Sweller et al., Reference Sweller, Ayres and Kalyuga2011, Reference Sweller, Van Merrienboer and Paas1998), using a scheme adopted from learning research (Mayer, Reference Mayer2014b, table 3.6). Effectively designed information should limit extraneous cognitive load (in which processing occurs that is unrelated to the comprehension goal) and support intrinsic and germane cognitive load (where information is being effectively comprehended) but without overloading working memory. This can be achieved using information design principles (“Measuring Comprehension Performance” section).
The distinction between essential and generative processing reflects comprehension at multiple levels of depth. According to the construction-integration model of comprehension (Kintsch, Reference Kintsch1988), comprehension can be understood at three levels.
-
• Surface-level comprehension (The “surface model” (Van Dijk and Kintsch, Reference Van Dijk and Kintsch1983; Kintsch, Reference Kintsch1992)) involves a basic recognition and understanding of the words and phrases in a text, as well as the linguistic relations between them. For images, this involves the shallow recognition of the form being depicted by lines on a page.
-
• Textbase comprehension. At this level, readers associate words and images with conceptual knowledge and draw meaningful relations between concepts. This level concerns the semantic and rhetorical structure of the information. It is analogous to essential processing (Table 1) that forms representations in working memory (Mayer, Reference Mayer2014b).
-
• At the deepest level of comprehension, the reader forms a situation model. This is a deep understanding of the information, where propositions from the textbase are integrated with prior knowledge. This is analogous to generative processing (Table 1) which involves integrating information with prior knowledge (Mayer, Reference Mayer2014b).
Table 1. Three demands on cognitive capacity during multimedia comprehension – adapted from Mayer (Reference Mayer2014b), table 3.6
At the textbase level, a distinction can be made between micro- and macrostructures (Kintsch and Van Dijk, Reference Kintsch and Van Dijk1978; Kintsch, Reference Kintsch1998). The former refers to the individual statements and propositions within a text and the latter refers to the global organization of the information into a hierarchy of points.
Information design principles for improving comprehension
To improve a designer’s ability to search through and comprehend documents, the goal is to guide them through the document effectively and support comprehension without overloading their working memory capacity. Broadly, this can be achieved through training, prompting or changes to the information source (Van Meter and Stepanik, Reference Van Meter and Stepanik2020). This last class of intervention is what we term “information design principles” or what Van Meter and Stepanik (Reference Van Meter and Stepanik2020) refer to as “materials-driven interventions.” These are manipulations of the “surface features” of text and images, that is, the manner in which information is presented. This can be contrasted with the “deep” structure or underlying semantic content that conveys meaning. We have created a bank of information design principles that are relevant for improving comprehension of text and images (Table 2). To improve text search and comprehension processes, we adopt research on text organizers (Lorch, Reference Lorch1989; Jean-Francois Rouet, Reference Rouet2006c) and to improve multimedia comprehension, we use multimedia design principles (Mayer, Reference Mayer2017, Reference Mayer2014b).
Table 2. Information design principles for improving comprehension
Text organizers are a class of information design principles that emphasize selective elements of the content or structure of a text without adding to or changing the content of the text (Lorch, Reference Lorch1989). Organizers can have a mixture of visual and verbal properties (Jean-Francois Rouet, Reference Rouet2006c).Footnote 1 Visual organizers signal information to the reader via the depictive properties of information, such as the typographic formatting and position on the page. Verbal organizers signal through verbal content that is superfluous to the main content of the text. Examples include overviews (such as the abstract of a research article) or function indicators (“it should be emphasized that…”). Some signals, such as section headings, have both visual and verbal properties. Document search can be improved by the use of headings (Hartley and Trueman, Reference Hartley and Trueman1985). Different text organizers influence different cognitive processes (Lorch, Reference Lorch1989). Document search can be improved by the use of headings (Hartley and Trueman, Reference Hartley and Trueman1985) and the use of typographical cues such as segmentation and indentation (Frase and Schwartz, Reference Frase and Schwartz1979). Text comprehension can be improved via the use of titles (Kozminsky, Reference Kozminsky1977; Daniel and Raney, Reference Daniel and Raney2007) and logical connectives*** (Sanders et al., Reference Sanders, Land and Mulder2007), and may be influenced by text segmentation (Lemarié et al., Reference Lemarié, Eyrolle and Cellier2008).
Multimedia principles are prescriptive guidelines for improving the ability of people to learn from or comprehend multimedia representations. These principles work by reducing extraneous cognitive load and facilitating effective cognitive processing (“Comprehension” section). Mayer (Reference Mayer2014b) provides a list of 15 multimedia principles (for other general reviews, see Mayer, Reference Mayer and Mayer2014a; Van Meter et al., Reference Van Meter, List, Lombardi and Kendeou2020). We propose that four of these principles are applicable to the modification of text- and image-based documents without modifying the wording of the content (see: Multimedia (text and image) principles (Table 2)). For example, extraneous processing can be reduced by placing relevant text and images beside one another, thereby reducing the time spent scanning back and forward (Ayres and Sweller, Reference Ayres and Sweller2005; Schroeder and Cenkci, Reference Schroeder and Cenkci2018). Essential processing can be managed by using pictures to supplement text-only content (Butcher, Reference Butcher2014), or by segmenting information into smaller chunks (Mayer and Pilegard, Reference Mayer, Pilegard and Mayer2014).
Measuring comprehension performance
Comprehension can be measured by assessing the extent to which the reader can recall information about a source and can transfer that knowledge to new scenarios that are not explicitly addressed in the source. As document search and comprehension are intertwined, it is first necessary to elicit document search and comprehension processes. This can be done through the use of test questions that engage the desired cognitive processes. To facilitate the elicitation and measurement of document search and comprehension, we established a taxonomy of question types (Table 3) by adapting an existing framework (Rickards, Reference Rickards1979) so that it can be used to target specific levels of comprehension (see the three levels in “Comprehension” section). The taxonomy has three parameters.
Forward and backward questions – forward questions are given to participants before reading the text. These can be used to guide the participant toward specific information in a long document. Backward (surprise) questions are given to participants after they have read a text. They can be used to measure the extent of comprehension.
Table 3. A classification of question types for eliciting search and comprehension processing
Note: HLQ, high-level question; LLQ, low-level question.
Degree of integration – this parameter is operationalized in two question types (Rouet et al., Reference Rouet, Vidal-Abarca, Erboul and Millogo2001). Low-level questions (LLQs) ask about specific pieces of information such as words or phrases. High-level questions (HLQs) ask about more general information that requires integration from disparate locations in the information source. HLQs facilitate deeper comprehension than LLQs (Rouet et al., Reference Rouet, Vidal-Abarca, Erboul and Millogo2001; Jean-François Rouet, Reference Rouet2006b), and so they collectively provide more granular control of comprehension elicitation.
Depth of comprehension – refers to the three levels of comprehension introduced in “Comprehension” section. As comprehension is a multilevel construct, multiple question types are required to measure it. Comprehension at the surface or textbase level can be measured using recognition or recall tests (Butcher, Reference Butcher2006). Recognition asks participants to identify words or images that were present in a text while ignoring distractors. Recall engages a deeper form of comprehension and asks participants to recall propositions conveyed by the media. Comprehension at the level of the situation model can be elicited by transfer tasks that task participants with applying knowledge to novel situations that were not explicitly stated in the initial material (Mayer, Reference Mayer2014c).
The taxonomy provides two main benefits. It enables researchers to selectively engage processes to reflect the real-world engineering design behaviors that they wish to study (see the implementation in “Questions and Scoring” and “Procedure” sections. It also allows for some homogenization of search processes by guiding participants toward specific information in the text.
Applying the approach to enhance and measure the comprehensibility of patent documents
The approach introduced in the previous section was applied to enhance and measure the comprehensibility of patent documents for professional engineering designers.
The constitution of a patent document
A patent right is a form of intellectual property that grants certain rights to the patent owner. Meanwhile, a granted patent document serves to define the subject of protection and typically consists of six key parts (WIPO, 2023, p. 33).
-
I. Abstract (and title) – provides a high-level summary of the patent.
-
II. Background – discloses the closest prior art considered during the examination and may include short statements about the limitations of prior art without disclosing the solution described later in the document.
-
III. Summary – an overview that should be sufficient for allowing a person skilled in the art to understand the invention.
-
IV. Detailed description – also known as the “preferred embodiment of invention” section or the “disclosed embodiment of the invention.” This section provides “a sufficient explanation of the invention for an ordinary person skilled in the art to make and understand the invention” (WIPO, 2023, p. 35).
-
V. Drawings – two- or three-dimensional images representing embodiments of the invention. The drawings help to understand the content of the claims.
-
VI. Claims – the coverage of the protections granted by the patent. Patent documents usually contain more than one claim. Each claim is represented by a single sentence and multiple claims are linked in hierarchical structures.
Within patent documents, a distinction may be made between the content of a patent document and how that content is presented, that is, its surface features. Content refers to the specific words, sentences and images in a patent document that represent the invention and the legal scope of the patent. The presentation of that content refers to its spatial arrangement on the page, the typographical formatting used, and the presence or absence of superfluous language that helps the reader understand the content. Patent documents always comprise text but often comprise images.
Hypothesis and study overview
In this study, we assess the feasibility of our approach for enhancing the comprehensibility of patent documents. The following hypothesis was tested:
-
• (Ha) after applying the approach outlined in “An Approach for Enhancing and Measuring Information Comprehensibility in Engineering Design”, patent documents remodeled with information design principles will be more comprehensible than unmodified patent documents. The null hypothesis (H0) is that there is no difference in comprehension scores.
Based on this hypothesis, we predicted that the comprehension scores for the modified patent documents would be statistically significantly higher than those for the control patent documents. We assumed that the modifications to the patents would result in improvements to both patent documents. An experiment was designed and conducted to compare the comprehensibility of unmodified patent documents (control) versus patents modified with information design principles. The context was a design information-sharing scenario in which a designer was given the goal of reading a patent application filed by a competitor and reporting about it in a team meeting. Each designer read two patent documents, one in its original form and one modified. They were asked to gain a general understanding of the invention with a focus on the preferred embodiment (defined in “The Constitution of a Patent Document” section) and to be prepared to answer surprise questions about it. Figure 2 illustrates the key conceptual elements of the study. The main outcome measure was comprehension; however, since search involves comprehension (Jean-François Rouet, Reference Rouet2006b), any improvement in comprehension performance may be a direct improvement of comprehensibility or an indirect facilitation from improved searchability.
Figure 2. The key conceptual elements of the experiment.
Design and analysis
A counterbalanced experiment design with crossed participants and stimuli (Judd et al., Reference Judd, Westfall and Kenny2016) was used to compare the comprehension performance of professional engineering designers when reading patent documents that had been remodeled with the information design principles (“modified”) versus the original patent documents that are publicly available in a patent search database (“control”). This design was chosen to maximize the experimental power from a limited sample of participants versus, for example, a between-groups randomized control trial. The participants (n = 28) were given four forward questions that guided them through the document and then seven surprise (backward) questions that tested their knowledge of the preferred embodiment. Collectively, these questions tested comprehension of the textbase and situation model (see Table 3). Comprehension performance was the sum of the scores for the textbase questions (based on a predefined coding scheme) and transfer questions (based on a novel, generic coding scheme for scoring open-ended responses).
The crossover design had two factors as shown in Table 4. The within-subjects factor was “condition”; whether the participant was given a patent in its original form (control, C) or its modified form (modified, M) using information design principles. The between-subjects factor was “patent”; which of two patent documents was being viewed, termed GB24 and GB25Footnote 2. This resulted in two stimulus configuration groups, where half of the participants received patent GB24 in the control condition and GB25 in the modified condition, and vice versa. The period in which the modified patent document was presented was also balanced across the sample, but the period of treatment was not included in the analysis and is not shown in the table for simplicity. Participants were allocated to configuration groups and modified via restricted randomized assignment (Bruce et al., Reference Bruce, Juszczak, Ogollah, Partlett and Montgomery2022).
Table 4. Experiment factors
Note: GB24 and GB25 designate the two patent documents.
The data were analyzed using a two-way mixed ANOVA to test (i) the working hypothesis (“Hypothesis and study overview” section), and the assumptions that (ii) group allocation would have no difference on comprehension scores, and (iii) the control versions of the patent documents would be equivalently comprehensible.
Participants
The participants (n = 28) were individuals with at least 1.5 years of professional experience (range 1.5–14 years, M = 6 years) and at least 1 year of experience in an engineering design role during that time. Participants came from a variety of Engineering Design domains, including product development, mechanical design, electrical engineering, and manufacturing. Professional engineering designers were selected instead of, for example, students to increase the likelihood that the participants would have the relevant domain knowledge to understand the technical concepts being communicated in the patent documents and so that they would have experience with reading technical documents and communicating them in professional collaborative settings.
A lower bound was placed on sample size via an a priori power analysis and an upper bound was determined by resource constraints. For the lower bound, a power calculation for a dependent samples t test was conducted using G Power 3 (Faul et al., Reference Faul, Erdfelder, Lang and Buchner2007) to estimate the number of participants required to detect a statistically significant effect of condition. In learning research, the three information design principles used in this study (Table 6) have median effect sizes in randomized control trials of d = 0.46 (signaling), 0.79 (spatial contiguity) and 1.67 (multimedia) (Mayer, Reference Mayer2017). We assumed that the information design principles would have cumulative positive effects and so we powered the experiment to detect large effects. A minimum of 15 participants was required to have an 80% chance of detecting a large effect size of d = 0.8. Recruitment continued to 28 participants and was stopped owing to resource limitations, providing an 80% chance of detecting an effect size of d = 0.55.
Stimuli
Four stimuli were used in the experiment (Figure 3, Table 5). Two patent documents were split into condition variants: control and modified. Both patent documents describe inventions that are vacuum cleaner attachments, filed by a large UK/Singaporean-based product design company. Patent “GB24” relates to a pet grooming device with hair removal functionality and patent “GB25” relates to a floor tool for a vacuum cleaner with improved maneuverability relative to prior products.
Figure 3. Images from the patent documents.
Table 5. Summary of the four stimuli and how they were created
Note: “Original” means the document as downloaded from Espacenet.
Selection of patent documents
A manual, iterative search was conducted through Espacenet (EPO.org, n.d.) to find two patent documents while satisfying four requirements (Rx).
-
• R1 – To control for extraneous variables, the patent documents should have maximally similar surface features (the volume, layout, and presentation of information). The documents have the same number of pages of text and figures, and the figures span the same number of pages. The content of the patent was in the same order (i.e., abstract, background, disclosure of the invention, claims) and the respective sections started on the same page or +/− one page relative to the other patent.
-
• R2 – To control for extraneous variables, the inventions and products being described by the patent documents should be as similar as possible. Both inventions were from the same EPO classification, from the same applicant, and were published after 2001. Patents were both mechanical, physical, and consumer-use products from the same patent classification code. Incidentally, the selected patents were European B1 patent applications, although this was not planned.
-
• R3 – To avoid interactions between patent content and prior knowledge, that is, where only some participants would be familiar with obscure inventions, the patents should be familiar to members of the general public (R4). Both patent documents both described vacuum cleaner attachments which were assumed to be common household items.
-
• R4 – To reduce carryover effects (reading one patent could confer performance benefits for the second patent), the patents should describe different inventions (R3). The patents described inventions with different working principles. One was about a mechanically actuated hair grooming brush for pets; the other was about a means of improving maneuverability for a wand-shaped floor tool.
Patent documents were selected to balance the needs of requirements 2 and 4, with the former benefiting from increased similarity and the latter benefitting from reduced similarity.
Creation of modified patent documents with information design principles
Two kinds of modifications were made to the original documents to create the experimental stimuli. To create the control stimuli, a minor modification was made to patent GB25 to make it more similar to GB24. This involved adding information about prior art found in search reports within the same patent family and ensuring both patent documents provide broader contextual information about the problem space. Patent GB24 was left unedited.
To create the modified stimuli, three text signals and three multimedia design principles were applied to each of the control patents (Table 6). The principles were selected by analyzing patent documents GB24 and GB25 against the principles in Table 2. Figure A1 in the Appendix shows patent GB24 in its original state, Figure A2 shows it in its modified state. Figure 4 shows an example page from the preferred embodiment of patent GB24 of the modified patent documents. All changes were made manually using Adobe InDesign.
Table 6. The information design principles applied to create the modified patent documents
Figure 4. An example of a page of the preferred embodiment in the modified version of patent GB24.
Materials
The experiment was carried out online and in the participant’s chosen setting using their computer. In all cases, this was either their home office or place of work. The study was conducted live with the experimenter via a video call. The presentation of instructions and stimuli took place on two screens. One “main screen” contained the patent documents to be read during the study and was required to be at least 13 inches long diagonally. The second screen contained the pre-study demographic questionnaire and the study instructions. This screen could be any size, such as a second monitor or a mobile phone.
The demographic questionnaire and study instructions were presented using the Qualtrics survey software. Participants were taken through the study by the experimenter who clarified the key instructions and read the task questions. Participants provided verbal responses to the comprehension questions, which were later transcribed.
Before beginning their first trial, the participants were briefed with a fictitious scenario to set the engineering design context that the experiment is designed to reflect.
You have recently been hired to an engineering start-up that is looking to disrupt the market for consumer and commercial air-flow products such as vacuum cleaners, air purifiers, hand dryers, fans, and heaters. After the success of a new vacuum cleaner, the management team have set the aim of developing a suite of attachments to improve the capabilities of the vacuum cleaner.
As part of your work duties, your team has been asked to conduct a technology and competitor review and create a report for management. Your specific task is to review and understand existing inventions in the patent database so that you can describe them to your team and answer questions about them at your next meeting.
Each stimulus (patent document) was provided to the participants in a. pdf document via an online file-sharing platform.
Questions and scoring
Questions
Participants were given a series of demographic questions before the study, and document search and comprehension were elicited using questions from the taxonomy in Table 3. The demographic questions were used to check for covariates. Participants were asked to record their university degree, current job role, and number of years in an engineering or design role in the industry. Their experience with patent use was captured in two measures. Patent familiarity was recorded on a five-point Likert-type scale, ranging from “not familiar at all” to “extremely familiar.” For recent frequency of patent use (hereafter “patent frequency”), participants were asked: “On average over the last 3 years, how frequently have you read patent documents?” Results spanned six options from “never” to “multiple times per week.” Both measures were based on the assumption that patent comprehensibility stems from poor information design and that familiarity with the format of patent documents could influence comprehension performance.
Comprehension was measured with a mixture of forward and backward (surprise) questions, eliciting document search, recall (HLQ and LLQ) and transfer, based on the classification of question types in Table 3. Table 7 lists the three types of questions that were used to elicit document search and comprehension: forward search, backward comprehension and transfer. The questions were created by the article authors and trialed and modified following a pilot study with six participants.
Table 7. A list of the questions that were given to the participants
Note: * Question 4 was used to guide the participants through the document and setup the backward questions, it was not scored directly
They were designed to elicit comprehension at two levels of comprehension (textbase and situation model) and of specific (LLQ) and disparate (HLQ) information (Table 3). The forward search questions were made the same for both patent documents so that all participants would have a chance to read the patent document before answering questions, thereby providing some control for differences in prior knowledge. The backward questions were used to measure their comprehension. The recall questions were matched in depth of comprehension, degree of information and the number of points available for a correct answer so as to control for question demands across the two patents.
-
• Forward, search questions. The four forward questions were all answerable based on the text in the document and asked about the kind of invention that was being described (Q1), existing products and their limitations (Q2) and how the disclosed invention overcomes those limitations (Q3). Q4 asked the participants to gain a general understanding of the preferred embodiment, which set them up to answer the seven surprise questions that they would be asked later.
-
• Backward, comprehension (recall) questions. The five backward recall questions were also answerable from the text and tested participants on their memory and understanding of the preferred embodiment. These questions were unique to each patent.
-
• Transfer (deep comprehension) questions. For the two transfer questions for each patent document, one was a failure-modes question and the other was a redesign question. These were based on the types of questions used previously in learning research (Austin, Reference Austin2009; Mayer, Reference Mayer2020). The failure modes questions ask about possible causes of faults that had occurred after months of good performance. The redesign questions asked participants about how the invention could be redesigned to increase airflow or suction force, even if their design changes negatively impacted other elements of the device.
After the main study, participants were asked if they would volunteer to answer two short follow-up questions. The first question was used to check for manipulation blinding and asked them if they noticed any differences between the two patent documents. The second question was used to assess the baseline comprehensibility of the patent documents and asked them if either stimulus was more difficult to comprehend than the other.
Scoring
Questions were scored using a predefined marking scheme for the recall questions and a novel coding and scoring scheme for the transfer questions (“Creation of Modified Patent Documents with Information Design Principles” section). Comprehension performance was quantified as the sum of their scores from all questions.
Tables A1 and A2 show the marking scheme for the recall questions. Questions 1 and 4 were not scored. The former was only intended to act as a warm-up question and to provide an opportunity for the experimenter to give participants feedback on their responses. The latter was used to guide them through the preferred embodiment and set them up to answer the backward questions. The first author and an independent judge scored the first 12 participants. Krippendorff’s alpha was. 797, which may be considered sufficient for tentative conclusions (Krippendorff, Reference Krippendorff2018). Disagreements between the judges were arbitrated and used to refine the coding procedure. The entire dataset was then recoded by the first author once data collection was complete.
The transfer questions were open-ended, and an unlimited number of points were available for each question. To code the responses, we applied the following process to each transfer question per patent document.
-
1. Coding schemes were defined that were deemed to be indicative of mental model formation. Points were awarded for the failure modes questions if the response noted a cause of a failure (1 point) and the part that would fail (1 point). Points were awarded for the redesign questions if the response noted a part that could be redesigned (1 point) and the variable that would be altered (1 point). Code categories are defined in Table 8.
Table 8. Code definitions
-
2. The transcribed responses were examined to look for coherent utterances that addressed the question. Those utterances were coded against the two coding schemes using NVivo.
-
3. Individual utterances were exported to Microsoft Excel and scored. Points were awarded for non-duplicate codes per participant per patent. Two constraints were placed on the coding.
-
a. No points were awarded for duplicate codes. For example, for failure-modes questions (Table 9) participants would be awarded two points for identifying a blockage in a specific component. If they mentioned a blockage in a different component, they would not be awarded again at the “cause” level for repeatedly mentioning a blockage, but they would be awarded additional points at the “part” level for listing other parts that could be blocked.
-
b. Generic responses that do not demonstrate comprehension of the preferred embodiment were not counted. For example, a generic reference to “a mechanism”
(Table 9B) would not receive a point at the part level. Reference to a generic modification like “play about with” or “change” would not receive a point at the variable level (Table 10B).
Table 9. Scoring examples for the failure modes transfer questions
Table 10. Scoring examples for the redesign transfer questions
-
Procedure
Participants completed the study in one session, comprising two trials (one for each patent document), each involving the same four phases (Figure 5). Patent documents were shown on the participants’ main screen, and questions were shown on their second screen, as indicated by the grey bars in the figure.
Figure 5. The study procedure as it was shown to the participants.
Participants completed four steps for each trial.
-
1. Presentation of forward questions and familiarization. Participants were presented with four questions, informed about the duration of the study period, and instructed to build up answers in their heads to the questions without writing anything down, opening a second instance of the. pdf document, or using interactive tools like the “find” tool.
-
2. Study the patent to build up answers to questions. The patent document was presented, and the participant was given 20 minutes to read it, in private, and to build up answers to the questions in their mind. The questions remained on their second screen.
-
3. Answer the first three general questions without looking at the patent. The participant was asked to minimize the patent document. The experimenter then read the first three of the four forward questions out loud, and the participant provided verbal responses.
-
4. Answer the backward questions without looking at the patent. The participant was presented with seven backward (surprise) questions that probed their knowledge of the preferred embodiment.
The procedure typically lasted 1 hour and 20–30 minutes. Participants were offered a break in between the two trials.
Results
Demographic variables
Demographic data were compared across the two groups to check the participant randomization (Figure A3). A Mann–Whitney U test showed that there were no statistically significant differences in the degree of industry experience U = 95.50, z = −.115, p = .910, familiarity U = 111, z = 0.672, p = .571 or frequency of patent use U = 101, z = 0.176, p = .874.
To look for demographic covariates, we examined the relationships between the three demographic variables and comprehension scores for the control and modified conditions. Overall, the results are shown in Table A3 and Figure A4. A Kruskal–Wallis H test showed that median comprehension scores increased with all familiarity in both conditions and for the sum totals, and total comprehension scores increased with frequency of patent use, but the differences were not statistically significant. Pearson’s correlations showed that there were no statistically significant associations between years of industry experience and comprehension scores. Thus, none of the demographic variables were taken to be covariates for comprehension performance in this study.
Effects of condition and patent document
A two-way mixed ANOVA was conducted with condition as a within-groups factor and group (which patent was modified) as a between-groups factor (Table 11, Table 12). The data was normally distributed, as assessed by Shapiro–Wilk’s test of normality (p > .05) (Table A4). There was homogeneity of variances (p > .05) and covariances (p = .866) as assessed by Levene’s test of homogeneity of variances and Box’s M test, respectively. We report interactions, main effects, and simple main effects without correcting for multiple comparisons across all tests. All results are from this ANOVA unless otherwise stated. The data are illustrated in Figure 6 shows six views for clarity, including the 2 × 2 analysis structure in plot A, interactions in plots B and C, and main effects in D–F.
Figure 6. Data from the patent experiment, showing the data grouped by patent and condition (A), summary data and 95% CIs highlighting the participant groups (B) and the two patent documents (C), and truncated violin plots for the main effects of condition (D), groups (E), and patent (F). Mean values are shown by a cross (+) on the boxplots.
Table 11. Summary data showing mean comprehension scores
Table 12. Results from the two-way mixed ANOVA
There was a statistically significant interaction between condition and group on comprehension scores F(1, 26) = 13.95, p < .001, partial η 2 = .349. This interaction corresponds to a main effect of “patent” if the data were rearranged, as shown in Figure 6C. This means that the comprehension scores for patent GB25 were statistically significantly less than those for GB24 when spread across two conditions (see also Figure 6F).
For the main effect of condition, the null hypothesis could not be rejected. Comprehension scores were 1.79 points higher for the modified patents (M = 21.71) than the control patents (M = 19.93) but the effect was not statistically significant F(1, 26) = 3.48, p = 0.073, 95% CI [−.180, 3.751]. As expected, group allocation did not affect comprehension scores. Comprehension scores were higher for group 2 (21.54) than group 1 (20.11) but the difference was not statistically significant F(1, 26) = 0.86, p = 0.362. The main effects should be qualified in the context of the statistically significant interaction reported above.
Visual inspection of the data and univariate analyses were used to investigate how the effects of condition and patent interact (Figure 6C). There was a statistically significant difference in comprehension scores for the two patents in the control condition, F(1, 26) = 8.60, p = 0.007, partial η 2 = .249, but not in the modified condition, F(1, 26) = 1.254, p = 0.273. Based on visual inspection, it appears that patent GB24 was associated with a negligible increase in comprehensibility (control = 22.43, modified = 22.61), but patent GB25 increased in comprehensibility by 18% (control = 17.43, modified = 20.64), even though an independent t test showed that this difference was not statistically significant t(26) = 1.705, p = .100, two-tailed.
Discussion
This article presents an approach for improving the comprehensibility of multimedia information sources in an engineering design context, without changing the wording of the information. The feasibility of the approach was tested in a crossover experiment on a sample of 28 professional engineering designers. The hypothesis was that the information design principles would improve patent document comprehensibility, but it was not possible to reject the null hypothesis. The modified patent documents were more comprehensible overall, but the difference was not statistically significant. Thus, we cannot yet claim that the approach can be used to enhance information comprehensibility. In the remainder of this discussion, we highlight the contributions of the approach and explore the implications of the results for measuring and improving the comprehensibility of patent documents in engineering design. We outline the limitations of the work and specify steps for further research.
Discussion of the approach and methodological novelty
The approach presented in this article is intended to enhance the comprehensibility of multimedia information sources by using theoretically and empirically supported information design principles, without changing the wording of the information. It provides engineering design researchers with a distilled overview of the theories and methods needed to conduct research into information comprehension in engineering design. The controlled experiment further demonstrates how the approach can be applied to remodel and measure the comprehensibility of multimedia information sources in an engineering design context.
The approach is uniquely applicable to an engineering design context for (i) eliciting both document search and comprehension processes, (ii) improving long, naturalistic multimedia information sources such as patent documents (rather than artificial experiment stimuli), and (iii) enabling the measurement of comprehension through transfer questions that are relevant for engineering design. Prior collections of theory and empirical examples (for compilations, see Jean-Francois Rouet Reference Rouet2006a; Mayer Reference Mayer and Mayer2014a; Van Meter et al. Reference Van Meter, List, Lombardi and Kendeou2020) provide a substantial body of reference material, but until now, engineering design researchers who are not familiar with these theories and techniques would still require substantial time and resource investment to apply this knowledge. A few prior studies address the integrated measurement of document search and comprehension but have not been tailored for an engineering design context (Rouet et al. Reference Rouet, Vidal-Abarca, Erboul and Millogo2001; Cerdán et al., Reference Cerdán, Vidal-Abarca, Martínez, Gilabert and Gil2009). Some studies have examined the improvement and measurement of engineering concepts, but only for short information sources such as presentations (Johnson et al., Reference Johnson, Ozogul, Moreno and Reisslein2013b, Reference Johnson, Butcher, Ozogul and Reisslein2013a) or product teardowns (Kearney et al., Reference Kearney, Starkey and Miller2022).
The approach enables researchers to selectively elicit search and comprehension processes by providing the taxonomy of question types (Table 3). The taxonomy is adapted from earlier work (Rickards, Reference Rickards1979) but has been remodeled to align with contemporary research on document search and comprehension and has been implemented in engineering design research for the first time. It allows for questions to be posed that vary in degree of integration (whether a question can be answered from a single location or requires integration of disparate text) and depth of comprehension. As shown in the experimental implementation of the approach (“Applying the Approach to Enhance and Measure the Comprehensibility of Patent Documents” section) this allows researchers to develop task scenarios that simulate collaborative information-sharing meetings. Additionally, the forward questions provide experimental control by inducing the participants to familiarize themselves with the document in a similar way, thereby providing some homogeneity to the search strategies employed by the participants in the 20-minute study period.
The two novel methodological contributions that make the approach readily applicable for engineering design are the transfer questions and accompanying coding schemes. Collectively, they enable the measurement of deep comprehension for domain-specific, engineering design knowledge – a novel contribution in the field. The transfer questions task participants with reasoning about potential failure-modes for a product and how the product could be redesigned to improve the performance of a specified parameter. The transfer questions are based on similar transfer questions used to measure student learning about lightning (Austin, Reference Austin2009; Mayer, Reference Mayer2020). However, this previous work is assessed with a rubric of acceptable answers. This scoring method may be useful for measuring learning about specific topics, but for the comprehension of, for example, patent documents it would require the creation of a unique set of acceptable answers for each invention, where professional engineering designers may be able to produce a wide variety of plausible responses.
Discussion and implications of the results
Results of the experiment
The feasibility of the approach was tested in a crossover experiment on a sample of 28 professional engineering designers, reading and answering questions about two patent documents that were selected to be as similar as possible without describing the same technology (“Selection of Patent Documents” section). The main hypothesis was that information design principles would improve patent comprehensibility in comparison with the original documents. An a priori power analysis provided as estimate of an 80% chance of detecting medium to large effects (Cohen’s d = 0.55), consistent with prior research. The confirmatory analysis failed to provide evidence to reject the null hypothesis. The modified patent documents were more comprehensible overall (8.93%), but the difference was not statistically significant. As such, we cannot yet claim that the approach can be used to improve the comprehensibility of patent documents.
The exploratory analyses of the simple main effects revealed two additional patterns in the data. First, the two patent documents were statistically significantly different in baseline comprehensibility; the original version of patent GB24 was more comprehensible than the original version of patent GB25. Secondly, although the simple main effects were not significant, there was a negligible improvement in comprehension scores for patent GB24 (0.80%), but an improvement in patent GB25 (18.4%, d = .645) that is consistent with the sizes of effects found for multimedia design principles, such as d = 0.46 for multimedia signaling, and d = 0.79 for spatial contiguity (Mayer, Reference Mayer2017). Indeed, the modified version of patent GB25 made it almost as comprehensible (mean comprehension score = 20.64) as the original version of GB24 (M = 22.43). Thus, overall there appeared to be an isolated improvement in patent GB25, from a position of relatively lower comprehensibility in the original condition.
Explanations for the lack of a significant main effect
The lack of a significant main effect could be attributed to the information design principles having a smaller cumulative effect than expected, or one of the patent documents being more amenable to improvement than the other. One explanation for the smaller-than-expected overall effects is that carryover effects introduced by the participants completing two trials may have “washed out” the potential improvement conferred by the information design principles. A second explanation is that some of the individual principles may have produced minimal or negative effects, limiting the overall effectiveness of the modifications to the patent documents. For example, a redundancy effect could have been introduced by the spatial contiguity principle. The redundancy effect refers to detrimental performance caused by the extraneous cognitive load that occurs when information is presented that is not needed for comprehension (Chandler and Sweller, Reference Chandler and Sweller1991; Ayres and Sweller, Reference Ayres and Sweller2005). Redundancy can occur when additional information that is intended to improve comprehension increases the total volume of information in the source. The spatial contiguity principle resulted in some images being repeated and the text being spread out over more pages. This may have reduced the benefits of the spatial contiguity principle or produced negative effects.
The main effect could also have been made harder to detect if there was something different about the content of the two patent documents that made patent GB25, but not GB24, amenable to improvement. For example, a boundary condition for the multimedia spatial contiguity principle (putting the images and text side by side) (Ayres and Sweller, Reference Ayres and Sweller2005) and the multimedia signaling principle (highlighting correspondences between the text and images) (Jeung et al., Reference Jeung, Chandler and Sweller1997; Mayer and Fiorella, Reference Mayer and Fiorella2014) is that the principles are most effective for learners with relatively lower prior knowledge. If the participants had different levels of prior knowledge about the concepts being described by each patent document, the information design principles may have differed in effectiveness for those patents. Responses to the transfer questions and voluntary follow-up questions support this explanation. For example, when asked about the possible root causes of blockages in both vacuum tools, one participant noted that “…I’ve had this problem with my vacuum deaner…”, showing how prior knowledge can influence participant responses. Furthermore, when asked about whether either of the two trials was more difficult than the other, one participant said: “Second one [the original version of patent GB24] was easier to understand in terms of the concepts, but the layout was worse”, showing that the participants were sensitive to differences in both the surface-level features of the patent documents and the underlying content.
The potential role of prior knowledge challenges a core assumption of our research. That is, the information design of patent documents is sufficiently bad that our approach and information design principles would provide a consistent improvement for all patent documents. We expected that by using professional engineering designers and carefully selecting the patent documents to be as similar as possible without being identical (“Selection of Patent Documents” section), we would preemptively mitigate any effects of prior knowledge. Yet, there may remain the case that the working principles of patent GB24 (a mechanical actuator and brush head) may be conceptually less complex, or generally more familiar, than those of GB25 (a reduction in pushing force caused by the control of air pressure in the vacuum conduit).
Applications for the approach to patent simplification and beyond
The lack of a statistically significant main effect means that the comprehension-enhancing aspect of the approach has not yet been validated through the present study. However, the approach is still useful for the elicitation and measurement of information comprehension in an engineering design context. An immediate application of the approach is in the measurement of patent simplification systems. Previously, patent simplification tools have been evaluated with subjective ratings of usability or lexical measures of comprehensibility that do not provide objective measures of human comprehension (“Introduction” section). This approach allows for more objective measures of comprehension performance.
The same approach can also be applied to other cognitively demanding sources of multimedia information in engineering design. Examples include lifecycle management software (Eigner, Reference Eigner2021), engineering dashboards (Fradi et al., Reference Fradi, Bricogne, Bosch-Mauchand, Louhichi and Eynard2017) extended reality displays in Industry 4.0 applications (Adriana Cárdenas-Robledo et al., Reference Adriana Cárdenas-Robledo, Hernández-Uribe, Reta and Antonio Cantoral-Ceballos2022), and long multimedia documents used in engineering design such as requirements specifications or design histories (Gruber and Russell, Reference Gruber and Russell1992). As shown in Table 6, the changes made to the patent documents are technically feasible and so the same methods have the potential to be generally applicable to other information sources.
Limitations and further development of the approach
The main limitation in testing the feasibility of the approach is the lack of statistical significance of the main effect of condition (p = 0.073), meaning we cannot draw conclusions about the efficacy of using the information design principles to enhance patent comprehensibility. This likely stems from a lack of statistical power and may have been exacerbated by a moderating effect of prior knowledge and redundancy effects introduced through the application of the information design principles (“Explanations for the Lack of a Significant Main Effect” section).
The ability to detect a significant main effect could be improved by improving statistical power, improving experimental controls, or by implementing more effective interventions that create larger effects that would be easier to detect. For statistical power, a larger sample would have enabled the detection of smaller main effects and interactions. Future research could also address the potential role of prior knowledge. Further controls could be implemented by using pretests to select stimuli that are not overly familiar to the participants. Alternatively, researchers could directly examine the role of prior knowledge by conducting parameter range exploration (Scheel et al., Reference Scheel, Tiokhin, Isager and Lakens2021); varying the degree of prior knowledge between patents and participants to examine the boundary conditions of any moderating effect. Finally, the information design principles could be made more effective by avoiding redundancy effects. Digital multimedia environments, such as patent viewing systems, provide more flexible options for implementing information design principles. For example, hover-over pop-up images can allow users to selectively introduce additional information, and dynamic side-by-side windows can show complementary text and images without increasing the length of the document. Researchers may also consider how patents could be displayed in different modalities to enhance comprehension. For example, some multimedia design principles provide recommendations for text and animations (Mayer, Reference Mayer2014b).
The approach could be modified in two ways when measuring comprehension in new applications. One modification would be to develop new transfer questions. At present, the transfer questions test knowledge transfer in only two kinds of design activity: failure modes and redesign. However, engineering designers use knowledge from information sources in a wide variety of activities, which can be seen in ontologies of design activities (Sim and Duffy, Reference Sim and Duffy2003; Štorga et al., Reference Štorga, Andreasen, Marjanovic, Mariŏmarioštorga and Marjanović2008; Cash and Kreye, Reference Cash and Kreye2017). New transfer questions could be based on activities involving analysis, decision-making, evaluation, or selection. A second consideration is that the transfer questions were not scored for correctness. We opted not to differentiate between correct and incorrect inferences as we deemed this to be a measure of engineering knowledge rather than document-specific comprehension. Future research could explore whether assessing and scoring for correctness could provide more granular or accurate assessments of an engineering designer’s comprehension of an information source.
If the approach is to be generalizable in practice, more work will be required to understand what makes some information sources more difficult to comprehend than others. Future work may benefit from going beyond theories of multimedia learning and comprehension. One direction is to incorporate research about conceptual knowledge, which is the understanding of the governing principles in a domain and how knowledge units relate to one another (Rittle-Johnson, Reference Rittle-Johnson2006). It has been proposed that conceptual knowledge is more difficult to acquire when concepts are abstract and unobservable (Chi, Reference Chi2005; Streveler et al., Reference Streveler, Litzinger, Miller and Steif2008; Borghi et al., Reference Borghi, Binkofski, Castelfranchi, Cimatti, Scorolli and Tummolini2017). This is consistent with our discussion about prior knowledge (“Explanations for the Lack of a Significant Main Effect” section). The mechanical action of a slicker brush (patent GB24) acting on pet hair is both easier to observe and interpretable through everyday experience of physical part interactions. In contrast, the technology in patent GB25 improves maneuverability through relatively unobservable changes in forces.
Conclusions
Computational summarization tools may be used to enhance the comprehensibility of technical information for engineering designers. However, to guide the development and evaluate the effectiveness of such tools, there is a need to (i) establish how a designer processes and understands information, (ii) determine what makes for “good” information presentation, and (iii) measure the designer’s performance. The approach established in this article addresses these three needs. The use of long, complex technical documents can be understood using cognitive models of document search and comprehension. This theoretical grounding allows researchers to identify information design principles that can be used to remodel information sources to make them more compatible with effective cognitive processing. The approach then provides a taxonomy of question types that can be used to elicit document search and comprehension and methods for measuring comprehension, thereby allowing for an experimental comparison of the original and remodeled information sources.
The feasibility of the approach was evaluated by applying the approach to enhance and measure the comprehensibility of patent documents. In a 2 (patent document) × 2 (condition: original and modified) experiment with 28 professional engineering designers, we tested the hypothesis that patent documents remodeled with theoretically relevant information design principles would be more comprehensible than in their original form. Two patent documents were selected from the worldwide patent database and carefully matched to be as similar as possible in surface-level features (the way the information is presented) and content (the technologies being described), without being identical. The results showed that patent comprehensibility improved overall, but that the effect was not statistically significant. Thus, we cannot yet claim that the approach can be used to enhance information comprehensibility. Based on further exploratory analyses, we suggest that there may have been an isolated improvement in one of the patent documents. We attribute this to either redundancy effects producing smaller-than-expected effects, or an interaction between prior knowledge and the content of the patent documents (despite the careful matching of patent documents during stimuli selection).
Overall, the approach has shown to be applicable for measuring the comprehensibility of long, complex information sources in engineering design. This provides a valuable resource for developers of patent simplification tools who have previously relied on subjective user satisfaction ratings to assess their tools. Although the ability of the approach to enhance information comprehensibility has not been validated, the worked example of the approach allows for further development and application in new contexts in engineering design.
Acknowledgments
The authors would like to thank Dr. Nigel S. Clarke for his advice, support, and feedback on the empirical study.
Funding
This research was funded by the Digital Engineering Technology and Innovation (DETI) initiative and carried out at the University of the West of England in Bristol. DETI was funded by the West of England Combined Authority & Local Enterprise Partnership through the Local Growth Fund, administered by the West of England Combined Authority. This funding of £5m was matched by industry and HVM Catapult. The funder had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interest
The authors declare none.
Appendix
Figure A1. Patent GB24 in its original form.
Figure A2. Patent GB24 after modifications.
Table A1. Scoring criteria for patent GB24
Note: Location = where the answer is stated in the text, format page.line.
Table A2. Scoring criteria for patent GB25
Note: Location = where the answer is stated in the patent document, format: page.line.
Figure A3. Differences between the stimuli configuration groups for frequency of patent use (A), familiarity of patent use (B) and years of industry experience (C).
Table A3. Demographic analyses
Note: Familiarity and frequency analyzed with Kruskal–Wallis H test, Industry experience analyzed with Pearson’s correlation.
Figure A4. Demographic data and comprehension scores.
Table A4. Results from Shapiro–Wilk’s test of normality
Note: Stimulus configuration group refers to Table 4.
Open access
