3.1. ellipses of narrative categories

Given these structural properties, we can now identify how omissions function in a narrative sequence. Some panels can delete easier than others (Cohn, Reference Cohn2014; Magliano, Kopp, Higgs, & Rapp, Reference Magliano, Kopp, Higgs and Rapp2017). Establishers can acceptably be deleted from drawn sequences with little consequence (Cohn, Reference Cohn2014), and this appears to persist in other modalities like film (Kraft, Cantor, & Gottdiener, Reference Kraft, Cantor and Gottdiener1991) and motion graphics (Barnes, Reference Barnes2017). Initials and Peaks are generally unacceptable to delete, though when they are, their semantic content may be inferred, as discussed below (Cohn & Kutas, Reference Cohn and Kutas2015; Cohn & Wittenberg, Reference Cohn and Wittenberg2015; Magliano et al., Reference Magliano, Larson, Higgs and Loschky2015). Finally, a sequence may make sense without a Release, but it will create a fairly abrupt ending. These tendencies have been supported by studies where participants choose panels to delete from a sequence, or recognize which panels may have been deleted (Cohn, Reference Cohn2014). Additional studies have found viewing times increase to panels following dispreferred omissions, as categorized in a complementary analytical framework (Magliano et al., Reference Magliano, Kopp, Higgs and Rapp2017). Overall, omission of different narrative categories can vary the comprehensibility of the sequence. The variance of these ellipses illustrates that inference motivated by omission is not uniform (McCloud, Reference McCloud1993), but is contingent on (and can be characterized by) the narrative structure itself.

3.2. peak drop

As discussed, deleting Peaks from a sequence lowers its comprehensibility and increases the inference required to understand it. Nevertheless, Peaks can be omitted intentionally with constrained regularity. Consider the two strips in Figure 4. Figure 4a shows a man skating backwards, who is then distracted by something in the second panel (cued by wide eyes and open lips). The final panel shows him lying in a broken window of an antique store, while the distracting element is revealed as an attractive woman. This final image shows the aftermath of an event (its coda), but the sequence never shows the climatic Peak: where he crashes into the window. Similarly, Figure 4b depicts an owl in the first panel, with a mouse commenting how cute it is in the second (both Initials, united via Environmental-Conjunction). The final panel shows the mouse’s tail in the owl’s mouth. Again, the primary event (the owl eating the mouse) is left inferred, while the coda is depicted.

Fig. 4. Two strips which both omit Peak panels which thereby cause the inference of climactic events prior to the final panel. (a) Actions Speak © 2002 Sergio Aragonés. (b) Savage Chickens © 2016 Doug Savage (www.savagechickens.com).

These examples use Peak Drop, the purposeful omission of a Peak which generates an inference for an omitted causal event from the combination of its preparatory (Initial) and aftermath states (Release). Because a Peak is essential to a narrative sequence, and such cases appear to be systematic, this may be a special construction which omits the Peak from the canonical narrative schema. This would differ from simple elision of Peaks from a canonical schema, in that Peak Drop invokes a schematic pattern that structurally intends for Peak to be missing and the events to be inferred.

A Peak Drop Construction thus uses a narrative phase with no Peak, which maps to semantic information of a causative event (head) where the resulting state (coda) maps to the Release. Figure 5a illustrates this relationship within VNG. The combination of the preparatory and resulting events thus generate the inference about the missing event (Strickland & Keil, Reference Strickland and Keil2011), though it is absent in the narrative structure. Such omission of a Peak elicits an updating process, evident in slower self-paced viewing times to panels following omitted Peaks, despite fairly high ratings of comprehensibility (Cohn & Wittenberg, Reference Cohn and Wittenberg2015; Hutson, Magliano, & Loschky, Reference Hutson, Magliano and Loschky2018; Magliano et al., Reference Magliano, Larson, Higgs and Loschky2015). Updating is also suggested by visual search processes aiming to find inferential cues to clarify the preceding absent event information (Hutson et al., Reference Hutson, Magliano and Loschky2018). Such inference generation appears to overlap with linguistic working memory systems, implying that such mechanisms may be domain-general (Magliano et al., Reference Magliano, Larson, Higgs and Loschky2015).

Fig. 5. Missing event information generated by (a) Peak Drop where the Peak panel is omitted, and (b) replacement of a Peak with an action star panel.

Nevertheless, the claim is that a narrative Peak is indeed missing from a sequence, not just that its semantic content is inferred. Such an interpretation is supported by studies of visual narrative where omitting Peak panels show a cost relative to those that contain a Peak, as indicated through longer viewing times (Cohn & Wittenberg, Reference Cohn and Wittenberg2015) or larger neural responses indicative of reanalysis (Cohn & Kutas, Reference Cohn and Kutas2015), even when both involve inference generation. Readers cognitively recognize a missing narrative Peak, above and beyond the inference of the missing semantic information (Cohn & Kutas, Reference Cohn and Kutas2015).

3.3. suppletive panels

Besides deletion, panels may be replaced with inference-generating information. Suppletive panels are a class of panels – often systematic – which replace information at various levels of explicitness. Suppletive panels mostly express three types of concepts: general events, fights, and transformations.

The most general suppletive panel is an action star, as in Figure 5b, which depicts an impact star (Figure 1c), blown up to the size of a full panel (Cohn, Reference Cohn2013b). This star shape is homonymous with a carrier showing loudness, i.e., a jagged speech balloon or sound effect (Cohn, Reference Cohn2013b; Forceville, Veale, & Feyaerts, Reference Forceville, Veale, Feyaerts, Goggin and Hassler-Forest2010), and indeed, action stars often co-occur with text, like Pow! (Manfredi, Cohn, & Kutas, Reference Manfredi, Cohn and Kutas2017). Thus, action stars are morphologically ambiguous as either a blown-up impact star or a jagged-edged carrier.

Nevertheless, action stars express a sudden and potentially impactful event, without depicting what that event actually is. In Figure 5b, the event structure of the action star is encoded as a sudden and violent unspecified event (notated as a variable: ‘X’) between the Agent (A) and Patient (P), and thus an inference is required to infer its meaning (i.e., punch, connected to the action star through the subscript ‘b’). Text like Pow! or Smack! could provide greater specificity to this ambiguity (discussed below). With or without text, a reader must infer the missing events, pulling from information in the prior and subsequent panels, as in Peak Drop. Indeed, as with omitting a Peak event, readers view panels following an action star slower than following an explicit depiction of an event, implying that the longer time is caused by inferential processing (Cohn & Wittenberg, Reference Cohn and Wittenberg2015).

Despite their impoverished semantics, action stars play a well-formed narrative role in the sequence. They replace Peak panels (Figure 5b), but less suitably replace other categories. Action stars are a sort of ‘pro-Peak’ in the way that pronouns act as nouns; they satisfy the categorical role, but lack semantic specificity. This structural role is supported by findings that readers spend less time viewing an action star compared to a fully blank panel at the Peak position, despite the blank being less visually rich (Cohn & Wittenberg, Reference Cohn and Wittenberg2015). Thus, action stars facilitate viewing times more than simply omitting event information, thereby suggesting a sequential role.

Other suppletive panels are more explicit. Fighting is replaced by fight clouds (Figure 6a), which typically show a large cloud, a tornado, or a jumble of lines, along with body parts, clothes, and/or other morphemes (spirals, stars, lightning bolts) emerging from the fray. Though action stars and fight clouds may both replace combat, they imply different dimensions of ‘aspectuality’ – i.e., how much time they convey (Cohn, Reference Cohn2013b). Action stars are a punctive event, implying a single momentary action. You cannot infer multiple punches or an exchange of punches in an action star. In contrast, fight clouds imply durative events, occurring across a span of time. Here, there must be multiple actions (hits, kicks, bites, etc.) and cannot be only a single action. Thus, both panels replace actions, but evoke subtly different inferences about those events. Nevertheless, like action stars, fight clouds as full panels can function as Peaks, signaling a high arousal event.

Fig. 6. Suppletions of panels for specific events with (a) a fight cloud for fighting and (b) a love cloud for sex, by Dwayne Godwin and Jorge Cham, and a poof cloud transformation in Rickety Skitch and the Gelatinous Goo © Ben Costa and James Parks. Also, (d) a climactic panel (Peak) replaced by a panel solely with text from Quantum and Woody Must Die © Steve Lieber.

A love cloud also uses a cloud shape, but to imply sex instead of violence. This may be fairly saccharine, with hearts emerging (instead of the stars, like a fight cloud), as in Figure 6b, or with more salacious body parts sticking out. The overall intent of fight and love clouds are similar: rather than show the durative events directly, they become implied through a suppletive panel.

Various depictions also substitute for transformations, subsumed within a class of transformative suppletion panels (TSPs). For example, Figure 6c replaces a poof cloud for a depiction of the imp transforming to look like a gnome. Here, we infer that the imp transformed, despite seeing only the prior and end states; the poof cloud provides a climactic unit for the transformation, relieving the artist from showing the process overtly. TSPs have several allomorphic depictions, with slight variance in manner implied between clouds, blobs, stars, tornadoes, and other graphic features. Like action stars, TSPs imply a sudden punctive event, despite that transformations may extend across a duration of time. They also play narrative roles as Peaks.

A multimodal suppletion occurs when text replaces a panel, particularly onomatopoeia (Pow!) or a ‘descriptive’ (Punch!) sound effect (Guynes, Reference Guynes2014; Manfredi et al., Reference Manfredi, Cohn and Kutas2017; Pratha, Avunjian, & Cohn, Reference Pratha, Avunjian and Cohn2016). Consider Figure 6d, where a woman defends herself with a stapler against a monster. The climactic stapling moment is not shown, replaced instead with the sound effect Pa-chunk! This onomatopoeia replaces, and thereby demands inference of, the (Peak) event. Text substitution often accompanies other suppletions, like action stars and TSPs (Figure 6c).

Text-substitution elicits inference both from the sound effect and its surrounding context. The text provides event cues as a ‘sound emission’ (Goldberg & Jackendoff, Reference Goldberg and Jackendoff2004), whereby the sound resulting from an event indexes the actual event. In language, similar inferences arise from constructions like Tom belched his way out of the room to mean that Tom went out of the room while belching (Goldberg & Jackendoff, Reference Goldberg and Jackendoff2004). Further, if the text mismatches the visual context of the preceding or subsequent images (example in Figure 6d, text of Smooch!), it will appear incongruous (Manfredi et al., Reference Manfredi, Cohn and Kutas2017). Thus, as with other suppletive devices, both content and context motivate the inference of the undepicted events.

Text-substitution thus connects a sound effect to the unseen event. This text leaves both the carrier (i.e., ‘balloon’) and the stem (the thing generating the sound) undepicted (Cohn, Reference Cohn2013a). This suppletion goes beyond the off-panel inference created by a tail pointing to the border of a panel (Figure 1e), instead omitting everything but the contents of the carrier. Again, the morphological connection between the text and its stem facilitates the inference, even if the actual carrier is not depicted (Cohn, Reference Cohn2013a).

Text-substitution also typically functions as a Peak, with its cues coming exclusively from the verbal, not visual modality (Manfredi et al., Reference Manfredi, Cohn and Kutas2017). Cross-modal substitution of a grammatical category (narrative, syntax) occurs between all modalities – i.e., text into image sequences, images into text sequences, gesture into speech, etc. (Cohn, Reference Cohn2016b).

3.4. mental-space evoking panels

Panels can also imply events and actions through metonymy or metaphor, where depicted information if understood literally would be incongruous. However, in context, such panels evoke larger semantic frames or connect mental spaces to create conceptual blends or metaphors to form a coherent interpretation (Fauconnier & Turner, Reference Fauconnier and Turner1998; Forceville, Reference Forceville and Cohn2016).

First, panels might show something related to an event, without showing the event itself. These replacements thus depict a metonymic relationship to the unseen events (Cohn, Reference Cohn2010; Joue et al., Reference Joue, Boven, Willmes, Evola, Demenescu, Hassemer and Habel2018; Kowalewski, Reference Kowalewski2018; Kukkonen, Reference Kukkonen2008). Consider Figure 7a, where a man in bed turns off the light. We are not shown an image of his hand turning off an actual lamp, but rather we see a darkened light bulb in full. The darkened light bulb is metonymic for the lamp’s light being turned off, relative to the brightness of the prior panel and the darkness of the subsequent one. Other metonymies might depict doves flying or a statue crying instead of depicting a murderous event.

Fig. 7. Panels which use mappings between mental spaces for (a) metonymy of a light being turned off in One Night by Tym Godek, and (b) metaphors for a sex scene in panels 1, 3, and 5; Deadpool is © Marvel Comics.

Panels might also use metaphors to imply unseen events. Metaphors abound in visual narratives, often embedded into their visual morphology, such as steam coming from characters’ ears to depict anger as if the head was a pressurized container (Cohn, Reference Cohn and Booij2018; Forceville, Reference Forceville and Cohn2016; Lakoff & Johnson, Reference Lakoff and Johnson1980). Yet, metaphors can also insinuate events (Cohn, Reference Cohn2010; Gavaler & Beavers, Reference Gavaler and Beavers2018). Consider Figure 7b, which comes from a full page where numerous panels use metaphors for sex (train through a tunnel, volcano erupting, etc.). Another metaphor would be a scene where someone becomes mad, then a panel of a tea kettle boiling over, then the other person with a black eye. The kettle would thus invoke the metaphor of anger as fluid in a pressurized container (Forceville, Reference Forceville and Cohn2016; Lakoff & Johnson, Reference Lakoff and Johnson1980).

Figure 8 illustrates metaphor panels in the parallel architecture of Visual Narrative Grammar. The Peak has been replaced with a panel depicting an explosion. This panel should be incongruous, featuring none of the characters or situation involved in the sequence. However, if understood metaphorically, the events of an explosion become mapped to the event of a punch (subscript ‘m’), evoking parallels between the puncher’s weapon (fist/bomb) destroying its target (boxer/bombed target). These correspondences arise from the inferences drawn between the conceptual spaces of the punching and bombing events.

Fig. 8. Metaphoric mapping of seemingly incongruous event to the primary event in a sequence of images.

As suppletive panels, metonymic and metaphoric panels exist outside the storyworld of the visual narrative. However, these mental space evoking relations can also incorporate elements from the narrative scene, which will be discussed below.

3.5. onlookers

Beyond omissions or replacements, inference can also occur by diverting focus to something else in a scene, like showing a person observing an event instead of the event itself. Such onlookers may vary in how explicitly they suggest what actions or events they watch. A Passive Onlooker is the least explicit voyeur, showing characters observing an action off-panel, but with a neutral unexpressive facial expression. Typically, the panel after the Passive Onlooker reveals an aftermath of what occurs out of the reader’s view. In Figure 9a, Charlie Brown is the Passive Onlooker in panel 3, only to reveal at panel 4 that he watched Linus retrieve the ball, not Snoopy. Because the Passive Onlooker gives no cues about the events taking place, the final panel demands a bridging inference to fill in this missing information (Cohn & Kutas, Reference Cohn and Kutas2015).

Fig. 9. Onlooker panels in a slightly manipulated Peanuts strip, where (a) depicts Charlie as a Passive Onlooker, (b) depicts him as a Reactive Onlooker, (c) depicts him with an exclamatory cue of a carrier, and (d) adds descriptive narration. Peanuts is © Peanuts Worldwide LLC.

This final image is a narrative Release, as it shows the punchline or aftermath of an event. But what of the Passive Onlooker? Here, ambiguity remains as to whether the unseen event happens at this panel, watched by the onlooker, or if it happens between this and the final panel. If the climactic event occurs simultaneous to the onlooker, it would imply a Peak, but would only be recognized as such at the subsequent Release. Indeed, the semantic cues in this panel do not signal anything about a climactic event and, in fact, gives few clues about events at all. Thus, given its semantic cues, panel 3 of Figure 9a is more characteristic of a Prolongation, extending the prior state, and the subsequent Release implies that this sequence uses Peak Drop.

Variations to onlookers hint more about an unseen event. A Reactive Onlooker has an emotional facial expression that signals some action occurs off-panel. For example, Figure 9b alters 9a only by making Charlie’s facial expression surprised. Though Reactive Onlookers’ emotions generally suggest a viewed event (looking surprised, exasperated, grossed out, happy, etc.), this expression gives no clues about the actual event. This might align with the ‘Spielberg face’ in film, a film shot where an onlooker is shown with eyes wide in wonderment at an off-camera event (Fortunato, Reference Fortunato2014).

Unlike Passive Onlookers, Reactive Onlookers signal a climactic event, and thus are narrative Peaks. A Reactive Onlooker is modeled in Figure 10a, with a Peak of a surprised referee, cued by the facial expression and exclamation mark that he is reacting to the off-panel event at this moment. His event (below, subscript b) is that of ‘seeing’, but what he sees is the whole event of the Agent and Patient (α). This index binds to the inferred event (the punch), meaning that he sees this whole event (α), while the reader does not.

Fig. 10. Missing event information generated by (a) a Reactive Onlooker where the Peak panel depicts a voyeur of the actions, and (b) selective framing of non-essential event information.

Both Passive and Reactive Onlookers require inference to understand the unseen event, and thus draw from context in the previous and subsequent panels. A fairly explicit Initial may allow a strong inference at the Reactive Onlooker, such as when a person shoots a basketball in an Initial, followed by a Peak of the onlooker crowd cheering. The actual event (ball through the hoop) would not be depicted, but the inference is highly constrained. An event with less clear forecasting would thus generate inference at a subsequent panel to the onlooker. As stated, whereas the Passive Onlooker provides no cues of an off-panel event at that moment, a Reactive Onlooker signals that the inferential event occurs at that narrative moment, as in Figure 9a and 9b. Thus, Reactive Onlookers trigger the necessity for an inference prior to Passive Onlookers in the flow of the narrative sequence (Cohn & Kutas, Reference Cohn and Kutas2015).

This contrast in cues between Passive Onlookers and Reactive Onlookers has consequences on their narrative structure. Because Passive Onlookers lack cues, they remain categorically ambiguous (likely a Prolongation), while cues in Reactive Onlookers can signal a Peak, except the climactic event is known to remain off-panel. Neurocognitive research has supported this interpretation because Reactive Onlookers (like Figure 9b) evoke greater neural responses consistent with mental model updating than Passive Onlookers (like Figure 9a), suggesting that these cues provided more explicit event information (Cohn & Kutas, Reference Cohn and Kutas2015). Yet, at the subsequent panel, greater updating processes were suggested for Passive Onlookers than Reactive Onlookers. Indeed, neural responses to panels following Reactive Onlookers did not differ from panels following Peaks which explicitly showed the climactic events. Thus, despite requiring the same inference as Passive Onlookers, neural responses to panels after Reactive Onlookers patterned with those of regular Peaks, thereby suggesting they had a similar structural role.

Onlookers can become more expressive if their body posture fully implies the off-panel events. Echoic Onlookers’ postures imitate actions taken by characters out of view. Consider Figure 11, where a character, Rob, attempts to pick up on women at a bar. His friend, Ramon, watches the encounter. In panel 2 (an Initial), Ramon looks apprehensive, as a Reactive Onlooker, with Rob implied by the tail of the balloon pointing to the panel’s edge. In panel 3, the Peak, Ramon grabs his crotch and winces – suggesting that his friend has been kicked below the belt (reinforced by the text-substitutions: Smack, pow, punt!). Thus, Ramon in panel 3 is an Echoic Onlooker, ‘echoing’ the actions undergone by Rob off-panel. Echoic Onlookers are the most explicit type of onlooker, as their actions fully cue the inferred event.

Fig. 11. Sequence with an Echoic Onlooker in the third panel. Butternutsquash © Ramon Perez and Rob Coughler.

Additional exclamatory cues can make onlookers more expressive. As in Figure 9c, adding upfixes to onlookers can provide non-bodily cues that signal an off-panel event. These include exclamation marks (!) and question marks (?), either floating as upfixes above a head, or placed in carriers (balloons or bubbles). Exclamatory cues of meaningful balloons or bubbles might say ‘Gasp!’ or some other general surprised statement by an onlooker. Like the bodily cues in Reactive or Echoic Onlookers, exclamatory cues signal the onlooker’s perception of an off-panel event at that moment. Such cues can motivate the panel as a Peak, and, as discussed above, can elicit different brain responses than panels with Passive Onlookers (Cohn & Kutas, Reference Cohn and Kutas2015).

Finally, an onlooker might convey more than a vague sense of surprise like with exclamatory cues, instead being completely explicit about the unseen events. Descriptive narration by an onlooker provides a play-by-play of what they see happening. In Figure 9d Charlie, the onlooker, now explicitly describes the unseen event, thereby making the final panel fairly unrevealing. Here, the event is not depicted visually, but it is overtly stated in the text. Thus, a comprehender would need to infer the visual properties of an event, but would need little inference from the holistic multimodal interaction.

3.6. selective framing

Other inference generating techniques may show elements in a scene related to an event, while not showing the actual event. Such cases use selective framing of non-central aspects of an event, while still depicting the entities involved. Consider Figure 12a, where a man threatens the woman, only in panel 3 for her to grab his groin. Instead of showing her action, this panel selectively frames his pained expression in response to it. Similar relations are modeled in Figure 10b, where most of the structure remains the same as Figure 2, only now the panel windows a narrower amount of information (as reflected in spatial structure), which now makes the event in the Peak less salient.

Fig. 12. Peak panels (a) implying a man being grabbed in the groin; Battle Chasers © Joe Madureira, (b) a gunshot through the metonymic selective framing with a bullet casing and the accompanying emergent text (Blam!); The Creech © Greg Capullo, and (c) implying a skull being cracked through the crashing of a pumpkin; Watchmen © DC Comics.

Research with eye-tracking has indeed suggested that certain areas of a panel are more informative for the depicted events than others. Viewers tend to fixate event-informative regions (Hutson et al., Reference Hutson, Magliano and Loschky2018), whether the narrative sequence is coherent or incoherent (Foulsham, Wybrow, & Cohn, Reference Foulsham, Wybrow and Cohn2016), though fixations may be more dispersed in incoherent sequences. When framed as a whole panel, this fixated information would be similar to the first panel in Figure 12b, which shows only the primary action (hand pulling out a plug) without any of the surrounding details. Selective framing would thus window ‘non-fixated’ information where their indicative quality pushes a reader to infer the events.

Examples of selective framing include a sex scene depicting curled toes or an orgasmic face instead of the explicit sex act, or in a fight scene, a pale exasperated face instead of showing a person being stabbed, shot, etc. (as in Figure 12a). Throughout, the key visual information cuing the event would be missing. In some ways, onlookers share this windowing quality of selective framing by showing a person watching the events, rather than essential actors themselves. While selective framing can replace an explicit Peak, constrained windowing of information can apply to all panels in the narrative (example Figure 12b).

Other selective framing can window something in the scene other than the events, while retaining a semantic connection to them. Metonymic selective framing shows elements of a scene related to an event instead of the entities or cues involved in the event. Figure 12b shows a bullet casing flying in the air with smoke to indicate the firing of a gun. The gun is not shown firing, but the bullet discharge metonymically indicates the firing as a selectively framed part of the gunshot action. Other metonymic selective framing might include blood splattering without showing the wound itself, showing bubbles underwater instead of a person drowning, or, for the recurring boxing example, showing a mouthpiece flying through the air as the Peak.

The Peaks in Figure 12a–c also motivate inference through onomatopoeia. Text selective framing uses speech balloons, thought bubbles, or sound effects to indicate an unseen event. This is similar to the text-substitution onomatopoeia, but here may not depict a ‘sound effect’ (such as someone screaming ‘Nooooo!’ while killed off-panel) and may not fully replace a Peak.

Finally, metaphoric selective framing can use elements in a scene to imply other actions. Consider Figure 12c from Watchmen. The crashing pumpkin metaphorically stands in for the crushed head of the character in the first panel (Gavaler, Reference Gavaler2017), accompanied by the off-panel affixing text of ‘HHUUGGHH’. The pumpkin appears in the background of the first panel of Figure 12c, largely to set up its metaphorical function in the subsequent Peak. Metonymic and metaphoric selective framing differ from metonymic or metaphoric suppletions (discussed above) by whether the elements exist in the actual scene. Metonymies and metaphors involving selective framing show elements that exist within the scene itself and relate to the primary events, while suppletions shift to depict abstract information outside the storyworld.