Systematically evaluating DNNs for biological vision

While this commentary identifies multiple shortcuts in DNNs that are commonly used in vision science, such as ResNet and AlexNet, it does not delve into the root causes of these issues or how widespread they are across different DNN architectures and training routines. We previously addressed these questions with ClickMe, a web-based game in which human participants teach DNNs how to recognize objects by highlighting category-diagnostic visual features (Linsley, Eberhardt, Sharma, Gupta, & Serre, Reference Linsley, Eberhardt, Sharma, Gupta and Serre2017; Linsley, Shiebler, Eberhardt, & Serre, Reference Linsley, Shiebler, Eberhardt and Serre2019). With ClickMe, we collected annotations of the visual features that humans rely on to recognize approximately 25% of ImageNet images (https://serre-lab.github.io/Harmonization/). Human feature importance maps from ClickMe reveal startling regularity: Animals were categorized by their faces, whereas inanimate objects like cars were categorized by their wheels and headlights (Fig. 1a). Human participants were also significantly more accurate at rapid object classification when basing their decisions on these features rather than image saliency. In contrast, while DNNs sometimes selected the same diagnostic features as humans, they often relied on “shortcuts” for object recognition (Geirhos et al., Reference Geirhos, Jacobsen, Michaelis, Zemel, Brendel, Bethge and Wichmann2020). For example, a DNN called the Vision transformer (ViT) relied on background features, like grass, to recognize a hare, whereas human participants focused almost exclusively on the animal's head (Fig. 1a). Even more concerning is that the visual features and strategies of humans and DNNs are becoming increasingly misaligned as newer DNNs become more accurate (Fig. 1b). We and others have observed similar trade-offs between DNN accuracy on ImageNet and their ability to explain various human behavioral data and psychophysics (Fel, Felipe, Linsley, & Serre, Reference Fel, Felipe, Linsley, Serre, Koyejo, Mohamed, Agarwal, Belgrave, Cho and Oh2022; Kumar, Houlsby, Kalchbrenner, & Cubuk, Reference Kumar, Houlsby, Kalchbrenner and Cubuk2022). Our work indicates that the mismatch between DNN and biological vision identified by Bowers et al. is pervasive and worsening.

Figure 1 (Linsley and Serre). A growing misalignment between biological vision and DNNs (adapted from Fel et al., Reference Fel, Felipe, Linsley, Serre, Koyejo, Mohamed, Agarwal, Belgrave, Cho and Oh2022). (a) Diagnostic features for object classification differ between humans and DNNs. (b) The Spearman correlation between human and DNN feature importance maps is decreasing as a function of DNN accuracy on ImageNet. This trade-off can be addressed with the neural harmonizer — a method for explicitly aligning DNN representations with humans for object recognition.

The next generation of DNNs for biological vision

Bowers et al. argue that the inability of DNNs to learn human-like visual strategies reflects architectural limitations. They are correct that there is a rich literature demonstrating how mechanisms inspired by neuroscience can improve the capabilities of DNNs, helping them learn perceptual grouping (Kim, Linsley, Thakkar, & Serre, Reference Kim, Linsley, Thakkar and Serre2020; Linsley, Kim, Ashok, & Serre, Reference Linsley, Kim, Ashok and Serre2019a; Linsley, Kim, Veerabadran, Windolf, & Serre, Reference Linsley, Kim, Veerabadran, Windolf, Serre, Bengio, Wallach, Larochelle, Grauman, Cesa-Bianchi and Garnett2018, Reference Linsley, Malik, Kim, Govindarajan, Mingolla, Serre, Ranzato, Beygelzimer, Dauphin, Liang and Vaughan2021), visual reasoning (Kim, Ricci, & Serre, Reference Kim, Ricci and Serre2018; Vaishnav et al., Reference Vaishnav, Cadene, Alamia, Linsley, VanRullen and Serre2022; Vaishnav & Serre, Reference Vaishnav and Serre2023), robust object recognition (Dapello et al., Reference Dapello, Marques, Schrimpf, Geiger, Cox, DiCarlo, Larochelle, Ranzato, Hadsell, Balcan and Lin2020), and to more accurately predict neural activity (Bakhtiari, Mineault, Lillicrap, Pack, & Richards, Reference Bakhtiari, Mineault, Lillicrap, Pack and Richards2021; Kubilius et al., Reference Kubilius, Schrimpf, Nayebi, Bear, Yamins and DiCarlo2018; Nayebi et al., Reference Nayebi, Bear, Kubilius, Kar, Ganguli, Sussillo and Yamins2018). The other fundamental difference between DNNs and biological organisms is how they learn; humans and DNNs learn from vastly different types of data with presumably different objective functions. We believe that the limitations raised by Bowers et al. result from a mismatch in data diets and objective functions because we were able to significantly improved the alignment of DNNs with humans by introducing ClickMe data into their training routines (“Neural harmonizer,” Fig. 1).

Biologically inspired data diets and objective functions

We believe that the power of DNNs for biological vision is from their ability to generate computational- and algorithmic-level hypotheses about vision, which will guide experiments to identify plausible circuits. For instance, the great success of gradient descent and backpropagation for training DNNs has inspired the search for biologically plausible approximations (Lillicrap, Santoro, Marris, Akerman, & Hinton, Reference Lillicrap, Santoro, Marris, Akerman and Hinton2020). Visual neuroscience is similarly positioned to benefit from DNNs if we can improve their alignment with biology.

The most straightforward opportunity for aligning DNNs with biological vision is to train them with more biologically plausible data and objective functions (Smith & Slone, Reference Smith and Slone2017 Richards et al., Reference Richards, Lillicrap, Beaudoin, Bengio, Bogacz, Christensen and Kording2019). There have been efforts to do this with first-person video, however, these efforts have failed to yield much benefit in computer vision or other aspects of biological vision (Orhan, Gupta, & Lake, Reference Orhan, Gupta, Lake, Larochelle, Ranzato, Hadsell, Balcan and Lin2020; Sullivan, Mei, Perfors, Wojcik, & Frank, Reference Sullivan, Mei, Perfors, Wojcik and Frank2021; Zhuang et al., Reference Zhuang, Yan, Nayebi, Schrimpf, Frank, DiCarlo and Yamins2021), potentially because the small scale of these datasets makes them ill-suited for training DNNs. An alternative approach is to utilize advances in three-dimensional (3D) computer vision, like neural radiance fields (Mildenhall et al., Reference Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng2020), to generate spatiotemporal (and stereo) datasets for training DNNs that are infinitely scalable and can be integrated with other modalities, such as somatosensation and language. It is also very likely that objective functions that will lead to human-like visual strategies and features from these datasets have yet to be discovered. However, promising directions include optimizing for slow feature analysis (Wiskott & Sejnowski, Reference Wiskott and Sejnowski2002) and predictive coding (Lotter, Kreiman, & Cox, Reference Lotter, Kreiman and Cox2016; Mineault, Bakhtiari, Richards, & Pack, Reference Mineault, Bakhtiari, Richards, Pack, Ranzato, Beygelzimer, Dauphin, Liang and Vaughan2021), which could help align DNNs with humans without relying on ClickMe data.

Aligned DNNs may be all we need

Bowers et al. point out a number of ways in which DNNs fail as models of biological vision. These problems are pervasive and likely caused by the standard image datasets and training routines of DNNs, which are guided by engineering rather than biology. Bowers et al. may well be right that an entirely new class of models is needed to account for biological vision, but at the moment there are no viable alternatives. Until other model classes can rival human performance on visual tasks, we suspect that the most productive path forward toward modeling biological vision and aligning DNNs with biological vision is to develop more biologically plausible data diets and objective functions.