Bowers et al. argue that we need to go beyond hypothesis-blind benchmarking in assessing models of vision. On these points we agree: Benchmarking massively complex models on small and arbitrary brain and behavioural datasets is unlikely to yield satisfying outcomes. The space of models and stimuli is too vast, many models score similarly (e.g., Storrs et al., Reference Storrs, Kietzmann, Walther, Mehrer and Kriegeskorte2021a), even bad models can score highly in constrained settings, and the approach gives little insight into what model get right or wrong about biological vision. Bowers et al. advocate for the importance of hypothesis-driven research, which is something with which we also agree. However, this is also fraught with challenges for which the authors offer little in the way of solutions. Where do principled hypotheses come from? What theoretical considerations should constrain the hypotheses we consider? We suggest that the different answers that have been proposed to such questions can provide insight into the role that deep neural networks (DNNs) might play in understanding perception and the brain more broadly.

One approach is to begin with what our visual systems seem to do and work backwards to “reverse engineer” the brain. This approach was advocated by Marr, and has been taken up by Bayesian approaches that treat visual processes a set of “natural tasks.” The idea is that natural selection shaped our brains to approximate “ideal observers” of some set of environmental properties. But natural selection can only act retrospectively; it provides no insight into the genesis of the “options” that it “chooses” between. In the case of vision, it could select between brains that were better at extracting some world properties than others, but it provides no insight into how brains discovered that those environmental properties exist. Something more is needed to explain how the “natural tasks” the brain putatively solves were discovered.

It is here we believe the idea of “data-driven” processes plays a fundamental role. The only known mechanism for getting knowledge about the world into our heads is through our senses. As our brains were not given a list of scene variables they need to estimate, they had to discover properties of the world based on the “diet” of images they experienced over the course of evolution and development. This simple (and seemingly tautological) assertion has a profound theoretical ramification: It implies that anything our brains extract about the world must be based on information contained in, or derivable from, the input. Two routes to defining “principled hypotheses” about visual function follow from this: (1) we should identify what that information is, that is, explore how what we experience about the world relates to what exists in the input; and (2) we should identify how sensitivity to that information was acquired (learned) over the course of evolution and development, that is, explore the mechanisms that underlie sensitivity to these quantities. These two ideas are not independent. Understanding how information can be learned from images can provide insight into what is learned, and understanding what information is used can constrain attempts to construct a learning process that could become sensitive to it.

To ground this idea, consider an example from our recent work. Psychophysical studies revealed that the subjective perception of surface gloss depends not only on the physical specular reflectance of surfaces, but also on other, physically independent scene properties such as shape and illumination (Ho, Landy, & Maloney, Reference Ho, Landy and Maloney2008). Further experiments revealed that that these perceptual errors were caused by differences in the spatial structure and distribution of specular reflections in the image, which relates our experience of a world property (gloss) to properties of images (Marlow, Kim, & Anderson, Reference Marlow, Kim and Anderson2012). We then showed that a system trained to recover “ground truth” (i.e., trained to learn a mapping between images and gloss) failed to predict human judgements. In contrast, unsupervised DNNs, designed to summarise and predict properties of the input, learned representations exhibiting the same pattern of successes and errors in perceived gloss as humans (Storrs, Anderson, & Fleming, Reference Storrs, Anderson and Fleming2021b). In essence, the unsupervised DNN partially – but imperfectly – disentangled the different scene variables (here, gloss, shape, and illumination) in similar ways as our visual system. DNNs thus provided insight into how such illusions could result from an (imperfect) learning process.

How might we go about discovering the information in images that our visual systems use, and what role might DNNs play in hypothesis formation and/or model development? Supervised DNNs of the variety Bowers et al. focus on are inherently teleological; they start with a goal, and coerce the system towards that goal through explicit training of the distinctions it wants the system to make (Yamins & DiCarlo, Reference Yamins and DiCarlo2016). Unsupervised or self-supervised DNNs are techniques for finding similarities and differences between general features based on statistical properties of the input. One view of such networks is that they are generalised “covariance detectors,” which are driven largely by how different image properties do or do not covary. The idea that the visual system derives information about scene properties from the way that different types of image structure covary has provided recent leverage in understanding how the brain extracts the shape and material properties of surfaces (Anderson & Marlow, Reference Anderson and Marlow2023; Marlow & Anderson, Reference Marlow and Anderson2021; Marlow, Mooney, & Anderson, Reference Marlow, Mooney and Anderson2019).

What role might DNNs take in evaluating different computational models of vision? We agree that the psychology and psychophysics literatures provide a wealth of excellent starting points for testing candidate computational models of vision. However, human ingenuity is not always well suited to designing stimuli or experiments that can differentiate between multiple computationally complex alternative models – and such complexity will be unavoidable if we seek to capture human vision in even broad strokes. Therefore, we also see value in using automated selection of complex stimuli to maximally differentiate complex models (e.g., Golan, Raju, & Kriegeskorte, Reference Golan, Raju and Kriegeskorte2020; Wang & Simoncelli, Reference Wang and Simoncelli2008). More broadly, deep learning provides a means to instantiate different hypotheses about how vision is acquired, and the impacts this has on the mature visual system. There is a lot more to deep learning than benchmarking.