Image classifiers implemented with deep convolutional neural networks (DCNNs) have been taken by many to tell against language-of-thought (LoT) architectures. Quilty-Dunn et al. argue that this is a mistake. These image classifiers exhibit deep structural disanalogies to human vision, so, whether or not they implement LoT architectures tells us little about human vision. This is perhaps unsurprising, because biological vision is plausibly not optimized solely for image classification (Bowers et al., Reference Bowers, Malhorta, Dujmović, Montero, Tsvetkov, Biscione and Blything2022, p. 10). Would training artificial vision under more ecologically realistic conditions produce a more realistic model of human vision? To make progress on this question, I describe some reinforcement-learning (RL) agents trained to navigate artificial three-dimensional environments on the basis of how things appear from their perspective, and explain why we might expect their vision to be more human-like. Interestingly, language-like representations seem to be especially helpful to these agents. They explore more effectively, more quickly learn novel tasks, and are even facilitated in downstream image classification. These models arguably provide some indirect evidence for the language-of-thought hypothesis (LoTH) about human vision, and may offer some clues as to why LoT architectures arose evolutionarily.

What is biological vision optimized for, and what would artificial vision that was similarly optimized be like? One answer to the first question is that biological vision is optimized for an agent's success in their environment. Success requires a number of competences that vision must contribute to simultaneously. Agents need to effectively explore, learn new behaviors, and act to achieve their goals, all while the environment changes in often surprising ways.

Recent work in RL arguably more closely approximates the optimization problem facing biological agents. Artificial RL agents can learn to do many complex tasks, across a variety of environments – most interestingly, in this context, exploring and pursuing goals in artificial three-dimensional environments like Habitat (Savva et al., Reference Savva, Kadian, Maksymets, Zhao, Wijmans, Jain and Batra2019), Matterport3D (Chang et al., Reference Chang, Dai, Funkhouser, Halber, Nießner, Savva and Zhang2017), Gibson Env (Xia et al., Reference Xia, Zamier, He, Sax, Malik and Savarese2018), Franka Kitchen (Gupta et al., Reference Gupta, Kumar, Lynch, Levine and Hausman2019), VizDoom (Kempka et al., Reference Kempka, Wydmuch, Runc, Toczek and Jaśkowski2016), Playroom (Tam et al., Reference Tam, Rabinowitz, Lampinen, Roy, Chan, Strouse and Hill2022), and City (Tam et al., Reference Tam, Rabinowitz, Lampinen, Roy, Chan, Strouse and Hill2022). One way of accomplishing this – especially in environments where environmental reward is sparse – is by making novelty intrinsically rewarding. These “curious agents” can learn, without supervision, representations that enable them to perform navigation tasks, interact with objects, and also perform better than baseline in image recognition tasks (Du, Gan, & Isola, Reference Du, Gan and Isola2021). As the authors put it, their agents are “learning a task-agnostic representation for different downstream interactive tasks” (Du et al., Reference Du, Gan and Isola2021, p. 10409).

One challenge these researchers face is how to characterize novelty. Superficial differences in viewing angle or pixel distribution can easily be rated as highly novel, leading to low-level exploration that does not serve learning conducive to achieving goals. A recent innovation is to equip RL agents with “prior knowledge, in the form of abstractions derived from large vision-language models” (Tam et al., Reference Tam, Rabinowitz, Lampinen, Roy, Chan, Strouse and Hill2022, p. 2). Doing so enables the state space over which novelty is defined to be characterized by abstract, semantic categories, such that novelty is defined in task-relevant ways (Mu et al., Reference Mu, Zhong, Raileanu, Jiang, Goodman, Rocktäschel and Grefenstette2022). This method has been shown to substantially improve performance across a variety of tasks and environments, compared to nonlinguistic ways of characterizing the state space (Mu et al., Reference Mu, Zhong, Raileanu, Jiang, Goodman, Rocktäschel and Grefenstette2022; Schwartz et al., Reference Schwartz, Tennenholtz, Tessler and Mannor2019; Tam et al., Reference Tam, Rabinowitz, Lampinen, Roy, Chan, Strouse and Hill2022). The improvements are especially pronounced for tasks involving relations between objects, for example, “Put an OBJECT on a {bed, tray}” (Tam et al., Reference Tam, Rabinowitz, Lampinen, Roy, Chan, Strouse and Hill2022, p. 2), reminiscent of work on relations reviewed in the target article (Hafri & Firestone, Reference Hafri and Firestone2021). As the authors note, their training on vision–language representations that encode “objects and relationship” instead of on ImageNet – optimized for classification – should be expected to be more successful (Tam et al., Reference Tam, Rabinowitz, Lampinen, Roy, Chan, Strouse and Hill2022, p. 10).

Why would linguistic categories facilitate performance? One possibility is that language compresses the state space in ways that facilitate successful actions. The semantic categories enshrined in natural language tend to abstract from action-irrelevant variation, and respect action-relevant variation. So, visual processing optimized relative to natural language categories is de facto optimized for action-relevant distinctions. The LoT architecture characteristic of object files and visual working memory seems well-suited to serving this function (though LoT plausibly is importantly different from natural languages; Green, Reference Green2020; Mandelbaum et al., Reference Mandelbaum, Dunham, Feiman, Firestone, Green, Harris and Quilty-Dunn2022). Predicating abstract properties of individual objects in a LoT is poised to guide action, because abstract semantic categories often determine the action affordances available for some individual object, independent of nuisance variation associated with, for example, viewing angle (though viewing angle is plausibly relevant for more fine-grained control tasks; Parisi et al., Reference Parisi, Rajeswaran, Purushwalkam and Gupta2022, p. 6). Such abstract, task-agnostic representations are also able to transfer to new tasks or environments, in which familiar kinds take on novel relevance for action.

These recent innovations in RL arguably offer indirect support for the LoTH as applied to humans. Of course, similar performance can be achieved by distinct underlying competence, and we should not exaggerate how similar even artificial RL agents’ performance actually is to humans at present. Nevertheless, language-like structures appear especially helpful for artificial agents when faced with rather more biologically plausible optimization problems than the one that faces image classifiers. Perhaps an LoT served our ancestors similarly in an evolutionary context. Language-like structures enabled creatures to encode abstract properties in a task-agnostic way, which nevertheless facilitated downstream performance on a wide variety of tasks, as the environment changed. It's not hard to imagine why evolution might see to it that such a system stuck around.