Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-669899f699-swprf Total loading time: 0 Render date: 2025-04-25T16:19:55.082Z Has data issue: false hasContentIssue false

5 - The interplay of Pavlovian and instrumental processes in devaluation experiments: a computational embodied neuroscience model tested with a simulated rat

from Part II - The use of artificial neural networks to elucidate the nature of perceptual processes in animals

Published online by Cambridge University Press:  05 July 2011

By
Francesco Mannella ,
Marco Mirolli  and
Gianluca Baldassarre
Edited by
Colin R. Tosh  and
Graeme D. Ruxton
Francesco Mannella
Affiliation:
Consiglio Nazionale delle Ricerche (LOCEN-ISTC-CNR)
Marco Mirolli
Affiliation:
Consiglio Nazionale delle Ricerche (LOCEN-ISTC-CNR)
Gianluca Baldassarre
Affiliation:
Consiglio Nazionale delle Ricerche (LOCEN-ISTC-CNR)
Colin R. Tosh
Affiliation:
University of Leeds
Graeme D. Ruxton
Affiliation:
University of Glasgow
Get access

Summary

Introduction

The flexibility and capacity of adaptation of organisms greatly depends on their learning capabilities. For this reason, animal psychology has devoted great efforts to the study of learning processes. In particular, in the last century a huge body of empirical data has been collected around the two main experimental paradigms of ‘classical conditioning’ (Pavlov, 1927; Lieberman, 1993) and ‘instrumental conditioning’ (Thorndike, 1911; Skinner, 1938; Balleine et al., 2003; Domjan, 2006).

Classical conditioning (also called ‘Pavlovian conditioning’) refers to an experimental paradigm in which a certain basic behaviour such as salivation or approaching (the ‘unconditioned response’ – UR), which is linked to a biologically salient stimulus such as food ingestion (the ‘unconditioned stimulus’ – US), becomes associated to a neutral stimulus like the sound of a bell (the ‘conditioned stimulus’ – CS), after the neutral stimulus is repeatedly presented before the appearance of the salient stimulus. Such acquired associations are referred to as ‘CS-US’ or ‘CS-UR’ associations (Pavlov, 1927; Lieberman, 1993).

Instrumental conditioning (also called ‘operant conditioning’) refers to an experimental paradigm in which an animal, given a certain stimulus/context such as a lever in a cage (the ‘stimulus’ – S), learns to produce a particular action such as pressing the lever (the ‘response’ – R), which produces a certain outcome such as the opening of the cage (the ‘action outcome’ – O), if this outcome is consistently accompanied by a reward such as the access to food.

Type
Chapter
Information
Modelling Perception with Artificial Neural Networks , pp. 93 - 113
Publisher: Cambridge University Press
Print publication year: 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Armony, J. L., Servan-Schreiber, D., Romanski, L. M. & LeDoux, D. J. J. E. 1997. Stimulus generalization of fear responses: effects of auditorycortexlesions in a computational model and in rats. Cereb Cortex 7(2), 157–165.CrossRefGoogle Scholar
Baldassarre, G. 2008. Self-organization as phase transition in decentralized groups of robots: a study based on Boltzmann entropy. In Advances in Applied Self-Organizing Systems (ed. M. Prokopenko), pp. 127–146. Springer-Verlag.Google Scholar
Balleine, B. W., Killcross, A. S. & Dickinson, A. 2003. The effect of lesions of the basolateral amygdala on instrumental conditioning. J Neurosci 23(2), 666–675.CrossRefGoogle ScholarPubMed
Balleine, B. W. & Killcross, S. 2006. Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci 29(5), 272–279.CrossRefGoogle ScholarPubMed
Barto, A., Singh, S. & Chentanez, N. 2004. Intrinsically motivated learning of hierarchical collections of skills. In International Conference on Developmental Learning (ICDL), LaJolla, CA.Google Scholar
Baxter, M. G. & Murray, E. A. 2002a. The amygdala and reward. Nat Rev Neurosci 3(7), 563–573.CrossRefGoogle ScholarPubMed
Baxter, M. G. & Murray, E. A. 2002b. The amygdala and reward. Nature Rev Neurosci 3(7), 563–573.CrossRefGoogle ScholarPubMed
Blair, H. T., Sotres-Bayon, F., Moita, M. A. P. & LeDoux, J. E. 2005. The lateral amygdala processes the value of conditioned and unconditioned aversive stimuli. Neuroscience 133(2), 561–569.CrossRefGoogle ScholarPubMed
Brody, C., Pouget, A., Shadlen, M. & Zador, A. (Eds.) 2004. Abstracts of Papers Presented at the 2004 Meeting on Computational & System Neuroscience. Cold Spring Harbor Laboratory.Google Scholar
Camazine, S., Deneubourg, J. L., Franks, N. R., Sneyd, J., Theraulaz, G. & Bonabeau, E. (Ed.) 2001. Self-organization in Biological Systems. Princeton University Press.Google Scholar
Cardinal, R. N., Parkinson, J. A., Hall, J. & Everitt, B. J. 2002. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 26(3), 321–352.CrossRefGoogle ScholarPubMed
Cisek, P. 2007. Cortical mechanisms of action selection: the affordance competition hypothesis. Phil Trans R Soc B 362(1485), 1585–1599.CrossRefGoogle ScholarPubMed
Clark, A. 1997. Being There: Putting Brain, Body and World Together Again. MIT Press.Google Scholar
Darwin, C. 1859. The Origin of Species. Retrieved from http://www.literature.org/authors/darwincharles/the-origin-of-species/index.html.
Dayan, P. & Balleine, B. 2002. Reward, motivation and reinforcement learning. Neuron 36, 285–298.CrossRefGoogle ScholarPubMed
Domjan, M. 2006. Principles of Learning and Behaviour. Thomson Wadsworth.Google Scholar
Haber, S. N., Fudge, J. L. & McFarland, N. R. 2000. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20(6), 2369–2382.CrossRefGoogle ScholarPubMed
Holland, O. & McFarland, D. 2001. Artificial Ethology. Oxford University Press.Google Scholar
Houk, J. C., Adams, J. L. & Andrew, G. B. 1995. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Models of Information Processing in the Basal Ganglia (ed. Houk, J. C., Davids, J. L. & D. G. Beiser), pp. 249–270. MIT Press.Google Scholar
Knight, D. C., Nguyen, H. T. & Bandettini, P. A. 2005. The role of the human amygdala in the production of conditioned fear responses. Neuroimage 26(4), 1193–1200.CrossRefGoogle ScholarPubMed
Kobayashi, Y. & Okada, K.-I. 2007. Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Ann N Y Acad Sci 1104, 310–323.
Langton, C. (Ed.) 1987. The First International Conference on the Simulation and Synthesis of Living Systems (ALifeI).
Lieberman, D. A. 1993. Behavior and Cognition. Brooks/Cole.Google Scholar
Mannella, F., Mirolli, M. & Baldassarre, G. 2007. The role of amygdala in devaluation: a model tested with a simulated robot. In Proceedings of the Seventh International Conference on Epigenetic Robotics (ed. Berthouze, L., Prince, C. G., Littman, M., Kozima, H. & Balkenius, C.), pp. 77–84. University of Lund.Google Scholar
Mannella, F., Zappacosta, S. & Baldassarre, G. 2008. A computational model of the amygdala nuclei's role in second order conditioning. In Proceedings of the Tenth International Conference on Simulation of Adaptive Behavior: From Amimals to Animals 10 (ed. Tani, M. A. J., Hallam, J. & Meyer, J.-A.). Springer-Verlag.Google Scholar
Maren, S. 2005. Building and burying fear memories in the brain. Neuroscientist 11(1), 89–99.CrossRefGoogle Scholar
McDonald, A. J. 1998. Cortical pathways to the mammalian amygdala. Prog Neurobiol 55(3), 257–332.CrossRefGoogle ScholarPubMed
Meyer, J.-A. & Wilson, S. W. (Ed.) 1991. From Animals to Animats 1: Proceedings of the First International Conference on Simulation of Adaptive Behaviour. MIT Press.
Mogenson, G. J., Jones, D. L. & Yim, C. Y. 1980. From motivation to action: functional interface between the limbic system and the motor system. Prog Neurobiol 14(2–3), 69–97.CrossRefGoogle ScholarPubMed
Morén, J. & Balkenius, C. 2000. A computational model of emotional learning in the amygdala. In From Animals to Animats 6: Proceedings of the 6th International Conference on the Simulation of Adaptive Behaviour (ed. Meyer, J.-A., Berthoz, A., Floreano, D., Roitblat, H. L. & Wilson, S. W.). MIT Press.Google Scholar
Newell, A. 1973. You can't play 20 questions with nature and win: projective comments on the papers of this symposium. In Visual Information Processing (ed. Chase, W. G.), pp. 283–308. Academic Press.
Niv, Y., Daw, N. D., Joel, D. & Dayan, P. 2007. Tonic dopamine: opportunity costs and the control of response vigor. J Psychopharmacol 191(3), 507–520.CrossRefGoogle ScholarPubMed
Nolfi, S. 2006. Behaviour as a complex adaptive system: on the role of self-organization in the development of individual and collective behaviour. ComplexUs 2(3–4), 195–203.CrossRefGoogle Scholar
Nolfi, S. & Floreano, D. 2000. Evolutionary Robotics: The Biology, Intelligence, and Technology. MIT Press.Google Scholar
O'Reilly, R., Frank, M., Hazy, T. & Watz, B. 2007. PVLV: The primary value and learned value pavlovian learning algorithm. Behav Neurosci 121, 31–49.CrossRefGoogle ScholarPubMed
Packard, M. G. & Knowlton, B. J. 2002. Learning and memory functions of the basal ganglia. Annu Rev Neurosci 25, 563–593.CrossRefGoogle ScholarPubMed
Parisi, D. 2004. Internal robotics. Connection Sci 16(4), 325–338.CrossRefGoogle Scholar
Parisi, D., Cecconi, F. & Nolfi, S. 1990. Econets: Neural networks that learn in an environment. Network 1, 149–168.CrossRefGoogle Scholar
Pavlov, I. P. 1927. Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex. Oxford University Press.Google Scholar
Pitkänen, A., Jolkkonen, E. & Kemppainen, S. 2000. Anatomic heterogeneity of the rat amygdaloid complex. Folia Morphol. (Warsz) 59(1), 1–23.Google ScholarPubMed
Prescott, T. J., Gonzalez, F. M., Humphries, M. & Gurney, K. 2003. Towards a methodology for embodied computational neuroscience. In Proceedings of the Symposium on Scientific Methods for the Analysis of Agent-Environment Interaction (AISB2003). AISB Press.Google Scholar
Prescott, T. J., Gonzalez, F. M. M., Gurney, K., Humphries, M. D. & Redgrave, P. 2006. A robot model of the basal ganglia: behavior and intrinsic processing. Neural Netw 19(1), 31–61.CrossRefGoogle ScholarPubMed
Price, J. L. 2003. Comparative aspects of amygdala connectivity. Ann N Y Acad Sci 985(1), 50–58.CrossRefGoogle ScholarPubMed
Redgrave, P., Prescott, T. J. & Gurney, K. 1999. The basal ganglia: a vertebrate solution to the selection problem?J Neurosci 89(4), 1009–1023.CrossRefGoogle ScholarPubMed
Rolls, E. T. 2005. Taste and related systems in primates including humans. Chem Senses 30 Suppl. 1, i76–i77.CrossRefGoogle ScholarPubMed
Schembri, M., Mirolli, M. & Baldassarre, G. 2007. Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In Proceedings of the 6th International Conference on Development and Learning (ICDL) (ed. Demiris, Y., Mareschal, D., Scassellati, B. & Weng, J.), pp. E1–6. Imperial College London.Google Scholar
Schultz, W. 2002. Getting formal with dopamine and reward. Neuron 36, 241–263.CrossRefGoogle ScholarPubMed
Sejnowski, T. J., Koch, C. & Churchland, P. S. 1988. Computational neuroscience. Science 241(4871), 1299–1306.CrossRefGoogle ScholarPubMed
Shi, C. & Davis, M. 1999. Pain pathways involved in fear conditioning measured with fear potentiated startle: lesion studies. J Neurosci 19(1), 420–430.CrossRefGoogle ScholarPubMed
Skinner, B. F. 1938. The Behavior of Organisms. Appleton-Century-Crofts.Google Scholar
Sutton, R. S. & Barto, A. G. 1981. Toward a modern theory of adaptive networks: Expectation and prediction. Psychol Rev 88, 135–140.CrossRefGoogle Scholar
Sutton, R. & Barto, A. 1998. Reinforcement Learning: An Introduction. MIT Press.Google Scholar
Thorndike, E. L. 1911. Animal Intelligence. Transaction Publishers.Google Scholar
Weng, J., McClelland, J., Pentland, A.et al. 2001. Autonomous mental development by robots and animals. Science 291, 599–600.CrossRefGoogle ScholarPubMed
Yin, H. H. & Knowlton, B. J. 2006. The role of the basal ganglia in habit formation. Nat Rev Neurosci 7, 464–476.CrossRefGoogle ScholarPubMed
Zlatev, J. & Balkenius, C. 2001. Introduction: Why epigenetic robotics? In Proceedings of the First International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems (ed. Balkenius, C., Zlatev, J., Kozima, H., Dautenhahn, K. & Breazeal, C.), University of Lund. pp. 1–4.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×