Hostname: page-component-78c5997874-xbtfd Total loading time: 0 Render date: 2024-11-20T01:07:59.816Z Has data issue: false hasContentIssue false
  • English
  • Français

Reservoir computing and the Sooner-is-Better bottleneck

Published online by Cambridge University Press:  02 June 2016

Stefan L. Frank  and
Hartmut Fitz
Stefan L. Frank
Affiliation:
Centre for Language Studies, Radboud University Nijmegen, 6500 HD Nijmegen, The Netherlands. [email protected]
Hartmut Fitz
Affiliation:
Max Planck Institute for Psycholinguistics, 6500 AH Nijmegen, The Netherlands. [email protected]/people/fitz-hartmut
Get access Rights & Permissions [Opens in a new window]

Abstract

Prior language input is not lost but integrated with the current input. This principle is demonstrated by “reservoir computing”: Untrained recurrent neural networks project input sequences onto a random point in high-dimensional state space. Earlier inputs can be retrieved from this projection, albeit less reliably so as more input is received. The bottleneck is therefore not “Now-or-Never” but “Sooner-is-Better.”

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 39 , 2016 , e73
Copyright
Copyright © Cambridge University Press 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bi, G. & Poo, M. (2001) Synaptic modification of correlated activity: Hebb's postulate revisited. Annual Review of Neuroscience 24:139–66.Google Scholar
Buonomano, D. V. & Maass, W. (2009) State-dependent computations: Spatiotemporal processing in cortical networks. Nature Reviews Neuroscience 10:113–25.Google Scholar
Christiansen, M. H. & Chater, N. (1999) Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23:157–205.Google Scholar
Dominey, P. F., Hoen, M., Blanc, J.-M. & Lelekov-Boissard, T. (2003) Neurological basis of language and sequential cognition: Evidence from simulation, aphasia and ERP studies. Brain and Language 86:207–25.CrossRefGoogle ScholarPubMed
Fitz, H. (2011) A liquid-state model of variability effects in learning nonadjacent dependencies. In: Proceedings of the 33rd Annual Conference of the Cognitive Science Society, Boston, MA, July 2011, ed. Carlson, L., Hölscher, C. & Shipley, T., pp. 897–902. Cognitive Science Society.Google Scholar
Frank, S. L. & Bod, R. (2011) Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science 22:829–34.CrossRefGoogle ScholarPubMed
Gerstner, W., Kistler, W. M., Naud, R. & Paninski, L. (2014) Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press.CrossRefGoogle Scholar
Hinaut, X. & Dominey, P. F. (2013) Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network simulation study using reservoir computing. PLOS ONE 8(2):e52946.CrossRefGoogle ScholarPubMed
Jaeger, H. & Haas, H. (2004) Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304:78–80.Google Scholar
Lukoševičius, M. & Jaeger, H. (2009) Reservoir computing approaches to recurrent neural network training. Computer Science Review 3:127–49.CrossRefGoogle Scholar
Maass, W., Natschläger, T. & Markram, H. (2002) Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation 14:2531–60.Google Scholar
Mongillo, G., Barak, O. & Tsodyks, M. (2008) Synaptic theory of working memory. Science 319:1543–46.Google Scholar
Petersson, K. M. & Hagoort, P. (2012) The neurobiology of syntax: Beyond string sets. Philosophical Transactions of the Royal Society B 367:1971–83.CrossRefGoogle ScholarPubMed
Rabinovich, M., Huerta, R. & Laurent, G. (2008) Transient dynamics for neural processing. Science 321:48–50.CrossRefGoogle ScholarPubMed
Rigotti, M., Barak, O., Warden, M. R., Wang, X. -J., Daw, N. D., Miller, E. K. & Fusi, S. (2013) The importance of mixed selectivity in complex cognitive tasks. Nature 497:585–90.Google Scholar
Singer, W. (2013) Cortical dynamics revisited. Trends in Cognitive Sciences 17:616–26.Google Scholar