Simple model for sequence prediction based on dendritic spatiotemporal integration
DOI:
https://doi.org/10.20535/SRIT.2308-8893.2018.4.11Keywords:
sequence prediction, dendritic nonlinearity, association memoryAbstract
Recent experiments on dendritic spatiotemporal integration reveal the much bigger computational potential of a single neuron. An individual dendritic branch can work as a coincidence detector due to a dendritic spike initiated with locally spatially and temporally activated synapses. Here, we investigate a proposed idea that dendrites can perform temporal integration on behavior timescale ~1s, thus weakening simultaneous activation constraint. We construct the model of the recurrent neural network where each neuron activates not as a weighted summation of inputs, but due to their coincident activation both in space and time. We show that with using sparse distributed representation and tracking activity of the network in a certain time window it is possible to achieve a high capacity prediction system. We perform the theoretical analysis and estimate the capacity for the different parameters of the model where even the network with 100 neurons can store millions of sequences. Such a capacity results in a biologically unrealistic high number of synapses, much more than 100×100. However, this mechanism of tracking space-time coincidences in sparse activation can be realized in a limited biological neural network but still with a good sequence transition memory.References
Clegg B.A. Sequence learning / B.A. Clegg, G.J. DiGirolamo, S.W. Keele // Trends Cogn. Sci. — 1998. — Vol. 2, N 8. — P. 275–281.
Bhalla U.S. Dendrites, Deep Learning, and Sequences in the Hippocampus / U.S. Bhalla // Hippocampus. — 2017. — Vol. 2014, N 6.
Sun R. Sequence learning: from recognition and prediction to sequential decision making / R. Sun, C.L. Giles // IEEE Intell. Syst. — 2001. — Vol. 16, N 4. — P. 67–70.
London M. Dendritic Computation / M. London, M. Häusser // Annu. Rev. Neurosci. — 2005. — Vol. 28, N 1. — P. 503–532.
Branco T. The single dendritic branch as a fundamental functional unit in the nervous system / T. Branco, M. Häusser // Curr. Opin. Neurobiol. — 2010. — Vol. 20, N 4. — P. 494–502.
Sjöström P.J. Dendritic Excitability and Synaptic Plasticity / P.J. Sjöström, A. Rancz, A. Roth et al. // Physiol. Rev. — 2008. — Vol. 88, N 2. — P. 769–840.
Kastellakis G. Synaptic clustering within dendrites: An emerging theory of memory formation / G. Kastellakis, D.J. Cai, S.C. Mednick et al. // Prog. Neurobiol. — 2015. — Vol. 126. — P. 19–35.
Branco T. Dendritic discrimination of temporal input sequences in cortical neurons / T. Branco, B.A. Clark, M. Häusser // Science. — 2010. — Vol. 329, N 5999. — P. 1671–1675.
Bhalla U.S. Synaptic input sequence discrimination on behavioral timescales mediated by reaction-diffusion chemistry in dendrites / U.S. Bhalla // Elife. — 2017. — Vol. 6. — P. 1–24.
Kastellakis G. Linking Memories across Time via Neuronal and Dendritic Overlaps in Model Neurons with Active Dendrites / G. Kastellakis, A.J. Silva, P. Poirazi // Cell Rep. — 2016. — Vol. 17, N 6. — P. 1491–1504.
Frank A.C. Hotspots of dendritic spine turnover facilitate clustered spine addition and learning and memory / A.C. Frank, S. Huang, M. Zhou et al. // Nat. Commun. — Vol. 9, N 1. — P. 422. — 2018/
He K. Distinct Eligibility Traces for LTP and LTD in Cortical Synapses / K. He, M. Huertas, S.Z. Hong et al. // Neuron. — 2015. — Vol. 88, N 3. — P. 528–538.
Singer Y. Sensory cortex is optimised for prediction of future input / Y. Singer, Y. Teramoto, B.D.B. Willmore et al. — 2017.
Dasgupta S. A neural algorithm for a fundamental computing problem / S. Dasgupta, C.F. Stevens, S. Navlakha // Science. — 2017. — Vol. 358, N 6364. — P. 793–796.
Bose J. An associative memory for the on-line recognition and prediction of temporal sequences / J. Bose, S.B. Furber, J.L. Shapiro // Proc. Int. Jt. Conf. Neural Networks. — 2005. — Vol. 2. — P. 1223–1228.
Kanerva P. Sparse Distributed Memory / P. Kanerva. — MIT Press, 1988.
Kanerva P. Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors / P. Kanerva // Cognit. Comput. — 2009. — Vol. 1, N 2. — P. 139–159.
Barlow H. Redundancy reduction revisited / H. Barlow // Netw. Comput. Neural Syst. — 2001. — Vol. 12, N 3. — P. 241–253.
Knill D.C. The Bayesian brain: The role of uncertainty in neural coding and computation / D.C. Knill, A. Pouget // Trends Neurosci. — 2004. — Vol. 27, N 12. — P. 712–719.
Clark A. Whatever next? Predictive brains, situated agents, and the future of cognitive science / A. Clark // Behav. Brain Sci. — 2013. — Vol. 36, N 3. — P. 181–204.
Song S. Competitive Hebbian learning through spike-timing-dependent synaptic plasticity / S. Song, K.D. Miller, L.F. Abbott // Nat. Neurosci. — 2000. — Vol. 3, N 9. — P. 919–926.
Brea J. Matching Recall and Storage in Sequence Learning with Spiking Neural Networks / J. Brea, W. Senn, J.-P. Pfister // J Neurosci. — 2013. — Vol. 33, N 23. — P. 9565–9575.
Izhikevich E.M. Polychronization: computation with spikes / E.M. Izhikevich // Neural Comput. — 2006. — Vol. 18, N 2. — P. 245–282.
Ahmad S. How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites / S. Ahmad, J. Hawkins // arXiv. — P. arXiv:1601.00720 [q-NC], Jan. 2016.
Hawkins J. Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in Neocortex / J. Hawkins, S. Ahmad // Front. Neural Circuits. — 2016. — Vol. 10, N March. — P. 1–20.
Lang K.J. The development of the time delay neural network architecture for speech recognition / K.J. Lang, G.E. Hinton // Technical Report CMU-CS-88-152, Carnegie Mellon, 1988.
