A robotic model of hippocampal reverse replay for reinforcement learning
Source abstract
Abstract
Abstract Hippocampal reverse replay, a phenomenon in which recently active hippocampal cells reactivate in the reverse order, is thought to contribute to learning, particularly reinforcement learning (RL), in animals. Here, we present a novel computational model which exploits reverse replay to improve stability and performance on a homing task. The model takes inspiration from the hippocampal-striatal network, and learning occurs via a three-factor RL rule. To augment this model with hippocampal reverse replay, we derived a policy gradient learning rule that associates place-cell activity with responses in cells representing actions and a supervised learning rule of the same form, interpreting the replay activity as a ‘target’ frequency. We evaluated the model using a simulated robot spatial navigation task inspired by the Morris water maze. Results suggest that reverse replay can improve performance stability over multiple trials. Our model exploits reverse reply as an additional source for propagating information about desirable synaptic changes, reducing the requirements for long-time scales in eligibility traces combined with low learning rates. We conclude that reverse replay can positively contribute to RL, although less stable learning is possible in its absence. Analogously, we postulate that reverse replay may enhance RL in the mammalian hippocampal-striatal system rather than provide its core mechanism.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Memory and Neural Mechanisms · Neuroscience and Neuropharmacology Research · Neuroinflammation and Neurodegeneration Mechanisms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (67)
Show source-provided references
1. Reverse replay of hippocampal place cells is uniquely modulated by changing reward10.1016/j.neuron.2016.07.047 ↗
2. Control of a humanoid NAO robot by an adaptive bioinspired cerebellar module in 3D motion tasks10.1155/2019/4862157 ↗
3. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength and postsynaptic cell type10.1523/JNEUROSCI.18-24-10464.1998 ↗
4. Running large-scale simulations on the neurorobotics platform to understand vision–the case of visual crowding10.3389/fnbot.2019.00033 ↗
5. Prospective coding by spiking neurons10.1371/journal.pcbi.1005003 ↗
6. Remembering the past and imagining the future: a neural model of spatial memory and imagery10.1037/0033-295X.114.2.340 ↗
7. Documentation for the MiRo-E robot
8. Sequence learning in a single trial: a spiking neurons model based on hippocampal circuitry10.1109/TNNLS.2021.3049281 ↗
9. Forward and reverse hippocampal place-cell sequences during ripples10.1038/nn1961 ↗
10. Disruption of ripple-associated hippocampal activity during rest impairs spatial learning in the rat10.1002/hipo.20707 ↗
11. Adaptation of short-term plasticity parameters via error-driven learning may explain the correlation between activity-dependent synaptic properties, connectivity motifs and target specificity10.3389/fncom.2014.00175 ↗
12. Reverse replay of behavioural sequences in hippocampal place cells during the awake state10.1038/nature04587 ↗
13. Neuromodulated spike-timing-dependent plasticity and theory of three-factor learning rules10.3389/fncir.2015.00085 ↗
14. D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons10.1126/science.2147780 ↗
15. Eligibility traces and plasticity on behavioral time scales: experimental support of NeoHebbian three-factor learning rules10.3389/fncir.2018.00053 ↗
16. Selective suppression of hippocampal ripples impairs spatial memory10.1038/nn.2384 ↗
17. Hippocampal reactivation extends for several hours following novel experience10.1523/JNEUROSCI.1950-18.2018 ↗
18. VTA neurons coordinate with the hippocampal reactivation of spatial experience10.7554/eLife.05360 ↗
19. Mechanisms for selection of basic motor programs–roles for the striatum and pallidum10.1016/j.tins.2005.05.004 ↗
20. A computational model of action selection in the basal ganglia. I. A new functional anatomy10.1007/PL00007984 ↗
21. A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour10.1007/PL00007985 ↗
22. Recurrent network model for learning goal-directed sequences through reverse replay10.7554/eLife.34171 ↗
23. Influence of dopamine on GABA release in striatum: evidence for D1–D2 interactions and non-synaptic influences10.1016/S0306-4522(96)00475-7 ↗
24. Dynamics of learning and recall at excitatory recurrent synapses and cholinergic modulation in rat hippocampal region CA310.1523/JNEUROSCI.15-07-05249.1995 ↗
25. Neural networks and physical systems with emergent collective computational abilities10.1073/pnas.79.8.2554 ↗
26. The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy and reward10.1016/j.pneurobio.2009.11.003 ↗
27. Awake hippocampal sharp-wave ripples support spatial memory10.1126/science.1217230 ↗
28. From grid cells and visual place cells to multimodal place cell: a new robotic architecture10.3389/fnbot.2015.00001 ↗
29. Alterations in acetylcholine release in the rat hippocampus during sleep-wakefulness detected by intracerebral dialysis10.1016/0024-3205(90)90300-G ↗
30. A systematic review on reinforcement learning-based robotics within the last decade10.1109/ACCESS.2020.3027152 ↗
31. Reinforcement learning in robotics: a survey10.1177/0278364913495721 ↗
32. (Reinforcement?) Learning to forage optimally10.1016/j.conb.2017.08.008 ↗
33. A survey of deep learning applications to autonomous vehicle control10.1109/TITS.2019.2962338 ↗
34. Modeling place cells and grid cells in multi-compartment environments: entorhinal–hippocampal loop as a multisensory integration circuit10.1016/j.neunet.2019.09.002 ↗
35. Obstacle avoidance using stereo vision and deep reinforcement learning in an animal-like robot
36. Signal neutrality, scalar property and collapsing boundaries as consequences of a learned multi-timescale strategy10.1371/journal.pcbi.1009393 ↗
37. The basal ganglia: focused selection and inhibition of competing motor programs10.1016/s0301-0082(96)00042-1 ↗
38. Biomimetic robots as scientific models: a view from the whisker tip
39. Miro: a robot “mammal” with a biomimetic brain-based control system
40. Place units in the hippocampus of the freely moving rat10.1016/0014-4886(76)90055-8 ↗
41. The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat10.1016/0006-8993(71)90358-1 ↗
42. Fast and flexible sequence induction in spiking neural networks via rapid excitability changes10.7554/eLife.44324 ↗
43. The ventral striatum in off-line processing: ensemble reactivation during sleep and modulation by hippocampal ripples10.1523/JNEUROSCI.0575-04.2004 ↗
44. Spatial representation for navigation in animats10.1177/105971239600400201 ↗
45. Memory and mental time travel in humans and social robots10.1098/rstb.2018.0025 ↗
46. A robot model of the basal ganglia: behavior and intrinsic processing10.1016/j.neunet.2005.06.049 ↗
47. Prescott · 2018
48. Obstacle avoidance through reinforcement learning
49. MiRo: social interaction and cognition in an animal-like companion robot
50. Phasic dopamine signaling in action selection and reinforcement learning
51. Democratic population decisions result in robust policy-gradient learning: a parametric study with GPU simulations10.1371/journal.pone.0018539 ↗
52. Transition between encoding and consolidation/replay dynamics via cholinergic modulation of can current: a modeling study10.1002/hipo.22429 ↗
53. A neural substrate of prediction and reward10.1126/science.275.5306.1593 ↗
54. Predictive reward signal of dopamine neurons10.1152/jn.1998.80.1.1 ↗
55. Is there a geometric module for spatial orientation? insights from a rodent navigation model10.1037/a0016170 ↗
56. Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience10.1126/science.271.5257.1870 ↗
57. Sutton · 2018
58. Neural networks with dynamic synapses10.1162/089976698300017502 ↗
59. Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail10.1371/journal.pcbi.1000586 ↗
60. Emergence of connectivity patterns from long-term and short-term plasticities
61. Emergence of connectivity motifs in networks of model neurons with short- and long-term plastic synapses10.1371/journal.pone.0084626 ↗
62. Egocentric and allocentric representations of space in the rodent brain10.1016/j.conb.2019.11.005 ↗
63. Can robots make good models of biological behaviour?10.1017/S0140525X01000127 ↗
64. Fast reverse replays of recent spatiotemporal trajectories in a robotic hippocampal model10.1007/978-3-030-64313-3_37 ↗
65. Simple statistical gradient-following algorithms for connectionist reinforcement learning10.1007/BF00992696 ↗
66. Reactivation of hippocampal ensemble memories during sleep10.1126/science.8036517 ↗
67. The ingredients of real-world robotic reinforcement learning