Animal behaviours are only reinforced by rewards that closely follow the behaviours within a narrow time window. The reward is coded by the phasic activity of dopaminergic neurons in the ventral tegmental area (VTA) which densely innervates the medium spiny neurons (MSNs) in the nucleus accumbens of the striatum, a key brain area for reinforcement learning in mammals6. Glutamatergic synapses on dendritic spines of the MSNs also receive sensorimotor inputs that evoke action potentials (APs)7. These concurrent neuronal activities of synaptic inputs and APs can induce spike-timing-dependent plasticity (STDP), a major Hebbian learning mechanism. However, it was unknown whether dopamine reinforces the preceding, but not subsequent, STDP within a specific time window to account for the reward actions. Here, we selectively stimulated dopaminergic and glutamatergic inputs on MSNs by optogenetic stimulation of dopaminergic fibres and two-photon uncaging of caged-glutamates paired with APs (STDP)8,9 in MSNs in acute slices of mouse brain. We found that dopamine markedly potentiated spine enlargement, but this only occurred within a narrow time window (0.3-2 s) closely following STDP, which is consistent with behavioural conditioning findings. Imaging of Ca2+, Ca2+/calmodulin-dependent protein kinase II (CaMKII), and protein-kinase A (PKA) revealed that dopamine only activated PKA when preceded by APs, and then PKA activated CaMKII through the dopamine- and cAMP-regulated phosphoprotein 32 kDa (DARPP-32); thus dopamine regulated the degree of structural plasticity. In thin distal dendrites, where spines localize, the time window for PKA activation was narrow due to greater activity of phosphodiesterases (PDEs), which prevented accumulation of cAMP, unless dopamine was preceded by APs. Our results clarify the biological basis of reinforcement learning at the single spine level and reveal the interrelationship between Hebbian and reinforcement plasticity.