elifeeLifeeLife2050-084XeLife Sciences Publications, Ltd10103610.7554/eLife.10103610.7554/eLife.101036.3version of recordResearch ArticleNeuroscienceAge-dependent predictors of effective reinforcement motor learning across childhoodHillNayo Mhttps://orcid.org/0000-0001-9710-0291hilln@kennedykrieger.org12TrippHaley Mhttps://orcid.org/0000-0001-9697-15462WolpertDaniel Mhttps://orcid.org/0000-0003-2011-2790345†MaloneLaura Ahttps://orcid.org/0000-0002-9836-822X267†BastianAmy Jhttps://orcid.org/0000-0001-6079-0997bastian@kennedykrieger.org1267†https://ror.org/037zgn354Department of Neuroscience, Johns Hopkins School of MedicineBaltimoreUnited Stateshttps://ror.org/05q6tgt32Kennedy Krieger InstituteBaltimoreUnited Stateshttps://ror.org/00hj8s172Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkUnited Stateshttps://ror.org/00hj8s172Department of Neuroscience, Columbia UniversityNew YorkUnited Stateshttps://ror.org/00hj8s172Kavli Institute for Brain Science, Columbia UniversityNew YorkUnited Stateshttps://ror.org/037zgn354Department of Neurology, Johns Hopkins School of MedicineBaltimoreUnited Stateshttps://ror.org/037zgn354Department of Physical Medicine and Rehabilitation, Johns Hopkins School of MedicineBaltimoreUnited StatesDekkerTessaReviewing Editorhttps://ror.org/02jx3x895University College LondonUnited KingdomMakinTamar RSenior Editorhttps://ror.org/013meh722University of CambridgeUnited Kingdom
Across development, children must learn motor skills such as drawing with a crayon. Reinforcement learning, driven by success and failure, is fundamental to such sensorimotor learning. It typically requires a child to explore movement options along a continuum (grip location on a crayon) and learn from probabilistic rewards (whether the crayon draws or breaks). We studied the development of reinforcement motor learning using online motor tasks to engage children aged 3–17 years and adults (cross-sectional sample, N=385). Participants moved a cartoon penguin across a scene and were rewarded (animated cartoon clip) based on their final movement position. Learning followed a clear developmental trajectory when participants could choose to move anywhere along a continuum and the reward probability depended on the final movement position. Learning was incomplete or absent in 3–8 year-olds and gradually improved to adult-like levels by adolescence. A reinforcement learning model fit to each participant identified two age-dependent factors underlying improvement across development: an increasing amount of exploration after a failed movement and a decreasing level of motor noise. We predicted, and confirmed, that switching to discrete targets and deterministic reward would improve 3–8 year-olds’ learning to adult-like levels by increasing exploration after failed movements. Overall, we show a robust developmental trajectory of reinforcement motor learning abilities under ecologically relevant conditions, that is, continuous movement options mapped to probabilistic reward. This learning may be limited by immature spatial processing and probabilistic reasoning abilities in young children and can be rescued by reducing task demands.
reinforcement learningmotor learningdevelopmentResearch organismHumanhttp://dx.doi.org/10.13039/100000002National Institutes of HealthT32 HD007414BastianAmy Jhttp://dx.doi.org/10.13039/100000002National Institutes of HealthR35 NS122266BastianAmy JNIH NICDH/NCMRRR25 HD105583HillNayo MThe funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.Author impact statementReinforcement motor learning of probabilistic tasks shows a protracted developmental trajectory in childhood due to high motor noise and low exploration, though performance deficits can be ameliorated in younger children by reducing task demands in spatial processing and probabilistic reasoning.publishing-routeprcIntroduction
In the game of Poohsticks, invented by A. A. Milne and described in The House at Pooh Corner, two children each drop a stick into a stream from the upstream side of a bridge (Milne and Shepard, 1928). They then race to the downstream side to see whose stick appears first, with the winner scoring a point. The game is repeated with each child trying to find the sweet spot to drop their stick in to win. Given the capricious nature of streams with their turbulent flow, dropping both sticks in exactly the same spot on two successive games can lead to different outcomes. To be an expert, a child must use probabilistic success and failure feedback to select a location along a continuous and infinite set of possible options (anywhere along the span of the bridge) to drop their stick to maximize reward. Despite the complexity of this probabilistic reinforcement task, the current world champion is 9 years old. Here, we examine how children develop the ability to learn such tasks from reinforcement feedback alone.
Reinforcement motor learning is essential for many sensorimotor tasks and is typically driven by (scalar) reward feedback in contrast to vector errors used in adaptation learning. Reward feedback can be simple binary feedback, such as success or failure in hitting the space bar on a keyboard, or continuous, such as the height achieved on a swing. Critically, the learner is not told what to do but must discover which movements produce reward by trying different options (Sutton and Barto, 2018). Therefore, a key component of reinforcement learning is exploring and evaluating feedback to maximize reward.
The basic capacity for reinforcement motor learning emerges early in life. For example, a 9-week-old infant will increase kicking frequency when a ribbon connects their foot to an overhead mobile that moves with their kicks (Rovee and Rovee, 1969). Three-month-olds can even learn to produce a specific kick height to move a mobile (Sargent et al., 2014). Both tasks have a deterministic relationship between the action and the outcome; a correct kick is always rewarded. In a more complex probabilistic reaching task, children aged 3 to 9 years old showed different sensitivities to reward probability (Stevenson and Weir, 1959). The youngest aged children were more likely to stick on a rewarded target even if the reward rate was low (e.g. 33%), whereas older children explored other targets. This suggests that younger children are less likely to explore new options and are more willing to accept lower reward rates.
Reinforcement learning has also been studied in cognitive tasks that require children to select from discrete options to receive reward. In a relatively simple task, 2-year-old children could accurately select a reinforced visual image from two choices (de Sousa et al., 2015). More complex tasks have been studied in older children and adolescents identifying age-related changes in learning ability (Xia et al., 2021; Cohen et al., 2020; Schulz et al., 2016; Mayor-Dubois et al., 2016; Master et al., 2020). In one task, participants selected from options in a probabilistic reward environment with hidden factors that could change the outcome. Although children (7–12 years old) and adolescents (13–17 years old) could identify the factors in the environment that changed the outcome, children were unable to use this information to optimize their choices. Similarly, Schulz et al., 2016 used a sequential learning task (8–25 year olds) to show that probabilistic reasoning improves with age. As a whole, this previous work identifies differences between younger and older children on choice-based selection tasks that require integration of reward feedback to learn successfully.
Sensorimotor tasks involve different combinations of motor and cognitive mechanisms (Chen et al., 2017; McDougle et al., 2016), and there are both implicit and explicit (cognitive) contributions to reinforcement motor learning (Holland et al., 2018; van Mastrigt et al., 2023). Cognitive tasks tend to be discrete with arbitrary assignment of reward probability to the explicit choice options (cf. Giron et al., 2023). In contrast, motor learning tasks typically have features that are not present in cognitive tasks. Movement choice options can be continuous (akin to the choice of the direction to kick a soccer ball), corrupted by motor noise, and have a spatial gradient of reward (akin to the probability of scoring as a function of kick direction).
Here, we examine motor learning under reinforcement in which we control two key experimental factors. First, rewards can either be deterministic, the same action leads to the same outcome (e.g. pressing the space bar on a keyboard), or probabilistic, the outcome is stochastic (e.g. the path of a soccer ball depends on the wind and surface on which it is kicked). Second, action options can be discrete (e.g. the space or shift key on a keyboard) or continuous (e.g. the direction of a soccer kick). We report on a series of experiments in which we control both factors —reinforcement feedback (deterministic vs. probabilistic) and the action options (discrete vs. continuous targets) to examine the development of reinforcement learning across childhood. These factors reflect the natural variations in tasks that children have to learn during everyday life and are important factors for rehabilitation interventions. The goal of these tasks is to understand how children at different ages adjust their movements in response to reward feedback. Our study builds on previous work in healthy adults examining center-out reaching movements under binary reward feedback (Cashaback et al., 2019; Therrien et al., 2016; Holland et al., 2018; van der Kooij and Overvliet, 2016).
We developed a remote video game paradigm for a cross-sectional study of 298 children aged 3–17 years and 87 adults from across the USA (locations and demographics shown in Figure 1—figure supplement 1, Tables 1 and 2). We hypothesized that children’s reinforcement learning abilities would improve with age and depend on the developmental trajectory of exploration variability, learning rate (how much people adjust their reach after success), and motor noise (here defined as all sources of noise associated with movement, including sensory noise, memory noise, and motor noise). We think that these factors depend on the developmental progression of neural circuits that contribute to reinforcement learning abilities (Raznahan et al., 2014; Nelson et al., 2000; Schultz, 1998). We found that younger children (3–8 years) failed to learn with a continuous target and probabilistic reward feedback. Reinforcement learning improved with age, enabling older children to find the optimal reward region. Using a mechanistic model, we show that this improvement is due to a developmental gradient of increasing exploration after failure and reducing motor noise with age. Importantly, we then show that use of deterministic feedback and discrete targets can dramatically improve learning abilities of younger children.
Participant demographics.
Kids
Adults
Task
CP
DP
CD
DD
CP
DP
CD
DD
n
111
106
40
41
33
33
10
11
% Female
41
57
38
39
67
58
60
82
% RH
95
96
93
78
97
97
100
91
Age (yrs)
Mean (std)
10 (4.2)
9.5 (4.1)
5.4 (1.7)
5.4 (1.5)
25.2 (4.6)
24.4 (3.9)
24.9 (3.7)
25.9 (4.5)
Median
10
10
5.5
5
24
24
25
27
Range
3–17
3–17
3–8
3–8
18–35
18–31
20–32
18–34
N per age bin
3–5 yrs
21
22
20
21
6–8 yrs
27
21
20
20
9–11 yrs
20
28
12–14 yrs
20
19
15–17 yrs
23
16
Device
Mouse
69
61
15
19
17
22
7
8
Trackpad
16
38
12
14
15
11
3
0
Touchscreen
26
7
13
8
1
0
0
3
Participants included for each task. Abbreviations: CP, continuous probabilistic; CD, continuous deterministic; DD, discrete deterministic; DP, discrete probabilistic; n, number; RH, right-handed; std, standard deviation; yrs, years.
Participant ethnicity and race.
Ethnicity
Race
Hispanic
Not Hispanic
Black
White
Asian
Multiple
Other
Kids n (%)
19 (6.4)
279 (93.6)
15 (5.0)
234 (78.5)
17 (5.7)
24 (8.1)
8 (2.7)
Adults n (%)
3 (3.5)
84 (96.6)
6 (6.9)
34 (39.1)
41 (47.1)
5 (5.8)
1 (1.2)
Ethnicity and race classifications were self-reported by participant/parent. Participants who identified as two or more categories of race (Black, White, and/or Asian) were classified as Multiple. Participants who specified Asian (Indian) or South Asian were classified as Asian. Participants who identified as one or more races other than Black, White, or Asian were classified as Other.
Results
Different groups of participants performed one of four tasks: a continuous probabilistic, a discrete probabilistic, a continuous deterministic, and a discrete deterministic task. We first focus on the continuous probabilistic task and its associated reinforcement learning model, as this is the most complex environment, before reporting the results of the other three tasks.
Continuous probabilistic task
In the continuous probabilistic task, we studied 111 children and adolescents, aged 3–17 years, and 33 adults as they attempted to learn to produce movements that maximized reward. For the key learning phase of the experiment, we created a probabilistic reward landscape in which the probability of reward after each movement depended on its endpoint. To implement this in a task that was engaging to children, we created a computer game that was played at home. Participants were required to move a cartoon penguin from a starting position to join a group of penguins arranged horizontally across the screen (Figure 1a - continuous). Participants were told that there was a slippery ice patch just before the group (dark blue area, Figure 1a - continuous) and that they should try to make the penguin cross at the location where they would not slip. In the instructions, participants were told that there was a location in the ice where they would never slip. The reward landscape (Figure 1b top left) determined the probability of success on each trial and was not known to the participant. There was a small 100% reward zone where the penguin would never slip, with the reward probability decreasing away from this zone. Successful trials led to a short Disney cartoon clip playing, whereas a failed trial led to the penguin falling over and the appearance of a static image of a sad penguin (Figure 1c).
Game environment.
(a) Screenshot of game environment and sample movement path (large text, arrows, and movement path were not displayed to participants). During the learning block, participants either experienced a continuous target (continuous groups) or seven discrete targets (discrete groups). (b) Reward landscape for the learning blocks for the different task paradigms. The x-axis represents normalized units based on the participant’s computer setup. Continuous probabilistic: continuous target with position-based reward probability gradient; discrete probabilistic: discrete targets with target-specific reward probabilities; continuous deterministic: continuous target with a single 100% rewarded zone; discrete deterministic: discrete targets with a single target giving 100% reward. (c) Outcome feedback for continuous probabilistic task. Success (top), movie clip (different for each trial), and pleasant sound plays, and video screen is outlined in blue. Failure (bottom), movie clip does not play, the penguin falls over and red text ‘You slipped!’ appears with a sad face image.
Map of participant locations.
Thirty-eight states of the United States of America are represented in this dataset. The map was generated in Excel on Microsoft 365.
Participants performed five blocks of trials (Figure 2a). Figure 2b shows examples from different aged participants for endpoint position over the experiment. In block 1, participants made 20 movements to a series of discrete targets, which were the same width as the 100% reward zone. The targets appeared at different locations across the scene and participants received reinforcement feedback at the end of each trial (sample paths are shown in Figure 2—figure supplements 1 and 2). The targets were presented in a randomized order, but each participant received the same 20 target locations. This baseline block allowed participants to experience reaching to many locations in the workspace and allowed us to assess accuracy and precision in hitting discrete targets. All participants but one performed well in the baseline block of the task, with endpoint position on average within the target zone (Figure 2c). This shows that participants could accurately hit discrete targets.
Paradigm, example behavior, and target accuracy in the continuous probabilistic task.
(a) Experimental design with baseline: single discrete target presented in randomized locations across the screen; learning: learning block with reward determined by endpoint position; success clamp: feedback clamped to 100% success independent of endpoint position; fail clamp: feedback clamped to 100% failure independent of endpoint position; single target: single discrete target presented in the middle of the screen. (b) Representative endpoint time series from various aged participants. Gray shaded zones indicate positions in the workspace where a reward is given 100% of the time (thin gray lines in first and last blocks are for the discrete targets). Green filled circles indicate rewarded trials while open circles indicate unrewarded trials. The horizontal colored bar on the x-axis indicates the trials corresponding with the experimental blocks outlined in (a). In the learning block (trials 21–120), rewards were given based on the continuous probabilistic landscape. (c) Mean baseline accuracy (average reach deviation from the discrete targets) by age. Adult data (n=33) are averaged and plotted to the right with standard error of the mean. The gray region shows the width of a discrete target. (d) Same as (c) for the single target in block 5. In (c) and (d), participants who completed the task in person (in lab; n=16) are indicated in white circles.
Example baseline paths for participants ages 3–11 years old.
Each trajectory begins at (0,0) and ends at Y=24 when the penguin crosses the back edge of the ice. The final X position of each trajectory corresponds to the interpolated final position of the movement (see Methods for additional details). As available, a sample for each age bin from each input device type is provided. Note that trajectories tend to be straighter for touchscreen input compared to other devices. The 20 squares represent the target centers. Note that the full reward zone is not shown due to overlap between targets. Unrewarded and reward paths are shown as dashed and solid lines, respectively.
Example baseline paths for participants ages 12–17 years old and adults.
Same format as Figure 2—figure supplement 1.
Path length ratios.
The path length ratio is a measure of path curvature (path length divided by distance from first to last point of movement) for the four tasks. Continuous probabilistic samples per age bin: 3-5yrs n=21, 6-8yrs n=27, 9-11yrs n=20, 12-14yrs n=20, 15-17yrs n=23, 18+ n=33. Discrete probabilistic samples per age bin: 3-5yrs n=22, 6-8yrs n=21, 9-11yrs n=28, 12-14yrs n=19, 15-17yrs n=16, 18+ n=33. Continuous deterministic samples per age bin: 3-5yrs n=20, 6-8yrs n=20, 18+ n=10. Discrete deterministic samples per age bin: 3-5yrs n=21, 6-8yrs n=20, 18+ n=11. Bars show mean and standard error of the mean. Significant pairwise comparisons between age bins indicated above plots are as follows: ✳ = p<0.05, + = p<0.01, and Δ = p<0.001.
Timing information.
Reaction time (time from when penguin appeared until the participant clicked on the penguin to start the trial), stationary time (time from click to start of movement), movement time (time from start to end of movement), and game time (time to complete the whole task in minutes) for the four tasks split by age bins. Refer to Figure 2—figure supplement 3 for samples per age bin for each task. Bars show mean and standard error of the mean. Significant pairwise comparisons between age bins indicated above plots as follows: ✳ = p<0.05, + = p<0.01, and Δ = p<0.001.
Participants then began the learning block, where they could move to any endpoint location on the continuous target. Note that the reach location for the first learning trial was not affected by (correlated with) the target position on the last baseline trial (p>0.3 for both children and adults, separately). Figure 2b shows the 5- and 7-year-old children moved to many endpoint locations (i.e. showed exploration) receiving reward based on the probability landscape. Interestingly, their exploration appeared to show little sensitivity to previous failure (open circles) versus success (filled circles) and endpoints did not converge on the 100% reward zone (gray area) by the end of the block. The older children, aged 11, 12, and 15, performed similarly to the adult, exploring early in the block, and then focusing movements on the unseen 100% reward zone. This indicates that they were able to explore successfully after failures and exploit the 100% reward zone. These patterns were also observed in group data (Figure 3, binned across 6 age groups) where, on average, the 3–5 and 6–8 year-olds did not find the 100% reward zone, but the older age groups did.
Continuous probabilistic task learning block time series.
Data and model (red, smooth curves) for each trial of the learning block grouped into age ranges. Data shows mean (solid line) and standard error of mean (shading) of participants’ endpoint. The model in red shows mean (dashed lines) and standard error (shading) from the model simulations. The gray region shows 100% reward zone.
After learning, participants transitioned seamlessly (no break or change in instructions) into two short blocks with clamped feedback to examine behavior in response to repeated success or repeated failure. The success clamp always rewarded movements, and the fail clamp never rewarded movements. The 5- and 7-year-olds explored similarly in both clamped blocks whereas the older children showed greater exploration in the fail clamp compared to the success clamp (Figure 2b).
In the final block, participants moved to a single discrete target in the center of the workspace to examine whether the precision changed over the course of the experiment (examples Figure 2b; group data Figure 2d). For participants aged 3–17 years, an ANOVA of accuracy (distance from discrete target center) by block (first vs. last) and age (5 bins) shows no effect of block (F1,106=0.282, p=0.597) and no interaction between age and block (F4,106=1.219, p=0.307), indicating that accuracy of movement to discrete targets was maintained throughout the experiment even for the younger children.
Note that participants were instructed to reach to the continuous target; there were no instructions regarding the path nor the precise timing of the movements. In younger children, movements tended to be more curved compared to older children, as indicated by larger path length ratios (Figure 2—figure supplement 3). Movement times were similar for all children (Figure 2—figure supplement 4), but younger children took longer to initiate a trial by clicking on the ball (longer Reaction Time) and start moving after initiating (longer Stationary Time). Since these elements of movement were not instructed and did not prevent participants from hitting discrete targets, we focus on the endpoint of the movement in our analyses.
Measures associated with learning in the continuous probabilistic task
We assessed how key features of behavior changed across childhood (Figure 4, with the average of adult performance for comparison). In the Baseline block, the endpoint variability relative to target center (Baseline precision) decreased significantly with age (Figure 4a; regression R2 = 0.153, F1,109 = 19.7, p<0.001). This is consistent with known developmental changes in motor variability (Takahashi et al., 2003; Deutsch and Newell, 2005) and may represent one component of motor noise that is distinct from exploration.
Variability and learning in the continuous probabilistic task.
(a) Baseline precision by age. Average adult variability shown for comparison. (b) Learning block performance by age. Learning measure is the distance from the target, measured as the absolute distance from the 100% reward zone. (c) Endpoint variability in the success clamp by age. (d) Endpoint variability in the failure clamp by age. Regression line with 95% confidence interval shown for children and error bars show standard error of the mean for adults (n=33). Participants who completed the task in person (in lab; n=16) are indicated in white symbols.
Children also improved learning as a function of age by reducing their endpoint distance from the 100% reward zone (distance from target). The distance from the target, averaged over the last 10 reaches, significantly reduced with age (Figure 4b; regression R2 = 0.256, F1,109 = 37.4, p<0.001). Younger children rarely reached the 100% reward zone (gray region), whereas children over 9 years old often did.
We used the clamp blocks to determine how participants responded to success versus failure, and if this could contribute to developing adult-like learning patterns. We would expect participants to explore more after failure versus success. To assess this, we calculated the standard deviation of movement endpoints in each clamp condition as a measure of movement variability (Figure 4c & d). Adults showed high variability after failure and low variability after success, as expected. In children, variability after failure was low in younger children and increased significantly with age (regression R2 = 0.17, F1,109 = 22.3, p<0.001), doubling across the age range to reach adult levels (Figure 4d). In contrast, variability after success was relatively stable across age (regression R2 = 0.012, F1,109 = 1.34, p=0.249). Overall, these results suggest that younger children do not explore as much as older children after failures, which is essential to finding the reward zone.
Comparing reinforcement learning models of the task
To understand how participants update their reach endpoint based on binary feedback, we developed a set of reinforcement learning models of the task. Several reinforcement models have been proposed for tasks similar to ours (Therrien et al., 2018; Therrien et al., 2016; Cashaback et al., 2019; Roth et al., 2023). In the full model, a participant maintains an estimate of their desired reach location (xt\begin{document}$x_{t}$\end{document}) which can be updated across trials. On each trial, the actual reach location (st\begin{document}$s_{t}$\end{document}) is the desired reach with the addition of (possibly) exploration variability (et\begin{document}$e_{t}$\end{document}), planning variability (pt\begin{document}$ p_{t}$\end{document}) and motor noise (mt\begin{document}$m_{t}$\end{document}). Motor noise, as we use it here, can include other sources of noise that do not benefit movement (e.g. sensory and memory noise). The distinction between exploration and planning variability is that exploration is only added if the last trial was unsuccessful (reward rt=1\begin{document}$r_{t}=1$\end{document} for successful trials, and 0 for unsuccessful trials) whereas planning variability is added on all trials:st=xt+(1−rt−1)et+pt+mt\begin{document}$$\displaystyle s_t = x_t + (1-r_{t-1}) e_t+ p_t + m_t$$\end{document}
where the sources of variability are all drawn from zero-mean Gaussians with standard deviations given by σe\begin{document}$ \sigma_{e}$\end{document}, σp\begin{document}$\sigma_{p}$\end{document} and σm\begin{document}$\sigma_{m}$\end{document}.
The probability p\begin{document}$ p$\end{document} of receiving a reward, rt\begin{document}$r_{t}$\end{document}, depends on the actual reach endpoint and the particular reward regime used such that the reward equation isp(rt=1)=f(st)\begin{document}$$\displaystyle p(r_{t}=1) = f(s_t)$$\end{document}
where f()\begin{document}$f()$\end{document} can represent different functions such as the continuous or discrete probabilistic and deterministic reward regimes. The desired reach can then be updated by both planning noise and any exploration noise but not by motor noise:xt+1=xt+ηert(1−rt−1)et+ηprtppt\begin{document}$$\displaystyle x_{t+1} = x_t+ \eta_e r_t (1-r_{t-1}) e_t + \eta_p r^p_t p_t$$\end{document}
where ηp\begin{document}$ \eta_{p}$\end{document} and ηe\begin{document}$ \eta_{e}$\end{document} are the learning rates controlling exploration and planning contribution to the update. In the full model rtp\begin{document}$r^{p}_{t}$\end{document} is the reward on that trial rt\begin{document}$ r_{t}$\end{document} so that both planning and exploration are used to update only after success. In some model variants rtp=1\begin{document}$r^{p}_{t}=1$\end{document} so that the desired reach is always updated by planning variability. By allowing different sources of variability, learning rates, and update rules, there are 30 reduced variants of the full model allowing us to fit all 31 models to the data (see Methods and Figure 5—figure supplement 1).
To identify the best model variant that could account for the data, we performed model selection. We fit the model to the 100 learning trials for each child (see Methods for details), so as not to contaminate model selection by behavior in the clamp blocks or of the adults. We performed BIC comparison combining the BICs from the continuous and discrete probabilistic tasks (which had the largest datasets). Model 11 was the preferred model with a ΔBIC = 216 to the next best model (Figure 5—figure supplement 1). The same model was preferred when we only used BICs from each of the two tasks separately for either the children alone or including the adults. We also considered a value-based model that tries to learn the value of each location. This model is based on the model by Giron et al., 2023 (See Methods). This model was not preferred to Model 11 (ΔBIC = 3018) when fitting the children in the Continuous probabilistic task.
Reinforcement learning model with exploration variability and motor noise
In the preferred model (Model 11 which, from now on, we will refer to simply as the model) there was no planning noise and the learning rate for exploration variability was unity. This model is a special case (with no exploration after success) of the model proposed in Therrien et al., 2018 and has the following output and update processes:st=xt+(1−rt−1e)et+mt\begin{document}$$\displaystyle s_t = x_t + (1-r^e_{t-1}) e_t+ m_t$$\end{document}xt+1=xt+rtet\begin{document}$$\displaystyle x_{t+1} = x_t+ r_t e_t $$\end{document}
In this model, if the previous trial was a failure (Figure 5a, top), the current reach is the desired reach with the addition of exploration variability and motor noise. The desired reach for the next trial is updated only if the current reach is successful. In contrast, if the previous reach was a success (Figure 5a, bottom), the participant does not explore so that the current reach is the desired reach with motor noise. As there is no exploration, the desired reach will remain the same for the next trial whatever the outcome.
Reinforcement learning model for the continuous probabilistic task.
(a) Model schematic. The participant maintains an estimate of the desired reach which they can update across the experiment. The actual reach on the current trial (pink box) depends on whether the previous trial (yellow box) was a failure (top) or success (bottom). After failure (top), the actual reach is the desired reach with the addition of exploration variability and motor noise (draws from zero mean Gaussian distributions with standard deviations σe\begin{document}$\sigma_{e}$\end{document} and σm\begin{document}$\sigma_{m}$\end{document}, respectively). In contrast, if the previous trial was a success (bottom), the participant does not explore so that the actual reach is the desired reach with only motor noise. The actual reach determines the probability of whether the current trial is a failure or a success. If the current trial is a success, the desired reach is updated for the next trial (blue box) by the exploration (if any). (b) Examples of model fits to three participants. The data are shown as circles, with success trials filled green and unsuccessful trials filled white. The estimated desired reach is shown as a thick black line, and the estimated exploration variability (orange line) and motor noise (blue line) connect the desired reach to the data. The simulation of the participant with the fit parameters is shown in the pink line with shading showing one standard deviation across the simulations. (c) Model fit parameters {σm,σe}\begin{document}$\{\sigma_{m},\sigma_{e}\}$\end{document}, by age for the continuous probabilistic group. The line is a regression fit (with 95% confidence interval in shading) to the data for participants younger than 18 years old. The correlation and p-value for each regression are shown in the bottom left corner of each plot (and exclude the adult data). Average adult (n=33) parameters are shown on the right with standard error of the mean.
Model comparison for the continuous and discrete probabilistic tasks.
Difference in Bayesian Information Criterion (BIC) between the preferred model (Model 11) and the other variants for the continuous probabilistic task. The variants depend on the presence (+) or absence (0) of each source of variability (σe,σp,σm\begin{document}$\sigma_e,\sigma_p,\sigma_m$\end{document}); whether the learning rates (ηe,ηp\begin{document}$\eta_e,\eta_p$\end{document}) are absent (.), fit (+), set to unity (1) or ηp=ηe\begin{document}$\eta_p=\eta_e$\end{document}; and whether rp\begin{document}$r_p$\end{document} is absent (.), set to rt\begin{document}$r_t$\end{document} or unity (1). Degrees of freedom of each model is shown by d.o.f. The number of participant who are best fit by each model are shown for the continuous (NC\begin{document}$N_C$\end{document}), discrete (Nd\begin{document}$N_d$\end{document}) and combined (Nc+Nd\begin{document}$N_c +N_d$\end{document}) tasks. When we restricted model selection to only the Continuous Probabilistic task, Model 11 was again preferred with ΔBIC of 155 and 90 for the children alone or all participants. When we restricted model selection to only the Discrete Probabilistic task, Model 11 was again preferred with ΔBIC of 6 and 53 for the children alone or all participants.
Model parameter recovery.
The recovered vs. true parameters for synthetic data generated by the model and then fit. Correlations are shown above the plots.
Example fits of the model to the continuous probabilistic task.
Fits to the same participants shown in Figure 2 in the same format as Figure 5b.
Fits to the success and failure clamp phases for the continuous probabilistic task.
(a) Success clamp standard deviation (s.d.) as a function of age for the data (blue) and model (red) with regression lines with 95% confidence interval shading. (b) Same as (a) for the fail clamp. (c) Model vs. empirical success clamp s.d. with variance explained and correlation with p-value. (d) Same as (c) for the fail clamp.
To gain more power for parameter estimates, we then used this model to fit the 120 trials in the experiment (learning and clamp phases). Having done this, we then examined how well our model fitting procedure could recover parameters from synthetic data. This analysis showed that both parameters (σe,σm\begin{document}$\sigma_{e},\sigma_{m}$\end{document}) were well recovered with correlations with the true parameters of at least 0.97 (Figure 5—figure supplement 2).
Example fits of the model are shown in Figure 5b for three participants. The first participant (top) is an example of a 5-year-old non-learner with minimal exploration variability and large motor noise. The desired reach (black line) is connected to the data (filled and hollow circles for rewarded and unrewarded trials) by the estimates of motor noise (blue vertical lines). The other participants (bottom) show learning. Here the desired reach is connected to the data by either motor noise after a successful reach, or by both motor noise and exploration variability (orange vertical lines) after an unsuccessful reach. The desired reach only changes after a successful trial that was preceded by a failure trial (hence exploration). The pink lines and shading show the mean feedforward simulation and standard deviation. Figure 5—figure supplement 3 shows individual fits to the same participants as in Figure 2.
We examined how the two parameters fit to the data varied with age (Figure 5c). This showed that both parameters varied significantly with age for the children (both p<0.001). Motor noise decreased by about 40% within the age range tested. Exploration variability increased with age, with an almost fourfold increase on average from the youngest to the oldest child. While this change is in the same direction to the variability in the fail clamp, the model allows us to examine exploration variability in the learning setting and separate it from motor noise. Overall, this model analysis suggests that the increased motor noise and reduced exploration limit the ability of the younger children to learn efficiently.
The simulations of the model with the fit parameters accounted for the differences in learning across the age group bins (Figure 3 red line and shading show model simulations). While the BIC analysis with the other model variants provides a relative goodness of fit, it is not straightforward to provide an absolute goodness of fit for probabilistic models such as the one we use here (as explained in the Methods). To provide an overall goodness of fit, we examined the traditional R2 between the average of all the children’s data during learning and the average simulation of the model (repeated 4000 times) for each child so as to reduce the stochastic variation. This analysis gave an R2=0.41\begin{document}$ R^{2}=0.41$\end{document}.
We compared the model fits for the reach variability with the empirical data in the two clamp blocks. Figure 5—figure supplement 4a and b show the variability expected from simulations of the model (red) and the measured movement variability of the children (blue) for the success and fail clamps as a function of age. We also examined the model fits against the empirical data (Figure 5—figure supplement 4c and d), which showed we could explain 16% and 42% of the variance for the success and fail clamp phases (note that we only have 9 trials to calculate each clamp block standard deviation).
Overall, the results from the continuous probabilistic task model results suggest that through development, children’s motor noise decreases, exploration variability increases, and upon success, the desired reach is fully updated by any exploration present.
Discrete probabilistic task
Thus far, we identified features that limit learning in young children within a continuous probabilistic landscape. In a second experiment, we asked if segmenting the continuous target into seven discrete targets could improve learning (Figure 1a – discrete). In this new task, participants were not able to move between targets but were required to definitively reach to a single target on each trial. We predicted that discrete targets could increase exploration by encouraging children to move to a different target after failure. We studied a new cohort of 106 children and 33 adults as they learned the discrete probabilistic task (Figure 1b top right).
Individual example data show that the older children tend to move to a new target after failure, whereas the 3-year-old often chose the same target after failure (Figure 6—figure supplement 1). Discretizing the target appeared to improve the learning for the two youngest age groups (Figure 6a). Learning, baseline precision, and fail clamp variability had similar trends across age as the continuous probabilistic group (Figure 6—figure supplement 2). That is, distance from target decreased with age (R2=0.074, F1,104 = 8.31, p=0.0048), baseline precision improved with age (R2=0.150, F1,104 = 18.4, p<0.001) and fail clamp variability increased with age (R2=0.162, F1,104 = 20.1, p<0.001). In addition, success clamp variability decreased with age (R2=0.048, F1,104 = 5.25, p=0.024).
Discrete probabilistic task learning block time series and parameter fits.
(a) Same format as Figure 3. (b) Panels in same format as Figure 5c with the regressions for the continuous probabilistic task overlaid in blue.
Example behavior and discrete target performance in the discrete probabilistic task.
Same format as Figure 2.
Variability and learning in the discrete probabilistic task.
Same format as Figure 4.
Example fits of the model to the discrete probabilistic task.
Fits to the same participants shown in Figure 6—figure supplement 1 in the same format as Figure 5b.
Fits to the success and fail clamp phases for the discrete probabilistic task.
Same format as Figure 5—figure supplement 4.
We assessed if there were differences in these measures between the discrete and continuous probabilistic groups by comparing regressions as a function of age, with the intercept set to the age of the youngest child (Figure 6—figure supplement 2 vs. Figure 4). This showed there was no significant difference in baseline precision or variability after success. However, the variability after failure had a higher intercept (increase of 41%; p=0.048) for the discrete probabilistic group, consistent with increasing exploration in the younger children. In addition, the intercept was lower for the distance to target (decrease of 41%; p=0.007), consistent with better learning in the younger children with discrete targets.
Modeling the discrete probabilistic task
We fit the model to the discrete probabilistic task. Exploration increased significantly with age (p<0.001; Figure 6b), but the motor noise change with age was not significant. To evaluate the fit parameters by age between the two tasks, we compared the linear regressions (Figure 6b blue line is continuous regression replotted on the discrete parameters). This showed that neither slope nor intercept (set to the youngest child’s age) was significantly different between the groups. In sum, the empirical data showed a significant increase in variability after failure for the younger children. The exploration parameter of the model changed in the expected direction but did not reach significance.
Deterministic reward tasks
In both experiments considered so far, the reward was probabilistic. While this provides gradient information that should guide the participant to the target, younger children may not use the same strategy for this reward landscape. In the final two tasks, we studied new cohorts of 3–8 year-old children since they showed poorest learning in the continuous and discrete probabilistic tasks. We assessed the effect of a deterministic reward landscape (Figure 1b, bottom row) on learning with the continuous and discrete targets.
Figure 7a compares the time courses for the 3–8 year olds across all four tasks. There was no significant difference in mean age between the four tasks (F3,168=1.072, p=0.362) nor was there a significant interaction between task and sex on learning (F3,164=0.97, p=0.409). An ANOVA of final learning (Figure 7 left panel) showed a significant effect of Target type (discrete better than continuous; F1,168=12.87, p<0.001) and Reward landscape (deterministic better than probabilistic; F1,168=43.66, p<0.001) with no interaction (F1,168=0.24, p=0.628). Learning was worst under the continuous probabilistic task, followed by the discrete probabilistic task and continuous deterministic task. The 3–8 year olds performed best on the discrete deterministic task. This shows that making the task discrete or deterministic improves learning and that these factors were additive.
Comparison of the four tasks for the 3- to 8-year-old children.
(a) Learning block performance for the continuous probabilistic, discrete probabilistic, continuous deterministic, and discrete deterministic tasks in the same format as Figure 3. (b) Comparative performance between tasks for precision in baseline, learning distance from target, and variability in the success and fail clamp. Refer to plot titles in (a) for sample size per task. Results from an ANOVA showed that learning significantly improved with discrete targets and deterministic reward feedback. Precision in baseline was not statistically different between tasks. (c) Estimates of motor noise and exploration variability standard deviation from the model. For both (b) and (c), bars show mean and standard error of the mean. Statistically significant pairwise comparisons indicated as follows: ✳ = p<0.05, + = p<0.01, and Δ = p<0.001. Abbreviations: CD = Continuous Deterministic; CP = Continuous Probabilistic; DD = Discrete Deterministic; DP = Discrete Probabilistic.
Model fit parameters by age for the deterministic tasks.
Panels in the same format as Figure 5c. For adults, mean (CD n=10; DD n=11) and standard error of mean plotted for each parameter.
Expected distance as a function of model parameters.
Expected distance to target as a function of motor noise and exploration variability for the four tasks. The green filled circle shows the optimal parameters to maximize reward. The white line shows the optimal exploration variability for different levels of motor noise. The gray shaded line shows the exploration variability vs. age regression line plotted against the motor noise vs. age regression with shading showing the 95% confidence interval. The gray shading shows the participant age in the regression. As the task goes from continuous to discrete, note how the youngest children (darkest end of bar) increase their exploration variability. As the task goes from probabilistic to deterministic, the youngest children decrease their motor noise and increase their exploration variability. Abbreviations: CP, continuous probabilistic; CD, continuous deterministic; DD, discrete deterministic; DP, discrete probabilistic.
This improvement in learning was not due to significant differences in baseline precision (Figure 7b 2nd panel; ANOVA gave Target type: F1,168 = 0.16, p=0.69, Reward landscape: F1,168 = 0.56, p=0.455, interaction: F1,168 = 1.71, p=0.192). For the success clamp block, there was no main effect of Target type (F1,168=0.07, p=0.79), but a main effect of Reward landscape (F1,168=6.71, p=0.01; less variability with a deterministic landscape) with no interaction (F1,168=3.58, p=0.06). This is consistent with there being no advantage to explore after success under a deterministic reward landscape, whereas there is under a probabilistic landscape (exploration can lead to locations that have higher probability of reward). For the fail clamp block, there was a main effect of both Target type (F1,168=29.93, p<0.001; greater variability for discrete targets) and Reward landscape (F1,168=9.26, p=0.003; greater variability with a deterministic landscape) with no interaction (F1,168=1.01, p=0.316). The increased variability for discrete targets is likely because participants must move to a new target to explore, resulting in a larger position change. Increased variability after failure in the deterministic landscape is likely because a failure at one location predicts there will never be success at that location (in contrast to the probabilistic tasks), thereby encouraging exploration. These show that even young children choose their exploration in a rational way based on task features.
Modeling the four tasks and the optimal model
Model fits to the 3–8 year olds are shown in Figure 7a (red lines and shading). Examination of the fit parameters for these groups (Figure 7c) showed that motor noise was significantly affected by Reward landscape (F1,168=35.5, p<0.001) but not by Target type. Exploration variability was significantly affected by Reward landscape (F1,168=60.1, p<0.001) and Target type (F1,168=16.6, p<0.001).
We can use the model to understand the optimal amount of exploration variability for different amounts of motor noise. To do this, we used the model to estimate the average distance to target across the learning phase for different levels of motor noise and exploration variability for the four tasks (Figure 7—figure supplement 2 colormap). These simulations showed that as motor noise increases, the optimal amount of exploration variability decreases (white lines; see Appendix 1 for a proof of this relation). Although the same trend is seen in the model fits to the children’s behavior (gray bars), in general, the children did not produce the optimal amount of exploration.
Task comprehension
Poor or absent learning in the youngest age group could arise from these participants not understanding the task. However, we can rule out task comprehension as an explanation for lack of learning, at least for the majority of younger participants. First, 16 participants completed the continuous probabilistic task in the lab, and this sample included five participants in the 3- to 8-year-old age range. The task was administered in the same way as the online version, without additional instructions or interaction with the experimenter during the task. However, after completion, the experimenter asked these children to explain what they did during the task as an informal comprehension check, and it was clear that this group understood the task. These in-lab participants were able to accurately describe the goal of the task. Performance for all participants tested in the lab was not significantly different from online completion, and these data points are indicated in Figures 2 and 4 with white symbols. Wilcoxon signed-rank tests showed no significant difference between in-lab and online participants in baseline mean (Z = 1.238, p=0.216), baseline precision (Z = –0.626, p=0.532), distance from target (Z = –0.348, p=0.728), success clamp variability (Z = –1.952, p=0.051), fail clamp variability (Z = 0.281, p=0.779), or single target mean (Z = –0.399, p=0.690).
Second, we used the reinforcement learning model to assess whether an individual’s performance is consistent with understanding the task. In the case of a child who does not understand the task, we expect that they simply have motor noise on their reach, and crucially, that they would not explore more after failure, nor update their reach after success. Therefore, we used a likelihood ratio test to examine whether the preferred model was significantly better at explaining each participant’s data compared to the model variant which had only motor noise (Model 1). Focusing on only the youngest children (age 3–5), this analysis showed that 43, 59, 65, and 86% of children (out of n=21, 22, 20, and 21) for the continuous probabilistic, discrete probabilistic, continuous deterministic, and discrete deterministic conditions, respectively, were better fit with the preferred model, indicating non-zero exploration after failure.
In the 3- to 5-year-old group for the discrete deterministic condition, 18 out of 21 had performance better fit by the preferred model, suggesting this age group understands the basic task of moving in different directions to find a rewarding location. The reduced numbers fit by the preferred model for the other conditions likely reflect differences in the task conditions (continuous and/or probabilistic) rather than a lack of understanding of the goal of the task.
Discussion
We studied a large, cross-sectional cohort of children and adolescents aged 3–17 years old and adults performing reinforcement-based motor learning tasks. Binary feedback—success or failure—was the only learning signal provided as participants moved to continuous or discrete targets under probabilistic or deterministic reward landscapes. The continuous target gave them unlimited movement choices across the target zone, whereas discrete targets constrained choices. The movement endpoint position was mapped to reward probability, with the probabilistic landscape varying from 25% to a small 100% reward region, and the deterministic landscapes had a single, small 100% reward zone. We found a developmental trajectory of reinforcement learning in the continuous probabilistic task, with older children learning more than younger children. This was paralleled by age-dependent increases in exploration after a failed trial, which is essential for learning and decreases in baseline movement precision. A mechanistic model of the task revealed that in addition to the increase in exploration, there was a reduction in motor noise with age.
In contrast to the children in the continuous probabilistic group, younger children learned better when the task had discrete targets or deterministic reward. These effects appeared to be additive—the 3- to 8-year-old children learned best with discrete targets and in a deterministic reward landscape. Thus, the youngest children had the fundamental mechanisms for interpreting binary reward signals to drive reinforcement learning of movement—this is not surprising given that this ability has been shown in infants (Rovee and Rovee, 1969). However, children aged 3–8 did not effectively utilize reward signals to learn in situations where they had to respond to probabilistic environments or where there were no spatial cues specifying distinct targets.
Our data suggest that the developmental trajectory we identified was not due to poor motor accuracy or lack of understanding of the task in the younger children. We designed the baseline block to ensure that children could accurately hit individual targets presented in different locations across the screen. The width of these targets was the same as that of the hidden 100% reward zone within the learning block, and children of all ages could hit these targets accurately. The youngest children could also learn similarly to adults in the discrete deterministic task. This shows that children were able to understand the concept of the task and how to be successful.
In reinforcement learning tasks, different sources of movement variability have been proposed, together with a range of models of how these sources are used to update the desired reach for future movements. For example, motor noise has been proposed in several models of control (e.g. Harris and Wolpert, 1998; Todorov and Jordan, 2002; Trommershäuser et al., 2003). Such noise is thought to be unintentional and unavoidable, and studies show that humans tend to plan movements to minimize the bad consequences of motor noise. Other sources of variability are typically considered to be beneficial as they can be used to update reaching behavior. For example, in some models, planning variability (van Beers, 2009) is always added to the movement independent of whether previous trials were successful or not. A third possible source of variability is exploration that is distinguished from planning variability in that it is only present after an unsuccessful movement. Several models have been proposed with different combinations of motor noise, planning, and exploration variability (e.g. Therrien et al., 2016; Therrien et al., 2018; Roth et al., 2023). We were able to unify these previous models as special cases of a more general model. This allowed us to compare 31 model variants, and we found that the preferred model was a special case (with no exploration after success) of the model proposed in Therrien et al., 2018. This suggests that variability in our task is due to motor noise as well as active exploration after failure. The desired reach is updated by the exploration present on a successful trial which was preceded by a failure trial (the necessary condition for exploration). The model differs from the preferred model in Roth et al., 2023 which also includes a learning rate for the update so that only part of exploration is incorporated into the next reach.
One major finding from this study is that exploration variability increases with age. Some other studies of development have shown that exploration can decrease with age, indicating that adults explore less compared to children (Schulz et al., 2019; Meder et al., 2021; Giron et al., 2023). We believe the divergence between our work and these previous findings is largely due to the experimental design of our study and the role of motor noise. In the paradigm used initially by Schulz et al., 2019 and replicated in different age groups by Meder et al., 2021 and Giron et al., 2023, participants push buttons on a two-dimensional grid to reveal continuous-valued rewards that are spatially correlated. Participants are unaware that there is a maximum reward available, and therefore children may continue to explore to reduce uncertainty if they have difficulty evaluating whether they have reached a maxima. In our task, by contrast, participants are given binary reward and told that there is a region in which reaches will always be rewarded. Motor noise is an additional factor that plays a key role in our reaching task but minimal, if any, role in the discretized grid task. As we show in simulations of our task, as motor noise goes down (as it is known to do through development), the optimal amount of exploration goes up (see Figure 7—figure supplement 2 and Appendix 1). Therefore, the behavior of our participants is rational in terms of increasing exploration as motor noise decreases.
A key result is that exploration in our task reflects sensitivity to failure. Older children make larger adjustments after failure compared to younger children to find the highly rewarded zone more quickly. Dhawale et al., 2017 discuss the different contexts in which a participant may explore versus exploit (i.e. stick at the same position). Exploration is beneficial when reward is low as this indicates that the current solution is no longer ideal, and the participant should search for a better solution. Konrad et al., 2025 have recently shown this behavior in a real-world throwing task where 6- to 12-year-old children increased throwing variability after missed trials and minimized variability after successful trials. This has also been shown in a postural motor control task where participants were more variable after non-rewarded trials compared to rewarded trials (van Mastrigt et al., 2020). In general, these studies suggest that the optimal amount of exploration is dependent on the specifics of the task.
We found that turning a continuous target into discrete buttons improved learning performance. In our tasks, a child needs to remember their previous endpoint position and whether it was rewarded to decide where to move at the start of the next trial. We suspect that the continuous target task relies more on spatial working memory, which is not fully developed (van Asselen et al., 2006), particularly in those under 10 years of age (Hitch et al., 1988; Pickering, 2001; Tsujii et al., 2009).
Younger children may benefit from clear markers in the environment to differentiate various spatial positions along the target. This is consistent with tasks that have assessed children’s preferences for reporting information; they find it easier to select from discrete choices on a Likert scale versus using a continuous visual analog scale (van Laerhoven et al., 2004; Shields et al., 2003). Finally, it is also possible that older children had the ability to use different working memory encoding mechanisms. It is known that phonological encoding using verbal labels develops later than visual encoding and both can be used to hold information in working memory (e.g. my last movement was near the left edge of the screen) (Pickering, 2001; Hitch et al., 1988). Future work could explore whether providing an explicit verbal strategy to younger children could improve their ability to explore more effectively with a continuous target, highlighting the interplay of cognitive and motor domains in reinforcement learning.
We also found that deterministic reward feedback improved learning. The deterministic landscape is less ambiguous than the probabilistic landscape by design—participants always fail if they are outside the 100% reward zone. The need to track and evaluate the reward frequency across target zones is eliminated, making the task less complex. Younger children have been reported to take longer to choose a reward maximization strategy in a probabilistic environment where all choices have the possibility of being rewarded. Plate et al., 2018 showed that when children and adults choose between eight options, they initially probability match (i.e. the frequency of selection closely matches the frequency of reward on the various task options). However, adults switch over to a maximizing strategy (i.e. choosing the highest reward option) more quickly than children (Plate et al., 2018). In a deterministic landscape, probability matching would result in the same behavior as reward maximizing and therefore the younger children’s behavior would appear nearly the same as adults. Young children’s behavior may also stem from a fundamental need to prioritize hypothesis testing and gathering information about the world (Schulz et al., 2019; Liquin and Gopnik, 2022; Nussenbaum and Hartley, 2019), a discovery process that facilitates increased knowledge about the environment’s causal structure (Sobel and Sommerville, 2010).
Our interpretation is that poorer learning performance in tasks besides discrete deterministic was due to an inability to effectively utilize probabilistic reward signals or find the high reward location without clearly delineated spatial cues. This has significant ramifications as most objects in the world do not have delineated targets on them; we learn which location on an object leads to reward by exploring different options (e.g. the best location to grab a mug or push a heavy door open). The world is also not deterministic, as it is rare that the same action will always give the same result. Movement outcomes are probabilistic due to both environmental variation and motor noise (e.g. the direction of a soccer ball when kicked on different fields or the location of a thrown dart on a dartboard). Eventually, children must learn how to move successfully to interact with their environments using such probabilistic signals.
The differential ability to incorporate reward signals into changes in behavior across childhood may stem from maturation of the reward centers in the brain. Structures important for these processes, such as the basal ganglia reward centers, dorsal lateral prefrontal cortex, posterior parietal cortex, and the anterior cingulate cortex develop throughout childhood (Raznahan et al., 2014; Nelson et al., 2000; Schultz, 1998). As a main underlying neural structure of reinforcement learning, the pallidum reaches peak volume at 9.5 years for females and 7.7 years for males while it takes the thalamus until 13.8 years for females and 17.4 years for males to reach peak volume (Raznahan et al., 2014). Older children have more mature brain circuits and may be better able to take advantage of strategies in probabilistic environments that younger children cannot (Schulz et al., 2019; Liquin and Gopnik, 2022). For example, older children might know to avoid a location as soon as a failure occurs, even if that location was previously rewarded. Younger children might continue sampling those locations, perhaps due to immature signaling in the brain. Indeed, it has been shown that brain activity in younger children can be similar after positive and negative rewards, whereas in older children and adults, it is more distinct (Mai et al., 2011; Eppinger et al., 2009; van Meel et al., 2005).
Online studies such as ours inevitably have some limitations. Given that this task was conducted remotely, we did not have control of the computer operating quality, internet speed, or testing environment for our participants (see Methods for exclusion criteria). As such, we were not able to control or analyze timing parameters of movement in detail, which can be done more easily in in-person experimentation. However, our key analyses compare across large groups that likely factor out these uncontrolled variables. Our results (comparison of in-lab vs. online and fit of a noise model vs. full reinforcement learning model) also confirm that even the youngest age group understood the task. We also recognize that other processes, such as memory and motivation, could affect performance on these tasks; however, our study was not designed to test these processes directly, and future work would benefit from exploring these other components more explicitly.
Our findings in typical development lay a foundation for better understanding of behavior during childhood and could help inform what function may be lost or impaired after injury. This underscores the need to consider not only disease process for interventions but also age, as there are developmental differences in motor learning capacity in individuals at different ages. Knowing how the sensorimotor system works at different ages can guide decisions on how to intervene or alter an environment and give children the best opportunity to use reinforcement learning mechanisms for rehabilitation outcomes.
Methods
Reagent type (species) or resource
Designation
Source or reference
Identifiers
Additional information
Software, algorithm
MATLAB
MATLAB
R2023a
Software, algorithm
Illustrator
Adobe
Version 11.0
Participants
Children and adults without neurological impairment or developmental delay were recruited to one of four experiments as outlined in Tables 1 and 2. A total of 385 complete datasets were included in the analysis. The continuous probabilistic task was designed first, and the other three tasks, discrete probabilistic, continuous deterministic, and discrete deterministic, were conducted to expand upon the results of the first task and further identify factors contributing to learning by reinforcement across the developmental spectrum. Participants were recruited from the Johns Hopkins University community through an online announcement portal, the greater Baltimore Maryland area through Research Match, and nationwide through the online platform Lookit which merged with Children Helping Science in 2023 (Sheskin et al., 2020; Scott and Schulz, 2017). Our sample includes participants from 38 states out of the 50 United States of America (Figure 1—figure supplement 1). Each participant was screened to rule out neurological impairment, developmental delay, and other developmental disorders that could affect motor and cognitive development. This study was approved by the Johns Hopkins School of Medicine Institutional Review Board and all participants, or their parent/legal guardian, provided informed consent prior to participation.
Task platform
All four tasks were completed on a web-based platform built with Javascript as previously reported by Malone et al., 2023 with modifications to the game environment and feedback type. This platform allowed creativity in the game environment design to be kid-friendly and engaging to young participants to foster sustained attention and increase the likelihood of completing the full task. The task platform allowed remote completion of the experiment by participants on their home computer or tablet. A small subset of participants in the continuous probabilistic task (n=16; 3 to 5yo n=2, 6 to 8yo n=3, 9 to 11yo n=5, 12 to 14yo n=5, 15 to 17yo n=1) completed the task in person in the research laboratory and the remainder of participants completed the task remotely. Participants used a mouse, trackpad, or touchscreen input to control a cartoon penguin game piece and move across the game environment. Movement trajectories were sampled at the polling rate of the selected input device, and data from each trial were uploaded and saved to Google Firebase Realtime Database at the end of each trial. Due to the remote data collection nature of this experiment, we were not able to control data sampling rates. Each trial was manually inspected and sampling rates of input devices had a mean of 37.363 ±4.184 Hz.
Procedure
The game environment is an immersive icy landscape with penguins. The overall goal of the task was to move a penguin from the starting position on one side of the ice (at the bottom of the computer or tablet screen closest to the participant) to a distance of 24 game units (GU) into the game (at the far edge of the screen away from the participant). If successful, a pleasant sound would play, the video screen above the ice would be outlined in blue, and a Disney video clip (different on each trial; gifs hosted on https://giphy.com) would play. Multiple signals of reward were provided to ensure that participants associated their behavior with the feedback provided and could clearly differentiate between a successful trial and a failure trial. The penguin game piece started as a blue color to indicate that it was not active. To activate the penguin, the participant had to click the left mouse button (mouse or trackpad) or touch and hold the penguin, and it would turn white to indicate that it was movable. Then the participant made a reaching movement to move the penguin across the ice. The trial would end when the Y position of the penguin exceeded 24 GU. To account for variability in input device sampling rates, the final X position was computed as an interpolation between the data points immediately before and after Y=24 GU such that every trial had a calculated X position at Y=24 GU.
Rewards were determined based upon the interpolated final X position of the game piece at the back edge of the ice and a task-specific reward landscape. All tasks included five blocks as outlined in Figure 2a. Baseline (20 trials): a single, discrete target (image of three dancing penguins) was presented at a randomized X position at the far edge of the ice. Participants were instructed to move accurately to the target. Learning (100 trials): participants were exposed to one of four learning environments with variations in the target type (continuous vs. discrete, Figure 1a) and the reward feedback (probabilistic vs. deterministic, Figure 1b). In the continuous conditions, participants could choose to move the penguin to any location on a continuous horizontal target (Figure 1a – continuous). In the discrete conditions, participants could move to one of seven targets spread horizontally (Figure 1a – discrete). For probabilistic feedback, the reward was determined by an unseen position-based probability gradient with a small 100% reward zone away from which reward probabilities decreased linearly to a baseline (Figure 1b – continuous probabilistic and discrete probabilistic). For deterministic feedback, reward was always given within a reward zone but not elsewhere (Figure 1b – continuous deterministic and discrete deterministic). Success Clamp (10 trials): every trial in this block was rewarded regardless of the final X position of the game piece. Fail Clamp (10 trials): no trials in this block were rewarded. Single Target (10 trials): the same single, discrete target as in baseline was presented in the center of the far edge of the ice, and participants were cued to move accurately towards it. A break for a new set of instructions was provided between Baseline and Learning as well as Fail Clamp and Single Target. Participants were unaware of the reward criteria transitions between Learning and Success Clamp and Success Clamp and Fail Clamp. The ideal/mature behavior in these tasks was to explore at the beginning of the learning block to find the area of most frequent reward and then exploit this behavior to maximize reward for the remaining trials of the block. Moreover, if the 100% reward zone has been found successfully, continuing to move to this reinforced position during the success clamp and then exploring again in the fail clamp are indicators of mature sensitivity to reward and failure.
Continuous probabilistic
In the Learning block for the continuous probabilistic task, participants were presented with a continuous target and probabilistic reward feedback (Figure 1b – continuous probabilistic). The probability reward landscape is defined by setting reward percentage probabilities at 5 specific x\begin{document}$ x$\end{document} locations with reward between these locations being linearly interpolated (and constant outside the range). We set X values of {−24,−14.5,−9.5,1.1875}\begin{document}$ \{-24,\,-14.5,\,-9.5,\,1.1875\}$\end{document} to reward probabilities {33,100,100,25}\begin{document}$ \{33,\,100,\,100,\,25\}$\end{document}.
Participants were warned that some of the ice was starting to melt, which would cause the penguin to slip, and were told that in some places they would slip a lot of the time, in some places they would slip some of the time, and in some places they would never slip. They were instructed to move the penguin across the ice without slipping to get the movie clip to play as often as possible. If they were not successful, the penguin would fall over, and they would see a sad face penguin image before the game reset for the next trial (Figure 1c). This task design builds from Cashaback et al., 2019 where participants were asked to reach to any location on a continuous line, with the probability of being rewarded dependent on the reach end location. In their task, there was a small (hidden) 100% reward zone with reward probability decreasing on either side away from this zone. They found that adult participants could learn from a probabilistic reward landscape and find the rewarding zone (Cashaback et al., 2019). We explored a similar task design in participants across development.
Discrete probabilistic
In the Learning block for the discrete probabilistic task, participants were presented with a set of targets, each associated with a specific probability of reward (Figure 1b – discrete probabilistic). We set the center of the seven targets at X values of {−18,−12,−6,0,6,12,18}\begin{document}$\{-18,\,-12,\,-6,\,0,\,6,\,12,\,18\}$\end{document} with target width of 5 and with reward percentage probabilities of the {66,100,66,33,25,25,25}\begin{document}$ \{66,\,100,\,66,\,33,\,25,\,25,\,25\}$\end{document}.
The discrete targets were visually the same as those used in the Baseline and Single Target blocks; however, all seven were presented at the same time in equally spaced positions across the edge of the ice. Participants were instructed to find the group of penguins that made the video clip play all the time by moving their penguin game piece across the ice. They were told to move to one group of penguins on each trial and that some penguins would make the movie clip play some of the time, but there was a group of penguins that played the clip all the time. If they were not successful, they would see a red X on the video screen and the video clip would not play before the game reset for the next trial. To ensure that participants could accurately distinguish the feedback associated with each target, there was a visible space between each target. If the participant moved between targets, the participant would receive a message to try again, and the trial would reset until one target was accurately hit.
Continuous deterministic
In the Learning block for the continuous deterministic task, participants were presented with a continuous target and deterministic reward feedback (Figure 1b – continuous deterministic). Participants were warned that some of the ice was starting to melt, which would cause the penguin to slip, and were told that in some places they would slip all of the time and in some places they would never slip. They were instructed to move the penguin across the ice without slipping to get the movie clip to play as often as possible. If they were not successful, the penguin would fall over, and they would see a sad face penguin image before the game reset for the next trial. This task was completed by a subset of participants aged three to eight years and adults.
Discrete deterministic
In the Learning block for the discrete deterministic task, participants were presented with a set of seven discrete targets and deterministic reward feedback (Figure 1b – discrete deterministic). Participants were instructed to move across the ice to one of the groups of penguins on each trial to get the movie clip to play. They were told that one group of penguins would make the video clip play all of the time. If they were not successful, they would see a red X on the video screen and the video clip would not play before the game reset for the next trial. As in the discrete probabilistic task, to ensure that participants could accurately distinguish the feedback associated with each target, there was a space between each target. If the participant moved between targets, the trial would reset until one target was accurately hit. This task was completed by a subset of participants aged three to eight years and adults.
Demo versions of tasks
Shortened versions of each task without data collection are provided at the following links. In these versions, there are 5 trials in baseline, 10 trials in learning, 2 trials in each clamp, and 3 trials of single target. To proceed beyond the information input screen, use an arbitrary six-digit code for the subjectID and participant information to sample the game environment.
We used several metrics to analyze performance and variability in different blocks and evaluate reinforcement learning over childhood. Baseline performance was defined as the signed distance from the target center averaged over each of the 20 baseline trials. Baseline precision was defined as the standard deviation of the baseline performance. Importantly, the width of these baseline targets was the same as the 100% reward zone in the learning block for the continuous target tasks. This allowed determination of both success and precision with moving to targets the width of the goal. Learning performance (Distance from target) is defined as the absolute value of the interpolated X position distance from the center of 100% reward zone (located at X = –12 in all tasks) averaged over the last 10 trials of the learning block. A Distance from target value closer to 0 indicates better learning. To quantify variability after success, we used the standard deviation of the interpolated X position in trials 2 through 10 of the success clamp block. To quantify variability after failure, we used the standard deviation of the interpolated X position in trials 2 through 10 of the fail clamp block.
Participant characteristics (sex and game play handedness) and game play parameters (device and browser) were analyzed in one way ANOVAs with dependent variables of Precision and Distance from target to determine whether these parameters influenced learning ability (Appendix 2—table 1). To determine whether there was a differential effect of sex, device, or handedness on learning performance in the four different tasks, additional two-way ANOVAs with dependent variable of Distance from target were used. There was not a significant interaction between task and sex (reported in Results), task and device (F3,168=0.62, p=0.717), or task and handedness (F3,168=1.2, p=0.312). Each trial was divided into three phases to describe the reaction, stationary, and movement times. Reaction time is defined as the time from the appearance of the penguin (start of the trial) to the time the penguin is clicked and activated. Stationary time is defined as the time from penguin click to movement onset (first non-zero position). Movement time is defined as movement onset to end of the trial when the penguin moves across the back edge of the ice. Trial timings for each phase of movement were extracted and averaged for each participant across the whole experiment and then averaged within individual age bins to evaluate across ages. Total game play time was also extracted and averaged by age bins. Path length ratios were calculated as the actual path length from the start to end of the trajectory divided by the ideal straight-line path from the first position in the trajectory to the last position in the trajectory. The path length ratio for all trials was averaged for each participant and then averaged within age bins for comparison between ages.
We used linear regression and one- and two-way ANOVAs to evaluate effects of age and other outcome variables on learning as well as compare performance between tasks. Significance level of 0.05 was used for all statistical tests. All raw data processing and statistical analyses were completed using custom scripts in MATLAB (version R2023a) (Hill et al., 2025).
Reinforcement learning model
Several reinforcement models have been proposed for tasks similar to ours (Therrien et al., 2018; Therrien et al., 2016; Roth et al., 2023). With some reparameterizing, all these models can be considered as special cases of a ‘full’ model. We fit this full model, as well as all 30 possible variants of the full model. These models can be considered mechanistic models as defined by Levenstein et al., 2023.
In these models, a participant maintains an estimate of their desired reach location (xt\begin{document}$ x_{t}$\end{document}; which reflects the estimated target location) which can be updated across trials. On each trial, the actual real location (st\begin{document}$ s_{t}$\end{document}) is the desired reach with the addition of (possibly) exploration variability (et\begin{document}$ e_{t}$\end{document}), planning variability (pt\begin{document}$ p_{t}$\end{document}) and motor noise (mt\begin{document}$ m_{t}$\end{document}). The distinction between exploration and planning variability/motor noise is that exploration variability is only added if the last trial was unsuccessful (reward rt=1\begin{document}$ r_{t}=1$\end{document} for successful trials, and 0 for unsuccessful trials), whereas planning variability and motor noise is added on all trials:st=xt+(1−rt−1)et+pt+mt\begin{document}$$\displaystyle s_t = x_t + (1-r_{t-1}) e_t+ p_t + m_t$$\end{document}
where the sources of variability are all zero mean Gaussian with standard deviations given by σe\begin{document}$\sigma_{e}$\end{document}, σp\begin{document}$\sigma_{p}$\end{document}, and σm\begin{document}$ \sigma_{m}$\end{document}. There can be other sources of noise (sensory, memory, etc.) and we regard them as being included as part of motor noise (since they are never used to update the reach). The probability p\begin{document}$ p$\end{document} of reward received, rt\begin{document}$r_{t}$\end{document}, depends on the reach location and the particular reward regime used such that the reward equation isp(rt=1)=f(st)\begin{document}$$\displaystyle p(r_{t}=1) = f(s_t) $$\end{document}
where f()\begin{document}$f()$\end{document} can represent different functions such as the continuous or discrete probabilistic and deterministic reward regimes. The desired reach can then be updated by both planning variability and any exploration noise but not by motor noise (this is the key distinction between planning variability and motor noise).xt+1=xt+ηert(1−rt−1)et+ηprtppt\begin{document}$$\displaystyle x_{t+1} = x_t+ \eta_e r_t (1-r_{t-1}) e_t + \eta_p r^p_t p_t$$\end{document}
where ηp\begin{document}$\eta_{p}$\end{document} and ηe\begin{document}$\eta_{e}$\end{document} are the learning rates controlling exploration and planning variability contribution to the update. In the full model rtp\begin{document}$ r^{p}_{t}$\end{document} is the reward on that trial rt\begin{document}$ r_{t}$\end{document} so that both exploration (if present) and planning variability are used to update after success. In some model variants rtp=1\begin{document}$r^{p}_{t}=1$\end{document} so that the desired reach is always updated by planning variability.
The variants of the model determine whether (i) each source of variability (et,pt,mt\begin{document}$e_{t},p_{t},m_{t}$\end{document}) are present or not, (ii) the settings of the learning rates (ηe\begin{document}$\eta_{e}$\end{document} and ηp\begin{document}$\eta_{p}$\end{document} independent, identical, ηe\begin{document}$ \eta_{e}$\end{document} or ηp\begin{document}$\eta_{p}$\end{document} or both set to unity) and (iii) whether rtp\begin{document}$r^{p}_{t}$\end{document} equals rt\begin{document}$r_{t}$\end{document} or 1.
This leads to 80 variants: exploration (2) × planning (2) × motor (2) × learning (5) × (2). However, many of these variants can be excluded, leaving 31 model variants. The exclusions are (i) models with no planning variability and no motor noise (as they show no variability in success clamp for example), (ii) models with no exploration variability but a fit or non-zero ηe\begin{document}$\eta_{e}$\end{document}, (iii) models with no planning variability but a fit or non-zero ηp\begin{document}$\eta_{p}$\end{document}, (iv) models with no planning variability but rtp\begin{document}$ r^{p}_{t}$\end{document} set to rt\begin{document}$r_{t}$\end{document}.
The model variants are shown in Figure 5—figure supplement 1 and some of these variants correspond to previously proposed models (after allowing renaming of variables; for example, a model with exploration variability present on every trial would be regarded as planning variability). Models 23, 22, 30, 10, and 6 are models 1–5, respectively, from Roth et al., 2023. Model 8 is the model from Therrien et al., 2016 and Model 28 is the model in Therrien et al., 2018. Model 20 is the model from Cashaback et al., 2019.
Model fitting
To fit these stochastic models to each participant’s data, we used a Kalman smoother. The full model has 5 parameters: θ={σm,σe,σp,ηe,ηp}\begin{document}$\theta=\{\sigma_{m},\sigma_{e},\sigma_{p},\eta_{e},\eta_{p}\}$\end{document}. We can reformulate the equations above into a standard Kalman update and output equations. The update equation is given by[xtetptmt]=[1ηert−1eηprt−1p0000000000000][xt−1et−1pt−1mt−1]+[000(1−rt−1)σe000σp000σm][vt1vt2vt3]\begin{document}$$\displaystyle \begin{bmatrix}x_t \\e_t \\p_t \\m_t\end{bmatrix}=\begin{bmatrix} 1 & \eta_e r^e_{t-1} & \eta_p r^p_{t-1} & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\\end{bmatrix}\begin{bmatrix}x_{t-1} \\e_{t-1} \\p_{t-1} \\m_{t-1} \\\end{bmatrix}+\begin{bmatrix} 0 & 0 & 0\\ (1-r_{t-1}) \sigma_e & 0 & 0\\ 0 & \sigma_p & 0\\ 0 & 0 & \sigma_m\\\end{bmatrix}\begin{bmatrix}v^1_t\\v^2_t\\v^3_t\\\end{bmatrix}$$\end{document}
where vti∼N(0,1)\begin{document}$v^{i}_{t}\sim\mathcal{N}(0,1)$\end{document}. The output equation is given byst=[1111][xtetptmt]\begin{document}$$\displaystyle s_t =\begin{bmatrix}1 & 1 & 1 & 1 \end{bmatrix}\begin{bmatrix}x_t \\e_t \\p_t \\m_t\end{bmatrix}$$\end{document}
We initialized the states as all zeros and the state covariance as[2020000σe20000σp20000σm2]\begin{document}$$\displaystyle \begin{bmatrix}20^2 & 0 & 0 &0 \\0& \sigma_e^2& 0 &0 \\0 & 0 & \sigma_p^2 &0 \\0 & 0 & 0 &\sigma_m^2 \\\end{bmatrix}$$\end{document}
where the wide prior on the initial reach direction allows the Kalman smoother to fit each participant’s initial reach. We used Matlab fminsearchbnd to fit the parameters to maximize the likelihood (Hill et al., 2025).
Model selection
We first fit each model to maximize the likelihood over the 100 learning trials for each child participant in the continuous probabilistic task. This ensured that our model selection was not contaminated by behavior in the clamp block. We used BIC to select the preferred model (the BIC provides a relative measure of goodness of fit accounting for differences in the number of parameters of each model). The same model (model 11) was preferred if we only considered the children or also included the adults. Moreover, the same model was preferred for both the children in the discrete probabilistic task (or again if we included the adults). We therefore focus on Model 11 which is a special case (with no exploration after success) of the model proposed in Therrien et al., 2018.
Preferred model
In the preferred model, the produced reach is the desired reach with the addition of the variability from motor noise and exploration (if any), given by the output equation:st=xt+(1−rt−1)et+mt\begin{document}$$\displaystyle s_t = x_t +(1-r_{t-1}) e_t+ m_t$$\end{document}
After each reach, the participant updates their desired reach location only if they were successful. They update the reach to incorporate all of the exploration (if any) so that the update equation isxt+1=xt+rt(1−rt−1)et\begin{document}$$\displaystyle x_{t+1} = x_t+ r_t (1-r_{t-1}) e_t$$\end{document}
We examined parameter recovery for the preferred model by generating simulated data (for each participant using their best fit parameters for the preferred model) and compared the fit values with the true values used to simulate the data. This showed correlations were above 0.96 for both parameters.
Having found the preferred model, we used that to fit the 120 trials for each participant in each of the four conditions so as to constrain the parameter fits using all the data. For the discrete tasks, we fit the data in the same way as the continuous (so that the model does not know about the discrete nature of the task).
Model simulations
To simulate a participant, we used the best fit parameter to generate Monte Carlo simulations. We used the estimated posterior of the initial state from the Kalman smoother to initialize both the desired and actual reach by sampling from the appropriate posterior 100,000 times and used these to generate the same number of Monte Carlo simulations. We ensured that the desired and actual reaches were constrained to the workspace by clipping values outside the range. We calculate the median (to be robust) of these runs as well as the standard deviation across runs for plotting.
The model simulations also allowed us to simulate the two clamp phases, and we calculated the average standard deviation of actual reaches in each clamp phase across simulations.
Goodness of fit
While the BIC analysis with the other model variants provides a relative goodness of fit, it is not straightforward to provide an absolute goodness of fit such as standard R2. There are two problems. First, there is no single model output. Each time the model is simulated with the fit parameters, it produces a different output (due to motor noise, exploration variability, and reward stochasticity). Second, the model is not meant to reproduce the actual motor noise, exploration variability, and reward stochasticity of a trial. For example, the model could fit pure Gaussian motor noise across trials (for a poor learner) by accurately fitting the standard deviation of motor noise but would not be expected to actually match each data point and would therefore have a traditional R2 of 0.
To provide an overall goodness of fit, we have to reduce the noise component, and to do so, we examined the traditional R2 between the average of all the children’s data and the average simulation of the model (repeated 1000 times per participant) so as to reduce the stochastic variation.
Optimal exploration variability as a function of motor noise
We used model simulations over a 50 × 50 grid of exploration variability and motor noise to assess the expected learning distance for the 4 tasks. To do this, for each grid location, we simulated each child participant 4000 times (as above, selecting the initial state stochastically for each child) and calculated the expected learning distance over the 100 learning trials across the children. This showed the optimal exploration variability increasing as a function of decreasing motor noise (as also observed with age in the data). For an intuition and proof of this relation, see Appendix 1.
Alternative value-based model
We considered a value-based model which tries to learn the value of each location. This model is based on the model by Giron et al., 2023. In this model, the participant represents the value of each location on each trial vt(x)\begin{document}$ v_{t}(x)$\end{document}. Given the reach locations and rewards received up until trials T, {x1,r1,x2,r2⋅xT,rT}\begin{document}$\{x_{1},r_{1},x_{2},r_{2}\cdot x_{T},r_{T}\}$\end{document} the value function is fit using a Gaussian process with squared exponential kernel and logistic likelihood to give vT(x)\begin{document}$v_{T}(x)$\end{document}. The kernel has two parameters, a length scale σl\begin{document}$\sigma_{l}$\end{document} which determines the generalization of learning to other locations and a strength σs\begin{document}$\sigma_{s}$\end{document} that controls the amount of learning (similar to a learning rate). The value function is passed through the softmax function with parameter b\begin{document}$ b$\end{document} that controls the explorationPt(x)∝expbv(x)\begin{document}$$\displaystyle P_t(x)\propto \exp^{b v(x)}$$\end{document}
where Pt(x)\begin{document}$P_{t}(x)$\end{document} is normalized so as to be the probability of moving to locations x\begin{document}$x$\end{document}.
To fit the models, three parameters {σl,σs,b}\begin{document}$\{\sigma_{l},\sigma_{s},b\}$\end{document} we discretized x\begin{document}$ x$\end{document} into 500 points and used fmincon to maximize the likelihood of the data. We found these optimized parameters were very sensitive to the initial parameter, so we chose 50 random initializations for each participant and selected the one with the highest likelihood. Although we were able to fit the data, unlike the noise-based model, often the parameters were not easily identifiable. For example, there are many ways a participant could fail to learn, such as a large σl\begin{document}$\sigma_{l}$\end{document} or small σs\begin{document}$\sigma_{s}$\end{document}. Since this was not the preferred model, we chose not to examine the parameters for the fits.
Additional informationCompeting interests
No competing interests declared
Author contributions
Conceptualization, Resources, Data curation, Software, Formal analysis, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing
Data curation, Visualization, Writing – review and editing
Conceptualization, Software, Formal analysis, Supervision, Visualization, Writing – original draft, Writing – review and editing
Conceptualization, Supervision, Writing – review and editing
Conceptualization, Resources, Supervision, Funding acquisition, Methodology, Writing – original draft, Writing – review and editing
Ethics
This study was approved by the Johns Hopkins School of Medicine Institutional Review Board and all participants, or their parent/legal guardian, provided informed consent prior to participation.
Additional filesData availability
Behavioral data and scripts used for analysis and modeling are available on Mendeley Data.
The following dataset was generated:
HillNTrippHWolpertDMaloneLBastianA2025Developmental Reinforcement Motor Learning DatasetMendeley Data10.17632/g9z67hkyz5.1
Acknowledgements
This work was supported by the following funding sources: grants from the National Institutes of Health T32 HD007414 and R35 NS122266 to AJB. DMW is a consultant to CTRL-Labs Inc, in the Reality Labs Division of Meta. This entity did not support or influence this work. NMH gained skills in open science and reproducibility as a learner in the 2024–2025 cohort of Reproducible Rehabilitation funded by NIH NICDH/NCMRR R25 HD105583.
ReferencesCashabackJGALaoCKPalidisDJColtmanSKMcGregorHRGribblePL2019The gradient of the reinforcement landscape influences sensorimotor learningPLOS Computational Biology15e100683910.1371/journal.pcbi.100683930830902ChenXMohrKGaleaJM2017Predicting explorative motor learning using decision-making and motor noisePLOS Computational Biology13e100550310.1371/journal.pcbi.100550328437451CohenAONussenbaumKDorfmanHMGershmanSJHartleyCA2020The rational use of causal inference to guide reinforcement learning strengthens with ageNPJ Science of Learning51610.1038/s41539-020-00075-333133638de SousaNMde A GilMSCMcIlvaneWJ2015Discrimination and reversal learning by toddlers aged 15-23 monthsThe Psychological Record65414710.1007/s40732-014-0084-125663716DeutschKMNewellKM2005Noise, variability, and the development of children’s perceptual-motor skillsDevelopmental Review2515518010.1016/j.dr.2004.09.001DhawaleAKSmithMAÖlveczkyBP2017The role of variability in motor learningAnnual Review of Neuroscience4047949810.1146/annurev-neuro-072116-03154828489490EppingerBMockBKrayJ2009Developmental differences in learning and error processing: evidence from ERPsPsychophysiology461043105310.1111/j.1469-8986.2009.00838.x19497006GironAPCirankaSSchulzEvan den BosWRuggeriAMederBWuCM2023Developmental changes in exploration resemble stochastic optimizationNature Human Behaviour71955196710.1038/s41562-023-01662-137591981HarrisCMWolpertDM1998Signal-dependent noise determines motor planningNature39478078410.1038/295289723616HillNTrippHWolpertDMaloneL2025Developmental reinforcement motor learning datasetMendeley Data10.17632/g9z67hkyz5.1HitchGJHallidaySSchaafstalAMSchraagenJMC1988Visual working memory in young childrenMemory & Cognition1612013210.3758/BF03213479HollandPCodolOGaleaJM2018Contribution of explicit processes to reinforcement-based motor learningJournal of Neurophysiology1192241225510.1152/jn.00901.2017KonradJDLohseKRMarrusNLangCE2025Trial-to-trial motor behavior during a reinforcement learning task in children ages 6 to 12Human Movement Science9910331710.1016/j.humov.2024.10331739667095LevensteinDAlvarezVAAmarasinghamAAzabHChenZSGerkinRCHasenstaubAIyerRJolivetRBMarzenSMonacoJDPrinzAAQuraishiSSantamariaFShivkumarSSinghMFTraubRNadimFRotsteinHGRedishAD2023On the role of theory and modeling in neuroscienceThe Journal of Neuroscience431074108810.1523/JNEUROSCI.1179-22.202236796842LiquinEGGopnikA2022Children are more exploratory and learn more than adults in an approach-avoid taskCognition21810494010.1016/j.cognition.2021.10494034715584MaiXTardifTDoanSNLiuCGehringWJLuoY-J2011Brain activity elicited by positive and negative feedback in preschool-aged childrenPLOS ONE6e1877410.1371/journal.pone.001877421526189MaloneLAHillNMTrippHWolpertDMBastianAJ2023A novel video game for remote studies of motor adaptation in childrenPhysiological Reports11e1576410.14814/phy2.1576437434268MasterSLEcksteinMKGotliebNDahlRWilbrechtLCollinsAGE2020Distentangling the systems contributing to changes in learning during adolescenceDevelopmental Cognitive Neuroscience4110073210.1016/j.dcn.2019.10073231826837Mayor-DuboisCZesigerPVan der LindenMRoulet-PerezE2016Procedural learning: A developmental study of motor sequence learning and probabilistic classification learning in school-aged childrenChild Neuropsychology2271873410.1080/09297049.2015.105834726144545McDougleSDIvryRBTaylorJA2016Taking aim at the cognitive side of learning in sensorimotor adaptation tasksTrends in Cognitive Sciences2053554410.1016/j.tics.2016.05.002MederBWuCMSchulzERuggeriA2021Development of directed and random exploration in childrenDevelopmental Science24e1309510.1111/desc.1309533539647MilneAAShepardEH1928The House at Pooh CornerE.P. Dutton & CompanyNelsonCAMonkCSLinJCarverLJThomasKMTruwitCL2000Functional neuroanatomy of spatial working memory in childrenDevelopmental Psychology3610911610.1037/0012-1649.36.1.109NussenbaumKHartleyCA2019Reinforcement learning across development: What insights can we draw from a decade of research?Developmental Cognitive Neuroscience4010073310.1016/j.dcn.2019.10073331770715PickeringSJ2001The development of visuo-spatial working memoryMemory942343210.1080/0965821014300018211594361PlateRCFulvioJMShuttsKGreenCSPollakSD2018Probability learning: changes in behavior across time and developmentChild Development8920521810.1111/cdev.1271828121026RaznahanAShawPWLerchJPClasenLSGreensteinDBermanRPipitoneJChakravartyMMGieddJN2014Longitudinal four-dimensional mapping of subcortical anatomy in human developmentPNAS1111592159710.1073/pnas.1316911111RothAMCalaloJALokeshRSullivanSRGrillSJekaJJvan der KooijKCarterMJCashabackJGA2023Reinforcement-based processes actively regulate motor exploration along redundant solution manifoldsProceedings. Biological Sciences2902023147510.1098/rspb.2023.147537848061RoveeCKRoveeDT1969Conjugate reinforcement of infant exploratory behaviorJournal of Experimental Child Psychology8333910.1016/0022-0965(69)90025-35804591SargentBSchweighoferNKuboMFettersL2014Infant exploratory learning: influence on leg joint coordinationPLOS ONE9e9150010.1371/journal.pone.009150024626202SchultzW1998Predictive reward signal of dopamine neuronsJournal of Neurophysiology8012710.1152/jn.1998.80.1.19658025SchulzEWuCMRuggeriA2016From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learningPsychological Science2784885810.1177/095679761663930127634005SchulzEWuCMRuggeriAMederB2019Searching for rewards like a child means less generalization and more directed explorationPsychological Science301561157210.1177/095679761986366331652093ScottKSchulzL2017Lookit (Part 1): A new online platform for developmental researchOpen Mind141410.1162/OPMI_a_00002SheskinMScottKMillsCMBergelsonEBonawitzESpelkeESFei-FeiLKeilFCGweonHTenenbaumJBJara-EttingerJAdolphKERhodesMFrankMCMehrSASchulzL2020Online developmental science to foster innovation, access, and impactTrends in Cognitive Sciences2467567810.1016/j.tics.2020.06.004ShieldsBJPalermoTMPowersJDGreweSDSmithGA2003Predictors of a child’s ability to use a visual analogue scaleChild2928129010.1046/j.1365-2214.2003.00343.xSobelDMSommervilleJA2010The importance of discovery in children’s causal learning from interventionsFrontiers in Psychology117610.3389/fpsyg.2010.00176StevensonHWWeirMW1959Variables affecting children’s performance in a probability learning taskJournal of Experimental Psychology5740341210.1037/h003925813654658SuttonRSBartoAG2018Reinforcement Learning: An IntroductionThe MIT PressTakahashiCDNemetDRose-GottronCMLarsonJKCooperDMReinkensmeyerDJ2003Neuromotor noise limits motor performance, but not motor adaptation, in childrenJournal of Neurophysiology9070371110.1152/jn.01173.200212904490TherrienASWolpertDMBastianAJ2016Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noiseBrain13910111410.1093/brain/awv32926626368TherrienASWolpertDMBastianAJ2018Increasing motor noise impairs reinforcement learning in healthy individualseNeuro5ENEURO.0050-18.201810.1523/ENEURO.0050-18.201830105298TodorovEJordanMI2002Optimal feedback control as a theory of motor coordinationNature Neuroscience51226123510.1038/nn96312404008TrommershäuserJMaloneyLTLandyMS2003Statistical decision theory and the selection of rapid, goal-directed movementsJournal of the Optical Society of America A20141910.1364/JOSAA.20.001419TsujiiTYamamotoEMasudaSWatanabeS2009Longitudinal study of spatial working memory development in young childrenNeuroreport2075976310.1097/WNR.0b013e32832aa97519352206van AsselenMKesselsRPCNeggersSFWKappelleLJFrijnsCJMPostmaA2006Brain areas involved in spatial working memoryNeuropsychologia441185119410.1016/j.neuropsychologia.2005.10.00516300806van BeersRJ2009Motor learning is optimally tuned to the properties of motor noiseNeuron6340641710.1016/j.neuron.2009.06.02519679079van der KooijKOvervlietKE2016Rewarding imperfect motor performance reduces adaptive changesExperimental Brain Research2341441145010.1007/s00221-015-4540-126758721van LaerhovenHvan der Zaag-LoonenHJDerkxBHF2004A comparison of Likert scale and visual analogue scales as response options in children’s questionnairesActa Paediatrica9383083510.1080/0803525041002657215244235van MastrigtNMSmeetsJBJvan der KooijK2020Quantifying exploration in reward-based motor learningPLOS ONE15e022678910.1371/journal.pone.022678932240174van MastrigtNMTsayJSWangTAvrahamGAbramSJvan der KooijKSmeetsJBJIvryRB2023Implicit reward-based motor learningExperimental Brain Research2412287229810.1007/s00221-023-06683-w37580611van MeelCSOosterlaanJHeslenfeldDJSergeantJA2005Telling good from bad news: ADHD differentially affects processing of positive and negative feedback during guessingNeuropsychologia431946195410.1016/j.neuropsychologia.2005.03.01815878183XiaLMasterSLEcksteinMKBaribaultBDahlREWilbrechtLCollinsAGE2021Modeling changes in probabilistic reinforcement learning during adolescencePLOS Computational Biology17e100852410.1371/journal.pcbi.100852434197447Appendix 1Optimal exploration variability as a function of motor noise
Simulations (Figure 7—figure supplement 2) showed that the optimal exploration variability increases as a function of decreasing motor noise (as also observed with age in the data). Here we provide an intuition and proof for this inverse relation in a simplified setting. The setting is one in which there is a discrete point target offset at +d\begin{document}$ +d$\end{document} from the desired reach (which without loss of generality we take to be at zero, xt=0\begin{document}$x_{t}=0$\end{document}; Appendix 1—figure 1 left panel, red line).
As in the preferred model, we have two independent zero-mean Gaussian noise sources for exploration variability and motor noise:e∼N(0,σe2),m∼N(0,σm2).\begin{document}$$\displaystyle e\sim\mathcal{N}(0,\sigma_{e}^{2}),\quad m\sim\mathcal{N}(0,\sigma_{m}^{2}).$$\end{document}
We will examine how the expected reward depends on motor and exploration variability for the possible scenarios underlying a reach, that is (i) a reach after an unsuccessful trial and (ii) a reach after a successful trial that had exploration (i.e. the desired reach is updated). There is a third possibility, which is a reach after a successful trial that had no exploration (i.e. the previous trial was successful). However, the reaches in such a scenario do not involve exploration variability (only motor noise) and are not germane to the interplay of motor and exploration variability.
Reach after an unsuccessful trial
After an unsuccessful reach, there is no updating of desired reach, and the next reach includes both exploration variability and motor noise, (st=et+mt\begin{document}$s_{t}=e_{t}+m_{t}$\end{document}). Therefore, the distribution of reach locations is given by N(0,σe2+σm2)\begin{document}$ \mathcal{N}(0,\sigma_{e}^{2}+\sigma_{m}^{2})$\end{document} (Fig. 8 left panel, blue line). The probability of hitting the target (success, rt=1\begin{document}$ r_{t}=1$\end{document}) is given byP(rt=1|st)=P(et+mt=d)\begin{document}$$\displaystyle P(r_{t}=1|s_{t})=P(e_{t}+m_{t}=d)$$\end{document}
In general, to maximize the density at d\begin{document}$ d$\end{document} away from the mean of a Gaussian distribution (Appendix 1—figure 1 left panel, green line shows target location), the standard deviation should be d\begin{document}$ d$\end{document}. Therefore, to maximize reward d2=σe2+σm2\begin{document}$d^{2}=\sigma_{e}^{2}+\sigma_{m}^{2}$\end{document}. Rearranging gives σe2=max(0,d2−σm2)\begin{document}$\sigma_{e}^{2}=\max(0,d^{2}-\sigma_{m}^{2})$\end{document} and hence an inverse relation between optimal exploration variability and motor noise, provided |d|>σm\begin{document}$|d| \gt \sigma_{m}$\end{document} (if |d|≤σm\begin{document}$|d|\leq\sigma_{m}$\end{document} it is better to have no exploration). This is the fundamental reason that exploration variability should decrease as motor noise increases, so as to maximize reward.
Distribution of desired reach and actual reach for a reach after an unsuccessful trial (left) and after a successful trial (right).
Plots show probability density across endpoint positions. In both cases, the desired reach on trial t\begin{document}$t$\end{document} is at xt=0\begin{document}$x_{t}=0$\end{document} (thin red line shows distribution as a delta function). After an unsuccessful reach (left panel), the actual reach (blue distribution) includes exploration variability and motor noise. The probability of reward depends on the height of this distribution at the target (d=10\begin{document}$d=10$\end{document}, shown by green line). For a successful trial with exploration (right panel), the distribution (xt+1\begin{document}$x_{t+1}$\end{document} and st\begin{document}$ s_{t}$\end{document}, thin lines) is the same as for the left panel. However, the desired reach is updated by exploration that led to success, which gives the distribution of the next desired reach (xt+1\begin{document}$x_{t+1}$\end{document}, thick red line) and the next actual reach is this distribution with the addition of motor noise (thick blue line). For this illustration, both motor and exploration standard deviations were set to 8 and the target was set to +10 (green lines).
Reach after a successful trial
The situation is more complicated if we consider what happens on the reach after a successful trial which had exploration. We need to consider (a) the update from the successful trial and (b) the probability of reward for the next trial.
(a) If the reach st=et+mt\begin{document}$ s_{t}=e_{t}+m_{t}$\end{document} (Appendix 1—figure 1 right panel, blue thin line) led to success (et+mt=d\begin{document}$ e_{t}+m_{t}=d$\end{document}) then the desired reach is updated by this exploration variability (xt+1=et\begin{document}$x_{t+1}=e_{t}$\end{document}). We can consider the conditional distribution of et\begin{document}$e_{t}$\end{document} given it led to success, which is known (from the simple properties of Gaussians) to be:(et|mt+et=d)=N(σe2σe2+σm2d,σm2σe2σe2+σm2)\begin{document}$$\displaystyle (e_t|m_t+e_t=d)= \mathcal{N}\left(\frac{\sigma_e^2}{\sigma_e^2+\sigma_m^2}d,\frac{\sigma_m^2\sigma_e^2}{\sigma_e^2+\sigma_m^2}\right)$$\end{document}
which is shown in Appendix 1—figure 1 (right panel, red thick line). Equation 15 shows that as σe\begin{document}$ \sigma_{e}$\end{document} increases, so does the average update.
(b) The subsequent reach is only corrupted by motor noise (as the previous reach was successful) giving st+1=et+mt+1\begin{document}$ s_{t+1}=e_{t}+m_{t+1}$\end{document}. The reach distribution (Appendix 1—figure 1 right panel, blue thick line) is therefore the same as Equation 15 with the addition of motor noise:N(σe2σe2+σm2d,σm2σe2σe2+σm2+σm2)\begin{document}$$\displaystyle \mathcal{N}\left(\frac{\sigma_e^2}{\sigma_e^2+\sigma_m^2}d,\frac{\sigma_m^2\sigma_e^2}{\sigma_e^2+\sigma_m^2}+\sigma_m^2\right)$$\end{document}
We can find the exploration variability that maximizes the reward, that is, maximizes the density at d\begin{document}$d$\end{document}. To do this, we can differentiate the Gaussian distribution with mean and variance given in Equation 16 with respect to σe\begin{document}$ \sigma_{e}$\end{document} to find the maximum. With a little algebra, this leads to the optimal exploration variability given by:σe2=16d4+σm4+4d2−3σm24\begin{document}$$\displaystyle \sigma_e^2=\frac{{\sqrt{16\,d^4+{\sigma _{m}}^4}+4\,d^2-3\,{\sigma_m}^2}}{4}$$\end{document}
For an optimal σe\begin{document}$\sigma_{e}$\end{document} to exist, this expression must be positive (or the optimal σe\begin{document}$\sigma_{e}$\end{document} is 0) and this can be shown to be true provided |d|>σm/3\begin{document}$|d| \gt \sigma_{m}/\sqrt{3}$\end{document}. How the optimal σe\begin{document}$\sigma_{e}$\end{document} changes with σm\begin{document}$\sigma_{m}$\end{document} depends on the sign of the derivative of Equation 17 with respect to motor noise. We need only consider the components of Equation 17 that depend on σm\begin{document}$ \sigma_{m}$\end{document}, that is:f(σm)=16d4+σm4−3σm2\begin{document}$$\displaystyle f(\sigma_{m})=\sqrt{16\,d^{4}+{\sigma_{m}}^{4}}-3\,{\sigma_{m}}^{2}$$\end{document}
The derivative needs to be less than zero for the inverse relation to be held (i.e. increasing σm\begin{document}$\sigma_{m}$\end{document} reduces the optimal σe\begin{document}$\sigma_{e}$\end{document}). That is:f′(σm)=2σm316d4+σm4−6σm<0\begin{document}$$\displaystyle f^{\prime}(\sigma_{m})=\frac{2\,{\sigma_{m}}^{3}}{\sqrt{16\,d^{4}+{\sigma_{m}}^{4}}}-6\,\sigma_{m} \lt 0$$\end{document}σm216d4+σm4<3\begin{document}$$\displaystyle \frac{{\sigma_{m}}^{2}}{\sqrt{16\,d^{4}+{\sigma_{m}}^{4}}} \lt 3$$\end{document}σm4<9(16d4+σm4)\begin{document}$$\displaystyle {\sigma_{m}}^{4} \lt 9(16\,d^{4}+{\sigma_{m}}^{4})$$\end{document}0<144d4+8σm4\begin{document}$$\displaystyle 0 \lt 144d^{4}+8{\sigma_{m}}^{4}$$\end{document}
which is true. Therefore, the inverse relation persists even if we only consider reaches after a successful trial.
When the distance to the target is small compared to motor noise, then exploration actually reduces reward. However, in general for a real experiment with multiple reaches and exploration variability that does not change during learning, the above carries over to the full simulations (Figure 7—figure supplement 2).
Appendix 2Supplementary Information
Statistical analysis of sex, handedness, device, and browser on behavior.
Results from one-way ANOVAs of participant-specific factors on precision from experimental block one (baseline) and distance from target from experimental block two (learning) for each of the four tasks. For all tasks, participant-specific factors did not significantly affect behavior.
ANOVA Results
Continuous Probabilistic
Baseline Precision
Sex
F1,142=0.251
p=0.617
Hand
F1,142=0.213
p=0.645
Device
F2,141=1.066
p=0.347
Browser
F3,140=0.414
p=0.743
Distance from Target
Sex
F1,142=0.740
p=0.391
Hand
F1,142=0.815
p=0.368
Device
F2,141=0.733
p=0.482
Browser
F3,140=0.313
p=0.816
Discrete Probabilistic
Baseline Precision
Sex
F1,137=0.431
p=0.513
Hand
F1,137=0.923
p=0.338
Device
F2,136=0.677
p=0.510
Browser
F3,135=0.192
p=0.901
Distance from Target
Sex
F1,137=0.096
p=0.758
Hand
F1,137=0.645
p=0.423
Device
F2,136=1.216
p=0.300
Browser
F3,135=0.998
p=0.396
Continuous Deterministic
Baseline Precision
Sex
F1,48=1.772
p=0.189
Hand
F1,48=0.544
p=0.464
Device
F2,47=0.337
p=0.716
Browser
F3,46=0.162
p=0.922
Distance from Target
Sex
F1,48=0.455
p=0.503
Hand
F1,48=0.770
p=0.385
Device
F2,47=1.482
p=0.238
Browser
F3,46=0.343
p=0.794
Discrete Deterministic
Baseline Precision
Sex
F1,50=0.076
p=0.783
Hand
F1,50=0.015
p=0.902
Device
F2,49=0.691
p=0.506
Browser
F2,49=0.154
p=0.857
Distance from Target
Sex
F1,50=2.002
p=0.163
Hand
F1,50=0.105
p=0.747
Device
F2,49=0.933
p=0.400
Browser
F2,49=0.318
p=0.729
10.7554/eLife.101036.3.sa0eLife AssessmentDekkerTessaReviewing EditorUniversity College LondonUnited KingdomConvincingImportant
This important study tests the development of motor reinforcement learning from toddlerhood to adulthood, using a large online sample. They show that learning improves with age in a task that, like real-life movement, involves a continuous range of response options and probabilistic rewards, and link this shift to reduced movement variability and more efficient feedback-based learning through behavioural modeling. Simplifying the task with discrete actions and deterministic outcomes boosted younger children's performance, suggesting early learning is limited by spatial and probabilistic processing. The evidence is convincing, although future work may investigate more naturalistic movement.
Here the authors address how reinforcement-based sensorimotor adaptation changes throughout development. To address this question, they collected many participants in ages that ranged from small children (3 years old) to adulthood (18+ years old). The authors used four experiments to manipulate whether binary and positive reinforcement was provided probabilistically (e.g., 30 or 50%) versus deterministically (e.g.,100%), and continuous (infinite possible locations) versus discrete (binned possible locations) when the probability of reinforcement varied along the span of a large redundant target. The authors found that both movement variability and the extent of adaptation changed with age.
Strengths:
The major strength of the paper is the number of participants collected (n = 385). The authors also answer their primary question, that reinforcement-based sensorimotor adaptation changes throughout development, which was shown by utilizing established experimental designs and computational modelling. They have compared an extensive number of potential models, finding the one that best fits the data while penalizing the number of free parameters.
In this study, Hill and colleagues use a novel reinforcement-based motor learning task ("RML"), asking how aspects of RML change over the course of development from toddler years through adolescence. Multiple versions of the RML task were used in different samples, which varied on two dimensions: whether the reward probability of a given hand movement direction was deterministic or probabilistic, and whether the solution space had continuous reach targets or discrete reach targets. Using analyses of both raw behavioral data and model fits, the authors report four main results: First, developmental improvements reflected 3 clear changes, including increases in exploration, an increase in the RL learning rate, and a reduction of intrinsic motor noise. Second, changes to the task that made it discrete and/or deterministic both rescued performance in the youngest age groups, suggesting that observed deficits could be linked to continuous/probabilistic learning settings. Overall, the results shed light on how RML changes throughout human development, and the modeling characterizes the specific learning deficits seen in the youngest ages.
Strengths:
(1) This impressive work addresses an understudied subfield of motor control/psychology - the developmental trajectory of motor learning. It is thus timely and will interest many researchers.
(2) The task, analysis, and modeling methods are very strong. The empirical findings are rather clear and compelling, and the analysis approaches are convincing. Thus, at the empirical level, this study has very few weaknesses.
(3) The large sample sizes and in-lab replications further reflect the laudable rigor of the study.
(4) The main and supplemental figures are clear and concise.
The study investigates the development of reinforcement learning across the lifespan with a large sample of participants recruited for an online game. It finds that children gradually develop their abilities to learn reward probability, possibly hindered by their immature spatial processing and probabilistic reasoning abilities. Motor noise and exploration after a failure all contribute to children's subpar performance.
Strengths:
Experimental manipulations of both the continuity of movement options and the probabilistic nature of the reward function enable the inference of what cognitive factors differ between age groups.
A large sample of participants is studied.
The model-based analysis provides further insights into the development of reinforcement learning ability.
Weaknesses:
The conclusion that immature spatial processing and probabilistic reasoning abilities limit reinforcement learning here still needs more direct evidence.
10.7554/eLife.101036.3.sa4Author responseHillNayo MAuthorKennedy Krieger InstituteBaltimoreUnited StatesTrippHaley MAuthorKennedy Krieger InstituteBaltimoreUnited StatesWolpertDaniel MAuthorColumbia UniversityNew YorkUnited StatesMaloneLaura AAuthorKennedy Krieger Institute/Johns Hopkins University School of MedicineBaltimoreUnited StatesBastianAmy JAuthorKennedy Krieger InstituteBaltimoreUnited States
The following is the authors’ response to the original reviews
Overview of changes in the revision
We thank the reviewers for the very helpful comments and have extensively revised the paper. We provide point-by-point responses below and here briefly highlight the major changes:
(1) We expanded the discussion of the relevant literature in children and adults.
(2) We improved the contextualization of our experimental design within previous reinforcement studies in both cognitive and motor domains highlighting the interplay between the two.
(3) We reorganized the primary and supplementary results to better communicate the findings of the studies.
(4) The modeling has been significantly revised and extended. We now formally compare 31 noise-based models and one value-based model and this led to a different model from the original being the preferred model. This has to a large extent cleaned up the modeling results. The preferred model is a special case (with no exploration after success) of the model proposed in Therrien et al. (2018). We also provide examples of individual fits of the model, fit all four tasks and show group fits for all, examine fits vs. data for the clamp phases by age, provide measures of relative and absolute goodness of fit, and examine how the optimal level of exploration varies with motor noise.
Reviewer #1 (Public review):
Summary:
Here the authors address how reinforcement-based sensorimotor adaptation changes throughout development. To address this question, they collected many participants in ages that ranged from small children (3 years old) to adulthood (1 8+ years old). The authors used four experiments to manipulate whether binary and positive reinforcement was provided probabilistically (e.g., 30 or 50%) versus deterministically (e.g., 100%), and continuous (infinite possible locations) versus discrete (binned possible locations) when the probability of reinforcement varied along the span of a large redundant target. The authors found that both movement variability and the extent of adaptation changed with age.
Thank you for reviewing our work. One note of clarification. This work focuses on reinforcementbased learning throughout development but does not evaluate sensorimotor adaptation. The four tasks presented in this work are completed with veridical trajectory feedback (no perturbation).
The goal is to understand how children at different ages adjust their movements in response to reward feedback but does not evaluate sensorimotor adaptation. We now explain this distinction on line 35.
Strengths:
The major strength of the paper is the number of participants collected (n = 385). The authors also answer their primary question, that reinforcement-based sensorimotor adaptation changes throughout development, which was shown by utilizing established experimental designs and computational modelling.
Thank you.
Weaknesses:
Potential concerns involve inconsistent findings with secondary analyses, current assumptions that impact both interpr tation and computational modelling, and a lack of clearly stated hypotheses.
(1) Multiple regression and Mediation Analyses.
The challenge with these secondary analyses is that:
(a) The results are inconsistent between Experiments 1 and 2, and the analysis was not performed for Experiments 3 and 4,
(b) The authors used a two-stage procedure of using multiple regression to determine what variables to use for the mediation analysis, and
(c)The authors already have a trial-by-trial model that is arguably more insightful.
Given this, some suggested changes are to:
(a) Perform the mediation analysis with all the possible variables (i.e., not informed by multiple regression) to see if the results are consistent.
(b) Move the regression/mediation analysis to Supplementary, since it is slightly distracting given current inconsistencies and that the trial-by-trial model is arguably more insightful.
Based on these comments, we have chosen to remove the multiple regression and mediation analyses. We agree that they were distracting and that the trial-by-trial model allows for differentiation of motor noise from exploration variability in the learning block.
(2) Variability for different phases and model assumptions:
A nice feature of the experimental design is the use of success and failure clamps. These clamped phases, along with baseline, are useful because they can provide insights into the partitioning of motor and exploratory noise. Based on the assumptions of the model, the success clamp would only reflect variability due to motor noise (excludes variability due to exploratory noise and any variability due to updates in reach aim). Thus, it is reasonable to expect that the success clamps would have lower variability than the failure clamps (which it obviously does in Figure 6), and presumably baseline (which provides success and failure feedback, thus would contain motor noise and likely some exploratory noise).
However, in Figure 6, one visually observes greater variability during the success clamp (where it is assumed variability only comes from motor noise) compared to baseline where variability would come from: (a) Motor noise.
(b) Likely some exploratory noise since there were some failures.
(c) Updates in reach aim.
Thanks for this comment. It made us realize that some of our terminology was unintentionally misleading. Reaching to discrete targets in the Baseline block was done to (a) determine if participants could move successfully to targets that are the same width as the 100% reward zone in the continuous targets and (b) determine if there are age dependent changes in movement precision. We now realize that the term Baseline Variability was misleading and should really be called Baseline Precision.
This is an important distinction that bears on this reviewer's comment. In clamp trials, participants move to continuous targets. In baseline, participants move to discrete targets presented at different locations. Clamp Variability cannot be directly compared to Baseline Precision because they are qualitatively different. Since the target changes on each baseline trial, we would not expect updating of desired reach (the target is the desired reach) and there is therefore no updating of reach based on success or failure. The SD we calculate over baseline trials is the endpoint variability of the reach locations relative to the target centers. In success clamp, there are no targets so the task is qualitatively different.
We have updated the text to clarify terminology, expand upon our operational definitions, and motivate the distinct role of the baseline block in our task paradigm (line 674).
Given the comment above, can the authors please:
(a) Statistically compare movement variability between the baseline, success clamp, and failure clamp phases.
Given our explanation in the previous point we don't think that comparing baseline to the clamp makes sense as the trials are qualitatively different.
(b) The authors have examined how their model predicts variability during success clamps and failure clamps, but can they also please show predictions for baseline (similar to that of Cashaback et al., 2019; Supplementary B, which alternatively used a no feedback baseline)?
Again, we do not think it makes sense to predict the baseline which as we mention above has discrete targets compared to the continuous targets in the learning phase.
(c) Can the authors show whether participants updated their aim towards their last successful reach during the success clamp? This would be a particularly insightful analysis of model assumptions.
We have now compared 31 models (see full details in next response) which include the 7 models in Roth et al. (2023). Several of these model variants have updating even after success with so called planning noise. We also now fit the model to the data that includes the clamp phases (we can't easily fit to success clamp alone as there are only 10 trials). We find that the preferred model is one that does not include updating after success.
(d) Different sources of movement variability have been proposed in the literature, as have different related models. One possibility is that the nervous system has knowledge of 'planned (noise)' movement variability that is always present, irrespective of success (van Beers, R.J. (2009). Motor learning is optimally tuned to the properties of motor noise. Neuron, 63(3), 406-417). The authors have used slightly different variations of their model in the past. Roth et al (2023) directly Rill compared several different plausible models with various combinations of motor, planned, and exploratory noise (Roth A, 2023, "Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds." Proceedings of the Royal Society B 290: 20231475: see Supplemental). Their best-fit model seems similar to the one the authors propose here, but the current paper has the added benefit of the success and failure clamps to tease the different potential models apart. In light of the results of (a), (b), and (c), the authors are encouraged to provide a paragraph on how their model relates to the various sources of movement variability and ther models proposed in the literature.
Thank you for this. We realized that the models presented in Roth et al. (2023) as well as in other papers, are all special cases of a more general model. Moreover, in total there are 30 possible variants of the full model so we have now fit all 31 models to our larger datasets and performed model selection (Results and Methods). All the models can be efficiently fit by Kalman smoother to the actual data (rather than to summary statistics which has sometimes been done). For model selection, we fit only the 100 learning trials and chose the preferred model based on BIC on the children's data (Figure 5—figure Supplement 1). After selecting the preferred model we then refit this model to all trials including the clamps so as to obtain the best parameter estimates.
The preferred model was the same whether we combined the continuous and discrete probabilistic data or just examin d each task separately either for only the children or for the children and adults combined. The preferred model is a pecial case (no exploration after success) of the one proposed in Therrien et al. (2018) and has exploration variability (after failure) and motor noise with full updating with exploration variability (if any) after success. This model differs from the model in the original submission which included a partial update of the desired reach after exploration this was considered the learning rate. The current model suggests a unity learning rate.
In addition, as suggested by another reviewer, we also fit a value-based model which we adapted from the model described in Giron et al. (2023). This model was not preferred.
We have added a paragraph to the Discussion highlighting different sources of variability and links to our model comparison.
(e) line 155. Why would the success clamp be composed of both motor and exploratory noise? Please clarify in the text
This sentence was written to refer to clamps in general and not just success clamps. However, in the revision this sentence seemed unnecessary so we have removed it.
(3) Hypotheses:
The introduction did not have any hypotheses of development and reinforcement, despite the discussion above setting up potential hypotheses. Did the authors have any hypotheses related to why they might expect age to change motor noise, exploratory noise, and learning rates? If so, what would the experimental behaviour look like to confirm these hypotheses? Currently, the manuscript reads more as an exploratory study, which is certainly fine if true, it should just be explicitly stated in the introduction. Note: on line 144, this is a prediction, not a hypothesis. Line 225: this idea could be sharpened. I believe the authors are speaking to the idea of having more explicit knowledge of action-target pairings changing behaviour.
We have included our hypotheses and predictions at two points in the paper In the introduction we modified the text to:
"We hypothesized that children's reinforcement learning abilities would improve with age, and depend on the developmental trajectory of exploration variability, learning rate (how much people adjust their reach after success), and motor noise (here defined as all sources of noise associated with movement, including sensory noise, memory noise, and motor noise). We think that these factors depend on the developmental progression of neural circuits that contribute to reinforcement learning abilities (Raznahan et al., 2014; Nelson et al., 2000; Schultz, 1998)."
In results we modified the sentence to:
"We predicted that discrete targets could increase exploration by encouraging children to move to a different target after failure.”
Reviewer #2 (Public review):
Summary:
In this study, Hill and colleagues use a novel reinforcement-based motor learning task ("RML"), asking how aspects of RML change over the course of development from toddler years through adolescence. Multiple versions of the RML task were used in different samples, which varied on two dimensions: whether the reward probability of a given hand movement direction was deterministic or probabilistic, and whether the solution space had continuous reach targets or discrete reach targets. Using analyses of both raw behavioral data and model fits, the authors report four main results: First, developmental improvements reflected 3 clear changes, including increases in exploration, an increase in the RL learning rate, and a reduction of intrinsic motor noise. Second, changes to the task that made it discrete and/or deterministic both rescued performance in the youngest age groups, suggesting that observed deficits could be linked to continuous/probabilistic learning settings. Overall, the results shed light on how RML changes throughout human development, and the modeling characterizes the specific learning deficits seen in the youngest ages.
Strengths:
(1) This impressive work addresses an understudied subfield of motor control/psychology - the developmental trajectory of motor learning. It is thus timely and will interest many researchers.
(2) The task, analysis, and modeling methods are very strong. The empirical findings are rather clear and compelling, and the analysis approaches are convincing. Thus, at the empirical level, this study has very few weaknesses.
(3) The large sample sizes and in-lab replications further reflect the laudable rigor of the study.
(4) The main and supplemental figures are clear and concise.
Thank you.
Weaknesses:
(1) Framing.
One weakness of the current paper is the framing, namely w/r/t what can be considered "cognitive" versus "non-cognitive" ("procedural?") here. In the Intro, for example, it is stated that there are specific features of RML tasks that deviate from cognitive tasks. This is of course true in terms of having a continuous choice space and motor noise, but spatially correlated reward functions are not a unique feature of motor learning (see e.g. Giron et al., 2023, NHB). Given the result here that simplifying the spatial memory demands of the task greatly improved learning for the youngest cohort, it is hard to say whether the task is truly getting at a motor learning process or more generic cognitive capacities for spatial learning, working memory, and hypothesis testing. This is not a logical problem with the design, as spatial reasoning and working memory are intrinsically tied to motor learning. However, I think the framing of the study could be revised to focus in on what the authors truly think is motor about the task versus more general psychological mechanisms. Indeed, it may be the case that deficits in motor learning in young children are mostly about cognitive factors, which is still an interesting result!
Thank you for these comments on the framing of our study. We now clearly acknowledge that all motor tasks have cognitive components (new paragraph at line 65). We also explain why we think our tasks has features not present in typical cognitive tasks.
(2) Links to other scholarship.
If I'm not mistaken a common observation in tudies of the development of reinforcement learning is a decrease in exploration over-development (e.g., Nussenbaum and Hartley, 2019; Giron et al., 2023; Schulz et al., 2019); this contrasts with the current results which instead show an increase. It would be nice to see a more direct discussion of previous findings showing decreases in exploration over development, and why the current study deviates from that. It could also be useful for the authors to bring in concepts of different types of exploration (e.g. "directed" vs "random"), in their interpretations and potentially in their modeling.
We recognize that our results differ from prior work. The optimal exploration pattern differs from task to task. We now discuss that exploration is not one size fits all, it's benefits vary depending upon the task. We have added the following paragraphs to the Discussion section:
"One major finding from this study is that exploration variability increases with age. Some other studies of development have shown that exploration can decrease with age indicating that adults explore less compared to children (Schulz et al., 2019; Meder et al., 2021; Giron et al., 2023). We believe the divergence between our work and these previous findings is largely due to the experimental design of our study and the role of motor noise. In the paradigm used initially by Schulz et al. (2019) and replicated in different age groups by Meder et al. (2021) and Giron et al. (2023), participants push buttons on a two-dimensional grid to reveal continuous-valued rewards that are spatially correlated. Participants are unaware that there is a maximum reward available and therefore children may continue to explore to reduce uncertainty if they have difficulty evaluating whether they have reached a maxima. In our task by contrast, participants are given binary reward and told that there is a region in which reaches will always be rewarded. Motor noise is an additional factor which plays a key role in our reaching task but minimal if any role in the discretized grid task. As we show in simulations of our task, as motor noise goes down (as it is known to do through development) the optimal amount of exploration goes up (see Figure 7—figure Supplement 2 and Appendix 1). Therefore, the behavior of our participants is rational in terms of R230 increasing exploration as motor noise decreases.
A key result in our study is that exploration in our task reflects sensitivity to failure. Older children make larger adjustments after failure compared to younger children to find the highly rewarded zone more quickly. Dhawale et al. (2017) discuss the different contexts in which a participant may explore versus exploit (i.e., stick at the same position). Exploration is beneficial when reward is low as this indicates that the current solution is no longer ideal, and the participant should search for a better solution. Konrad et al. (2025) have recently shown this behavior in a real-world throwing task where 6 to 12 year old children increased throwing variability after missed trials and minimized variability after successful trials. This has also been shown in a postural motor control task where participants were more variable after non-rewarded trials compared to rewarded trials (Van Mastrigt et al., 2020). In general, these studies suggest that the optimal amount of exploration is dependent on the specifics of the task."
(3) Modeling.
First, I may have missed something, but it is unclear to me if the model is actually accounting for the gradient of rewards (e.g., if I get a probabilistic reward moving at 45°, but then don't get one at 40°, I should be more likely to try 50° next then 35°). I couldn't tell from the current equations if this was the case, or if exploration was essentially "unsigned," nor if the multiple-trials-back regression analysis would truly capture signed behavior. If the model is sensitive to the gradient, it would be nice if this was more clear in the Methods. If not, it would be interesting to have a model that does "function approximation" of the task space, and see if that improves the fit or explains developmental changes.
The model we use (similar to Roth et al. (2023) and Therrien et al. (2016, 2018)) does not model the gradient. Exploration is always zero-mean Gaussian. As suggested by the reviewer, we now also fit a value-based model (described starting at line 810) which we adapted from the model presented in Giron et al. (2023). We show that the exploration and noise-based model is preferred over the value-based model.
The multiple-trials-back regression was unsigned as the intent was to look at the magnitude and not the direction of the change in movement. We have decided to remove this analysis from the manuscript as it was a source of confusion and secondary analysis that did not add substantially to the findings of these studies.
Second, I am curious if the current modeling approach could incorporate a kind of "action hysteresis" (aka perseveration), such that regardless of previous outcomes, the same action is biased to be repeated (or, based on parameter settings, avoided).
In some sense, the learning rate in the model in the original submission is highly related to perseveration. For example if the learning rate is 0, then there is complete perseveration as you simply repeat the same desired movement. If the rate is 1, there is no perseveration and values in between reflect different amounts of perseveration. Therefore, it is not easy to separate learning rate from perseveration. Adding perseveration as another parameter would likely make it and the learning unidentifiable. However, we now compare 31 models and those that have a non-unity learning rate are not preferred suggesting there is little perseveration.
(4) Psychological mechanisms. There is a line of work that shows that when children and adults perform RL tasks they use a combination of working memory and trial-by-trial incremental learning processes (e.g., Master et al., 2020; Collins and Frank 2012). Thus, the observed increase in the learning rate over development could in theory reflect improvements in instrumental learning, working memory, or both. Could it be that older participants are better at remembering their recent movements in short-term memory (Hadjiosif et al., 2023; Hillman et al., 2024)?
We agree that cognitive processes, such as working memory or visuospatial processing, play a role in our task and describe cognitive elements of our task in the introduction (new paragraph at line 65). However, the sensorimotor model we fit to the data does a good job of explaining the variation across age, which suggests that that age-dependent cognitive processes probably play a smaller role.
Reviewer #3 (Public review):
Summary:
The study investigates reinforcement learning across the lifespan with a large sample of participants recruited for an online game. It finds that children gradually develop their abilities to learn reward probability, possibly hindered by their immature spatial processing and probabilistic reasoning abilities. Motor noise, reinforcement learning rate, and exploration after a failure all contribute to children's subpar performance.
Strengths:
(1) The paradigm is novel because it requires continuous movement to indicate people's choices, as opposed to discrete actions in previous studies.
(2) A large sample of participants were recruited.
(3) The model-based analysis provides further insights into the development of reinforcement learning ability.
Thank you.
Weaknesses:
(1) The adequacy of model-based analysis is questionable, given the current presentation and some inconsistency in the results.
Thank you for raising this concern. We have substantially revised the model from our first submission. We now compare 31 noise-based models and 1 value-based model and fit all of the tasks with the preferred model. We perform model selection using the two tasks with the largest datasets to identify the preferred model. From the preferred model, we found the parameter fits for each individual dataset and simulated the trial by trial behavior allowing comparison between all four tasks. We now show examples of individual fits as well as provide a measure of goodness of fit. The expansion of our modeling approach has resolved inconsistencies and sharpened the conclusions drawn from our model.
(2) The task should not be labeled as reinforcement motor learning, as it is not about learning a motor skill or adapting to sensorimotor perturbations. It is a classical reinforcement learning paradigm.
We now make it clear that our reinforcement learning task has both motor and cognitive demands, but does not fall entirely within one of these domains. We use the term motor learning because it captures the fact that participants maximize reward by making different movements, corrupted by motor noise, to unmarked locations on a continuous target zone. When we look at previous ublications, it is clear that our task is similar to those that also refer to this as reinforcement motor learning Cashaback et al. (2019) (reaching task using a robotic arm in adults), Van Mastrigt et al. (2020) (weight shifting task in adults), and Konrad et al. (2025) (real-world throwing task in children). All of these tasks involve trial-by-trial learning through reinforcement to make the movement that is most effective for a given situation. We feel it is important to link our work to these previous studies and prefer to preserve the terminology of reinforcement motor learning.
Recommendations for the authors:
Reviewing Editor Comments:
Thank you for this summary. Rather than repeat the extended text from the responses to the reviewers here, we point the Editor to the appropriate reviewer responses for each issue raised.
The reviewers and editors have rated the significance of the findings in your manuscript as "Valuable" and the strength of evidence as "Solid" (see eLife evalutation). A consultancy discussion session to integrate the public reviews and recommendations per reviewer (listed below), has resulted in key recommendations for increasing the significance and strength of evidence:
To increase the Significance of the findings, please consider the following:
(1) Address and reframe the paper around whether the task is truly getting at a motor learning process or more generic cognitive decision-making capacities such as spatial memory, reward processing, and hypothesis testing.
We have revised the paper to address the comments on the framing of our work. Please see responses to the public review comments of Reviewers #2 and #3.
(2) It would be beneficial to specify the differences between traditional reinforcement algorithms (i.e., using softmax functions to explore, and build representations of state-action-reward) and the reinforcement learning models used here (i.e., explore with movement variability, update reach aim towards the last successful action), and compare present findings to previous cognitive reinforcement learning studies in children.
Please see response to the public review comments of Reviewer #1 in which we explain the expansion of our modeling approach to fit a value-based model as well as 31 other noise-based models. In our response to the public review comments of Reviewer #2, we comment on our expanded discussion of how our findings compare with previous cognitive reinforcement learning studies.
To move the "Strength of Evidence" to "Convincing", please consider doing the following:
(1) Address some apparently inconsistent and unrealistic values of motor noise, exploration noise, and learning rate shown for individual participants e.g., Figure 5b; see comments reviewers 1 and take the following additional steps: plotting r squares for individual participants, discussing whether individual values of the fitted parameters are plausible and whether model parameters in each age group can extrapolate to the two clamp conditions and baselines.
We have substantially updated our modeling approach. Now that we compare 31 noise-based models, the preferred model does not show any inconsistent or unrealistic values (see response to Reviewer #3). Additionally, we now show example individual fits and provide both relative and absolute goodness of fit (see response to Reviewer #3).
(2) Relatedly, to further justify if model assumptions are met, it would be valuable to show that the current learning model fits the data better than alternative models presented in the literature by the authors themselves and by others (reviewer 1). This could include alternative development models that formalise the proposed explanations for age-related change: poor spatial memory, reward/outcome processing, and exploration strategies (reviewer 2).
Please see response to public review comments of Reviewer #1 in which we explain that we have now fit a value-based model as well as 31 other noise-based models providing a comparison of previous models as well as novel models. This led to a slightly different model being preferred over the model in the original submission (updated model has a learning rate of unity). These models span many of the processes previously proposed for such tasks. We feel that 32 models span a reasonable amount of space and do not believe we have the power to include memory issues or heuristic exploration strategies in the model.
(3) Perform the mediation analysis with all the possible variables (i.e., not informed by multiple regression) to see if the results are more consistent across studies and with the current approach (see comments reviewer 1).
Please see response to public review comments of Reviewer #1. We chose to focus only on the model based analysis because it allowed us to distinguish between exploration variability and motor noise.
Please see below for further specific recommendations from each reviewer.
Reviewer #1 (Recommendations for the author):
(1) In general, there should be more discussion and contextualization of other binary reinforcement tasks used in the motor literature. For example, work from Jeroen Smeets, Katinka van der Kooij, and Joseph Galea.
Thank you for this comment. We have edited the Introduction to better contextualize our work within the reinforcement motor learning literature (see line 67 and line 83).
(2) Line 32. Very minor. This sentence is fine, but perhaps could be slightly improved. “select a location along a continuous and infinite set of possible options (anywhere along the span of the bridge)"
Thank you for this comment. We have edited the sentence to reflect this suggestion.
(3) Line 57. To avoid some confusion in successive sentences: Perhaps, "Both children over 12 and adolescents...".
Thank you for this comment. We have edited the sentence to reflect this suggestion.
(4) Line 80. This is arguably not a mechanistic model, since it is likely not capturing the reward/reinforcement machinery used by the nervous system, such as updating the expected value using reward predic tion errors/dopamine. That said, this phenomenological model, and other similar models in the field, do very well to capture behaviour with a very simple set of explore and update rules.
We use mechanistic in the standard use in modeling, as in Levenstein et al. (2023), for example. The contrast is not with neural modeling, but with normative modeling, in which one develops a model to optimize a function (or descriptive models as to what a system is trying to achieve). In mechanistic modeling one proposes a mechanism and this can be at a state-space level (as in our case) or a neural level (as suggested my the reviewer) but both are considered mechanistic, just at different levels. Quoting Levenstein "... mechanistic models, in which complex processes are summarized in schematic or conceptual structures that represent general properties of components and their interactions, are also commonly used." We now reference the Levenstein paper to clarify what we mean by mechanistic.
(5) Figure 1. It would be useful to state that the x-axis in Figure 1 is in normalized units, depending on the device.
Thank you for this comment. We have added a description of the x-axis units to the Fig. 1 caption.
(6) Were there differences in behaviour for these different devices? e.g., how different was motor noise for the mouse, trackpad, and touchscreen?
Thank you for this question. We did not find a significant effect of device on learning or precision in the baseline block. We have added these one way ANOVA results for each task in Supplementary Table 1.
(7) Line 98. Please state that participants received reinforcement feedback during baseline.
Thank you for this comment. We have updated the text to specify that participants receive reward feedback during the baseline block.
(8) Line 99. Did the distance from the last baseline trial influence whether the participant learned or did not learn? For example, would it place them too far from the peak success location such that it impacted learning?
Thank you for this question. We looked at whether the position of movement on the last baseline block trial was correlated with the first movement position in the learning block. We did not find any correlations between these positions for any of the tasks. Interestingly, we found that the majority of participants move to the center of the workspace on the first trial of the learning block for all tasks (either in the presence of the novel continuous target scene or the presentation of 7 targets all at once). We do not think that the last movement in the baseline block "primed" the participant for the location of the success zone in the learning block. We have added the following sentence to the Results section:
"Note that the reach location for the first learning trial was not affected by (correlated with) the target position on the last baseline trial (p > 0.3 for both children and adults, separately)."
(9) The term learning distance could be improved. Perhaps use distance from target.
Thank you for this comment. We appreciate that learning distance defined with 0 as the best value is counter intuitive. We have changed the language to be "distance from target" as the learning metric.
(10) Line 188. This equation is correct, but to estimate what the standard deviation by the distribution of changes in reach position is more involved. Not sure if the authors carried out this full procedure, which is described in Cashaback et al., 2019; Supplemental 2.
There appear to be no Supplemental 2 in the referenced paper so we assume the reviewer is referring to Supplemental B which deals with a shuffling procedure to examine lag-1 correlations.
In our tasks, we are limited to only 9 trials to analyze in each clamp phase so do not feel a shuffling analysis is warranted. In these blocks, we are not trying to 'estimate what the standard deviation by the distribution of changes in reach position' but instead are calculating the standard deviation of the reach locations and comparing the model fit (for which the reviewer says the formula is correct) with the data. We are unclear what additional steps the reviewer is suggesting. In our updated model analysis, we fit the data including the clamp phases for better parameter estimation. We use simulations to estimate s.d. in the clamp phase (as we ensure in simulations the data does not fall outside the workspace) making the previous analytic formulas an approximation that are no longer used.
(11) Line 197-199. Having done the demo task, it is somewhat surprising that a 3-year-old could understand these instructions (whose comprehension can be very different from even a 5-year old).
Thank you for raising this concern. We recognize that the younger participants likely have different comprehension levels compared to older participants. However, we believe that the majority of even the youngest participants were able to sufficiently understand the goal of the task to move in a way to get the video clip to play. We intentionally designed the tasks to be simple such that the only instructions the child needed to understand were that the goal was to get the video clip to play as much as possible and the video clip played based on their movement. Though the majority of younger children struggled to learn well on the probabilistic tasks, they were able to learn well on the deterministic tasks where the task instructions were virtually identical with the exception of how many places in the workspace could gain reward. On the continuous probabilistic task, we did have a small number (n = 3) of 3 to 5 year olds who exhibited more mature learning ability which gives us confidence that the younger children were able to understand the task goal.
(12) Line 497: Can the authors please report the F-score and p-value separately for each of these one-way ANOVA (the device is of particular interest here).
Thank you for this request. We have added ina upplementarytable (Supplementary Table 1) with the results of these ANOVAs.
(13) Past work has discussed how motivation influences learning, which is a function of success rate (van der Kooij, K., in 't Veld, L., & Hennink, T. (2021). Motivation as a function of success frequency. Motivation and Emotion, 45, 759-768.). Can the authors please discuss how that may change throughout development?
Thank you for this comment. While motivation most probably plays a role in learning, in particular in a game environment, this was out of the scope of the direct focus of this work and not something that our studies were designed to test. We have added the following sentence to the discussion section to address this comment:
"We also recognize that other processes, such as memory and motivation, could affect performance on these tasks however our study was not designed to test these processes directly and future work would benefit from exploring these other components more explicitly."
(14) Supplement 6. This analysis is somewhat incomplete because it does not consider success.
Pekny and collegues (2015) looked at 3 trials back but considered both success and reward. However, their analysis has issues since successive time points are not i.i.d., and spurious relationships can arise. This issue is brought up by Dwahale (Dhawale, A. K., Miyamoto, Y. R., Smith, M. A., & R475 Ölveczky, B. P. (2019). Adaptive regulation of motor variability. Current Biology, 29(21), 3551-3562.). Perhaps it is best to remove this analysis from the paper.
Thank you for this comment. We have decided to remove this secondary analysis from the paper as it was a source of confusion and did not add to the understanding and interpretation of our behavioral results.
Reviewer #2 (Recommendations for the author):
(1) the path length ratio analyses in the supplemental are interesting but are not mentioned in the main paper. I think it would be helpful to mention these as they are somewhat dramatic effects
Thank you for this comment. Path length ratios are defined in the Methods and results are briefly summarized in the Results section with a point to the supplementary figures. We have updated the text to more explicitly report the age related differences in path length ratios.
(2) The second to last paragraph of the intro could use a sentence motivating the use ofthe different task features (deterministic/probabilistic and discrete/continuous).
Thank you for this comment. We have added an additional motivating sentence to the introduction.
Reviewer #3 (Recommendations for the author):
The paper labeled the task as one for reinforcement motor learning, which is not quite appropriate in my opinion. Motor learning typically refers to either skill learning or motor adaptation, the former for improving speed-accuracy tradeoffs in a certain (often new) motor skill task and the latter for accommodating some sensorimotor perturbations for an existing motor skill task. The gaming task here is for neither. It is more like a
decision-making task with a slight contribution to motor execution, i.e., motor noise. I would recommend the authors label the learning as reinforcement learning instead of reinforcement motor learning.
Thank you for this comment. As noted in the response to the public review comments, we agree that this task has components of classical reinforcement learning (i.e. responding to a binary reward) but we specifically designed it to require the learning of a movement within a novel game environment. We have added a new paragraph to the introduction where we acknowledge the interplay between cognitive and motor mechanisms while also underscoring the features in our task that we think are not present in typical cognitive tasks.
My major concern is whether the model adequately captures subjects' behavior and whether we can conclude with confidence from model fitting. Motor noise, exploration noise, and learning rate, which fit individual learning patterns (Figure 5b), show some quite unrealistic values. For example, some subjects have nearly zero motor noise and a 100% learning rate.
We have now compared 31 models and the preferred model is different from the one in the first submission. The parameter fits of the new model do not saturate in any way and appear reasonable to us. The updates to the model analysis have addressed the concern of previously seen unrealistic values in the prior draft.
Currently, the paper does not report the fitting quality for individual subjects. It is good to have an exemplary subject's fit shown, too. My guess is that the r-squared would be quite low for this type of data. Still, given that the children's data is noisier, it might be good to use the adult data to show how good the fitting can be (individual fits, r squares, whether the fitted parameters make sense, whether it can extrapolate to the two clamp phases). Indeed, the reliability of model fitting affects how we should view the age effect of these model parameters.
We now show fits to individual subjects. But since this is a Kalman smoother it fits the data perfectly by generating its best estimate of motor noise and exploration variability on each trial to fully account for the data — so in that sense R2 is always 1 so that is not helpful.
While the BIC analysis with the other model variants provides a relative goodness of fit, it is not straightforward to provide an absolute goodness of fit such as standard R2 for a feedforward simulation of the model given the parameters (rather than the output of the Kalman smoother). There are two problems. First, there is no single model output. Each time the model is simulated with the fit parameters it produces a different output (due to motor noise, exploration variability and reward stochasticity). Second, the model is not meant to reproduce the actual motor noise, exploration variability and reward stochasticity of a trial. For example, the model could fit pure Gaussian motor noise across trials (for a poor learner) by accurately fitting the standard deviation of motor noise but would not be expected to actually match each data point so would have a traditional R2 of O.
To provide an overall goodness of fit we have to reduce the noise component and to do so we exam ined the traditional R2 between the average of all the children's data and the average simulation of the model (from the median of 1000 simulations per participant) so as to reduce the stochastic variation. The results for the continuous probabilistic and discrete probabilistic task are R2 of 0.41 and 0.72, respectively.
Not that variability in the "success clamp" doe not change across ages (Figure 4C) and does not contribute to the learning effect (Figure 4F). However, it is regarded as reflecting motor noise (Figure SC), which then decreases over age from the model fitting (Figure 5B). How do we reconcile these contradictions? Again, this calls the model fitting into question.
For the success clamp, we only have 9 trials to calculate variability which limits our power to detect significance with age. In contrast, the model uses all 120 trials to estimate motor noise. There is a downward trend with age in the behavioral data which we now show overlaid on the fits of the model for both probabilistic conditions (Figure 5—figure Supplement 4) and Figure 6—figure Supplement 4. These show a reasonable match and although the variance explained is 1 6 and 56% (we limit to 9 trials so as to match the fail clamp), the correlations are 0.52 and 0.78 suggesting we have reasonable relation although there may be other small sources of variability not captured in the model.
Figure 5C: it appears one bivariate outlier contributes a lot to the overall significant correlation here for the "success clamp".
Recalculating after removing that point in original Fig 5C was still significant and we feel the plots mentioned in the previous point add useful information to this issue. With the new model this figure has changed.
It is still a concern that the young children did not understand the instructions. Nine 3-to-8 children (out of 48) were better explained by the noisy only model than the full model. In contrast, ten of the rest of the participants (out of 98) were better explained by the noisy-only model. It appears that there is a higher percentage of the "young" children who didn't get the instruction than the older ones.
Thank you for this comment. We did take participant comprehension of the task into consideration during the task design. We specifically designed it so that the instructions were simple and straight forward. The child simply needs to understand the underlying goal to make the video clip play as often as possible and that they must move the penguin to certain positions to get it to play. By having a very simple task goal, we are able to test a naturalistic response to reinforcement in the absence of an explicit strategy in a task suited even for young children.
We used the updated reinforcement learning model to assess whether an individual's performance is consistent with understanding the task. In the case of a child who does not understand the task, we expect that they simply have motor noise on their reach, and crucially, that they would not explore more after failure, nor update their reach after success. Therefore, we used a likelihood ratio test to examine whether the preferred model was significantly better at explaining each participant's data compared to the model variant which had only motor noise (Model 1). Focusing on only the youngest children (age 3-5), this analysis showed that that 43, 59, 65 and 86% of children (out of N = 21, 22, 20 and 21) for the continuous probabilistic, discrete probabilistic, continuous deterministic, and discrete deterministic conditions, respectively, were better fit with the preferred model, indicating non-zero exploration after failure. In the 3-5 year old group for the discrete deterministic condition, 18 out of 21 had performance better fit by the preferred model, suggesting this age group understands the basic task of moving in different directions to find a rewarding location.
The reduced numbers fit by the preferred model for the other conditions likely reflects differences in the task conditions (continuous and/or probabilistic) rather than a lack of understanding of the goal of the task. We include this analysis as a new subsection at the end of the Results.
Supplementary Figure 2: the first panel should belong to a 3-year-old not a 5-year-old? How are these panels organized? This is kind of confusing.
Thank you for this comment. Figure 2—figure Supplement 1 and Figure 2—figure Supplement 2 are arranged with devices in the columns and a sample from each age bin in the rows. For example in Figure 2—figure Supplement 1, column 1, row 1 is a mouse using participant age 3 to 5 years old while column 3, row 2 is a touch screen using participant age 6 to 8 years old. We have edited the labeling on both figures to make the arrangement of the data more clear.
Line 222: make this a complete sentence.
This sentence has been edited to a complete sentence.