The human brain segments continuous experience into discrete events, with theoretical accounts proposing two distinct mechanisms: creating boundaries at points of high prediction error (mismatch between expected and observed information) and high prediction uncertainty (reduced precision in predictions). Using fMRI and computational modeling, we investigated the neural correlates of error-driven and uncertainty-driven boundaries. We developed computational models that generate boundaries based on prediction error or prediction uncertainty, and examined how both types of boundaries, and human-identified boundaries, related to fMRI pattern shifts and evoked responses. Multivariate analysis revealed a specific temporal sequence of neural pattern changes around human boundaries: early pattern shifts in anterior temporal regions (–11.9 s), followed by shifts in parietal areas (–4.5 s), and subsequent whole-brain pattern stabilization (+11.8 s). The core of this dynamic response was associated with both error-driven and uncertainty-driven boundaries. Critically, both error- and uncertainty-driven boundaries were associated with unique pattern shifts. Error-driven boundaries were associated with early pattern shifts in ventrolateral prefrontal areas, followed by pattern stabilization in prefrontal and temporal areas. Uncertainty-driven boundaries were linked to shifts in parietal regions within the dorsal attention network, with minimal subsequent stabilization. In addition, within the core regions responsive to both types of boundaries, the timing differed significantly. These findings provide evidence for two overlapping brain networks that maintain and update representations of the environment, controlled by two distinct prediction quality signals: prediction error and prediction uncertainty.
neural mechanismevent segmentationcomputational modelingResearch organismHumanhttps://ror.org/00rk2pe57Office of Naval ResearchN00014-17-1-2961ZacksJeffrey Mhttps://ror.org/01cwqze88National Institutes of HealthR01AG06243801ZacksJeffrey MThe funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.Author impact statementThe brain uses separate prediction error and prediction uncertainty signals to trigger event boundaries.publishing-routeprcIntroduction
To act effectively in dynamic environments, humans use memory and knowledge to predict how activity will unfold over time (Clark, 2013; Friston et al., 2016; Knott and Takac, 2021; Niv and Schoenbaum, 2008). Contemporary cognitive models propose that perception is guided by event models—stable representations of the current situation (Zacks, 2020). These event models integrate sensory information with prior schematic knowledge—structured representations that encode knowledge about event classes (Anderson, 1978; Bartlett, 1932; Graesser and Nakamura, 1982). For example, when serving coffee, an event model helps predict that the recipient will pick up the cup and how they will move to do so. However, for such models to remain effective, they must update appropriately when one event ends and another begins—once the coffee has been taken, a coffee-serving model becomes unhelpful (DuBrow et al., 2017); this is known as event segmentation. There has been considerable recent interest in the features that predict event segmentation (Bailey and Zacks, 2015; Brunec et al., 2018; Magliano et al., 1999; Su and Swallow, 2024; Wang and Egner, 2022), in the effects of segmentation on immediate access to information in working memory (Radvansky et al., 2010; Radvansky and Copeland, 2006), and in the role of segmentation in shaping long-term memory (Clewett et al., 2019; Ding and Zacks, 2025; DuBrow and Davachi, 2013; Sargent et al., 2013; Swallow et al., 2009). Individual and group differences in segmentation are robust predictors of memory and action performance. However, an important open question is: What control signal or signals does the cognitive system use to determine when to update event models?
One potential signal to control event model updating is prediction error—the difference between the system’s prediction and what actually occurs. A transient increase in prediction error is a valid indicator that the current model no longer adequately captures the current activity. Event Segmentation Theory (EST; Zacks et al., 2007) proposes that event models are updated when prediction error increases beyond a threshold, indicating that the current model no longer adequately captures ongoing activity. A related but computationally distinct proposal is that prediction uncertainty (also termed ‘unpredictability’) can serve as a control signal (Baldwin and Kosie, 2021). The advantage of relying on prediction uncertainty to detect event boundaries is that it is inherently proactive: the cognitive system can start looking for cues about what might come next before the next event starts (Baldwin and Kosie, 2021; Kuperberg, 2021). Building on this idea, Gumbsch et al., 2022, developed a computational model in which uncertainty dynamically directs the model’s gaze, which successfully predicted infant gaze allocation.
In naturalistic activity, prediction error and prediction uncertainty tend to co-occur. In a recent study (Nguyen et al., 2024), we used a large corpus and a formal computational model to assay the joint potential contributions of prediction error and prediction uncertainty to human event segmentation. We found that both error-driven and uncertainty-driven updating signals aligned with human segmentation and categorization judgments, but that uncertainty-driven segmentation aligned significantly better. This result argues for a unique role for prediction uncertainty in event model updating. However, it also opens up the possibility that multiple control mechanisms could govern updating of stable brain representations. One possibility is that representational updates in some brain areas may be triggered by either prediction error spikes or prediction uncertainty spikes. Another possibility (not mutually exclusive) is that updates in some brain areas may be uniquely triggered by prediction error or prediction uncertainty.
A body of fMRI research has shown that event boundaries evoke responses in multiple brain regions. Early univariate analyses found transient increases in blood oxygenation level-dependent (BOLD) activity at event boundaries within visual and parietal areas, including medial visual cortex, the precuneus, and intraparietal sulcus (Speer et al., 2007; Zacks et al., 2010; Zacks et al., 2001). Moreover, regions within the DMN showed decreased activity at musical and movie event boundaries (Burunat et al., 2024; Zacks et al., 2010). Boundary-related activity is predictive of subsequent memory, supporting a causal role for segmentation in memory formation (Ben-Yakov and Dudai, 2011). Recent research has further revealed that hippocampal responses at movie event boundaries show a graded profile, wherein greater univariate activity is associated with greater perceived boundary ‘salience’, measured by agreement across a separate group of individuals that a boundary occurred (Ben-Yakov and Henson, 2018).
More recently, multivariate approaches have provided insights into neural representations during event segmentation. One prominent approach uses hidden Markov models (HMMs) to detect moments when the brain switches from one stable activity pattern to another (Baldassano et al., 2017) during movie viewing; these periods of relative stability were referred to as ‘neural states’ to distinguish them from subjectively perceived events. Sensory regions, such as visual and auditory cortex, showed faster transitions between neural states. Multimodal regions, such as the posterior medial cortex, angular gyrus, and intraparietal sulcus, showed slower neural state shifts, and these shifts aligned with subjectively reported event boundaries. Geerligs et al., 2022; Geerligs et al., 2021, employed a different analytical approach called Greedy State Boundary Search (GSBS) to identify neural state boundaries. Their findings echoed the HMM results: short-lived neural states were observed in early sensory areas (visual, auditory, and somatosensory cortex), while longer-lasting states appeared in multimodal regions, including the angular gyrus, posterior middle/inferior temporal cortex, precuneus, anterior temporal pole, and anterior insula. Particularly prolonged states were found in higher-order regions, such as lateral and medial prefrontal cortex (PFC). They found that transitions between neural states in the default mode network (DMN) significantly coincide with human-identified event boundaries. Geerligs et al., 2022, further mapped the specific brain regions showing statistically significant overlap between neural state boundaries and perceived event boundaries, identifying the anterior cingulate cortex, dorsal medial PFC, left superior and middle frontal gyrus, and anterior insula as areas with particularly strong alignment. These findings suggest that the subjective experience of event boundaries may be supported by shifts in neural activity patterns in these brain regions.
The previous evidence about evoked responses at event boundaries indicates that these are dynamic phenomena evolving over many seconds, with different brain areas showing different dynamics (Ben-Yakov and Henson, 2018; Burunat et al., 2024; Kurby and Zacks, 2018; Speer et al., 2007; Zacks et al., 2010). Less is known about the dynamics of pattern shifts at event boundaries (e.g. whether shifts observed in higher-order regions precede or follow shifts observed in lower-level regions), because the HMM and GSBS analysis methods do not directly provide moment-by-moment measures of pattern shifts. Both the spatial and temporal aspects of evoked responses and pattern shifts at event boundaries have the potential to provide evidence about two potential control processes (error-driven and uncertainty-driven) for event model updating. However, this requires an approach that can model each potential mechanism independently and predict when event model updating driven by each mechanism should happen. Here, we combined fMRI with quantitative computational models of event segmentation that had been previously validated against human behavior (Nguyen et al., 2024). The models make predictions about when error-driven updating or uncertainty-driven updating should lead to fMRI-evoked responses and pattern shifts. This approach allowed us to characterize the spatiotemporal profiles of neural dynamics associated with event segmentation.
Our study addresses three main research questions. First, how do neural activity and neural activity patterns change before, during, and after human-identified event boundaries? We aimed to replicate and extend previous findings using both univariate and multivariate analyses. Second, are human-identified boundaries predicted by both error- and uncertainty-driven computational models? We compared the alignment between our model-derived boundaries and the segmentation judgments of human observers, testing whether both error and uncertainty signals contribute uniquely to the subjective experience of event boundaries. Third, and most importantly, do error- and uncertainty-driven boundaries engage distinct brain networks with different temporal dynamics? By comparing brain regions whose neural pattern changes are related to error-driven or uncertainty-driven boundaries and the temporal profiles of neural pattern changes, we can determine whether these computational processes rely on separate neural systems with distinct temporal signatures.
Results
In this study, we examined brain responses around event boundaries during naturalistic video viewing. Forty-five participants watched four videos depicting everyday activities while undergoing fMRI scanning (see Methods). For each activity, a separate group of 30 participants had previously segmented each movie to identify fine-grained event boundaries (Bezdek et al., 2023). The mean event length was 21.4 s (median 22.2 s, SD 16.1 s). Mean event lengths for uncertainty-driven model and error-driven model were 28.96 s and 24.7 s, respectively (Nguyen et al., 2024). We applied finite impulse response (FIR) models to analyze brain activity changes and neural pattern shifts within a window of approximately 20 s before and after human boundaries, error-driven boundaries, and uncertainty-driven boundaries. For all analyses, we described activity within regions defined by the Schaefer 400-parcel atlas (Schaefer et al., 2018).
Our analyses below addressed three main questions about the neural mechanisms of event segmentation. We first characterized how the magnitude and pattern of neural activity change around human-identified event boundaries. Next, we tested whether human event segmentation might be supported by one or both candidate cognitive mechanisms by examining the extent to which error-driven and uncertainty-driven boundaries could predict human boundaries. Finally, we investigated whether these two potential control mechanisms engage separate neural systems by comparing the brain networks and temporal dynamics associated with error-driven and uncertainty-driven boundaries.
Event boundaries were associated with increased activity in regions within the visual and dorsal attention networks, and decreased activity in regions within the default network
We first analyzed the BOLD response around human-identified event boundaries. We analyzed two effects: changes in BOLD magnitude (this section) and in local patterns of activity (following section). The magnitude analyses are a replication of previous studies (Burunat et al., 2024; Kurby and Zacks, 2018; Speer et al., 2007; Zacks et al., 2010): We time-locked the magnitude of the evoked response to the identified boundaries. Whereas previous studies were analyzed at the voxel level, here, we took advantage of modern functional anatomic parcellation (Schaefer et al., 2018) to improve sensitivity. The pattern-based analyses represent a significant new view of the brain’s response to event boundaries. In the next section, we time-locked the rate of pattern change to the event boundary. This constitutes a valuable complement to methods based on the HMM (Baldassano et al., 2017) or GSBS (Geerligs et al., 2021) by providing a continuous timecourse of pattern change around the boundary.
To analyze changes in BOLD magnitude, we employed an FIR model over a time window spanning approximately 20 s before and 20 s after human-identified event boundaries (see Methods). This analysis revealed brain regions whose BOLD activity changes significantly (FDR with alpha <0.001, corrected-p-value<0.001) around event boundaries. The strongest changes were in areas within the visual and dorsal attention networks; the precuneus and medial PFC within the default network, as well as the lateral PFC within the control network, also had significant activity changes (Figure 1). The spatial distribution of these responses is similar to previous results (Burunat et al., 2024; Kurby and Zacks, 2018; Speer et al., 2007; Zacks et al., 2010).
Regions where blood oxygenation level-dependent (BOLD) activity changed significantly (corrected p<0.001) around human boundaries.
Finite impulse response (FIR) models were used to predict parcel-wise BOLD activity within a time window spanning approximately 20 s before and 20 s after human-identified event boundaries. The z-statistics revealed significant activity changes across multiple brain networks, with particularly strong changes in the visual and dorsal attention networks, and moderate changes in the precuneus and medial prefrontal cortex (PFC) within the default model network, and lateral PFC within the control network.
We then examined these evoked responses in detail, extracting FIR-estimated BOLD activity timecourses for each parcel showing significant change over time. Parcels in the visual, dorsal, and control networks showed an increase in BOLD activity around event boundaries, whereas many default network parcels showed decreases in BOLD activity around event boundaries (Figure 2). Interestingly, some regions, such as the precuneus, displayed a biphasic response characterized by an initial increase in activity at the event boundary followed by a subsequent decrease (Figure 2, bottom). Again, these findings are consistent with prior studies examining brain activity at event boundaries (Burunat et al., 2024; Kurby and Zacks, 2018; Speer et al., 2007; Zacks et al., 2010), and suggest that event segmentation is a process that unfolds over more than 20 s, involving multiple brain networks.
Brain activity relative to estimated baseline activity –11.9 s before event boundary (top), at event boundary (middle), and 11.8 s after event boundary (bottom).
Brain maps display blood oxygenation level-dependent (BOLD) signal changes across multiple networks, with warm colors (red/yellow) indicating increased activity and cool colors (blue) indicating decreased activity. Visual, dorsal attention, and control network parcels tend to show increased BOLD activity around event boundaries, while default network regions exhibit decreased activity. Timecourses (right panels) illustrate activity patterns in specific parcels (representative timecourses observed) within control network and default network (shades indicate ±2 standard errors). Parcel labels are from the Schaefer 400×7 parcellation atlas (Schaefer et al., 2018); e.g., LH_Cont_PFCl_7 indicates a parcel which is in the left lateral prefrontal cortex and is a component of the Control network. Some regions, notably the precuneus (bottom line graph), display a biphasic response characterized by initial activity increase at the boundary followed by subsequent decrease. Frame-by-frame brain maps and BOLD activity around boundaries for all parcels can be found at https://openneuro.org/datasets/ds005551 in the derivatives/figures/brain_maps_and_timecourses/ directory.
Prefrontal and anterior temporal regions show early pattern shifts, followed by parietal shifts near boundaries, and whole-brain stabilization post-boundary
To model the effect of event boundaries on shifts in the local pattern of activity, we first calculated the amount of frame-to-frame change in the pattern of activity within each parcel and then fit FIR models to this pattern dissimilarity measure. Pattern dissimilarity was calculated as 1r, where r is the Pearson correlation between the voxel-wise activity patterns in the parcel for the two successive timepoints (see Methods for details). High pattern dissimilarity indicates that the neural patterns are more unstable (pattern shift), whereas low pattern dissimilarity indicates that neural patterns are more stable (pattern stabilization). As with the magnitude analyses, we fit the FIR model to a time window spanning approximately 20 s before and 20 s after human-identified boundaries for each parcel. This analysis revealed brain regions whose neural patterns shifted (increased dissimilarity) or stabilized (decreased dissimilarity) significantly (FDR with alpha <0.001, corrected-p-value<0.001) around event boundaries; these brain regions belong to the control (lateral PFC), default (medial PFC, posterior cingulate cortex, and temporal areas), dorsal attention (superior parietal lobule and posterior cortex), and visual (temporal occipital areas) networks (Figure 3).
Regions where pattern dissimilarity changed significantly (corrected p<0.001) around event boundaries.
Finite impulse response (FIR) models were used to predict pattern dissimilarity within a time window spanning approximately 20 s before and 20 s after human-identified event boundaries. Results show the degree of changes in pattern dissimilarity relative to estimated baseline pattern dissimilarity in parcels across multiple brain networks, including the control network (lateral PFC), default network (medial PFC, posterior cingulate cortex, and temporal areas), dorsal attention network (superior parietal lobule and posterior cortex), and visual network (temporal occipital areas).
Next, we examined the same FIR model output for timecourses of pattern dissimilarity around event boundaries. Figure 4 shows changes in pattern dissimilarity relative to estimated baseline dissimilarity over time. The analysis revealed three salient phenomena. First, early pre-boundary temporal pole pattern shifts: At the early pre-boundary timepoint (–11.9 s), pattern dissimilarity in the anterior temporal pole and orbitofrontal cortex significantly increased compared to estimated baseline dissimilarity, indicating pattern shifts in these regions. Second, immediate pre-boundary parietal pattern shifts: At the immediate pre-boundary timepoint (–4.5 s), pattern dissimilarity in postcentral and superior parietal lobule areas (part of the dorsal attention network) increased significantly compared to estimated baseline dissimilarity, reflecting notable pattern shifts; moreover, the magnitude of immediate pre-boundary pattern shifts (–4.5 s) is stronger than that of pattern shifts at the early pre-boundary timepoint (–11.9 s). Third, post-boundary whole-brain pattern stabilization: At the post-boundary timepoint (+11.8 s), pattern dissimilarity across the whole brain decreased the most compared to estimated baseline dissimilarity, indicating pattern stabilization. This period of stability may represent the establishment and maintenance of a new event model. The result that patterns started to stabilize after boundaries and reached their peak of stabilization after about 10 s suggests that the brain might take several seconds to build a new stable representation of the environment. The widespread nature of this stabilization suggests a global reconfiguration of brain states, consistent with theories proposing that event boundaries mark transitions between distinct neural states representing different events.
Event boundaries are characterized by increases in pattern dissimilarity, followed by post-boundary stability.
Brain surface maps show deviations from baseline pattern dissimilarity (orange: increased dissimilarity/pattern shifts; blue: decreased dissimilarity/pattern stabilization) at three points in time. Line plots for selected regions show pattern dissimilarity for selected parcels over time, illustrating the representative timecourses observed (shades indicate ±2 standard errors). Baseline dissimilarity levels are indicated by horizontal purple lines. Higher values reflect greater pattern shifts, while lower values indicate more stable patterns. Parcel labels followed the Schaefer 400×7 parcellation atlas (Schaefer et al., 2018). Three key phenomena emerge: (1) At early pre-boundary (–11.9 s), anterior temporal pole regions and orbitofrontal cortex showed significantly increased dissimilarity relative to baseline dissimilarity, indicating pattern shifts. (2) At immediate pre-boundary (–4.5 s), superior parietal lobule and postcentral areas within the dorsal attention network exhibited increased pattern dissimilarity, indicating pattern shifts. (3) At post-boundary (+11.8 s), pattern dissimilarity across the whole brain decreased, indicating widespread pattern stabilization. Brain maps and pattern dissimilarity around boundaries for all parcels can be found at https://openneuro.org/datasets/ds005551 in the derivatives/figures/brain_maps_and_timecourses/ directory.
The sequence of changes we observed—from temporal areas to parietal regions and finally to global stabilization—may reflect different stages of event model updating. The engagement of multiple brain regions at different timepoints also suggests that event segmentation may involve several distinct cognitive processes, each potentially implemented by different brain networks. This possibility sets the stage for our subsequent investigation of two potential event model updating control mechanisms: error-driven updating and uncertainty-driven updating. By examining the neural correlates of error-driven and uncertainty-driven updating separately, we can potentially identify dissociable neural mechanisms underlying human event segmentation.
Error-driven boundaries and uncertainty-driven boundaries uniquely predict human boundaries
To investigate error-driven and uncertainty-driven event segmentation, we developed two computational models—one in which event model updating is triggered by an increase in prediction error and another in which event model updating is triggered by an increase in prediction uncertainty (Nguyen et al., 2024). The models generated error-driven or uncertainty-driven boundaries (see Methods). Discrete human and model event boundaries were transformed into boundary densities by applying a Gaussian density kernel (bandwidth = 4.45 seconds). Model-derived boundary densities were then employed to jointly predict human boundary density (see Methods), an analysis not performed in the previous work (Nguyen et al., 2024). This analysis revealed that boundary densities from both uncertainty-driven and error-driven models uniquely predicted human boundary density (t-statistics are 14.32 and 5.14, respectively, p<0.001, df = 1636), even though error-driven boundary density is correlated with uncertainty-driven boundary density (mean correlation: 0.499, 95% CI: [0.401, 0.607]). This finding suggests that humans might rely on both types of prediction quality signals—errors in prediction and uncertainty about predictions—when segmenting events in continuous experience. Quantitatively, the combination of error-driven and uncertainty-driven boundaries explained approximately 20% of the variance in human-identified boundaries. To contextualize this result, human behavioral boundary identification typically has a reliability of approximately 0.6, which sets an upper limit of 36% for explainable variance. The relatively high explained variance suggests that our model-estimated error-driven and uncertainty-driven boundaries are good candidates for representing human error-driven and uncertainty-driven event updating processes. Model boundary densities and human boundary densities for four videos are depicted in Figure 5; boundary density peaks were used for all FIR analyses (see Methods). Based on these findings, we proceeded to use our model-estimated error-driven and uncertainty-driven boundaries as proxies for human cognitive processes in subsequent analyses. This approach allows us to investigate the neural correlates of error-driven and uncertainty-driven event segmentation separately, potentially revealing distinct brain networks associated with these two complementary processes.
Error-model, uncertainty-model, and human boundary density and peaks in the four movie stimuli.
Error-model boundary density and uncertainty-model boundary density both uniquely predict human boundary density. Boundary peaks were used for all finite impulse response (FIR) analyses.
Error-driven and uncertainty-driven boundaries are associated with distinct brain networks and temporal dynamics
Building on the observation that human-identified boundaries are associated with pattern shifts in prefrontal, parietal, and temporal regions, and on our hypothesis that these regions might be responsible for distinct event model updating mechanisms, we sought to disentangle the neural correlates of error-driven and uncertainty-driven event model updating. Our first aim was to identify whether two qualitatively different types of boundaries are linked to pattern shifting and stabilization in overlapping or distinct brain areas. Thus, for each brain parcel, we fitted two FIR models to predict pattern dissimilarity: one using only error-driven boundaries (error FIR) and another using only uncertainty-driven boundaries (uncertainty FIR), and compared their performance.
This analysis revealed a dissociation between brain regions showing significant (FDR with alpha <0.001, corrected-p-value<0.001) neural pattern shift or stabilization around error-driven versus uncertainty-driven boundaries (Figure 6). Regions exhibiting strong neural pattern shift or stabilization around error-driven boundaries included the ventrolateral PFC (Figure 6, reddish regions). Regions exhibiting strong neural pattern shift or stabilization around uncertainty-driven boundaries included the postcentral gyrus, particularly those associated with the dorsal attention network, areas in the mid cingulate cortex associated with the ventral attention network, and areas within the visual network (Figure 6, bluish regions). Regions exhibiting strong neural pattern shift or stabilization around both types of boundaries included the medial PFC and temporal components of the default network (Figure 6, magenta regions). These results reveal distinct brain networks whose neural pattern shifted or stabilized around error-driven or uncertainty-driven boundaries, each partially aligning with regions showing neural pattern shift or stabilization around human boundaries (Figure 3). The identification of two dissociable sets of brain regions associated with each boundary type, and the finding that both sets partially overlapped with regions involved in human-identified boundaries, suggests that both error-driven and uncertainty-driven signals play roles in event model updating. This neural evidence corroborates and extends the behavioral findings that both types of boundaries uniquely predict human-identified boundaries. To account for the correlation between uncertainty-driven boundaries and error-driven boundaries, we also fitted an FIR model that predicted pattern dissimilarity from both types of boundaries (combined FIR) for each parcel. Then, we performed two likelihood ratio tests: combined FIR to error FIR, which measures the unique contribution of uncertainty boundaries to pattern dissimilarity, and combined FIR to uncertainty FIR, which measures the unique contribution of error boundaries to pattern dissimilarity. The analysis also revealed two dissociable sets of brain regions associated with each boundary type (see Figure 6—figure supplement 1).
Regions showing significant (corrected p<0.001) changes in pattern dissimilarity relative to estimated baseline dissimilarity for error-driven and/or uncertainty-driven boundaries.
Regions in red show stronger changes in pattern dissimilarity relative to estimated baseline pattern dissimilarity for error-driven boundaries, while regions in blue show stronger changes in pattern dissimilarity relative to estimated baseline pattern dissimilarity for uncertainty-driven boundaries, and regions in magenta show strong changes in pattern dissimilarity for both boundary types.
Regions showing unique contributions of error-driven and uncertainty-driven boundaries to changes in pattern dissimilarity (corrected p<0.001).
Unique effects were estimated using likelihood ratio tests comparing the combined finite impulse response (FIR) model with models including only one boundary type. Regions in red show stronger unique contributions of error-driven boundaries, while regions in blue show stronger unique contributions of uncertainty-driven boundaries, and regions in magenta show strong unique contributions of both boundary types.
Next, we used the two FIR models to investigate the timecourse of changes in pattern dissimilarity around error-driven and uncertainty-driven boundaries. This analysis is crucial for understanding the temporal sequence of cognitive processes involved in event segmentation and how they might differ between these two mechanisms. It revealed distinct spatial and temporal changes in pattern dissimilarity, which partially overlap with the changes observed for human-identified boundaries discussed previously. Some regions showed early pre-boundary (–11.9 s) neural pattern shifts: For error-driven boundaries, increased pattern dissimilarity was observed in the ventrolateral PFC, which is part of the control network (Figure 7A, top). For uncertainty-driven boundaries, we observed increased pattern dissimilarity (indicating pattern shifts) in temporal areas and the PFC (Figure 7B, top). These early pattern shifts associated with error-driven and uncertainty-driven boundaries corresponded to subcomponents of the early shifts observed before human-identified boundaries (Figure 4, top). Another group of regions showed immediate pre-boundary (–4.5 s) pattern shifts: error-driven boundaries were associated with increased pattern dissimilarity primarily in the left ventrolateral PFC and anterior temporal pole (Figure 7A, middle). Uncertainty-driven boundaries were associated with increased pattern dissimilarity in parietal areas, occipital areas, temporal areas, and prefrontal areas, with strong shifts in regions within the dorsal attention network (Figure 7B, middle). These pattern shifts for uncertainty-driven boundaries corresponded to our second observation from human boundaries, for which parietal regions showed significant pattern shifts (Figure 4, middle). Moreover, pattern shifts for both error-driven boundaries and uncertainty-driven boundaries were higher and involved more brain areas at immediate pre-boundary (–4.5 s) compared to early pre-boundary (–11.9 s). Finally, many regions showed post-boundary (+11.8 s) pattern stabilization: error-driven boundaries were followed by widespread decreases in pattern dissimilarity in prefrontal, temporal, and occipital parcels, with strongest stabilization in the PFC (Figure 7A, bottom). The pattern stabilization following error-driven boundaries corresponds with our third observation from human boundaries, where pattern stabilization was strongest for the PFC (Figure 4, bottom). For uncertainty-driven boundaries, we only observed decreased pattern dissimilarity in part of the medial PFC (Figure 7B, bottom).
Changes in pattern dissimilarity relative to baseline dissimilarity around boundaries identified by (<bold>A</bold>) error-driven model and (<bold>B</bold>) uncertainty-driven model.
Brain surface maps show deviations from baseline pattern dissimilarity (orange: increased dissimilarity/pattern shifts; blue: decreased dissimilarity/pattern stabilization). Line plots for selected regions show pattern dissimilarity over time, with baseline dissimilarity levels indicated by horizontal blue (uncertainty-driven) or red (error-driven) lines (shades indicate ±2 standard errors). Higher dissimilarity values reflect greater pattern shifts, while lower dissimilarity values indicate more stable patterns. Parcel labels followed the Schaefer 400×7 parcellation atlas (Schaefer et al., 2018). At early pre-boundary (–11.9 s), error-driven boundaries corresponded to increased pattern dissimilarity primarily in ventrolateral prefrontal cortex (PFC), while uncertainty-driven boundaries corresponded to more widespread pattern shifts in temporal areas, dorsomedial and dorsolateral prefrontal regions, and anterior temporal cortex. At immediate pre-boundary (–4.5 s), error-driven boundaries corresponded to pattern shifts predominantly in the left ventrolateral PFC and anterior temporal pole, whereas uncertainty-driven boundaries corresponded to more extensive pattern shifts across parietal, occipital, temporal, and prefrontal areas, particularly strong within the dorsal attention network. At post-boundary timepoints (+11.8 s), error-driven boundaries corresponded to widespread pattern stabilization (decreased dissimilarity) across prefrontal, temporal, and occipital areas with strongest effects in prefrontal cortex, while uncertainty-driven boundaries showed limited pattern stabilization restricted primarily to portions of the medial PFC. Brain maps and pattern dissimilarity around error-driven boundaries or uncertainty-driven boundaries for all parcels can be found at https://openneuro.org/datasets/ds005551 in the derivatives/figures/brain_maps_and_timecourses/ directory. Brain maps and neural activity around error-driven and uncertainty-driven boundaries for all parcels can also be found in the same directory.
The above analysis revealed shared brain regions whose changes in pattern dissimilarity were associated with both error-driven and uncertainty-driven boundaries (magenta regions in Figure 6). Although these two boundary types evoked neural pattern shift or stabilization in overlapping brain regions, they may drive distinct temporal dynamics of pattern shift and stabilization. To test this hypothesis, for each parcel that showed significant changes in pattern dissimilarity for both boundary types (FDR with alpha <0.001, corrected-p-value<0.001 for both error-driven and uncertainty-driven FIR models), we computed correlations between pattern dissimilarity timecourses for error-driven and uncertainty-driven boundaries and compared them against a null distribution. This null distribution was constructed from within-boundary-type correlations, using bootstrapped timecourses for both error-driven and uncertainty-driven boundaries separately. The resulting z-statistic maps (Figure 8) revealed statistically significant (FDR with alpha <0.001, corrected-p-value<0.001) differences in temporal dynamics between the two boundary types, particularly in prefrontal and temporal regions. For example, two parcels in the left medial PFC (LH_Default_PFCm_4) and in the right anterior temporal pole (RH_Default_Temp_1), whose changes in pattern dissimilarity were associated with both error-driven and uncertainty-driven boundaries, had different temporal dynamics for each boundary type (Figure 7A, middle and Figure 7B, top). This result suggests that even though error-driven and uncertainty-driven event model updating were associated with the same brain regions, the two updating mechanisms were associated with distinct temporal dynamics of pattern shift and stabilization in these shared regions.
Comparison of pattern dissimilarity timecourses between error-driven and uncertainty-driven boundaries.
Z-statistics maps show regions where pattern dissimilarity timecourses differ between the two boundary types, with higher values (more orange) indicating stronger differences in temporal dynamics. The most pronounced differences were observed in the prefrontal cortex, posterior parietal regions, and portions of the temporal cortex.
Discussion
Humans and other animals navigate a world that is complex and dynamic, but that is also characterized by stable dynamical regimes during which sequential activity is predictable. To cope with the complexity of experience and to leverage the intervals of predictability, the nervous system constructs stable models of the current situation on a hierarchy of timescales from seconds to tens of minutes (Baldassano et al., 2017; Hasson et al., 2015; Richmond and Zacks, 2017). The present study addressed a central question about these event models: What control mechanism determines when they are updated?
We first characterized the neural signatures of human event segmentation by examining both univariate activity changes and multivariate pattern changes around subjectively identified event boundaries. Using multivariate pattern dissimilarity, we observed a structured progression of neural reconfiguration surrounding human-identified event boundaries. The largest pattern shifts were observed near event boundaries (~4.5 s before) in dorsal attention and parietal regions; these correspond to regions identified by Geerligs et al. as shifting their patterns on a fast to intermediate timescale (2022). We also observed smaller pattern shifts roughly 12 s prior to event boundaries in higher-order regions within anterior temporal cortex and PFC, and these are slow-changing regions identified by Geerligs et al., 2022. This is puzzling. One prevalent proposal, based on the idea of a cortical hierarchy of increasing temporal receptive windows (TRWs), suggests that higher-order regions should update representations after lower-order regions do (Chang et al., 2021). In this view, areas with shorter TRWs (e.g. word-level processors) pass information upward, where it is integrated into progressively larger narrative units (phrases, sentences, events). This proposal predicts neural shifts in higher-order regions to follow those in lower-order regions. By contrast, our findings indicate the opposite sequence. Our findings suggest that the brain might engage in top-down event representation updating, with changes in coarser-grain representations propagating downward to influence finer-grain representations (Friston, 2005; Kuperberg, 2021). For example, in a narrative where the main goal is achieved midway—such as a detective solving a mystery before the story formally ends—higher-order regions might update the overarching event representation at that point, and this updated model could then cascade down to reconfigure how lower-level regions process the remaining sensory and contextual details. In the period after a boundary (around +12 s), we found widespread stabilization of neural patterns across the brain, suggesting the establishment of a new event model. Future work could focus on understanding the mechanisms behind the temporal progression of neural pattern changes around event boundaries.
We then took a computational cognitive neuroscience approach to disentangle the contributions of two potential updating mechanisms—prediction error and prediction uncertainty—in driving event model updating. Building on the theoretical framework proposed by Baldwin and Kosie, 2021, we developed two computational models to generate boundaries driven by prediction error or prediction uncertainty. Both models uniquely explained variance in human boundary judgments. Together, these model-derived boundaries explained more than half of the explainable variance in human segmentation. This result provides evidence that both prediction error and prediction uncertainty contribute to how observers parse ongoing experience into events. More importantly, error-driven and uncertainty-driven boundaries were associated with distinct brain networks exhibiting different temporal dynamics. Error-driven boundaries were linked to early neural pattern shifts in ventrolateral prefrontal areas, followed by pronounced pattern stabilization in prefrontal and temporal regions. In contrast, uncertainty-driven boundaries were associated with pattern shifts across wide areas of the brain, especially in parietal regions within the dorsal attention network, and with minimal subsequent stabilization in part of the medial PFC. Interestingly, both boundary types evoked pattern shifts in overlapping regions within the default network, but with different temporal dynamics. The dissociations in both spatial and temporal neural signatures provide evidence for the theoretical distinction proposed by Baldwin and Kosie, 2021, that humans rely on both prediction error and prediction uncertainty to parse continuous activities into meaningful events. Our findings suggest that event segmentation is supported by two separate but complementary brain networks, each responsive to a different prediction quality signal. This dual-network architecture represents a significant advancement in our understanding of event segmentation, extending the claim of EST that declines in prediction quality trigger event model updating (Zacks et al., 2007) to encompass these two aspects of prediction quality: accuracy and certainty.
Previous studies using HMMs (Baldassano et al., 2017) and GSBS Geerligs et al., 2021; Geerligs et al., 2022 have identified neural state boundaries in multimodal regions that align closely with subjectively perceived event boundaries. Consistent with these findings, our multivariate analyses also revealed pattern shifts around human boundaries in regions previously identified, including areas within the DMN (medial PFC, posterior cingulate cortex, and anterior temporal regions) and dorsal attention network. Despite the advances made in this study, several limitations and open questions remain. Although our computational models successfully captured a substantial portion of variance in human boundary judgments, approximately 16% of the explainable variance remains unaccounted for. This suggests that additional mechanisms beyond prediction error and prediction uncertainty may contribute to event segmentation. One candidate mechanism comes from latent cause inference models (Kuperberg, 2021; Shin and DuBrow, 2021). Whereas Zacks et al., 2007, and Baldwin and Kosie, 2021, assume the cognitive system maintains an event model and uses prediction error or prediction uncertainty from that model to segment events, latent cause inference models start from a different assumption. These models posit that the cognitive system monitors a probability distribution over unobservable latent causes that are most likely to generate current sensory observations. Event boundaries in this framework correspond to moments of high uncertainty over which latent cause is currently active. Future studies could develop computational models based on latent cause inference principles and compare their neural signatures with those of error-driven and uncertainty-driven boundaries identified in our study.
Although our error-driven and uncertainty-driven models segmented and categorized continuous experience in a manner that aligns with human behavior (Nguyen et al., 2024), the correspondence is far from perfect. Consequently, the event boundaries identified by these two models might not fully capture the actual error-driven and uncertainty-driven processes that humans employ. The current evidence for more than one model updating mechanism opens the door for an architecture in which a handful of different triggers may govern different components of event model updating. Such an architecture would be inelegant—but evolution’s engineering solutions are sometimes complex and kludgy rather than simple and sleek.
In conclusion, this study provides behavioral and neural evidence for two overlapping brain networks that maintain and update representations of the environment, controlled by either prediction error or prediction uncertainty. Error-driven boundaries were uniquely associated with representational shifts in ventrolateral PFC and subsequent pattern stabilization in temporal and prefrontal areas. Uncertainty-driven boundaries were uniquely linked to representational shifts in the dorsal attention network with minimal subsequent stabilization. Both boundary types were associated with pattern shifts in regions within the DMN, such as medial PFC and temporal areas, which have previously been hypothesized to maintain and update event representations. By identifying the neural correlates of theoretically distinct updating mechanisms, these results form a basis for an integrative account of the computational and neurophysiological aspects of how the brain maintains stable representations of complex everyday activity.
MethodsHuman and model segmentation
Human event boundaries for four activities were obtained from a norming study in which 30 online participants (from Mechanical Turk) watched each activity and marked event boundaries (Bezdek et al., 2023). Error-driven boundaries and uncertainty-driven boundaries were generated by two computational models developed previously (Nguyen et al., 2024). Both models maintain an active event representation that continuously predicts how activity will unfold. The error-driven model triggers event boundary detection when prediction error—operationalized as the Euclidean distance between observed and predicted scene vectors—exceeds a threshold. In contrast, the uncertainty-driven model triggers event boundary detection based on prediction uncertainty, conceptualized as epistemic uncertainty that reflects the model’s confidence in its own predictions given its current knowledge state. This prediction uncertainty is computed by generating 32 different predictions using random dropout on the model weights and measuring the variance across these predictions (Gal and Ghahramani, 2016; Kendall and Gal, 2017). When their respective thresholds are exceeded, each model initiates an inference process to evaluate alternative event representations. Event boundaries are identified at moments when this inference process results in a switch from one event representation to another, reflecting the model’s decision that the current event representation no longer adequately captures ongoing activity. To account for model random initialization, we ran 16 simulations for each model, and we used event boundaries from all 16 simulations. The error-driven and uncertainty-driven models were tuned such that their median number of boundaries was as close as possible to the human fine-grained median number of boundaries (Nguyen et al., 2024). To analyze the relationships between the three kinds of boundaries, we applied a Gaussian kernel density function (bandwidth = 4.45 s) on the human, error-driven, and uncertainty-driven event boundaries to obtain continuous event boundary densities. In later analyses, error-driven and uncertainty-driven boundary densities were used as continuous predictors of the normed human boundary density.
To model the effects of human and model event boundaries on fMRI magnitude and pattern dissimilarity, we identified normative boundaries by selecting peaks in the event boundary densities. To identify peaks from continuous boundary densities, we ran a greedy algorithm to iteratively select one peak at a time with the highest density value, with a condition that the current peak is more than 7 s away from already selected peaks. The number of peaks for each movie was set to the median number of boundaries identified by the Bezdek et al., 2023, study participants for each movie (Figure 5). These peaks were used in the FIR analyses as reference points, such that each ‘trial’ in the analysis represented a window around the event boundary (see below).
Imaging sessionfMRI participants
Imaging data were collected from 47 participants (age mean = 23.3 years, SD = 4.3, range 18–35; 32 female, 15 male). Their self-reported demographic groupings were 14 Asian, 2 Black/African-American, 26 White (5 Hispanic/Latino), and 5 More Than One Race. Imaging data from two additional participants (both White males, one Hispanic/Latino, 19 and 23 years) could not be obtained (equipment failure, incidental finding) and are not included here. All participants were right-handed, fluent English speakers, and without MRI safety contraindications or neurological impairments. Participants provided written informed consent in accordance with the Institutional Review Board at Washington University in St. Louis and received $125 compensation for completion of all sessions.
Acquisition parameters
All scans were acquired using a 3T Siemens Prisma with a 64-channel head coil in the East Building MR Facility of the Washington University Medical Center, between May 2021 and May 2022. Imaging files were sent from the scanner to the Washington University Central Neuroimaging Data Archive (Gurney et al., 2017) XNAT (RRID:SCR_003048) database for permanent archiving. FIRMM (Dosenbach et al., 2017) was used to monitor participant motion throughout the imaging session. Scan duration was not modified by FIRMM, but if it indicated excessive movement, the experimenter notified the participant and repositioned them, as needed.
Each imaging session began with T1- and T2-weighted high-resolution ABCD MPRAGE+PMC structural scans (T1: 1 mm isotropic voxels; 2.5 s TR, 2.9 ms TE, 1.07 s TI, 8° flip angle; T2: 1 mm isotropic voxels; 3.2 s TR, 0.565 s TE, 120° flip angle), followed by four functional task runs. The functional (BOLD) scans were acquired with CMRR multiband sequences (University of Minnesota Center for Magnetic Resonance Research) (Feinberg et al., 2010; Xu et al., 2013), multiband (simultaneous multislice) factor 4, without in-plane acceleration (iPat = none), resulting in 2.08 mm isotropic voxels, 1.483 s TR (TE 37 ms, flip angle 52°; protocol sheets at https://openneuro.org/datasets/ds005551 under derivatives/ Scanning_parameters_NP1157new_May8_2021.pdf). All functional runs were collected with anterior-to-posterior (‘AP’) encoding direction, and proceeded by a pair (AP, PA) of spin-echo field maps and a single-band reference (‘SBRef’) image.
Task runs
All participants completed four task fMRI runs, each of which consisted of watching a silent video. The four video stimuli (and accompanying normative event boundaries used for later analyses) are from the Multi-angle Extended Three-dimensional Activity (META) stimulus set (Bezdek et al., 2023). Each video depicts a different actor carrying out different everyday activities (exercising, cleaning a room, making breakfast, bathroom grooming), all filmed from a fixed camera position.
A different video was shown in each fMRI run, with all participants viewing the same four videos in the same order (Table 1). Participants were instructed, ‘As you watch, please pay attention and try to remember what occurs, as your memory will be tested later.’ A brief (8–11 s) fixation cross preceded the start of each film and reappeared when the film completed.
Video stimuli characteristics and fMRI scanning parameters for each run.
Run 1
Run 2
Run 3
Run 4
Video file name
1.2.3_C1_trim.mp4
6.3.9_C1_trim.mp4
3.1.3_C1_trim.mp4
2.4.1_C1_trim.mp4
Video file location
https://osf.io/7hqr9
https://osf.io/73yqe
https://osf.io/pdg43
https://osf.io/bm589
Video duration
9:44 min
11:14 min
9:43 min
10:34 min
fMRI run duration
405 TRs; 10 min
467 TRs; 12 min
404 TRs; 10 min
444 TRs; 11 min
Image processing
Image preprocessing was performed by fMRIPrep 20.2.3 (Esteban et al., 2018a; Esteban et al., 2018b; RRID:SCR_016216), which is based on Nipype 1.6.1 (Gorgolewski et al., 2011; Gorgolewski et al., 2018; RRID:SCR_002502); boilerplate text is provided in the Supplemental Information. Both the preprocessed images (spatially normalized to the MNI152NLin2009cAsym template) and realignment parameters (_desc-confounds_timeseries.tsv) were used in later analyses. Image quality was evaluated by reviewing temporal mean and standard deviation images (Etzel, 2023) and deemed acceptable in all participants. Frames with more than 0.5 mm FD were censored. Participants were retained only if more than half their data was usable; two participants had two or more runs at the 20% censoring threshold and so were dropped from analysis. Quality control summary files are available at https://openneuro.org/datasets/ds005551 under derivatives/QCKnitrs/directory.
Preprocessed fMRI data was further processed by regressing drifts (linear, polynomial, and cubic) and six motion confounds from every voxel BOLD time series. Next, a standard parcellation was used: the Schaefer 400 parcels (17-network labels) for cortex (Schaefer et al., 2018). All analyses were performed at the parcel level. All voxel BOLD time series were shifted by four TR (approximately 5.93 s) to account for hemodynamic lag.
Parcel BOLD activity and parcel pattern dissimilarity
To quantify parcel BOLD activity, we averaged across all voxels in every parcel, resulting in a single value for each parcel in each timepoint (TR) of the four functional runs.
To quantify parcel neural pattern shifts, we calculated temporal pattern dissimilarity for each brain parcel using the Schaefer 400-parcel atlas (Schaefer et al., 2018). For each timepoint (TR t), we computed the dissimilarity between patterns before and after that timepoint. Specifically, we first created stable pattern representations by averaging voxel activities across three consecutive TRs preceding the timepoint (TRs t-3, t-2, and t-1) and three consecutive TRs following the timepoint (TRs t+1, t+2, and t+3). This average approach reduces noise inherent in single-TR measurements. We then calculated the Pearson correlation between these two averaged patterns and subtracted the correlation value from 1 to obtain a dissimilarity measure.
FIR analysis
For each parcel, we employed FIR analysis to examine the temporal dynamics of BOLD activity (average across voxels) and pattern shifts around event boundaries. The FIR model is a flexible extension of the general linear model (GLM) commonly used in fMRI analysis. Whereas GLM approaches to fMRI modeling often assume a fixed hemodynamic response function (HRF), the FIR model makes no assumptions about the shape of the response, instead estimating separate coefficients for each timepoint (lag) relative to event boundaries. The coefficients from the FIR model indicate changes relative to baseline, which can be conceptualized as the expected value when far from event boundaries. This approach is particularly valuable for investigating neural responses to event boundaries, where the temporal profile may differ from canonical HRF shapes and vary across brain regions. In our study, there were three types of boundary: human-annotated boundaries and computationally derived boundaries (error-driven and uncertainty-driven models). We constructed separate FIR models for each boundary type, allowing us to compare how different types of event segmentation relate to brain responses in different parcels. This approach enabled us to distinguish between neural responses associated with computationally derived error-driven versus uncertainty-driven boundaries.
Individual FIR models were fit for each subject (first-level analysis) to estimate coefficients for each time lag relative to each type of boundary onsets. There were 10 time lags before the event boundary and 20 time lags after the event boundary, and successive time lags are 1.483 s apart (1 TR). FIR models were applied separately to both BOLD activity and pattern dissimilarity measures. After first-level analysis, we performed second-level analysis across subjects. Z-statistics were calculated by comparing the full FIR model to an intercept-only model that assumed no changes in BOLD activity or pattern dissimilarity in response to boundaries. These z-statistics were used to identify brain regions that showed significant BOLD activity changes and/or pattern shifts in response to different boundary types. Instead of considering only across-subject variance when estimating standard errors for coefficients, we incorporated both within-subject and across-subject variance components to better estimate the standard errors associated with coefficients at different lags. The custom R code to estimate both within-subject and across-subject variance is available at: https://openneuro.org/datasets/ds005551 under derivatives/scripts/fir_analyses.Rmd.
Additional informationCompeting interests
No competing interests declared
Author contributions
Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing – original draft, Writing – review and editing
Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing – review and editing
Conceptualization, Data curation, Software, Formal analysis, Investigation, Visualization, Methodology, Writing – review and editing
Conceptualization, Data curation, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Project administration, Writing – review and editing
Ethics
Participants provided written informed consent and consent to publish in accordance with the Institutional Review Board at Washington University in St. Louis, and received $125 compensation for completion of all sessions.
Additional filesData availability
BIDS data for this project can be found at https://openneuro.org/datasets/ds005551 under sub <participant_id> directories. All the data and analysis scripts used for this project can be found at https://openneuro.org/datasets/ds005551 under derivatives/scripts/ directory .
The following dataset was generated:
BezdekMZacksJNguyenTTEtzelJ2025Testing neural mechanisms of event segmentation with fMRIOpenNeurods005551
Acknowledgements
The authors would like to thank Sarah Morse and Grace Zhou for fMRI data collection, Todd S Braver for helpful feedback on data analyses, Hayoung Song for feedback on the manuscript, and the members of the Dynamic Cognition Laboratory for their thoughtful input.
ReferencesAbrahamAPedregosaFEickenbergMGervaisPMuellerAKossaifiJGramfortAThirionBVaroquauxG2014Machine learning for neuroimaging with scikit-learnFrontiers in Neuroinformatics81410.3389/fninf.2014.0001424600388AndersonRC1978Schema-directed processes in language comprehensionLesgoldAMPellegrinoJWCognitive Psychology and InstructionSpringer678210.1007/978-1-4684-2535-2_8AvantsBBEpsteinCLGrossmanMGeeJC2008Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brainMedical Image Analysis12264110.1016/j.media.2007.06.00417659998BaileyHRZacksJM2015Situation model updating in young and older adults: Global versus incremental mechanismsPsychology and Aging3023224410.1037/a003908125938248BaldassanoCChenJZadboodAPillowJWHassonUNormanKA2017Discovering event structure in continuous narrative perception and memoryNeuron9570972110.1016/j.neuron.2017.06.04128772125BaldwinDAKosieJE2021How does the mind render streaming experience as events?Topics in Cognitive Science137910510.1111/tops.1250232529736BartlettFC1932Remembering: A Study in Experimental and Social PsychologyCambridge University Press10.1017/CBO9780511759185BehzadiYRestomKLiauJLiuTT2007A component based noise correction method (CompCor) for BOLD and perfusion based fMRINeuroImage379010110.1016/j.neuroimage.2007.04.04217560126Ben-YakovADudaiY2011Constructing realistic engrams: poststimulus activity of hippocampus and dorsal striatum predicts subsequent episodic memoryThe Journal of Neuroscience319032904210.1523/JNEUROSCI.0702-11.201121677186Ben-YakovAHensonRN2018The hippocampal film editor: sensitivity and specificity to event boundaries in continuous experienceThe Journal of Neuroscience38100571006810.1523/JNEUROSCI.0524-18.201830301758BezdekMANguyenTTHallCSBraverTSBobickAFZacksJM2023The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognitionBehavior Research Methods553629364410.3758/s13428-022-01980-836217005BrunecIKMoscovitchMBarenseMD2018Boundaries shape cognitive representations of spaces and eventsTrends in Cognitive Sciences2263765010.1016/j.tics.2018.03.01329706557BurunatILevitinDJToiviainenP2024Breaking (musical) boundaries by investigating brain dynamics of event segmentation during real-life music-listeningPNAS121e231945912110.1073/pnas.231945912139186645ChangCHCNastaseSAHassonU2021Information flow across the cortical timescales hierarchy during narrative constructionbioRxiv10.1101/2021.12.01.470825ClarkA2013Whatever next? Predictive brains, situated agents, and the future of cognitive scienceThe Behavioral and Brain Sciences3618120410.1017/S0140525X1200047723663408ClewettDDuBrowSDavachiL2019Transcending time in the brain: How event memories are constructed from experienceHippocampus2916218310.1002/hipo.2307430734391CoxRWHydeJS1997Software tools for analysis and visualization of fMRI dataNMR in Biomedicine1017117810.1002/(SICI)1099-1492(199706/08)10:4/5<171::AID-NBM453>3.0.CO;2-LDingYZacksJM2025Temporal order memory in naturalistic events is scaffolded by semantic knowledge and hierarchical event structurePsyArXiv10.31234/osf.io/scaf8_v1DosenbachNUFKollerJMEarlEAMiranda-DominguezOKleinRLVanANSnyderAZNagelBJNiggJTNguyenALWesevichVGreeneDJFairDA2017Real-time motion analytics during brain MRI improve data quality and reduce costsNeuroImage161809310.1016/j.neuroimage.2017.08.02528803940DuBrowSDavachiL2013The influence of context boundaries on memory for the sequential order of eventsJournal of Experimental Psychology. General1421277128610.1037/a003402423957281DuBrowSRouhaniNNivYNormanKA2017Does mental context drift or shift?Current Opinion in Behavioral Sciences1714114610.1016/j.cobeha.2017.08.00329335678EstebanOBlairRMarkiewiczCJBerleantSLMoodieCMaF2018aFMRIPrepZenodohttps://doi.org/10.5281/zenodo.852659EstebanOMarkiewiczCJBlairRWMoodieCAIsikAIErramuzpeAKentJDGoncalvesMDuPreESnyderMOyaHGhoshSSWrightJDurnezJPoldrackRAGorgolewskiKJ2018bfMRIPrep: a robust preprocessing pipeline for functional MRINature Methods1611111610.1038/s41592-018-0235-430532080EtzelJA2023Efficient evaluation of the Open QC task fMRI datasetFrontiers in Neuroimaging2107027410.3389/fnimg.2023.107027437554632FeinbergDAMoellerSSmithSMAuerbachERamannaSGuntherMGlasserMFMillerKLUgurbilKYacoubE2010Multiplexed echo planar imaging for sub-second whole brain FMRI and fast diffusion imagingPLOS ONE5e1571010.1371/journal.pone.001571021187930FonovVSEvansACMcKinstryRCAlmliCRCollinsDL2009Unbiased nonlinear average age-appropriate brain templates from birth to adulthoodNeuroImage47S10210.1016/S1053-8119(09)70884-5FristonK2005A theory of cortical responsesPhilosophical Transactions of the Royal Society B36081583610.1098/rstb.2005.1622FristonKFitzGeraldTRigoliFSchwartenbeckPDohertyJOPezzuloG2016Active inference and learningNeuroscience & Biobehavioral Reviews6886287910.1016/j.neubiorev.2016.06.022GalYGhahramaniZ2016Dropout as a bayesian approximation: representing model uncertainty in deep learningProceedings of the 33rd International Conference on Machine LearningGeerligsLvan GervenMGüçlüU2021Detecting neural state transitions underlying event segmentationNeuroImage23611808510.1016/j.neuroimage.2021.11808533882350GeerligsLGözükaraDOetringerDCampbellKLvan GervenMGüçlüU2022A partially nested cortical hierarchy of neural states underlies event segmentation in the human braineLife11e7743010.7554/eLife.7743036111671GorgolewskiKBurnsCDMadisonCClarkDHalchenkoYOWaskomMLGhoshSS2011Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in pythonFrontiers in Neuroinformatics51310.3389/fninf.2011.0001321897815GorgolewskiKJEstebanOMarkiewiczCJZieglerEEllisDGNotterMPJareckaD2018NipypeZenodohttps://doi.org/10.5281/zenodo.596855GraesserACNakamuraGV1982The impact of a schema on comprehension and memoryPsychology of Learning and Motivation165910910.1016/S0079-7421(08)60547-2GreveDNFischlB2009Accurate and robust brain image alignment using boundary-based registrationNeuroImage48637210.1016/j.neuroimage.2009.06.06019573611GumbschCAdamMElsnerBMartiusGButzMV2022Developing hierarchical anticipations via neural network-based event segmentationIEEE International Conference on Development and Learning (ICDL)London, United Kingdom1810.1109/ICDL53763.2022.9962224GurneyJOlsenTFlavinJRamaratnamMArchieKRansfordJHerrickRWallaceLClineJHortonWMarcusDS2017The washington university central neuroimaging data archiveNeuroImage14428729310.1016/j.neuroimage.2015.09.06026439514HassonUChenJHoneyCJ2015Hierarchical process memory: memory as an integral component of information processingTrends in Cognitive Sciences1930431310.1016/j.tics.2015.04.00625980649JenkinsonMSmithS2001A global optimisation method for robust affine registration of brain imagesMedical Image Analysis514315610.1016/s1361-8415(01)00036-611516708JenkinsonMBannisterPBradyMSmithS2002Improved optimization for the robust and accurate linear registration and motion correction of brain imagesNeuroImage1782584110.1006/nimg.2002.113212377157KendallAGalY2017What uncertainties do we need in bayesian deep learning for computer visionAdvances in Neural Information Processing SystemsKnottATakacM2021Roles for event representations in sensorimotor experience, memory formation, and language processingTopics in Cognitive Science1318720510.1111/tops.1249732190996KuperbergGR2021Tea with milk? a hierarchical generative framework of sequential event comprehensionTopics in Cognitive Science1325629810.1111/tops.1251833025701KurbyCAZacksJM2018Preserved neural event segmentation in healthy older adultsPsychology and Aging3323224510.1037/pag000022629446971LanczosC1964Evaluation of noisy dataJournal of the Society for Industrial and Applied Mathematics Series B Numerical Analysis1768510.1137/0701007MaglianoJPTrabassoTGraesserAC1999Strategic processing during comprehensionJournal of Educational Psychology9161562910.1037//0022-0663.91.4.615NguyenTTBezdekMAGershmanSJBobickAFBraverTSZacksJM2024Modeling human activity comprehension at human scale: prediction, segmentation, and categorizationPNAS Nexus345910.1093/pnasnexus/pgae45939445050NivYSchoenbaumG2008Dialogues on prediction errorsTrends in Cognitive Sciences1226527210.1016/j.tics.2008.03.00618567531PowerJDMitraALaumannTOSnyderAZSchlaggarBLPetersenSE2014Methods to detect, characterize, and remove motion artifact in resting state fMRINeuroImage8432034110.1016/j.neuroimage.2013.08.04823994314RadvanskyGACopelandDE2006Walking through doorways causes forgetting: Situation models and experienced spaceMemory & Cognition341150115610.3758/BF0319326117128613RadvanskyGATamplinAKKrawietzSA2010Walking through doorways causes forgetting: Environmental integrationPsychonomic Bulletin & Review1790090410.3758/PBR.17.6.90021169587RichmondLLZacksJM2017Constructing experience: event models from perception to actionTrends in Cognitive Sciences2196298010.1016/j.tics.2017.08.00528899609SargentJQZacksJMHambrickDZZacksRTKurbyCABaileyHREisenbergMLBeckTM2013Event segmentation ability uniquely predicts event memoryCognition12924125510.1016/j.cognition.2013.07.00223942350SatterthwaiteTDElliottMAGerratyRTRuparelKLougheadJCalkinsMEEickhoffSBHakonarsonHGurRCGurREWolfDH2013An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity dataNeuroImage6424025610.1016/j.neuroimage.2012.08.05222926292SchaeferAKongRGordonEMLaumannTOZuoXNHolmesAJEickhoffSBYeoBTT2018Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRICerebral Cortex283095311410.1093/cercor/bhx17928981612ShinYSDuBrowS2021Structuring memory through inference-based event segmentationTopics in Cognitive Science1310612710.1111/tops.1250532459391SpeerNKZacksJMReynoldsJR2007Human brain activity time-locked to narrative event boundariesPsychological Science1844945510.1111/j.1467-9280.2007.01920.x17576286SuXSwallowKM2024People can reliably detect action changes and goal changes during naturalistic perceptionMemory & Cognition521093111110.3758/s13421-024-01525-838315292SwallowKMZacksJMAbramsRA2009Event boundaries in perception affect memory encoding and updatingJournal of Experimental Psychology. General13823625710.1037/a001563119397382TustisonNJAvantsBBCookPAZhengYEganAYushkevichPAGeeJC2010N4ITK: improved N3 bias correctionIEEE Transactions on Medical Imaging291310132010.1109/TMI.2010.204690820378467WangYCEgnerT2022Switching task sets creates event boundaries in memoryCognition22110499210.1016/j.cognition.2021.10499234929522XuJMoellerSAuerbachEJStruppJSmithSMFeinbergDAYacoubEUğurbilK2013Evaluation of slice accelerations using multiband echo planar imaging at 3 TNeuroImage83991100110.1016/j.neuroimage.2013.07.05523899722ZacksJMBraverTSSheridanMADonaldsonDISnyderAZOllingerJMBucknerRLRaichleME2001Human brain activity time-locked to perceptual event boundariesNature Neuroscience465165510.1038/8848611369948ZacksJMSpeerNKSwallowKMBraverTSReynoldsJR2007Event perception: a mind-brain perspectivePsychological Bulletin13327329310.1037/0033-2909.133.2.27317338600ZacksJMSpeerNKSwallowKMMaleyCJ2010The brain’s cutting-room floor: segmentation of narrative cinemaFrontiers in Human Neuroscience416810.3389/fnhum.2010.0016820953234ZacksJM2020Event perception and memoryAnnual Review of Psychology7116519110.1146/annurev-psych-010419-05110131905113ZhangYBradyMSmithS2001Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithmIEEE Transactions on Medical Imaging20455710.1109/42.90642411293691Appendix 1Supplemental information
Error-driven and uncertainty-driven boundaries are associated with distinct brain networks and temporal dynamics.
fMRIprep boilerplateAnatomical data preprocessing
A total of 1 T1-weighted (T1w) images were found within the input BIDS dataset. The T1-weighted (T1w) image was corrected for intensity non-uniformity with N4BiasFieldCorrection (Tustison et al., 2010), distributed with ANTs 2.3.3 (Avants et al., 2008, RRID:SCR_004757), and used as T1w-reference throughout the workflow. The T1w-reference was then skull-stripped with a Nipype implementation of the antsBrainExtraction.sh workflow (from ANTs), using OASIS30ANTs as target template. Brain tissue segmentation of cerebrospinal fluid (CSF), white matter (WM), and gray matter (GM) was performed on the brain-extracted T1w using fast (FSL 5.0.9, RRID:SCR_002823, Zhang et al., 2001). Volume-based spatial normalization to one standard space (MNI152NLin2009cAsym) was performed through nonlinear registration with antsRegistration (ANTs 2.3.3), using brain-extracted versions of both T1w reference and the T1w template. The following template was selected for spatial normalization: ICBM 152 Nonlinear Asymmetrical template version 2009c (Fonov et al., 2009, RRID:SCR_008796; TemplateFlow ID: MNI152NLin2009cAsym).
Functional data preprocessing
For each of the 4 BOLD runs found per subject (across all tasks and sessions), the following preprocessing was performed. First, a reference volume and its skull-stripped version were generated by aligning and averaging 1 single-band references (SBRefs). Susceptibility distortion correction (SDC) was omitted. The BOLD reference was then co-registered to the T1w reference using flirt (FSL 5.0.9, Jenkinson and Smith, 2001) with the boundary-based registration (Greve and Fischl, 2009) cost-function. Co-registration was configured with nine degrees of freedom to account for distortions remaining in the BOLD reference. Head-motion parameters with respect to the BOLD reference (transformation matrices, and six corresponding rotation and translation parameters) are estimated before any spatiotemporal filtering using mcflirt (FSL 5.0.9, Jenkinson et al., 2002). BOLD runs were slice-time corrected using 3dTshift from AFNI 20160207 (Cox and Hyde, 1997, RRID:SCR_005927). First, a reference volume and its skull-stripped version were generated using a custom methodology of fMRIPrep. The BOLD time-series (including slice-timing correction when applied) were resampled onto their original, native space by applying the transforms to correct for head-motion. These resampled BOLD time-series will be referred to as preprocessed BOLD in original space, or just preprocessed BOLD. The BOLD time-series were resampled into standard space, generating a preprocessed BOLD run in MNI152NLin2009cAsym space. First, a reference volume and its skull-stripped version were generated using a custom methodology of fMRIPrep. Several confounding time-series were calculated based on the preprocessed BOLD: framewise displacement (FD), DVARS, and three region-wise global signals. FD was computed using two formulations following Power (absolute sum of relative motions, Power et al., 2014) and Jenkinson (relative root mean square displacement between affines, Jenkinson et al., 2002). FD and DVARS are calculated for each functional run, both using their implementations in Nipype (following the definitions by Power et al., 2014). The three global signals are extracted within the CSF, the WM, and the whole-brain masks. Additionally, a set of physiological regressors were extracted to allow for component-based noise correction (CompCor, Behzadi et al., 2007). Principal components are estimated after high-pass filtering the preprocessed BOLD time-series (using a discrete cosine filter with 128 s cutoff) for the two CompCor variants: temporal (tCompCor) and anatomical (aCompCor). tCompCor components are then calculated from the top 2% variable voxels within the brain mask. For aCompCor, three probabilistic masks (CSF, WM, and combined CSF+WM) are generated in anatomical space. The implementation differs from that of Behzadi et al. in that instead of eroding the masks by 2 pixels on BOLD space, the aCompCor masks are subtracted a mask of pixels that likely contain a volume fraction of GM. This mask is obtained by thresholding the corresponding partial volume map at 0.05, and it ensures components are not extracted from voxels containing a minimal fraction of GM. Finally, these masks are resampled into BOLD space and binarized by thresholding at 0.99 (as in the original implementation). Components are also calculated separately within the WM and CSF masks. For each CompCor decomposition, the k components with the largest singular values are retained, such that the retained components’ time series are sufficient to explain 50 percent of variance across the nuisance mask (CSF, WM, combined, or temporal). The remaining components are dropped from consideration. The head-motion estimates calculated in the correction step were also placed within the corresponding confounds file. The confound time series derived from head motion estimates and global signals were expanded with the inclusion of temporal derivatives and quadratic terms for each (Satterthwaite et al., 2013). Frames that exceeded a threshold of 0.5 mm FD or 1.5 standardised DVARS were annotated as motion outliers. All resamplings can be performed with a single interpolation step by composing all the pertinent transformations (i.e. head-motion transform matrices, susceptibility distortion correction when available, and co-registrations to anatomical and output spaces). Gridded (volumetric) resamplings were performed using antsApplyTransforms (ANTs), configured with Lanczos interpolation to minimize the smoothing effects of other kernels (Lanczos, 1964). Non-gridded (surface) resamplings were performed using mri_vol2surf (FreeSurfer).
Many internal operations of fMRIPrep use Nilearn 0.6.2 (Abraham et al., 2014, RRID:SCR_001362), mostly within the functional processing workflow. For more details of the pipeline, see the section corresponding to workflows in fMRIPrep’s documentation.
Copyright waiver
The above boilerplate text was automatically generated by fMRIPrep with the express intention that users should copy and paste this text into their manuscripts unchanged. It is released under the CC0 license.
This valuable study tests whether prediction error or prediction uncertainty controls how the brain segments continuous experience into events. The paper uses validated models that predict human behavior to analyze multivariate neural pattern changes during naturalistic movie watching. The authors provide solid evidence that there are overlapping but partially distinct brain dynamics for each signal.
This paper investigates the control signals that drive event model updating during continuous experience. The authors apply predictions from previously published computational models to fMRI data acquired while participants watched naturalistic video stimuli. They first examine the time course of BOLD pattern changes around human-annotated event boundaries, revealing pattern changes preceding the boundary in anterior temporal and then parietal regions, followed by pattern stabilization across many regions. The authors then analyze time courses around boundaries generated by a model that updates event models based on prediction error and another that uses prediction uncertainty. These analyses reveal overlapping but partially distinct dynamics for each boundary type, suggesting that both signals may contribute to event segmentation processes in the brain.
Strengths:
The question addressed by this paper is of high interest to researchers working on event cognition, perception, and memory. There has been considerable debate about what kinds of signals drive event boundaries, and this paper directly engages with that debate by comparing prediction error and prediction uncertainty as candidate control signals.
The authors use computational models that explain significant variance in human boundary judgments, and they report the variance explained clearly in the paper.
The authors' method of using computational models to generate predictions about when event model updating should occur is a valuable mechanistic alternative to methods like HMM or GSBS, which are data-driven.
The paper utilizes an analysis framework that characterizes how multivariate BOLD pattern dissimilarity evolves before and after boundaries. This approach offers an advance over previous work focused on just the boundary or post-boundary points.
Weaknesses:
Boundaries derived from prediction error and uncertainty are correlated for the naturalistic stimuli. This raises some concerns about how well their distinct contributions to brain activity can be separated. While the authors attempt to look at the unique variance, there is a limit to how effectively this can be done without experimentally dissociating prediction error and uncertainty.
The authors reports an average event length of ~20 seconds, and they also look +20 and -20 seconds around each event boundary. Thus, it's unclear how often pre- and post-boundary timepoints are part of adjacent events. This complicates the interpretations of the reported timecourses.
Tan et al. examined how multivoxel patterns shift in time windows surrounding event boundaries caused by both prediction errors and prediction uncertainty. They observed that some regions of the brain show earlier pattern shifts than others, followed by periods of increased stability. The authors combine their recent computational model to estimate event boundaries that are based on prediction error vs. uncertainty and use this to examine the moment-to-moment dynamics of pattern changes. I believe this is a meaningful contribution that will be of interest to memory, attention, and complex cognition research.
Strengths:
The authors have shown exceptional transparency in terms of sharing their data, code, and stimuli which is beneficial to the field for future examinations and to the reproduction of findings. The manuscript is well written with clear figures. The study starts from a strong theoretical background to understand how the brain represents events and have used a well-curated set of stimuli. Overall, the authors extend the event segmentation theory beyond prediction error to include prediction uncertainty which is an important theoretical shift that has implications in episodic memory encoding, use of semantic and schematic knowledge and to attentional processing.
Weaknesses:
(1) I am not fully satisfied with the author's explanation of pattern shifts occurring 11.9s prior to event boundaries. The average length of time for an event was 21.4 seconds. The window around the identified event boundaries was 20 seconds on either side. The earliest identified pattern shift peaks occur at 11.9s prior to the actual event boundary. This would mean on average, a pattern shift is occurring approximately at the midway point of the event (11.9s prior to a boundary of a 21.4s event is approx. the middle of an event). The authors offer up an explanation in which top down regions signal an update that propagates to lower order regions closer to the boundary. To make this interpretation concrete, they added an example: "in a narrative where a goal is reached midway-for instance, a mystery solved before the story formally ends-higher-order regions may update the event representation at that point, and this updated model then cascades down to shape processing in lower-level regions". This might make sense in a one-off case of irregular storytelling, but it is odd to think this would generalize. If an event is occurring and a given collection of regions represent that event, it doesn't follow the accepted convention of multivariate representational analysis that that set of regions would undergo such a large shift in patterns in the middle of an event. The stabilization of these patterns taking so long is also odd to me. I suspect some of these findings may be due to the stimuli used in this experiment and I am not confident this would generalize and invite the authors to disagree and explain. In the case of the exercise routine video, I try to imagine going from the push-up event to the jumping jack event. The actor stops doing pushups, stands up, and moves minimally for 16 seconds (these lulls are not uncommon). At that point they start doing jumping jacks. It is immediately evident from that moment on that jumping jacks will be the kind of event you are perceiving which may explain the long delay in event pattern stabilisation. Then about 11.9s prior to the end of the event, when the person is still performing jumping jacks (at this point they have been performing jumping jacks for 6 seconds), I would expect the brain to still be expecting this " jumping jacks event". For some reason at this point multivariate patterns in higher order regions shift. I do not understand what kind of top down processing is happening here and the reviewers need to be more concrete in their explanation because as of right now it is ill-defined. I also recognize that being specific to jumping jacks is maybe unfair, but this would apply to the push-ups, granola bar eating, or table cleaning events in the same manner. I suspect one possibility is that the participants realize that the stereotyped action of jumping jacks is going to continue and, thus, mindwander to other thoughts while waiting for novel, informative information to be presented. This explanation would challenge the more active top down processing assumed by the authors.
I had provided a set of concerns to the authors that were not part of the public review and were not addressed. I was unaware of the exact format of the eLife approach, but I think they are worth open discussion so I am adding them here for consideration. Apologies for any confusion.
(2) Why did the authors not examine event boundary activity magnitude differences from the uncertainty vs error boundaries? I see that the authors have provided the data on the openneuro. However, it seems like the difference in activity maps would not only provide extra contextualization of the findings, but also be fairly trivial. Just by eye-balling the plots, it appears as though there may be activity differences in the mPFC occurring shortly after a boundary between the two. Given this regions role in prediction error and schema, it would be important to understand whether this difference is merely due to thresholding effects or is statistically meaningful.
(3) Further, the authors omitted all subcortical regions some of which would be especially interesting such as the hippocampus, basal ganglia, ventral tegmental area. These regions have a rich and deep background in event boundary activity, and prediction error. Univariate effects in these regions may provide interesting effects that might contextualize some of the pattern shifts in the cortex.
(3) I see that field maps were collected, but the fmriprep methods state that susceptibility distortion correction was not performed. Is there a reason to omit this?
The aim of this study was to investigate the temporal progression of the neural response to event boundaries in relation to uncertainty and error. Specifically, the authors asked 1. How neural activity changes before and after event boundaries 2. If uncertainty and error both contribute to explaining the occurrence of event boundaries and 3. If uncertainty and error have unique contributions to explaining the temporal progression of neural activity.
Strengths:
One strength of this paper is that it builds on an already validated computational model. It relies on straightforward and interpretable analysis techniques to answer the main question, with a smart combination of pattern similarity metrics and FIR. This combination of methods may also be an inspiration to other researchers in the field working on similar questions. The paper is well written and easy to follow. The paper convincingly shows that 1. There is a temporal progression of neural activity change before and after an event boundary 2. Event boundaries are predicted best by the combination of uncertainty and error signals.
Weaknesses:
Regarding question 3, the results are less convincing. Although the analyses in Figure S1 show that there are some unique contributions of uncertainty and error, it is unclear to what extent the results in Figure 7 are driven by shared variance. Therefore, it is not clear to what extent the main claim in the abstract is due to shared or unique variance. More specific comments are provided below.
The other issue is the distance between events is short compared to the pre-onset effects that are observed. Halfway the distance between two events there are already neural signatures of change relating to the upcoming event boundary. I wonder if methodological issues could explain this effect and if not, what could allow participants to notice the impending event boundary.
Impact:
If these comments can be addressed sufficiently, I expect that this work will impact the field in its thinking on what drives event boundaries and spur interest in understanding the mechanisms behind the temporal progression of neural activity around these boundaries.
Comments
(1) The correlation between uncertainly and prediction error is very high, which makes it challenging to disentangle the effects of both on the neural response. The analysis in Figure S1 shows that the two predictors indeed have dissociable contributions. However, the results mainly reported in the discussion section and abstract still rely on models where only one of these factors is included at a time. This makes it debatable whether these specific networks mentioned really reflect unique contributions of each of these components. I specifically refer to this statement in the abstract: "Error-driven boundaries were associated with early pattern shifts in ventrolateral prefrontal areas, followed by pattern stabilization in prefrontal and temporal areas. Uncertainty-driven boundaries were linked to shifts in parietal regions within the dorsal attention network, with minimal subsequent stabilization. ". I would encourage repeating all analyses (also the ones in figure 7) with a models that includes both predictors and showing both results in the manuscript, so it is clear which regions really show unique variance related to one of the predictors. I also wonder why it is necessary to look at model comparisons between the combined and unique models, rather than simply reporting the significance of each predictor in the combined model.
(2) The distance between event boundaries ranges between 20 and 30 seconds. The early pre-boundary effect that are observed in the manuscript occur at -12 seconds. This means that these effects occur roughly halfway between the previous and current event. This seems much earlier than expected. That is why I worry that the FIR analyses might not be able to distinguish effects of the previous event from effects of the upcoming event. What evidence is there that the FIR analyses can actually properly show the return to baseline? One way to address this might be to randomize the locations of the event boundaries while preserving the distance between them and rerun the models. This will give a null-model with the same event distances and should be able to distinguish this temporal overlap from the true effects of event boundaries.
(3) If the analyses in point 2 confirm that there is indeed an event-boundary related change that occurs 12 seconds before event onset, it is important to consider what might cause these changes. Are there cues in the movie that indicate that an event boundary is coming? It would be interesting to investigate whether uncertainty and error are higher than expected at 12 seconds pre-onset.
10.7554/eLife.107955.3.sa4Author responseNguyenTan TAuthorWashington University in St. LouisSaint LouisUnited StatesEtzelJo AAuthorWashington University in St. LouisSaint LouisUnited StatesBezdekMatthew AAuthorWashington University in St. LouisSaint LouisUnited StatesZacksJeffreyAuthorWashington University in St. LouisSt. LouisUnited States
The following is the authors’ response to the original reviews.
Public Reviews:
Reviewer #1 (Public review):
Summary:
This paper investigates the control signals that drive event model updating during continuous experience. The authors apply predictions from previously published computational models to fMRI data acquired while participants watched naturalistic video stimuli. They first examine the time course of BOLD pattern changes around human-annotated event boundaries, revealing pattern changes preceding the boundary in anterior temporal and then parietal regions, followed by pattern stabilization across many regions. The authors then analyze time courses around boundaries generated by a model that updates event models based on prediction error and another that uses prediction uncertainty. These analyses reveal overlapping but partially distinct dynamics for each boundary type, suggesting that both signals may contribute to event segmentation processes in the brain.
Strengths:
(1) The question addressed by this paper is of high interest to researchers working on event cognition, perception, and memory. There has been considerable debate about what kinds of signals drive event boundaries, and this paper directly engages with that debate by comparing prediction error and prediction uncertainty as candidate control signals.
(2) The authors use computational models that explain significant variance in human boundary judgments, and they report the variance explained clearly in the paper.
(3) The authors' method of using computational models to generate predictions about when event model updating should occur is a valuable mechanistic alternative to methods like HMM or GSBS, which are data-driven.
(4) The paper utilizes an analysis framework that characterizes how multivariate BOLD pattern dissimilarity evolves before and after boundaries. This approach offers an advance over previous work focused on just the boundary or post-boundary points.
We appreciate this reviewer’s recognition of the significance of this research problem, and of the value of the approach taken by this paper.
Weaknesses:
(1) While the paper raises the possibility that both prediction error and uncertainty could serve as control signals, it does not offer a strong theoretical rationale for why the brain would benefit from multiple (empirically correlated) signals. What distinct advantages do these signals provide? This may be discussed in the authors' prior modeling work, but is left too implicit in this paper.
We added a brief discussion in the introduction highlighting the complementary advantages of prediction error and prediction uncertainty, and cited prior theoretical work that elaborates on this point. Specifically, we now note that prediction error can act as a reactive trigger, signaling when the current event model is no longer sufficient (Zacks et al., 2007). In contrast, prediction uncertainty is framed as proactive, allowing the system to prepare for upcoming changes even before they occur (Baldwin & Kosie, 2021; Kuperberg, 2021). Together, this makes clearer why these two signals could each provide complementary benefits for effective event model updating.
"One potential signal to control event model updating is prediction error—the difference between the system’s prediction and what actually occurs. A transient increase in prediction error is a valid indicator that the current model no longer adequately captures the current activity. Event Segmentation Theory (EST; Zacks et al., 2007) proposes that event models are updated when prediction error increases beyond a threshold, indicating that the current model no longer adequately captures ongoing activity. A related but computationally distinct proposal is that prediction uncertainty (also termed "unpredictability") can serve as a control signal (Baldwin & Kosie, 2021). The advantage of relying on prediction uncertainty to detect event boundaries is that it is inherently proactive: the cognitive system can start looking for cues about what might come next before the next event starts (Baldwin & Kosie, 2021; Kuperberg, 2021). "
(2) Boundaries derived from prediction error and uncertainty are correlated for the naturalistic stimuli. This raises some concerns about how well their distinct contributions to brain activity can be separated. The authors should consider whether they can leverage timepoints where the models make different predictions to make a stronger case for brain regions that are responsive to one vs the other.
We addressed this concern by adding an analysis that explicitly tests the unique contributions of prediction error– and prediction uncertainty–driven boundaries to neural pattern shifts. In the revised manuscript, we describe how we fit a combined FIR model that included both boundary types as predictors and then compared this model against versions with only one predictor. This allowed us to identify the variance explained by each boundary type over and above the other. The results revealed two partially dissociable sets of brain regions sensitive to error- versus uncertainty-driven boundaries (see Figure S1), strengthening our argument that these signals make distinct contributions.
"To account for the correlation between uncertainty-driven boundaries and error-driven boundaries, we also fitted a FIR model that predicted pattern dissimilarity from both types of boundaries (combined FIR) for each parcel. Then, we performed two likelihood ratio tests: combined FIR to error FIR, which measures the unique contribution of uncertainty boundaries to pattern dissimilarity, and combined FIR to uncertainty FIR, which measures the unique contribution of error boundaries to pattern dissimilarity. The analysis also revealed two dissociable sets of brain regions associated with each boundary type (see Figure S1)."
(3) The authors refer to a baseline measure of pattern dissimilarity, which their dissimilarity measure of interest is relative to, but it's not clear how this baseline is computed. Since the interpretation of increases or decreases in dissimilarity depends on this reference point, more clarity is needed.
We clarified how the FIR baseline is estimated in the methods section. Specifically, we now explain that the FIR coefficients should be interpreted relative to a reference level, which reflects the expected dissimilarity when timepoints are far from an event boundary. This makes it clear what serves as the comparison point for observed increases or decreases in dissimilarity.
"The coefficients from the FIR model indicate changes relative to baseline, which can be conceptualized as the expected value when far from event boundaries."
(4) The authors report an average event length of ~20 seconds, and they also look at +20 and -20 seconds around each event boundary. Thus, it's unclear how often pre- and post-boundary timepoints are part of adjacent events. This complicates the interpretations of the reported time courses.
This is related to reviewer's 2 comment, and it will be addressed below.
(5) The authors describe a sequence of neural pattern shifts during each type of boundary, but offer little setup of what pattern shifts we might expect or why. They also offer little discussion of what cognitive processes these shifts might reflect. The paper would benefit from a more thorough setup for the neural results and a discussion that comments on how the results inform our understanding of what these brain regions contribute to event models.
We thank the reviewer for this advice on how better to set the context for the different potential outcomes of the study. We expanded both the introduction and discussion to better set up expectations for neural pattern shifts and to interpret what these shifts may reflect. In the introduction, we now describe prior findings showing that sensory regions tend to update more quickly than higher-order multimodal regions (Baldassano et al., 2017; Geerligs et al., 2021, 2022), and we highlight that it remains unclear whether higher-order updates precede or follow those in lower-order regions. We also note that our analytic approach is well-suited to address this open question. In the discussion, we then interpret our results in light of this framework. Specifically, we describe how we observed early shifts in higher-order areas such as anterior temporal and prefrontal cortex, followed by shifts in parietal and dorsal attention regions closer to event boundaries. This pattern runs counter to the traditional bottom-up temporal hierarchy view and instead supports a model of top-down updating, where high-level representations are updated first and subsequently influence lower-level processing (Friston, 2005; Kuperberg, 2021). To make this interpretation concrete, we added an example: in a narrative where a goal is reached midway—for instance, a mystery solved before the story formally ends—higher-order regions may update the event representation at that point, and this updated model then cascades down to shape processing in lower-level regions. Finally, we note that the widespread stabilization of neural patterns after boundaries may signal the establishment of a new event model.
Excerpt from Introduction:
“More recently, multivariate approaches have provided insights into neural representations during event segmentation. One prominent approach uses hidden Markov models (HMMs) to detect moments when the brain switches from one stable activity pattern to another (Baldassano et al., 2017) during movie viewing; these periods of relative stability were referred to as "neural states" to distinguish them from subjectively perceived events. Sensory regions like visual and auditory cortex showed faster transitions between neural states. Multi-modal regions like the posterior medial cortex, angular gyrus, and intraparietal sulcus showed slower neural state shifts, and these shifts aligned with subjectively reported event boundaries. Geerligs et al. (2021, 2022) employed a different analytical approach called Greedy State Boundary Search (GSBS) to identify neural state boundaries. Their findings echoed the HMM results: short-lived neural states were observed in early sensory areas (visual, auditory, and somatosensory cortex), while longer-lasting states appeared in multi-modal regions, including the angular gyrus, posterior middle/inferior temporal cortex, precuneus, anterior temporal pole, and anterior insula. Particularly prolonged states were found in higher-order regions such as lateral and medial prefrontal cortex.
The previous evidence about evoked responses at event boundaries indicates that these are dynamic phenomena evolving over many seconds, with different brain areas showing different dynamics (Ben-Yakov & Henson, 2018; Burunat et al., 2024; Kurby & Zacks, 2018; Speer et al., 2007; Zacks, 2010). Less is known about the dynamics of pattern shifts at event boundaries (e.g. whether shifts observed in higher-order regions precedes or follow shifts observed in lower-level regions), because the HMM and GSBS analysis methods do not directly provide moment-by-moment measures of pattern shifts. Both the spatial and temporal aspects of evoked responses and pattern shifts at event boundaries have the potential to provide evidence about two potential control processes (error-driven and uncertainty-driven) for event model updating.”
Excerpt from Discussion:
“We first characterized the neural signatures of human event segmentation by examining both univariate activity changes and multivariate pattern changes around subjectively identified event boundaries. Using multivariate pattern dissimilarity, we observed a structured progression of neural reconfiguration surrounding human-identified event boundaries. The largest pattern shifts were observed near event boundaries (~4.5s before) in dorsal attention and parietal regions; these correspond with regions identified by Geerligs et. al as shifting their patterns on a fast to intermediate timescale (2022). We also observed smaller pattern shifts roughly 12 seconds prior to event boundaries in higher-order regions within anterior temporal cortex and prefrontal cortex, and these are slow-changing regions identified by Geerligs et. al (2022). This is puzzling. One prevalent proposal, based on the idea of a cortical hierarchy of increasing temporal receptive windows (TRWs), suggests that higher-order regions should update representations after lower-order regions do (Chang et al., 2021). In this view, areas with shorter TRWs (e.g., word-level processors) pass information upward, where it is integrated into progressively larger narrative units (phrases, sentences, events). This proposal predicts neural shifts in higher-order regions to follow those in lower-order regions. By contrast, our findings indicate the opposite sequence. Our findings suggest that the brain might engage in top-down event representation updating, with changes in coarser-grain representations propagating downward to influence finer-grain representations. (Friston, 2005; Kuperberg, 2021). For example, in a narrative where the main goal is achieved midway—such as a detective solving a mystery before the story formally ends—higher-order regions might update the overarching event representation at that point, and this updated model could then cascade down to reconfigure how lower-level regions process the remaining sensory and contextual details. In the period after a boundary (around +12 seconds), we found widespread stabilization of neural patterns across the brain, suggesting the establishment of a new event model. Future work could focus on understanding the mechanisms behind the temporal progression of neural pattern changes around event boundaries.”
Reviewer #2 (Public review):
Summary:
Tan et al. examined how multivoxel patterns shift in time windows surrounding event boundaries caused by both prediction errors and prediction uncertainty. They observed that some regions of the brain show earlier pattern shifts than others, followed by periods of increased stability. The authors combine their recent computational model to estimate event boundaries that are based on prediction error vs. uncertainty and use this to examine the moment-to-moment dynamics of pattern changes. I believe this is a meaningful contribution that will be of interest to memory, attention, and complex cognition research.
Strengths:
The authors have shown exceptional transparency in terms of sharing their data, code, and stimuli, which is beneficial to the field for future examinations and to the reproduction of findings. The manuscript is well written with clear figures. The study starts from a strong theoretical background to understand how the brain represents events and has used a well-curated set of stimuli. Overall, the authors extend the event segmentation theory beyond prediction error to include prediction uncertainty, which is an important theoretical shift that has implications in episodic memory encoding, the use of semantic and schematic knowledge, and attentional processing.
We thank the reader for their support for our use of open science practices, and for their appreciation of the importance of incorporating prediction uncertainty into models of event comprehension.
Weaknesses:
The data presented is limited to the cortex, and subcortical contributions would be interesting to explore. Further, the temporal window around event boundaries of 20 seconds is approximately the length of the average event (21.4 seconds), and many of the observed pattern effects occur relatively distal from event boundaries themselves, which makes the link to the theoretical background challenging. Finally, while multivariate pattern shifts were examined at event boundaries related to either prediction error or prediction uncertainty, there was no exploration of univariate activity differences between these two different types of boundaries, which would be valuable.
The fact that we observed neural pattern shifts well before boundaries was indeed unexpected, and we now offer a more extensive interpretation in the discussion section. Specifically, we added text noting that shifts emerged in higher-order anterior temporal and prefrontal regions roughly 12 seconds before boundaries, whereas shifts occurred in lower-level dorsal attention and parietal regions closer to boundaries. This sequence contrasts with the traditional bottom-up temporal hierarchy view and instead suggests a possible top-down updating mechanism, in which higher-order representations reorganize first and propagate changes to lower-level areas (Friston, 2005; Kuperberg, 2021). (See excerpt for Reviewer 1’s comment #5.)
With respect to univariate activity, we did not find strong differences between error-driven and uncertainty-driven boundaries. This makes the multivariate analyses particularly informative for detecting differences in neural pattern dynamics. To support further exploration, we have also shared the temporal progression of univariate BOLD responses on OpenNeuro (BOLD_coefficients_brain_animation_pe_SEM_bold.html and BOLD_coefficients_brain_animation_uncertainty_SEM_bold.html in the derivatives/figures/brain_maps_and_timecourses/ directory; https://doi.org/10.18112/openneuro.ds005551.v1.0.4) for interested researchers.
Reviewer #3 (Public review):
Summary:
The aim of this study was to investigate the temporal progression of the neural response to event boundaries in relation to uncertainty and error. Specifically, the authors asked (1) how neural activity changes before and after event boundaries, (2) if uncertainty and error both contribute to explaining the occurrence of event boundaries, and (3) if uncertainty and error have unique contributions to explaining the temporal progression of neural activity.
Strengths:
One strength of this paper is that it builds on an already validated computational model. It relies on straightforward and interpretable analysis techniques to answer the main question, with a smart combination of pattern similarity metrics and FIR. This combination of methods may also be an inspiration to other researchers in the field working on similar questions. The paper is well written and easy to follow. The paper convincingly shows that (1) there is a temporal progression of neural activity change before and after an event boundary, and (2) event boundaries are predicted best by the combination of uncertainty and error signals.
We thank the reviewer for their thoughtful and supportive comments, particularly regarding the use of the computational model and the analysis approaches.
Weaknesses:
(1) The current analysis of the neural data does not convincingly show that uncertainty and prediction error both contribute to the neural responses. As both terms are modelled in separate FIR models, it may be that the responses we see for both are mostly driven by shared variance. Given that the correlation between the two is very high (r=0.49), this seems likely. The strong overlap in the neural responses elicited by both, as shown in Figure 6, also suggests that what we see may mainly be shared variance. To improve the interpretability of these effects, I think it is essential to know whether uncertainty and error explain similar or unique parts of the variance. The observation that they have distinct temporal profiles is suggestive of some dissociation,but not as convincing as adding them both to a single model.
We appreciate this point. It is closely related to Reviewer 1's comment 2; please refer to our response above.
(2) The results for uncertainty and error show that uncertainty has strong effects before or at boundary onset, while error is related to more stabilization after boundary onset. This makes me wonder about the temporal contribution of each of these. Could it be the case that increases in uncertainty are early indicators of a boundary, and errors tend to occur later?
We also share the intuition that increases in uncertainty are early indicators of a boundary, and errors tend to occur later. If that is the case, we would expect some lags between prediction uncertainty and prediction error. We examined lagged correlation between prediction uncertainty and prediction error, and the optimal lag is 0 for both uncertainty-driven and error-driven models. This indicates that when prediction uncertainty rises, prediction error also simultaneously rises.
(3) Given that there is a 24-second period during which the neural responses are shaped by event boundaries, it would be important to know more about the average distance between boundaries and the variability of this distance. This will help establish whether the FIR model can properly capture a return to baseline.
We have added details about the distribution of event lengths. Specifically, we now report that the mean length of subjectively identified events was 21.4 seconds (median 22.2 s, SD 16.1 s). For model-derived boundaries, the average event lengths were 28.96 seconds for the uncertainty-driven model and 24.7 seconds for the error-driven model.
" For each activity, a separate group of 30 participants had previously segmented each movie to identify fine-grained event boundaries (Bezdek et al., 2022). The mean event length was 21.4 s (median 22.2 s, SD 16.1 s). Mean event lengths for uncertainty-driven model and error-driven model were 28.96s, and 24.7s, respectively (Nguyen et al., 2024)."
(4) Given that there is an early onset and long-lasting response of the brain to these event boundaries, I wonder what causes this. Is it the case that uncertainty or errors already increase at 12 seconds before the boundaries occur? Or if there are other makers in the movie that the brain can use to foreshadow an event boundary? And if uncertainty or errors do increase already 12 seconds before an event boundary, do you see a similar neural response at moments with similar levels of error or uncertainty, which are not followed by a boundary? This would reveal whether the neural activity patterns are specific to event boundaries or whether these are general markers of error and uncertainty.
We appreciate this point; it is similar to reviewer 2’s comment 2. Please see our response to that comment above.
(5) It is known that different brain regions have different delays of their BOLD response. Could these delays contribute to the propagation of the neural activity across different brain areas in this study?
Our analyses use ±20 s FIR windows, and the key effects we report include shifts ~12s before boundaries in higher-order cortex and ~4.5s pre-boundary in dorsal attention/parietal areas. Given the literature above, region-dependent BOLD delays are much smaller (~1–2s) than the temporal structure we observe (Taylor et al., 2018), making it unlikely that HRF lag alone explains our multi-second, region-specific progression.
(6) In the FIR plots, timepoints -12, 0, and 12 are shown. These long intervals preclude an understanding of the full temporal progression of these effects.
For page length purposes, we did not include all timepoints. We uploaded a brain animation of all timepoints and coefficients for each parcel in Openneuro (PATTERN_coefficients_brain_animation_human_fine_pattern.html and PATTERN_coefficients_lines_human_fine.html in the derivatives/figures/brain_maps_and_timecourses/ directory; https://doi.org/10.18112/openneuro.ds005551.v1.0.4) for interested researchers.
References
Taylor, A. J., Kim, J. H., & Ress, D. (2018). Characterization of the hemodynamic response function across the majority of human cerebral cortex. NeuroImage, 173, 322–331. https://doi.org/10.1016/j.neuroimage.2018.02.061