Using the resting-state fMRI timeseries from the HCP dataset, we found that among the kernels constructed from HMMs, the linear Fisher kernel had the highest prediction accuracy on average across the range of behavioural variables and CV folds and iterations, as shown in Figure 2a. Compared to the other linear kernels (which do not respect the geometry of the HMM parameters), the linear Fisher kernel (mean r κ_Fl: 0.192) was significantly more accurate than the linear naïve kernel (mean r κ_Nl: 0.05, t_rkCV=2.631, p_BH=0.031). The comparison with the linear naïve normalised kernel was not significant (mean r κ_NNl: 0.153, t_rkCV=1.022, p_BH=0.192). This indicates a positive effect of using a tangent space embedding rather than incorrectly treating the HMM parameters as Euclidean, but that this effect can be mitigated by normalising the parameters before constructing the kernel.

Among the Gaussian kernels, the Gaussian Fisher kernel also had the highest average prediction accuracy (mean r κ_Fg: 0.166), though the comparisons with the other kernels were not significant (mean r κ_Ng: 0.107, t_rkCV=1.277, p_BH=0.173; mean r κ_NNg: 0.094, t_rkCV=1.466, p_BH=0.145; mean r κ_KL: 0.163, t_rkCV=0.069, p_BH=0.482). Comparing prediction accuracies of the linear with the Gaussian Fisher kernel was not significant (t_rkCV=0.993, p_BH=0.192).

Compared to the methods using time-averaged FC for prediction, the linear Fisher kernel significantly outperformed the Selected Edges method (mean r Selected Edges: 0.081, t_rkCV=2.417, p_BH=0.031). The average prediction accuracy of the linear Fisher kernel is also higher than the log-Euclidean kernel (mean r: 0.140, t_rkCV=1.456, p_BH=0.145) and the Ridge regression model (mean r: 0.111, t_rkCV=1.906, p_BH=0.085), but outperformed by time-averaged KL divergence (mean r: 0.194, t_rkCV=−0.044, p_BH=0.482) and Ridge regression in Riemannian space (mean r Ridge Riem.: 0.223, t_rkCV=−0.999, p_BH=0.192), though these comparisons were all not significant. Analogous to the effect of tangent space embedding for the HMM parameters (linear Fisher kernel compared to linear naïve kernel), using a tangent space embedding on the time-averaged covariance matrices (Ridge regression in Riemannian space compared to Ridge regression) also significantly improved the prediction accuracy (t_rkCV=−2.537, p_BH=0.0313). The Elastic Net and Elastic Net in Riemannian space were not used for statistical comparisons since they failed to converge in a substantial portion of runs (>20%). The Elastic Nets showed similar performance to the Ridge regression models. We also observed that the non-kernel-based Euclidean time-averaged FC models, that is the Ridge regression and the Selected Edges model, make predictions at a smaller range than the actual variables, close to the variables’ means. This leads to weak relationships between predicted and actual variables but smaller errors, as shown in Figure 2—figure supplement 1. The distributions of correlation coefficients for the different methods are shown in Figure 2a and the explained variance in Figure 2b. The performance of all methods is also summarised in Supplementary file 1b.

We found that the linear Fisher kernel also predicted more accurately than other HMM-based kernels when fitting the HMM to only the first resting-state session of each subject, as shown in Figure 2—figure supplement 2. However, the overall accuracies of all kernels were lower in this case, indicating that predicting traits benefits from a large amount of available data per subject. Similarly, the Fisher kernel outperformed the other kernels when HMM states were defined only in terms of covariance (not mean) at comparable accuracy, as shown in Figure 2—figure supplement 3.

As shown in Figure 3, there are differences in how well these demographic or behavioural variables can be predicted from a model of brain dynamics. Certain variables may be more related to static or structural measures (Liégeois et al., 2019; Vidaurre et al., 2021), or just be difficult to predict in general. Overall, age could be best predicted, followed by language-related cognitive items (PicVocab, ReadEng both age adjusted and unadjusted).

In summary, the linear Fisher kernel has the highest prediction accuracy of the time-varying methods, significantly outperforming the linear naïve kernel which does not take the geometry of the HMM parameters into account. The linear Fisher kernel also has a higher prediction accuracy than several methods using time-averaged FC features, but it is outperformed by time-averaged methods that work in tangent space.

We now show empirically that the linear Fisher kernel is more reliable than other kernels, both in terms of risk of large errors and in terms of robustness over CV iterations.

The linear versions of the Fisher kernel and the naïve normalised kernel had the overall lowest risk of large errors among the time-varying methods, as shown in Figure 2c. We assessed the risk of large errors (NMAXAE >10), very large errors (NMAXAE >100), and extreme errors (NMAXAE >1,000), corresponding to one, two, and three orders of magnitude of the range of the actual variable. For the Fisher kernel, the risk of large errors is low: 0% in the linear version κ_Fl and 0.029% in the Gaussian version κ_Fg. That means that the linear Fisher kernel never makes large errors exceeding the range of the actual variable by orders of magnitude. In the naïve kernel, both the linear κ_Nl and the Gaussian version κ_Ng have a low risk of large errors at 0.057% for the linear version and 0.029% for the Gaussian version. While the linear naïve normalised kernel κ_NNl has a 0% risk of large errors, its Gaussian version κ_NNg has the overall highest risk of large errors at 1.229%, a risk of very large errors at 0.143%, and even a risk of extreme errors at 0.029%. The KL divergence model κ_KL has a 0.686% risk of large errors and a 0.029% risk of very large errors. The time-averaged KL divergence model performs slightly better than the time-varying KL divergence, but also has a risk of large errors at 0.600%. The other time-averaged models had no risk of excessive errors. The maximum error distributions are shown in Figure 2c.

A reason for the higher risk of large errors in the Gaussian kernels is likely that the radius τ of the radial basis function needs to be selected (using cross-validation), introducing an additional factor of variability and leaving more room for error. Figure 2—figure supplement 4 shows the relation between the estimated hyperparameters (the regularisation parameter λ and the radius τ of the radial basis function) and how large errors in the predictions may be related to poor estimation of these parameters.

With respect to robustness, we found that the linear Fisher kernel κ_Fl had the most robust performance among the time-varying methods, and the Ridge regression model in Riemannian space among the time-averaged methods, on average across the range of variables tested, as shown in Figure 2d. Robustness was quantified as the standard deviation of the correlation between model-predicted and actual values over 100 iterations and 10 folds of CV. A low standard deviation indicates high robustness since the method’s performance does not differ greatly depending on the specific subjects it was trained and tested on.

Among the time-varying methods, the linear Fisher kernel was the most robust method (κ_Fl mean S.D. r: 0.088), though the comparison with the other linear kernels was not significant (linear naïve kernel κ_Nl mean S.D. r: 0.096, t=−2.351, p_BH=0.099; linear naïve normalised kernel κ_NNl mean S.D. r: 0.090, t=–1.745, p_BH=0.180). Similarly, the comparison between the Gaussian Fisher kernel (κ_Fg mean S.D. r: 0.089) and the other Gaussian kernels was not significant (Gaussian naïve kernel κ_Ng mean S.D. r: 0.089, t=0.311, p_BH=0.826; Gaussian naïve normalised kernel κ_NNg mean S.D. r: 0.093, t=−1.802, p_BH=0.180; KL divergence κ_KL mean S.D. r: 0.093, t=−1.758, p_BH=0.180). Compared to the time-averaged methods, the linear Fisher kernel was significantly more robust than the time-averaged KL divergence model (mean S.D. r: 0.100, t=−3.798, p_BH=0.003) and the Selected Edges model (mean S.D. r: 0.100, t=−5.446, p_BH <0.0001), while the other comparisons were not significant (log-Euclidean mean S.D. r: 0.091, t=−1.317, p_BH=0.277; Ridge reg. mean S.D. r: 0.091, t=−1.451, p_BH=0.268; Ridge reg. Riem. mean S.D. r: 0.088, t=0.059, p_BH=0.953). This large variation in model performance depending on the CV fold structure in the time-averaged KL divergence model and the Selected Edges method is problematic. Among the time-averaged methods, the Ridge Regression model in Riemannian space was the most robust method. The ranges in model performance across CV iterations for each variable of the different kernels are shown in Figure 3.

Overall, the linear Fisher kernel was the most reliable method among the time-varying methods, and the Ridge regression model in Riemannian space was the most reliable among the time-averaged methods. For both methods, there was no risk of large errors and the variability over CV iterations was the smallest. The Gaussian kernels had higher risks of large errors, and both the time-varying and the time-averaged KL divergence model risked producing large errors, indicating that their performance was less reliable. The Gaussian naïve normalised kernel was the most problematic in terms of reliability with a risk of extreme errors ranging up to four orders of magnitude of the actual variable’s range. Two of the time-averaged methods, the time-averaged KL divergence model and the Selected Edges method, showed problems with robustness, indicating considerable susceptibility to changes in CV folds.

To understand which features drive the prediction, we next simulated timeseries of two groups of subjects that were different either in the mean amplitude of one state or in the transition probabilities. As shown in Figure 4a, when we simulated two groups of subjects that are different in terms of the mean amplitude, the Fisher kernel was able to recover this difference in all runs with 0% error, meaning that it identified all subjects correctly in all runs. The Fisher kernel significantly outperformed the other two kernels (Fisher kernel κ_Fl vs. naïve kernel κ_Nl: p=0.0003, vs. naïve normalised kernel κ_NNl: p=0.0001). There was no significant difference between the naïve and the naïve normalised kernel (p=1). However, when we simulated differences in transition probabilities between two groups, neither of the kernels were able to reliably recover this difference. In this simulation, the Fisher kernel performed significantly worse than the other two kernels on average (compared to naïve kernel: p=0.006, compared to naïve normalised kernel: p=0.004), as shown in Figure 4b. As in the previous simulation, the naïve kernel and the naïve normalised kernel did not significantly differ from each other (p=1).

The above results suggest that all kernels, but particularly the Fisher kernel, are most sensitive to differences in state parameters rather than differences in transition probabilities. To understand whether the difference in transition probabilities can be recovered when it is not overshadowed by the more dominant state parameters, we ran the second case of simulations again, where we introduced a group difference in terms of transition probabilities, but this time we exclude the state parameters when we constructed the kernels. As shown in Figure 4c, the Fisher kernel was now able to recover the group difference with minimal errors, while the naïve normalised kernel improved but did not perform as well as the Fisher kernel. The naïve kernel performed below chance. The Fisher kernel significantly outperformed both the naïve kernel (p=0.0001) and the naïve normalised kernel (p=0.02), and the naïve normalised kernel was significantly more accurate than the naïve kernel (p=0.0004). This shows that the Fisher kernel can recover the group difference in transition probabilities when this difference is not overshadowed by the state parameters. The features and kernel matrices of example runs for all three simulations are shown in Figure 4a–c middle and right panels.

When using real data, the features driving the prediction may differ from trait to trait: For some traits, state parameters may be more relevant, while for other traits, transitions may be more relevant. Given our above findings on simulated data, we therefore compared the effects of systematically removing state features or removing transition features in all traits in the real data.

We found that state parameters were the most relevant features for the Fisher kernel predictions in all traits: As shown in Figure 5a, the prediction accuracy of the Fisher kernel was significantly diminished when state features were removed (t_rkCV = 2.922, p_BH = 0.016), while removing transition features had no significant effect (t_rkCV = –0.173, p_BH = 0.480). We observed the same effect in the naïve normalised kernel (no state features: t_rkCV = 2.460, p_BH = 0.031; no transition features: t_rkCV = 0.264, p_BH = 0.480). For the naïve kernel, removing features did not have any significant effects (no state features: t_rkCV = 0.501, p_BH = 0.462; no transition features: t_rkCV=0.050, p_BH = 0.480). One reason for the dominance of state parameters may simply be that the state parameters outnumber the other parameters: In the full kernels, we have 15,300 state features (300 features associated with the state means and 15,000 features associated with the state covariances), but only 42 transition features (6 features associated with the state probabilities and 36 features associated with the transition probabilities). To compensate for this imbalance, we also constructed a version of the kernels where state parameters were reduced to the same amount as transition features using Principal Component Analysis (PCA), so that we have 84 features in total (42 transition features and the first 42 PCs of the 15,300 state features). These PCA-kernels performed better than the ones where state features were removed, but worse than the kernels including all features at the original dimensionality, although not significantly (Fisher kernel: t_rkCV = 1.434, p_BH = 0.170; naïve kernel: t_rkCV = 0.860, p_BH = 0.351; naïve normalised kernel: t_rkCV = 1.555, p_BH = 0.170). This indicates that the fact that state parameters are more numerous than transition parameters does not explain why kernels including state features performed better. Instead, the content of the state features is more relevant for the prediction than the content of the transition features.

When looking at the performance separately for each variable (Figure 5b), we found that all variables were better predicted by the version of the kernel which included state features than the ones where state features were removed, while it did not seem to matter whether transition features were included. This indicates that the simulation case described above, where the relevant changes are in the transition probabilities, did not occur in the real data. In certain variables, reducing state features using PCA improved the accuracy compared to the full kernels. This is not unexpected since feature dimensionality reduction is known to be able to improve prediction accuracy by removing redundant features (Mwangi et al., 2014).

Empirically, we thus found that the Fisher kernel is most sensitive to individual differences in state parameters, both in simulated timeseries and in the real data. This means that predictions are driven more by what an individual’s states look like, rather than by how they transition between states. However, the Fisher kernel can be modified to recover differences in transition probabilities if these are relevant for specific traits.

In the results above, we have constructed kernels from HMMs fit to all subjects in the dataset and only separated them into training and test set at the regression (and deconfounding) step. In machine learning, the gold standard is considered to be a full separation between training and test set at all stages of (pre-)processing to avoid data leakage from the test set into the training set (Kapoor and Narayanan, 2023; Poldrack et al., 2020). However, a recent study in neuroimaging has found minimal effects of data leakage for breaches of separation between training and test set not involving the target variable or feature selection (Rosenblatt et al., 2024). To test whether training the HMM on all subjects may have inflated prediction accuracies, we repeated the HMM-based predictions for kernels that were constructed from HMMs trained only on training subjects. Consistent with the results in Vidaurre et al., 2021, for all kernels, both linear and Gaussian versions, the separation of training and test subjects before vs. after fitting the HMM had no significant effects on prediction accuracies (training together vs. separate: linear Fisher kernel: t_kCV = 0, p_BH = 1; Gaussian Fisher kernel: t_kCV = 0, p_BH = 1; linear naïve kernel: t_kCV = −0.139, p_BH = 1; Gaussian naïve kernel: t_kCV = −0.018, p_BH = 1; linear naïve normalised kernel: t_kCV = −0.030, p_BH = 1; Gaussian naïve normalised kernel: t_kCV = 0.040, p_BH = 1). Figure 6a shows the distribution of accuracies across target variables and CV folds for all kernels and training schemes, confirming that at this sample size (N=1001), the effect is negligible.

Importantly, in the Fisher kernel, individual subjects’ features are defined in reference to a group ‘average’. While this has no effect in situations where training and test subjects are taken from the same distribution, it may introduce a bias where test subjects are taken from a different distribution (e.g. the case where one might want to train a model on healthy controls and test on patients). To illustrate this effect, we simulated timeseries for two groups of subjects with varying degrees of between-group difference and noise on the target variable. As shown in Figure 6b, this does not affect the kernels when training the HMM on all subjects (training scheme: together, middle panel), but the performance of the Fisher kernel worsens as group difference increases when training the HMM only on the training set (training scheme: separate, right panel). This is because in the Fisher kernel, subjects’ similarity is determined by their difference from the group-level HMM. When the test subjects are included in this group-level HMM, their scores will be similar to the training subjects’ scores, but when they are excluded from the group-level HMM, their scores may be overestimated since they are all different from the group-level model. This between-group difference may then overshadow the more subtle differences related to the target variable, as shown in the example kernels in Figure 6c. Whether or not this behaviour is desired will depend on the use case.

We used data from the open-access Human Connectome Project (HCP) S1200 release (Smith et al., 2013a; Van Essen et al., 2013), which contains MR imaging data and various demographic and behavioural data from 1200 healthy, young adults (age 22–35). All data described below, that is timecourses of the resting-state fMRI data and demographic and behavioural variables, are publicly available at https://db.humanconnectome.org.

Specifically, we used resting state fMRI data of 1001 subjects, for whom any of the behavioural variables of interest were available. Each participant completed four resting state scanning sessions of 14 min and 33 s duration each. This resulted in 4800 timepoints per subject. For the main results, we used all four resting state scanning sessions of each participant to fit the model of brain dynamics (but see Figure 2—figure supplement 2 for results with just one session). The acquisition parameters are described in the HCP acquisition protocols and in Van Essen et al., 2013; Van Essen et al., 2012. Briefly, structural and functional MRI data were acquired on a 3T MRI scanner. The resting state fMRI data was acquired using multiband echo planar imaging sequences with an acceleration factor of 8 at 2 mm isotropic spatial resolution and a repetition time (TR) of 0.72 s. The preprocessing and timecourse extraction pipeline is described in detail in Smith et al., 2013a. For the resting state fMRI scans, preprocessing consisted of minimal spatial preprocessing and surface projection (Glasser et al., 2013), followed by temporal preprocessing. Temporal preprocessing consisted of single-session Independent Component Analysis (ICA; Beckmann, 2012) and removal of noise components (Griffanti et al., 2014; Salimi-Khorshidi et al., 2014). Data were high-pass-filtered with a cut-off at 2000s to remove linear trends.

The parcellation was estimated from the data using multi-session spatial ICA on the temporally concatenated data from all subjects. Note that this means that there is no strict divide between the subjects used for training and the subjects for testing the later predictive models, so that there is potential for leakage of information between training and test set. However, since this step does not concern the target variable, but only the preprocessing of the predictors, the effect can be expected to be minimal (Rosenblatt et al., 2024). Using this approach, a data-driven functional parcellation with 50 parcels was estimated, where all voxels are weighted according to their activity in each parcel, resulting in a weighted, overlapping parcellation. While other parcellations are available for the resting-state fMRI HCP dataset, we chose this parcellation because dynamic changes in FC have been shown to be better detected in this parcellation compared to other functional or anatomical parcellations or more fine-grained parcellations (Ahrends et al., 2022). Timecourses were extracted using dual regression (Beckmann et al., 2009), where group-level components are regressed onto each subject’s fMRI data to obtain subject-specific versions of the parcels and their timecourses. We normalised the timecourses of each subject to ensure that the model of brain dynamics and, crucially, the kernels were not driven by (averaged) amplitude and variance differences between subjects.

Subjects in the HCP study completed a range of demographic and behavioural questionnaires. Following Vidaurre et al., 2021, we here focus on a subset of those items, including age and various cognitive variables. The cognitive variables span items assessing memory, executive function, fluid intelligence, language, processing speed, spatial orientation, and attention. The full list of the 35 behavioural variables used here, as well as their categorisation within the HCP dataset can be found in Supplementary file 1a.

To estimate patterns of time-varying amplitude and FC, we here use the Hidden Markov Model (Vidaurre et al., 2016; Vidaurre et al., 2017). However, the kernels, which are explained in detail in the following section, can be constructed from any generative probabilistic model.

The Hidden Markov model (HMM) is a generative probabilistic model, which assumes that an observed time-series, such as BOLD signal in a given parcellation, was generated by a sequence of ‘hidden states’ (Baum and Eagon, 1967; Baum and Petrie, 1966). We here model the states as multivariate Gaussian distributions, defined both in terms of mean and covariance —which can be interpreted as distinct patterns of amplitude and FC (Vidaurre et al., 2017). For comparison, we also considered a second variety of the HMM where state means were pinned to zero (given that the data was demeaned and the global average is zero) and only the covariance was allowed to vary across states (Vidaurre et al., 2017), which is shown in Figure 2—figure supplement 3. This is equivalent to using a Wishart state model, and therefore focuses more specifically on time-varying FC.

The HMM is described by a set of parameters θ, containing the state probabilities π, the transition probabilities A, the mean vectors μ of all states (if modelled), and the covariance matrices Σ of all states:θ=[π,A,μ,Σ]π∈R1xK, A∈RKxK, μ∈RKxM,Σ∈RKxMxM

where K is the number of states and M is the number of parcels in the parcellation. The entire set of parameters θ is estimated from the data. The number of states can be understood as the level of detail or granularity with which we describe the spatiotemporal patterns in the data, akin to a dimensionality reduction, where a small number of states will lead to a very general, coarse description and a large number of states will lead to a very detailed, fine-grained description. Here, we chose a small number of states, K=6, to ensure that the group-level HMM states are general enough to be found in all subjects, since a larger number of states increases the chances of certain states being present only in a subset of subjects. The exact number of states is less relevant in this context, since the same HMM estimation is used for all kernels.

The HMM is a probabilistic generative model, as the generative process works by sampling probabilistically in this case from a Gaussian distribution with mean μ_k and covariance Σ_k when state k is active:Xt|qt=k∼N(μk,Σk)

where X_t is the timeseries at timepoint t and q_t is the currently active state. Which state is active depends on the previous state q_t-1 and is determined by the transition probabilities A, so that the generated state sequence is sampled from a categorical distribution with parameters:qt|qt-1=k~Cat(Ak)

where A_k indicates the k-th row of the transition probability matrix.

The space of parameters θ forms a Riemannian manifold R_θ, where the relationships between the different parameters of the HMM are acknowledged by construction. The Fisher kernel, as described below, is built upon a projection on this manifold, so predictions based on this kernel account for the mathematical structure of the HMM.

Here, we fit the HMM to the concatenated timeseries of all N subjects (see Figure 1, step 1). We refer to the group-level estimate as HMM⁰, which is defined by the parameters θ0 (see Figure 1, step 2):θ0=[π0,A0,μ0,Σ0]

To use the information from the HMM to predict subjects’ phenotypes, we estimate subject-specific versions of the group-level HMM (see Figure 1, step 3) through dual estimation (Vidaurre et al., 2017). Dual estimation refers to the process of fitting the previously estimated group-level model again to a single subject’s timeseries, so that the parameters from the group-level model HMM⁰ are adapted to fit the individual. We will refer to the subject-specific estimate for subject n as HMMⁿ, with parameters θn.

These subject-specific HMM parameters are the features from which we construct the kernels. To understand which features are most important for the predictions, we also construct versions of the kernels that include only subsets of the features. Specifically, we can group the features into two subsets: 1. the state features, describing what states look like, containing the mean vectors μ, and the covariance matrices Σ of all states, and 2. the transition features, describing how individuals transition between these states, containing the initial state probabilities π and the transition probabilities A. By removing one or the other set of features and evaluating how model performance changes compared to the full kernels, we can draw conclusions about the importance of these two different types of changes for the predictions. Since the state features are considerably more numerous than the transition features (15,300 state features compared to 42 transition features in this case), we also construct a version of the kernels where state features have been reduced to the same number as the transition features using PCA, that is we use all 42 transition features and the first 42 PCs of the state features. This allowed us to perform a fairer comparison of what elements in the model are more predictive of the subject traits.

Kernels (Shawe-Taylor and Cristianini, 2004) are a convenient approach to accommodate nonlinearity and to work with high-dimensional, complex features, such as parameters from a model of brain dynamics. In general, kernels are similarity functions, and they can be used straightforwardly in a prediction algorithm. While feature matrices can be very high dimensional, a kernel is represented by a (no. of subjects by no. of subjects) matrix. Kernel methods can readily be adapted to deal with nonlinear decision boundaries in prediction, by projecting the data into a high-dimensional (possibly infinite-dimensional) space through an embedding x→ϕx; then, by estimating a linear separating hyperplane on this space, we can effectively have a nonlinear estimator on the original space (Shawe-Taylor and Cristianini, 2004). In practice, instead of working explicitly in a higher-dimensional embedding space, the so-called kernel trick uses a kernel function κ(n,m) containing the similarity between data points n and m (here, subjects) in the higher-dimensional embedding space (Schölkopf et al., 2002; Shawe-Taylor and Cristianini, 2004), which can be simpler to calculate. Once κ(⋅,⋅) is computed for each pair of subjects, this is all that is needed for the prediction. This makes kernels computationally very efficient, since in most cases the number of subjects will be smaller than the number of features —which, in the case of HMMs, can be very large (potentially, in the order of millions). However, finding the right kernel can be a challenge because there are many available alternatives for the embedding.

Here, in combination with a linear predictive model, we apply a kernel that is specifically conceived to be used to compare instances of generative models such as the HMM. We expected this to result in better predictions than existing methods. Using the same HMM estimate, we compare three different kernels, which map the HMM parameters into three distinct spaces, corresponding to different embeddings (see Figure 1, step 4): the naïve kernel, the naïve normalised kernel, and the Fisher kernel.

While the first two kernels (naïve and naïve normalised kernel) do not take into account constraints imposed by the HMM on how the model parameters can change with respect to each other, the Fisher kernel does. The Fisher kernel achieves this by calculating how, and by how much, one should change the group level HMM parameters to make the model generate data that is more like the data from a particular subject.

We then construct linear and Gaussian versions of the different kernels (see Figure 1, step 4), which take the general form κl(n,m)=ϕxnTϕxm for the linear kernel and κg(n,m)=exp(−‖ϕxn−ϕxm‖22τ2) for the Gaussian kernel. We compare these to a kernel constructed using Kullback-Leibler divergence, previously used for predicting behavioural phenotypes (Vidaurre et al., 2021).

The Fisher kernel (Jaakkola et al., 1999; Jaakkola and Haussler, 1998) is specifically designed to preserve the structure of a generative probabilistic model (here, the HMM). This can be thought of as a ‘proper’ projection on the manifold, as illustrated in Figure 1, step 4b. Similarity between subjects is here defined in reference to a group-level model of brain dynamics. The mapping x→ϕx is given by the ‘Fisher score’, which indicates how (i.e., in which direction in the Riemannian parameter space) we would have to change the group-level model to better explain a particular subject’s timeseries. The similarity between subjects can then be described based on this score, so that two subjects are defined as similar if the group-level model would have to be changed in a similar direction for both, and dissimilar otherwise.

More precisely, the Fisher score is given by the gradient of the log-likelihood with respect to each model parameter:g(θ0,xn)=(∂logLθ0(xn)∂θ0)g∈R1x(K+K∗K+K∗M+K∗M∗M)

where xn is the timeseries of subject n, and Lθ(x)=P(x|θ) represents the likelihood of the timeseries x given the model parameters θ. This way, the Fisher score maps an example (i.e. a subject’s timeseries) xn into a point in the gradient space of the Riemannian manifold Rθ defined by the HMM parameters.

The invariant Fisher kernel κF- is the inner product of the Fisher score g, scaled by the Fisher information matrix F, which gives a local metric on the Riemannian manifold Rθ:κF−(n,m)=g(θ0,xn)TFRθ−1g(θ0,xm)κF−∈RNxN

for subjects n and m. FRθ is the Fisher information matrix, defined asFRθ=Ex[g(θ0,x)g(θ0,x)T]

where the expectation is with respect to x under the distribution P(x|θ). The Fisher information matrix FRθ can be approximated empirically:F^Rθ=1N∑i=1Ng(θ0,xi)g(θ0,xi)T

which is simply the covariance matrix of the gradients g. Using F^Rθ essentially serves to whiten the gradients; therefore, given the large computational cost associated with F^Rθ, we here disregard the Fisher information matrix and reduce the invariant Fisher kernel to the so-called practical Fisher kernel (Jaakkola et al., 1999; Jaakkola and Haussler, 1998; van der Maaten, 2011), for which the linear version κFl takes the form:κFl(n,m)=⟨g(θ0,xn),g(θ0,xm)⟩κFl∈RNxN

In this study, we will use the practical Fisher kernel for all computations.

One issue when working with the linear Fisher kernel is that the gradients of typical examples (i.e. subjects whose timeseries can be described by similar parameters as the group-level model) are close to zero, while gradients of atypical examples (i.e. subjects who are very different from the group-level model) can be very large. This may lead to an underestimation of the similarity between two typical examples because their inner product is very small even though they are very similar. To mitigate this, we can plug the gradient features (i.e. the Fisher scores g) into a Gaussian kernel, which essentially normalises the kernel. For subjects n and m, where xn is the timeseries of subject n and xm is the timeseries of subject m, the Gaussian Fisher kernel κFg is defined asκFg(n,m)=exp⁡(−‖g(θ0,xn)−g(θ0,xm)‖22τ2)κFg∈RNxN

where ‖g(θ0,xn)−g(θ0,xm)‖ is the distance between examples n and m in the gradient space, and τ is the width of the Gaussian kernel.

The Kullback-Leibler (KL) divergence is an information-theoretic distance measure which estimates divergence between probability distributions —in this case between subject-specific versions of the HMM. Here, KL divergence of subject n from subject m, KL(HMMn||HMMm) can be interpreted as how much new information the HMM of subject n contains if the true distribution was the HMM of subject m. KL divergence is not symmetric, that is KL(HMMn||HMMm) is different than KL(HMMn||HMMm). We here use an approximation of KL divergence as in Do, 2003 and Vidaurre et al., 2021. That is, given two models HMMn from HMMm for subject n and subject m, we haveKL(HMMn||HMMm)=∑kvkKL(Akn, Akm)+vkKL(Gkn, Gkm)

where Akn are the transition probabilities from state k into any other state according to HMMn and Gkn are the state Gaussian distributions for state k and HMMn (respectively Akm and Gkm for HMMm); see MacKay et al., 2003. Since the transition probabilities are Dirichlet-distributed and the state distributions are Gaussian distributed, KL divergence for those has a closed-form solution. Variables vk can be computed numerically such that,vAn=vlimx→∞πnAnx=ν

To be able to use KL divergence as a kernel, we symmetrise the KL divergence matrix asDKL(n,m)=0.5KL(HMMn||HMMm)+0.5KL(HMMn||HMMm)DKL∈RNxN

This symmetrised KL divergence can be plugged into a radial basis function, analogous to the Gaussian kernels to obtain a similarity matrix κKLκKL(n,m)=exp⁡(−(DKL(n,m))22τ2)κKL∈RNxN

The resulting KL similarity matrix can be used in the predictive model in a similar way as the kernels described above.

Similarly to Vidaurre et al., 2021, we use kernel ridge regression (KRR) to predict demographic and behavioural variables from the different kernels (other kernel-based prediction models or classifiers such as a support vector machine are also possible). KRR is the kernelised version of ridge regression (Saunders et al., 1998):y^=hαα∈RStrainx1, h∈RStestxStrain

where α are the regression weights; h is the (number of subjects in test set by number of subjects in training set) kernel matrix between the subjects in the training set and the subjects in the test set; y^ are the predictions in the (out-of-sample) test set; Strain are the number of subjects in the training set; and Stest are the number of subjects in the test set. The regression weights α can be estimated using the kernels specified above asα=(κ+λI)−1∗yκ∈RStrainxStrain

where λ is a regularisation parameter that we can choose through cross-validation; I is the identity matrix; κ is the (Strain by Strain) kernel matrix of the subjects in the training set; and y are the training examples.

We use KRR to separately predict each of the 35 demographic and behavioural variables from each of the different methods, removing subjects with missing entries from the prediction. We used k-fold nested cross-validation (CV) to select and evaluate the models. We used 10 folds for both the outer loop (used to train and test the model) and the inner loop (used to select the optimal hyperparameters) such that 90% were used for training and 10% for testing. The optimal hyperparameters λ (and τ in the case of the Gaussian kernels) were selected using grid-search from the vectors λ=[0.0001, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1] and τ=[1/5, 1/3, 1/2, 1, 2, 3, 5]. In both the outer and the inner loop, we accounted for family structure in the HCP dataset so that subjects from the same family were never split across folds (Winkler et al., 2015). Within the CV, we regressed out sex and head motion confounds, that is we estimated the regression coefficients for the confounds on the training set and applied them to the test set (Snoek et al., 2019). We repeated the nested 10-fold CV 100 times, so that different combinations of subjects were randomly assigned to the folds at each new CV iteration to obtain a distribution of model performance values for each variable. This is to explicitly show how susceptible each model was to changes in the training folds, which we can take as a measure of the robustness of the estimators, as described below. We generated the 100 random repetitions of the 10 outer CV folds once, and then used them for training and prediction of all methods, so that all methods were fit to the same partitions.

To compare our approach’s performance to simpler methods that do not take dynamics into account, we compared them to seven different regression models based on time-averaged FC features. For each subject, time-averaged FC was computed as covariance between each pair of regions. The first time-averaged model is, analogous to the HMM-derived KL divergence model, a time-averaged KL divergence model. The model is described in detail in Vidaurre et al., 2021. Briefly, we construct symmetrised KL divergence matrices of each subject’s time-averaged FC and predict from these matrices using the KRR pipeline described above. We refer to this model as time-averaged KL divergence. We next used a kernel constructed from a geodesic distance (i.e. a metric defined on the Riemannian manifold) of time-averaged covariance matrices. Specifically, we used a Gaussian kernel on the log-Euclidean distance, which is the Frobenius norm of the logarithm map of the time-averaged covariance matrices (Jayasumana et al., 2013). We refer to this model as log-Euclidean. The other five time-averaged FC benchmark models do not involve kernels but predict directly from the features instead. Namely, we use two variants of a Ridge Regression and an Elastic Net model (Zou and Hastie, 2005), one using the unwrapped time-averaged FC matrices as input (i.e. in Euclidean space), and one using the time-averaged FC matrices in Riemannian space, where covariance matrices are projected into tangent space (Barachant et al., 2013; Smith et al., 2013b), which we refer to as Ridge Reg. and Ridge Reg. (Riem), and Elastic Net and Elastic Net (Riem.), respectively. Finally, we compare our models to the approach taken in Rosenberg et al., 2016, where relevant edges of the time-averaged FC matrices are first selected, and then used as predictors in a regression model. We refer to this model as Selected Edges. All time-averaged FC models are fitted using the same (nested) cross-validation strategy as described above (10-fold CV using the outer loop for model evaluation and the inner loop for model selection using grid-search for hyperparameter tuning, accounting for family structure in the dataset, and repeated 100 times with randomised folds).

All code used in this paper, including scripts to reproduce the figures and additional application examples of the Fisher Kernel can be found in the repository https://github.com/ahrends/FisherKernel (copy archived at Ahrends, 2025). The HMM-Fisher kernel pipeline in Matlab is also publicly available within the repository of the HMM-MAR toolbox at https://github.com/OHBA-analysis/HMM-MAR, (copy archived at OHBA Analysis Group, 2025). A Python-version of the Fisher kernel is also available at https://github.com/vidaurre/glhmm (copy archived at Vidaurre, 2025a). Code for Ridge Regression and Elastic Net prediction from time-averaged FC features is available at https://github.com/vidaurre/NetsPredict/blob/master/nets_predict5.m (copy archived at Vidaurre, 2025b). The procedure for Selected Edges time-averaged FC prediction is described in detail in Shen et al., 2017 and code is provided at https://www.nitrc.org/projects/bioimagesuite/behavioralprediction.m.