To examine the heavy-tailed structure of the observed point-trajectory velocity distributions, we pool horizontal and vertical velocity components together for an example movie, bees8-full, and compare this histogram to a Gaussian distribution with the same variance (Figure 2A). Plotted on a log-scale to highlight the tails, the empirical frequency falls off nearly linearly away from zero, while the Gaussian probability falls off quadratically. The same is true for the other movies in our dataset, pooled by category and all together (Figure 4A–C). Velocity distributions from animal and plant movies tend to have heavier tails, while those of water movies are closer to Gaussian.

When multiple variables are involved, heavy tails may be associated with a nonlinear form of dependency, as observed in the spatial structure of natural images (Wainwright et al., 2001). The same is true for the two velocity components in our data. We illustrate this for bees8-full, but results are similar for all other movies. In Figure 2B, we show a heat map of the joint histogram of horizontal and vertical velocities, u\begin{document}$u$\end{document} and v\begin{document}$v$\end{document}. It is nearly radially symmetric. (For other movies with unequal variance in the two components, distributions are elliptic.) The lack of tilt in the histogram indicates that the two velocity components are uncorrelated (in this context, correlation between velocity components would indicate a tendency for objects within a scene to move along some diagonal axis relative to the camera). However, they are far from independent: when we shuffle the data to break any association between u\begin{document}$u$\end{document} and v\begin{document}$v$\end{document}, the resulting histogram is no longer radially symmetric but is instead diamond-shaped (Figure 2C). This is, in fact, to be expected, since only Gaussian distributions may be both radially symmetric and separable. We demonstrate this dependency more clearly by plotting the conditional distribution of v\begin{document}$v$\end{document} for each value of u\begin{document}$u$\end{document}, normalizing by the peak value at each u\begin{document}$u$\end{document} for visualization purposes (Figure 2D). The resulting ‘bow-tie’ shape (Schwartz and Simoncelli, 2001) indicates that the variance of v\begin{document}$v$\end{document} conditioned on u\begin{document}$u$\end{document} increases with the magnitude of u\begin{document}$u$\end{document}. A simple, quantitative indicator of this nonlinear dependence is the Pearson correlation of the component magnitudes (ρ=0.33\begin{document}$\rho=0.33$\end{document}, p≪0.01\begin{document}$p\ll 0.01$\end{document}), which will be zero for any pair of independent random variables.

The form of the velocity distributions observed above suggests that they can be modeled as GSM distributions. As the name suggests, a GSM distribution is obtained by combining (zero-mean) Gaussian distributions of different scales, parameterized by a positive scalar random variable S\begin{document}$S$\end{document}. Let Y\begin{document}$Y$\end{document} be a Gaussian random variable with mean zero and variance σY2\begin{document}$\sigma_{Y}^{2}$\end{document}. If S\begin{document}$S$\end{document} is a known quantity s\begin{document}$s$\end{document}, then X=Ys\begin{document}$X=Ys$\end{document} is simply a Gaussian random variable with mean zero and variance s2σY2\begin{document}$s^{2}\sigma_{Y}^{2}$\end{document}. The conditional distribution is given byp(x|s)=N(x;0,s2σy2)\begin{document}$$\displaystyle p(x\vert s) = \mathcal{N}\left(x;0,s^2 \sigma_y^2\right)$$\end{document}

When S\begin{document}$S$\end{document} is unknown, X=YS\begin{document}$X=YS$\end{document} follows a GSM distribution given byp(x)=∫0∞p(x|s)p(s)ds,\begin{document}$$\displaystyle p(x)=\int_0^{\infty} p(x|s)p(s)ds,$$\end{document}

where p(s)\begin{document}$p(s)$\end{document} is a distribution with positive support. A convenient choice is to let S=exp⁡(Z)\begin{document}$S=\exp(Z)$\end{document}, where Z\begin{document}$Z$\end{document} is Gaussian random variable, which we will refer to as the scale generator, with mean zero and variance σZ2\begin{document}$\sigma_{Z}^{2}$\end{document}. Then, S\begin{document}$S$\end{document} follows a log-normal distribution, which simplifies the inference problem significantly, despite the fact that the resulting GSM distribution does not have a closed form. The choice of a log-normal distribution can also be justified by a maximum entropy argument (Van der Straeten and Beck, 2008). See Wainwright and Simoncelli, 2000; Wainwright et al., 2001, for a discussion of the GSM model in the context of wavelet analysis of natural images. Parameters were estimated using a variant of the expectation-maximization (EM) algorithm (Dempster et al., 1977) (see Materials and methods).

For an individual velocity component as in Figure 2A, the GSM model captures the shape of the distribution well, with only two parameters: σY\begin{document}$\sigma_{Y}$\end{document}, controlling the overall scale, and σZ\begin{document}$\sigma_{Z}$\end{document}, controlling the heaviness of the tail. The variance of X\begin{document}$X$\end{document} is related to these parameters byσX2=σY2exp⁡(2σZ2).\begin{document}$$\displaystyle \sigma_X^2=\sigma_Y^2\exp\left(2\sigma_Z^2\right).$$\end{document}

The kurtosis, which is the standard measurement of tail heaviness, depends only on σZ\begin{document}$\sigma_{Z}$\end{document}:κX=3exp⁡(4σZ2).\begin{document}$$\displaystyle \kappa_X=3\exp\left(4\sigma_Z^2\right)\,.$$\end{document}

The kurtosis of X\begin{document}$X$\end{document} thus grows exponentially with the variance of Z\begin{document}$Z$\end{document} and matches the Gaussian kurtosis of 3 if and only if σZ=0\begin{document}$\sigma_{Z}=0$\end{document}.

To model the joint distribution, as in Figure 2B, clearly we cannot use independent GSM models for each component, since this corresponds to the shuffled distribution in Figure 2C. Instead, we consider a model in which two independent Gaussian random variables, Y1\begin{document}$Y_{1}$\end{document} and Y2\begin{document}$Y_{2}$\end{document}, are multiplied by a shared scale variable S\begin{document}$S$\end{document}:X1=Y1S,\begin{document}$$\displaystyle X_1=Y_1S\,,$$\end{document}X2=Y2S.\begin{document}$$\displaystyle X_2=Y_2S\,.$$\end{document}

Note that we will maintain the general notation for the model for clarity. Applied to the velocity data, we have(X1,X2)=(U,V),\begin{document}$$\displaystyle (X_1,X_2)=(U,V)\,,$$\end{document}

and (Y1,Y2)\begin{document}$(Y_1, Y_2)$\end{document} are the corresponding scale-normalized velocity components. The model is depicted in Figure 3, and it captures the radially symmetric (or more generally, when the variances are not equal, elliptic) shape of the joint distribution. Note that this model reveals why the Pearson correlation of the component magnitudes captures the nonlinear dependence between U and V. We have, for the covariance (the numerator of the correlation coefficient),E[(|X1|−E[|X1|])(|X2|−E[|X2|])]=E[|Y1|]E[|Y2|](E[S2]−E[S]2),\begin{document}$$\displaystyle E\left[(|X_1| - E[|X_1|]) (|X_2| - E[|X_2|])\right]= E[|Y_1|]\, E[|Y_2|]\, \left(E[S^2] - E[S]^2\right),$$\end{document}

which is positive whenever S\begin{document}$S$\end{document} has nonzero variance.

This model has only three parameters: σY1\begin{document}$\sigma_{Y_{1}}$\end{document}, σY2\begin{document}$\sigma_{Y_{2}}$\end{document}, and σZ\begin{document}$\sigma_{Z}$\end{document}. (A more complicated model could have two correlated scale variables with different standard deviations. This does not appear to be necessary since the scale generator standard deviations fit independently to each component are nearly identical for most movies, and the elliptic shapes of the distributions indicate that the scale correlations across components are near one.) We observe a wide range of scale generator standard deviations σZ\begin{document}$\sigma_{Z}$\end{document} both within and across categories (Figure 4D). The trend across categories – namely that animal and plant movies tend to have higher scale standard deviations than water movies – agrees with the relative heaviness of the tails for the pooled data (Figure 4C). On the other hand, σY1\begin{document}$\sigma_{Y_{1}}$\end{document} and σY2\begin{document}$\sigma_{Y_{2}}$\end{document} need not be similar, and many movies had a larger standard deviation of motion on the horizontal axis than vertical (the ratio σY2/σY1\begin{document}$\sigma_{Y_{2}}/\sigma_{Y_{1}}$\end{document} tended to be less than one, Figure 4E). The AIC calculated for the two-dimensional GSM model with a common scale variable indicates that it is a better fit to the data than a two-dimensional, independent Gaussian model (Figure 4F).

While the time-independent statistics of velocity are important, a full description of how objects move must include how the velocity evolves over time along point trajectories. (This description should ideally also include how multiple points on the same object evolve over time, allowing us to capture rotations, contractions, and expansions; we do not attempt this more ambitious analysis here and limit our discussion to local translations.) This motivates our point tracking analysis, which provides information that cannot be gleaned from motion estimates at fixed locations alone. From the raw data, we know the velocity is highly correlated at short time lags, but it is not clear how the scale variable enters into play. We again inspect the joint velocity distribution for an example movie, now across neighboring time points for one velocity component (Figure 5A). The tilt indicates strong linear correlation across time in the velocity, as expected, and we note that the overall shape is elliptic despite having heavy-tailed marginals, as in Figure 2B. In Figure 5B, we condition on the velocity at one time step and observe a tilted version of the same bow-tie shape as in Figure 2D. With the addition of an overall tilt, these are precisely the same observations that led us to the GSM model above. Thus, two forms of dependence – linear correlation and the nonlinear dependence due to the scale variable – coexist in the time domain.

We next ask whether this scale variable is constant in time (varying only from trajectory to trajectory) or dynamic (varying in time within a given trajectory). If it is constant, the joint distribution of the two components will not depend on the alignment of the two components in time, so long as they are from the same trajectory. In Figure 5D, we examine these joint distributions after shifting one component relative to the other by a time lag, for a range of lags. The distributions gradually shift from the radially symmetric zero-lag distribution to a diamond shape similar to the shuffled distribution. Similarly, the correlation between velocity component magnitudes, an indicator of nonlinear dependence discussed above, decreases as the lag increases (Figure 5C). In other words, at zero lag, the two velocity components are governed by a single scale variable; at nonzero lags, they are governed by two correlated scale variables, and this correlation decreases with lag. This indicates that the underlying scale variable fluctuates over time. Notably, we can detect these fluctuations within the ∼1-s-long trajectories to which we limit our analysis. This would not be the case if the scale were to change only on a very long timescale or only across different point trajectories within a scene.

We would like to capture this dynamic scale variable in our model of natural motion. It is straightforward to make the GSM model dynamic by replacing each Gaussian variable with an autoregressive Gaussian process, and we call this new model the ARGSM model. We illustrate it schematically in Figure 6 by generating example traces for one Gaussian velocity component Y\begin{document}$Y$\end{document} and the scale generator Z\begin{document}$Z$\end{document}. Note that the autoregressive scale generator variable is the temporal equivalent to the spatial Markov random fields explored in the image domain (Wainwright et al., 2001; Roth and Black, 2009; Lyu and Simoncelli, 2009). Given this model, we perform estimation of the parameters using a stochastic approximation variant of the EM algorithm (see Materials and methods). (Code for fitting the ARGSM model is available at https://github.com/sepalmer/motion-scale-mixture, Palmer and Salisbury, 2026) Example traces illustrating the results of this model are shown in Figure 7A–C. The estimated autoregression coefficients determine the correlation functions of the underlying Gaussian velocity and scale generator processes, which we plot for each movie in Figure 7D and E, respectively. The fact that some velocity correlation functions and many scale generator correlation functions do not go to zero over the length of the trajectories could indicate a nonzero mean component that varies from trajectory to trajectory, but this is beyond the scope of the present analysis. Average correlation functions across categories are shown in Figure 7F–G. We also report the time to 0.5 correlation for each movie for the velocity in Figure 7H–I. Akaike information criterion (AIC) scores (Figure 7K) indicate that the full ARGSM is a better fit to the data compared to the AR model. It is also a better fit compared to the ARGSM model with a static Z\begin{document}$Z$\end{document} value for each trajectory, indicating that a dynamic scale variable is essential for describing the data. In the context of visual tracking of moving objects, the timescales of these correlation functions are extremely important. On one hand, the velocity correlation time determines how far into the future motion can be extrapolated. On the other hand, the scale correlation time determines the timescale on which adaptation must take place in order to efficiently process motion signals with limited dynamic range.

Finally, we ask whether the full ARGSM model is necessary to carry out scale normalization in practice for our trajectory data. Our model fitting provides an estimate of the scale at each time point, which we use to normalize the raw data. To quantify normalization performance, we calculate the kurtosis, or fourth standardized moment, which measures how heavy-tailed a distribution is. The standard reference is a Gaussian random variable, which has a kurtosis of 3. In Figure 8, we compare the kurtosis of the velocity before and after dividing by a point estimate of the scale under three models of increasing complexity. If normalization is successful, the distribution of the resulting normalized velocity should be approximately Gaussian. Under the time-independent model, the normalized velocity consistently has kurtosis less than 3, indicating that the scale tends to be overestimated (Figure 8A). In contrast, for a model with correlated velocity and constant scale for each trajectory, the kurtosis is consistently larger than 3, indicating that the scale tends to be underestimated (Figure 8B). Only the full model, with correlated velocity and a dynamic, correlated scale variable, yields a kurtosis around 3 for each movie, even with highly kurtotic data (Figure 8C).

This exercise of using the ARGSM model to estimate the scale at each time point, then dividing the velocity by this scale, serves as a proxy for what the nervous system can achieve through adaptation mechanisms. An important caveat is that the model has access to the full trajectory, while the nervous system must operate in an online, causal setting.

Prediction is an important problem both for compression via predictive coding and for overcoming sensory and motor delays during behavior. Prediction is built into the ARGSM framework since the regression coefficients of the AR models are optimal for predicting the next time step of Yt\begin{document}$Y_{t}$\end{document} and Zt\begin{document}$Z_{t}$\end{document} given their past values. Let Y^t\begin{document}$\hat{Y}_{t}$\end{document} and Z^t\begin{document}$\hat{Z}_{t}$\end{document} denote the predicted values and υt\begin{document}$\upsilon_{t}$\end{document} and ζt\begin{document}$\zeta_{t}$\end{document} the respective errors:Yt=Y^t+υt,\begin{document}$$\displaystyle Y_t=\hat{Y}_t+\upsilon_t,$$\end{document}Zt=Z^t+ζt.\begin{document}$$\displaystyle Z_t=\hat{Z}_t+\zeta_t.$$\end{document}

The variance explained by a prediction X^t\begin{document}$\hat{X}_{t}$\end{document} is given byR2=E[Xt2]−E[(Xt−X^t)2]E[Xt2].\begin{document}$$\displaystyle R^2=\frac{E\left[X_t^2\right]-E\left[\left(X_t-\hat{X}_t\right)^2\right]}{E\left[X_t^2\right]}\,.$$\end{document}

For the Gaussian process Yt\begin{document}$Y_{t}$\end{document}, this simplifies toRY2=σY^2σY2=1−συ2σY2.\begin{document}$$\displaystyle R_Y^2=\frac{\sigma_{\hat{Y}}^2}{\sigma_Y^2}=1-\frac{\sigma_\upsilon^2}{\sigma_Y^2}\,.$$\end{document}

Assuming knowledge of the histories of both Xt\begin{document}$X_{t}$\end{document} and Zt\begin{document}$Z_{t}$\end{document}, the prediction for Xt\begin{document}$X_{t}$\end{document} isX^t=Y^texp⁡(Z^t),\begin{document}$$\displaystyle \hat{X}_t=\hat{Y}_t\exp(\hat{Z}_t),$$\end{document}

The associated variance explained isRX|Z2=RY2(2exp⁡12σζ2−exp⁡2σζ2).\begin{document}$$\displaystyle R_{X|Z}^2=R_Y^2\left(2\exp\frac{1}{2}\sigma_\zeta^2-\exp2\sigma_\zeta^2\right)\,.$$\end{document}

This is an upper bound on the performance of any predictor with access only to Xt\begin{document}$X_{t}$\end{document}.

Notably, RX|Z2\begin{document}$R_{X|Z}^{2}$\end{document} is independent of σZ2\begin{document}$\sigma_{Z}^{2}$\end{document}: the variance explained under the ARGSM model for X\begin{document}$X$\end{document} is equal to the variance explained for Y\begin{document}$Y$\end{document}, multiplied by a function of the variance of the driving noise for Z\begin{document}$Z$\end{document}, σζ2\begin{document}$\sigma_{\zeta}^{2}$\end{document}, that slowly decreases from one to zero. Since σζ2\begin{document}$\sigma_{\zeta}^{2}$\end{document} is small for the estimated models, indicating that the scale fluctuations are highly predictable at the level of single time steps, we expect this term to have little effect. In Figure 7J, we compare the variance explained by applying naive autoregression to Xt\begin{document}$X_{t}$\end{document} (denoted RX2\begin{document}$R_{X}^{2}$\end{document}) to RY2\begin{document}$R_{Y}^{2}$\end{document} and RX|Z2\begin{document}$R_{X|Z}^{2}$\end{document} using the estimated model parameters. The variance explained is close to one for all movies except three depicting insects (a consequence of their very short velocity correlation times). Values of RX2\begin{document}$R_{X}^{2}$\end{document} tend to be only slightly smaller than RX|Z2\begin{document}$R_{X|Z}^{2}$\end{document} (except in two instances where it was slightly larger). We conclude that heavy-tailed statistics have little effect on the predictability of natural motion, although scale estimation is necessary for estimating the variance associated with the prediction, i.e., the variance of the posterior distribution p(xt|x←t−1)≈N(xt;x^t,συ2exp⁡(2zt^))\begin{document}$p\left(x_{t}|\overleftarrow{\mathbf{x}}_{t-1}\right)\approx\mathcal{N}\left(x_ {t};\hat{x}_{t},\sigma_{\upsilon}^{2}\exp\left(2\hat{z_{t}}\right)\right)$\end{document}.

Long correlation times and high values of R2\begin{document}$R^{2}$\end{document} indicate that the velocity time series of natural motion are highly predictable. One way to make use of this predictability is through predictive coding, in which only prediction errors (with variance σX2−σX^2\begin{document}$\sigma_{X}^{2}-\sigma_{\hat{X}}^{2}$\end{document}, as opposed to σX2\begin{document}$\sigma_{X}^{2}$\end{document} for the original signal) are sent through a channel. However, this may be a challenge for the visual system, since motion is encoded in spatial arrays of neurons rather than individual channels. A second use is actually carrying out the prediction to compensate for delays in perception or to drive motor output. (Note that since the position q\begin{document}$q$\end{document} of a point is the integral of its velocity, the prediction of position by means of correlations in the velocity is given by q^(t+α)=q(t)+∫0αv^(t+τ)dτ\begin{document}$\hat{q}(t+\alpha)=q(t)+\int_{0}^{\alpha}\hat{v}(t+\tau)d\tau$\end{document}, where v^(t+τ)\begin{document}$\hat{v}(t+\tau)$\end{document} is the prediction of the velocity at time t+τ\begin{document}$t+\tau$\end{document} given its history up to time t\begin{document}$t$\end{document}.) The results of this section indicate that this is feasible with or without taking the fluctuating scale into account.

We compute short trajectories using the PointTracker function in MATLAB’s Computer Vision toolbox. The function employs a Kanade-Lucas-Tomasi (Lucas and Kanade, 1981; Tomasi and Kanade, 1991) feature tracking algorithm, which uses multi-scale image registration under a translational motion model to track individual points from frame to frame. Briefly, given an image patch I(x,y,t)\begin{document}$I(x,y,t)$\end{document} centered on some seeded initial position, the algorithm finds the displacement (Δx,Δy)\begin{document}$(\Delta x,\Delta y)$\end{document} that minimizes the squared error,ϵ2=∬[I(x,y,t)−I(x+Δx,y+Δy,t+Δt)]2dxdy,\begin{document}$$\displaystyle \epsilon^2=\iint\left[I(x,y,t)-I(x+\Delta x,y+\Delta y,t+\Delta t)\right]^2dxdy \,,$$\end{document}

and updates the seed position on the next frame. Our strategy is to collect as many high-quality, short trajectories (64 frames) as possible from each movie, then subsample these down to a reasonable number of trajectories (8192) for statistical analysis. Initial points are seeded using the detectMinEigenFeatures function, which detects image features that can be tracked well under the motion model (Shi and Tomasi, 1994). From the initial seeds, we run the tracking algorithm forward and backward for 32 frames each, rather than running it in one direction for 64 frames. This increases the chances of capturing short-lived trajectories bounded by occlusion or image boundaries. Points are seeded on each frame, so the resulting set of trajectories is highly overlapping in time. On most movies, we employ the built-in forward-backward error checking method (Kalal et al., 2010), with a threshold of 0.25 pixels, to automatically detect tracking errors. The exceptions are three movies depicting water (water3, water5, and water6) where the small threshold leads to rejecting most trajectories, so we use a threshold of 8 pixels. In these cases, there are not well-defined objects, so relaxing this strict criterion is justified. The algorithm uses a multi-resolution pyramid and computes gradients within a neighborhood at each level. We use the default values of 3 pyramid levels and a neighborhood size of 31 by 31 pixels for all movies except the 3 water movies, where we find we can decrease the amount of erroneously large jumps in trajectories by increasing the neighborhood size to 129 by 129 pixels and using only 1 pyramid level (at a cost of greater computation time).

This method automatically tracks the stationary background points, which may be erroneously ‘picked up’ by a moving object as it traverses that location. To ensure that the trajectories we analyze are full of motion, we define a speed (velocity magnitude) threshold of 0.1 pix/frame and discard trajectories in which 16 or more time steps are below this threshold. We validated our method on a simple synthetic movie animated by an ARGSM process (Appendix 2).

The velocity time series is simply the first difference of the point positions along each trajectory. Within each ensemble, we subtract the ensemble mean from each velocity component (this is typically very close to zero, except for some water movies with a persistent flow). We then slightly rotate the horizontal and vertical velocity components to remove small correlations between them (these arise if, for example, objects tended to move along a slight diagonal relative to the camera’s sensor). All visualizations and calculations are carried out after these minor preprocessing steps. Note that we do not subtract the average velocity within each trajectory, as this introduces an artificial anticorrelation at long lags.

The basic one-dimensional GSM model is described in the main text. Note that for some choices of the distribution for S\begin{document}$S$\end{document}, the distribution for X\begin{document}$X$\end{document} has a closed-form solution. For example, the well-known Student’s t\begin{document}$t$\end{document}-distribution is formed when S\begin{document}$S$\end{document} follows an inverse χ\begin{document}$\chi$\end{document}-distribution, and the Laplace distribution is formed when S\begin{document}$S$\end{document} follows a Rayleigh distribution. In this work, we assume S\begin{document}$S$\end{document} follows a log-normal distribution, which does not yield a closed-form distribution for X\begin{document}$X$\end{document}. This choice makes modeling correlations straightforward, as will be made clear below. In practice, the lack of a closed-form p(x)\begin{document}$p(x)$\end{document} is not a drawback, since we do not need to normalize the posterior distribution, p(s|x)∝p(x|s)p(s)\begin{document}$p(s|x)\propto p(x|s)p(s)$\end{document}, in order to sample from it.

When considering multiple variables, a shared scale variable introduces a nonlinear form of dependence between them. Suppose X1=Y1S\begin{document}$X_{1}=Y_{1}S$\end{document} and X2=Y2S\begin{document}$X_{2}=Y_{2}S$\end{document}. If Y1\begin{document}$Y_{1}$\end{document} and Y2\begin{document}$Y_{2}$\end{document} are uncorrelated, then X1\begin{document}$X_{1}$\end{document} and X2\begin{document}$X_{2}$\end{document} are conditionally independent given S\begin{document}$S$\end{document}:p(x1,x2|s)=p(x1|s)p(x2|s).\begin{document}$$\displaystyle p(x_1,x_2|s)=p(x_1|s)p(x_2|s).$$\end{document}

However, X1\begin{document}$X_{1}$\end{document} and X2\begin{document}$X_{2}$\end{document} are not, in general, independent:p(x1,x2)=∫0∞p(x1|s)p(x2|s)p(s)ds≠p(x1)p(x2).\begin{document}$$\displaystyle p(x_1,x_2)=\int_0^\infty p(x_1|s)p(x_2|s)p(s)ds\neq p(x_1)p(x_2)\,.$$\end{document}

This nonlinear dependence manifests itself in the elliptic level sets of p(x1,x2)\begin{document}$p(x_{1},x_{2})$\end{document}, in contrast to the diamond-shaped level sets of p(x1)p(x2)\begin{document}$p(x_{1})p(x_{2})$\end{document}. Note that this nonlinear dependence can coincide with the usual linear dependence if Y1\begin{document}$Y_{1}$\end{document} and Y2\begin{document}$Y_{2}$\end{document} are correlated, and that a weaker form of nonlinear dependence may be present if X1=Y1S1\begin{document}$X_{1}=Y_{1}S_{1}$\end{document} and X2=Y2S2\begin{document}$X_{2}=Y_{2}S_{2}$\end{document}, where S1\begin{document}$S_{1}$\end{document} and S2\begin{document}$S_{2}$\end{document} are not independent.

Autoregressive models (Yule, 1927; Walker, 1931) are a well-established and flexible way to capture correlations in time series data by supposing a linear relationship between the current value of a random variable with its previous values. Given a time series, {X1,…,XT}\begin{document}$\{X_{1},\dots,X_{T}\}$\end{document}, the kth order autoregressive, or AR(k\begin{document}$k$\end{document}), model is given byXt=∑i=1kϕiXt−i+ξt\begin{document}$$\displaystyle X_t=\sum_{i=1}^k\phi_iX_{t-i}+\xi_t$$\end{document}

where {ϕ1,…,ϕk}\begin{document}$\{\phi_{1},\dots,\phi_{k}\}$\end{document} are regression coefficients and ξt\begin{document}$\xi_{t}$\end{document} is Gaussian noise with variance σ2\begin{document}$\sigma^{2}$\end{document}.

The order k\begin{document}$k$\end{document} is typically chosen by cross-validation to avoid overfitting. This makes sense from the standpoint of finding a model that generalizes well to new data. However, our primary aim here is simply to measure the autocovariances of the hidden variables, since their timescales are relevant to prediction and adaptation in the nervous system. For this reason, we choose k\begin{document}$k$\end{document} to be as high as possible: k=31\begin{document}$k=31$\end{document} time steps, since k\begin{document}$k$\end{document} must be less than T/2\begin{document}$T/2$\end{document}.

Typically, the model parameters are fit by standard linear regression (after organizing the data appropriately) (Hamilton, 2020). However, this method gives maximum likelihood estimates only if the initial k\begin{document}$k$\end{document} time steps are considered fixed. If the initial data are assumed to be drawn from the stationary distribution defined by the parameters, the problem becomes nonlinear. The EM algorithm described below requires parameter estimates to be maximum likelihood, and since we would like the initial k\begin{document}$k$\end{document} time steps (where k\begin{document}$k$\end{document} is large) to be modeled by the stationary distribution, we must pursue this more difficult course. We calculate the maximum likelihood estimates numerically, following Miller, 1995. See Appendix 3 for a full description of this method.

The dynamic scale-mixture model generalizes the two-dimensional, shared scale variable GSM model described above to time series, assuming the underlying Gaussian random variables, Y1\begin{document}$Y_{1}$\end{document} and Y2\begin{document}$Y_{2}$\end{document}, and the generator, Z\begin{document}$Z$\end{document}, of the scale variable are all AR(k\begin{document}$k$\end{document}) processes. Specifically, let X1,t=Y1,tSt\begin{document}$X_{1,t}=Y_{1,t}S_{t}$\end{document} and X2,t=Y2,tSt\begin{document}$X_{2,t}=Y_{2,t}S_{t}$\end{document}. Written as T\begin{document}$T$\end{document}-dimensional vectors, we have X1=Y1⊙S\begin{document}$\mathbf{X}_{1}=\mathbf{Y}_{1}\odot\mathbf{S}$\end{document} and X2=Y2⊙S\begin{document}$\mathbf{X}_{2}=\mathbf{Y}_{2}\odot\mathbf{S}$\end{document}, where ⊙\begin{document}$\odot$\end{document} is element-wise multiplication. The AR process assumptions implyp(y1)=N(y1;0,ΣY1)\begin{document}$$\displaystyle p(\mathbf{y}_1)=\mathcal{N}\left(\mathbf{y}_1;\mathbf{0},\mathbf{\Sigma}_{Y_1}\right)$$\end{document}p(y2)=N(y2;0,ΣY2)\begin{document}$$\displaystyle p(\mathbf{y}_2)=\mathcal{N}\left(\mathbf{y}_2;\mathbf{0},\mathbf{\Sigma}_{Y_2}\right)$$\end{document}p(z)=N(z;0,ΣZ)\begin{document}$$\displaystyle p(\mathbf{z})=\mathcal{N}\left(\mathbf{z};\mathbf{0},\mathbf{\Sigma}_Z\right)$$\end{document}

where the covariance matrices are determined by the parameters of independent AR(k\begin{document}$k$\end{document}) models as described above. S\begin{document}$\mathbf{S}$\end{document} is related to Z\begin{document}$\mathbf{Z}$\end{document} by element-wise application of the exponential function, S=exp⁡(Z)\begin{document}$\mathbf{S}=\exp(\mathbf{Z})$\end{document}. When Z\begin{document}$\mathbf{Z}$\end{document} is known, we havep(x1|z)=N(x1;0,DsΣY1Ds)\begin{document}$$\displaystyle p(\mathbf{x}_1|\mathbf{z})=\mathcal{N}\left(\mathbf{x}_1;\mathbf{0},\mathbf{D}_\mathbf{s}\mathbf{\Sigma}_{Y_1}\mathbf{D}_\mathbf{s}\right)$$\end{document}p(x2|z)=N(x2;0,DsΣY2Ds)\begin{document}$$\displaystyle p(\mathbf{x}_2|\mathbf{z})=\mathcal{N}\left(\mathbf{x}_2;\mathbf{0},\mathbf{D}_\mathbf{s}\mathbf{\Sigma}_{Y_2}\mathbf{D}_\mathbf{s}\right)$$\end{document}

where Ds\begin{document}$\mathbf{D}_{\mathbf{s}}$\end{document} is the matrix with the elements of s\begin{document}$\mathbf{s}$\end{document} along the diagonal and zeros elsewhere.

The classic EM algorithm is a useful tool for finding (local) maximum likelihood estimates of parameters of hidden variable models (Dempster et al., 1977). Let θ={ϕY1,ϕY2,ϕZ,σY12,σY22,σZ2}\begin{document}$\theta=\left\{\boldsymbol{\phi}_{Y_{1}},\boldsymbol{\phi}_{Y_{2}},\boldsymbol{ \phi}_{Z},\sigma_{Y_{1}}^{2},\sigma_{Y_{2}}^{2},\sigma_{Z}^{2}\right\}$\end{document} be the collection of parameters of the ARGSM model, where each ϕ\begin{document}$\boldsymbol{\phi}$\end{document} is the vector of AR coefficients and each σ2\begin{document}$\sigma^{2}$\end{document} is the driving noise variance for each variable. The observed data, D={x1,n,x2,n},1≤n≤N\begin{document}$\mathcal{D}=\{\mathbf{x}_{1,n},\mathbf{x}_{2,n}\},1\leq n\leq N$\end{document}, are the N\begin{document}$N$\end{document} pairs of T\begin{document}$T$\end{document}-dimensional vectors corresponding here to the horizontal and vertical velocity along each trajectory. The hidden variables, H={zn},1≤n≤N\begin{document}$\mathcal{H}=\{\mathbf{z}_{n}\},1\leq n\leq N$\end{document}, are the Gaussian generators of the time-varying scale associated with each trajectory. The likelihoodL=log⁡p(D|θ)=∑n=1Nlog⁡∫RTp(x1,n,x2,n|z,θ)p(z|θ)Dz\begin{document}$$\displaystyle \begin{array}{ll}L&=\log p(\mathcal{D}|\theta)\\ &=\sum_{n=1}^N\log\int_{\mathbb{R}^T}p(\mathbf{x}_{1,n},\mathbf{x}_{2,n}|\mathbf{z},\theta)p(\mathbf{z}|\theta)D\mathbf{z}\end{array}$$\end{document}

is intractable to maximize due to the high-dimensional integral. The EM algorithm finds a local maximum iteratively. Starting with an initial guess for the parameters, θ0\begin{document}$\theta_{0}$\end{document}, at each step, one computes the expectation with respect to the probability distribution of the hidden variables given the data and the current parameter estimate θt\begin{document}$\theta_{t}$\end{document} of the complete data log-likelihood,Q(θ|θt)=Ep(H|D,θt)[log⁡p(D,H|θ)],\begin{document}$$\displaystyle Q(\theta|\theta_t)=E_{p(\mathcal{H}|\mathcal{D},\theta_t)}\left[\log p(\mathcal{D},\mathcal{H}|\theta)\right]\,,$$\end{document}

then updates the parameters to maximize this function,θt+1=arg⁡maxθQ(θ|θt).\begin{document}$$\displaystyle \theta_{t+1}=\arg\max_\theta Q(\theta|\theta_t)\,.$$\end{document}

In our setting, we haveQ(θ|θt)=−12∑n=1NE[y1,n⊤ΣY1−1y1,n+y2,n⊤ΣY2−1y2,n+zn⊤ΣZ−1zn]+K=−12tr(CY1ΣY1−1+CY2ΣY2−1+CZΣZ−1)+K,\begin{document}$$\displaystyle \begin{array}{ll}Q(\theta|\theta_t)&=-\frac{1}{2}\sum_{n=1}^NE\left[\mathbf{y}_{1,n}^\top\mathbf{\Sigma}_{Y_1}^{-1}\mathbf{y}_{1,n}+\mathbf{y}_{2,n}^\top\mathbf{\Sigma}_{Y_2}^{-1}\mathbf{y}_{2,n}+\mathbf{z}_n^\top\mathbf{\Sigma}_Z^{-1}\mathbf{z}_n\right]+K \\ &=-\frac{1}{2}\mathrm{tr}\left(\mathbf{C}_{Y_1}\mathbf{\Sigma}_{Y_1}^{-1}+\mathbf{C}_{Y_2}\mathbf{\Sigma}_{Y_2}^{-1}+\mathbf{C}_Z\mathbf{\Sigma}_Z^{-1}\right)+K\,, \end{array}$$\end{document}

whereCZ=∑n=1NE[znzn⊤]\begin{document}$$\displaystyle \mathbf{C}_Z=\sum_{n=1}^NE\left[\mathbf{z}_n\mathbf{z}_n^\top\right]$$\end{document}

and similarly for CY1\begin{document}$\mathbf{C}_{Y_{1}}$\end{document} and CY2\begin{document}$\mathbf{C}_{Y_{2}}$\end{document}. Note that in this context, the y\begin{document}$\mathbf{y}$\end{document} variables are merely shorthand for y1,n=x1,n⊘sn\begin{document}$\mathbf{y}_{1,n}=\mathbf{x}_{1,n}\oslash\mathbf{s}_{n}$\end{document} and y2,n=x2,n⊘sn\begin{document}$\mathbf{y}_{2,n}=\mathbf{x}_{2,n}\oslash\mathbf{s}_{n}$\end{document}, where ⊘\begin{document}$\oslash$\end{document} is element-wise division. Given the C\begin{document}$\mathbf{C}$\end{document} matrices, the maximum likelihood estimates of the AR parameters can be calculated as described in Appendix 3.

Unfortunately, the expectation values in the C\begin{document}$\mathbf{C}$\end{document} matrices are also intractable, but they can be approximated through sampling methods. A variant of the EM algorithm, called stochastic approximation EM, was developed to address this problem (Delyon et al., 1999). Given a sample from the distribution p(H|D,θt)\begin{document}$p(\mathcal{H}|\mathcal{D},\theta_{t})$\end{document}, one calculates the sample matrices C^\begin{document}$\hat{\mathbf{C}}$\end{document}, then updates the stochastic approximations asCt=Ct−1+ηt(C^t−Ct−1).\begin{document}$$\displaystyle \mathbf{C}_t=\mathbf{C}_{t-1}+\eta_t\left(\hat{\mathbf{C}}_t-\mathbf{C}_{t-1}\right)\,.$$\end{document}

The sequence of parameters ηt\begin{document}$\eta_{t}$\end{document} is given byηt={11≤t≤α(t−α)−βt>α.\begin{document}$$\displaystyle \eta_t=\begin{cases}1&1\leq t\leq\alpha \\(t-\alpha)^{-\beta}&t \gt \alpha\end{cases}\,.$$\end{document}

We choose α = 2500 or 5000, so that the algorithm runs in a fully stochastic mode until the parameter estimates are nearly stationary, and β=1\begin{document}$\beta=1$\end{document}, so that after this initial period, the algorithm converges by simply taking a running average of the samples of the C^\begin{document}$\hat{\mathbf{C}}$\end{document} matrices. Importantly, the samples do not need to be independent across iterations for the algorithm to converge (Kuhn and Lavielle, 2004). This means that, when performing the Gibbs sampling described below, we only need to update each hidden variable element once for each iteration, rather than updating many times and throwing out samples to achieve independence. Since each M-step (the AR model MLE algorithm described above) is much faster than each E-step (calculating the C\begin{document}$\mathbf{C}$\end{document} matrices through sampling), this results in a more sample-efficient algorithm (Neal and Hinton, 1998).

We also estimate the expectation of the hidden variables {zn}\begin{document}$\{\mathbf{z}_{n}\}$\end{document} in an identical fashion. This is equivalent to a Bayesian point estimate where the estimated parameters form a forward model and prior. These estimates are then used to remove the scale from the velocity, in order to examine the kurtosis under different model assumptions (Figure 4).

The EM algorithm, and its stochastic approximation variant, converges to a local maximum of the likelihood function that depends on the initial conditions. We find that, in practice, it is important to introduce the scale variable gradually to the model. We initialize the model with AR parameters fit to the raw data for the Y1\begin{document}$Y_{1}$\end{document} and Y2\begin{document}$Y_{2}$\end{document} components, and let Z\begin{document}$Z$\end{document} be uncorrelated with very small variance (regression coefficients ϕZ=0\begin{document}$\boldsymbol{\phi}_{Z}=\mathbf{0}$\end{document} and driving noise variance σZ2=0.052(γY1,0+γY2,0)/2\begin{document}$\sigma_{Z}^{2}=0.05^{2}(\gamma_{Y_{1},0}+\gamma_{Y_{2},0})/2$\end{document}).

We use a combination of Gibbs and rejection sampling to sample from the posterior of the hidden variables given the data and the current parameter estimates (Bishop, 2006). In Gibbs sampling, an initial vector z\begin{document}$\mathbf{z}$\end{document} is used to generate a new sample by sampling each element individually, conditioned on the remaining elements. Since the conditional distribution is intractable, we use rejection sampling, which allows us to sample from an arbitrary, unnormalized distribution by sampling from a proposal distribution (in this case a Gaussian with parameters chosen to envelope the conditional distribution) and rejecting some draws in order to shape it into the target distribution. See Appendix 3 for a detailed description of the sampling algorithm.