We analyzed multiple datasets from two visual cortical regions (MT and IT cortex). All datasets were collected in awake, behaving macaques (MT: Macaca mulatta; IT: Macaca mulatta and Macaca fascicularis). The data were originally collected at the University of Pittsburgh (Ruff and Cohen, 2016b; Ruff et al., 2016a), Rockefeller University (Ebihara, 2015), and the German Primate Center in Goettingen (Li et al., 2016). All procedures were conducted in accordance with applicable US, German, and European guidelines or regulations and were approved by the relevant compliance committees at the relevant institutions (US guidelines: National Institutes of Health (NIH) Pub. No. 86-23, Revised 1985; Pittsburgh MT grating datasets: Institutional Animal Care and Use Committees of the University of Pittsburgh and Carnegie Mellon University (Protocol #: 20067560, PHS Assurance Number: D16-00118); IT datasets: The Rockefeller University Institutional Animal Care and Use Committee (Protocol #: 21,104 H USDA, PHS Assurance Number: A3081-01); Goettingen MT dot patch dataset: [Niedersächsisches Landesamtfür Verbraucherschutzund Lebensmittelsicherheit (LAVES)] permit numbers 33.42502/08-07.02 and 33.14.42502-04-064/07). All surgical procedures were conducted aseptically and with suitable anesthetics and analgesics.

Our goal was to assess fluctuating neural activity in multiple visual areas and in response to multiple stimuli (Table 1). In all datasets, fixation was controlled while the stimuli were presented (0.5–1.2 degree fixation window radius, depending on the dataset). In some datasets, monkeys performed a behavioral task in which attention was directed away from the stimuli, such as to the fixation position itself or to an additional stimulus in the ipsilateral hemifield. In the fixation-only tasks, it should be noted that the absence of experimental control over attention could have led to the monkeys covertly attending the receptive field stimuli from time to time.

For area MT, we used three datasets. The first dataset included single- and multi-unit responses to drifting grating stimuli positioned adjacent to one another (Figure 1a.1 and 2; Ruff and Cohen, 2016b). The experiment involved simultaneous recordings from V1, V4, and MT. As we have previously identified fluctuating response patterns in V1 and V4 (Jun et al., 2022), this MT dataset provides an opportunity to compare the presence of fluctuating activity across visual areas under the same experimental conditions (and some of the same recording sessions) as in our previous work. However, this dataset was comparatively small (as detailed further below), making additional data desirable.

Accordingly, we included a second dataset from a previously published study involving single- and multi-unit responses to adjacent random dot stimuli (Figure 1b; Li et al., 2016). Both of these datasets were recorded while monkeys performed a behavioral task in which attention was directed away from the stimuli in the MT receptive field: toward the opposite hemifield (Ruff and Cohen, 2016b) or to the fixation point (Li et al., 2016). Fixation was maintained during stimulus presentation to within 0.5 degrees of the fixation point for the Ruff and Cohen dataset and 1.2 degrees for the Li et al. dataset.

In both of these MT datasets, the stimuli can be considered as distinct and distinguishable objects from each other, regardless of whether they are gratings or patches of moving dots, because they are presented at separate locations. Our third MT dataset provides a comparison to these first two. In this dataset, individual gratings were presented at the same location in space, such that when they were presented together, they fused to form a plaid stimulus (Figure 1a.1 and 3). This dataset was originally published in Ruff et al., 2016a and involved simultaneous V1 and V4 recordings previously analyzed for fluctuating activity in Jun et al., 2022. In this dataset, monkeys performed a simple fixation task.

The specific stimulus details for these MT datasets are as follows. In the adjacent gratings dataset, the grating sizes were selected to lie within the MT receptive field under study and did not exceed 1 degree in diameter. The gratings used in the superimposed dataset were larger, ranging between 7 and 13 degrees in diameter, and covered the MT receptive field. The sizes of the random dot patches are not available, but they were depicted at adjacent positions within the MT receptive field (Li et al., 2016).

For the IT cortex, we included datasets involving single-unit recordings in two areas of the face patch system, areas MF and AL, during a fixation task (Figure 1c; fixation maintained within 1 degree) (Ebihara, 2015). We have previously reported fluctuating activity in face patch MF (Caruso et al., 2018), but here we have re-analyzed the MF dataset (see ‘Analysis: response periods and inclusion criteria’) to provide detailed results for different types of stimuli (face–face vs. face–object pairs), allowing a comparison of results in the two face patches (MF and AL) in the same experimental conditions.

For both MF and AL, we assessed neural responses to image pairs drawn from three visual stimuli selected from a pool of faces and objects: a ‘preferred face’ (strong neural response), a ‘non-preferred face’ (little to no neural response), and a ‘non-preferred object’ (little to no neural response). These stimuli were randomly presented either alone or in pairs, with each pair including the preferred face, positioned at the center of the neuron’s receptive field, and one of the non-preferred stimuli, presented adjacent to the preferred face at one of eight equidistant locations. Stimuli were 4 × 4 degrees in size.

As in Caruso et al., 2018, we did not incorporate the exact location of the non-preferred face/object into the present analysis, but any excessive heterogeneity in the responses of the non-preferred stimuli due to location would likely result in exclusion of the condition from further analysis (see ‘Analysis: response periods and inclusion criteria’).

Our analysis of fluctuating activity patterns relies on comparing the activity evoked on trials in which two stimuli, A and B, are presented simultaneously (AB) to trials in which only the A or only the B stimulus was presented. We refer to the corresponding A-alone, B-alone, and combined AB conditions as ‘triplets’.

To be included for analysis, triplets were required to meet several criteria. First, the minimum number of included trials for each of the A, B, and AB components was set at five trials for each, in keeping with our prior work and assessments of the sensitivity of the analysis to the number of trials involved (Caruso et al., 2018; Mohl et al., 2020; Jun et al., 2022). The average numbers of trials of the included triplets across datasets are illustrated in Figure 1—figure supplement 1. The two MT datasets involving adjacent stimuli had the lowest average trial counts (average of the A, B, and AB trial counts was ~18 trials for the gratings and ~6 trials for the dot patches), whereas the MT ‘plaid’ dataset had the most (average ~37 trials; all values computed after imposition of the full set of exclusion criteria, including those described below).

Second, spike count distributions for the A- and B-alone trials in a triplet were required to be sufficiently well described by a Poisson distribution (with a variance roughly equal to the mean). The spike count distributions for A and B trials for triplets in all five datasets exhibited variance-to-mean ratios (Fano factors) with a mode of about 1, but there was some spread around that value (Figure 1—figure supplement 1B). Accordingly, we discarded triplets for which the average variance-to-mean ratio in the A and B conditions was greater than 3. This value was chosen to exclude the worst outliers while preserving adequate data across all datasets to allow for appropriate comparisons. Screening with a more stringent criterion (variance-to-mean ratio <2) reduced the overall quantity of data but did not change the key results we report here (Figure 4—figure supplement 1). Notably, screening with the variance-to-mean ratio is a deviation from our previous approach involving an approximate chi-square goodness of fit test with Monte Carlo p-value calculation (Caruso et al., 2018; Jun et al., 2022). The former approach had the undesired effect of being more likely to exclude conditions with larger amounts of data, whereas the present method is neutral to dataset size.

Finally, spike count distributions for the individual A and B stimuli in a triplet were required to be sufficiently separable from one another. This is necessary so that it is possible to ascertain with some confidence how the AB spike count distributions compare to each of the corresponding single-stimulus A and B distributions. We set the inclusion criterion as a 95% likelihood that the A and B spike counts were drawn from different Poisson distributions (specifically, the logarithm of the intrinsic Bayes factor was required to be greater than 3, given an a priori 50–50 chance that the two distribution means are the same vs. different). The greater numbers of trials in some datasets than in others allowed for inclusion of triplets at smaller differences in the A vs. B response distributions (Figure 1—figure supplement 1), but this did not appear to impact the overall conclusions (see Figure 1—figure supplement 1 and associated legend).

A full description of our statistical method has been published previously (Caruso et al., 2018; Mohl et al., 2020; Jun et al., 2022). Our method centers on modeling spike counts based on Poisson distributions, a common technique for handling non-negative count data in neuroscience and other fields.

Briefly, we compare the distribution of spike counts in response to a stimulus pair (AB) to the distributions of spike counts in response to the component stimuli presented individually (A and B). We consider four possibilities for the relationship between the AB response and the individual A and B responses (Figure 2). The first three attempt to fit the AB responses with a single Poisson distribution, as follows:

We first evaluated the responses in visual area MT to stimuli that appeared as distinct objects. The first dataset involved adjacent drifting gratings, both of which were positioned within the receptive field of the MT neuron under study. These data were recorded simultaneously with recordings in V1, in which we have previously reported the presence of fluctuating activity in response to these stimuli (Jun et al., 2022). On the trials included for analysis, monkeys performed a task in which attention was directed elsewhere.

As in V1, we found that some units exhibited bimodal AB spike count distributions whose modes approximately match the individual A and B means, suggesting that the AB responses fluctuate between the A and B response rates across trials. An example of such a unit is shown in Figure 3a: the red and blue dashed lines indicate the means of the A- and B-alone response distributions, and the black curve provides a smoothed depiction of the distribution of spike counts across the combined AB trials. This distribution appears to reflect an admixture of the A- and B-alone response distributions, indicating that the unit fluctuates between each single response rate from trial to trial.

We next considered a second independent dataset from MT (Li et al., 2016) that shares similar properties to the adjacent gratings dataset. In this study, two separate patches of dots, each moving in different directions, were presented within the receptive field of the neurons under study. For the included trials, the monkey was performing a task in which attention was directed to a fixation point away from the dot patches. The combined stimulus AB trials in this dataset were previously reported to show fluctuating activity patterns using a different analysis method (Li et al., 2016), and the dataset was made publicly available at Li et al., 2016. As in the adjacent gratings MT dataset, we observed bimodal spike count distributions on combined AB trials with modes that appeared to match the mean responses observed on A- and B-alone trials (Figure 3b).

To evaluate the prevalence of such fluctuating activity patterns at the population level, we deployed the statistical comparison described in Materials and methods, in which spike count distributions on combined AB trials were evaluated in comparison to four possible options: ‘mixture’, ‘intermediate’, ‘single’, and ‘outside’ (Figure 2). Figure 3c shows the results of this analysis for the adjacent gratings and dot patches datasets. In these two datasets, we only identified ‘mixtures’ and ‘intermediates’, but no ‘singles’ or ‘outsides’. There was little difference between these two datasets (light blue vs. dark blue stacked bars in Figure 3c). Overall in these two datasets, ‘mixtures’ constituted about 45% of the categorized conditions (Figure 3e, bar labeled ‘distinct objects’).

In contrast, evidence of fluctuating activity was much lower in our third MT dataset, which involved gratings that were either presented individually (A- and B-alone) or superimposed to produce a fused percept of a ‘plaid’ or checkerboard-like pattern. As in the first dataset, the stimuli were presented in the receptive field of the MT neuron under study, but this time they were presented at the same exact location, rather than adjacent to one another. The monkeys were performing a fixation task; no explicit attentional report was required. In this dataset, ‘mixture’ responses were very rare, constituting only about 1.4% of the conditions that were well categorized (Figure 3d, e). The majority of conditions were categorized as ‘outside’ (~78%), with ‘single’ and ‘intermediate’ accounting for a further ~14% and ~6%, respectively.

The difference in the proportions of ‘mixture’ response patterns in the first two MT datasets involving distinct objects (~45%) vs. this last dataset in which the two stimuli fuse to form a new third object (~1.4%) was statistically significant (Figure 3e, blue vs. green bars, chi-squared test p < 10⁻⁹). This finding is consistent with the interpretation that the prevalence of fluctuating activity is related to whether two stimuli are perceptually distinguishable vs. fuse to form a single object.

To further explore the generality of these fluctuating activity patterns, we next turned to a higher order area of the visual pathway – the face patch system of IT cortex. Face patches are exquisitely sensitive to the identity of individual faces, and respond preferentially to faces as compared to other objects. This affords the opportunity to test whether fluctuating activity is evoked solely when the two stimuli are both faces, as might be expected because the non-face objects are largely thought to be encoded by neural populations in other regions of IT, or if both face–face and face–object pairs evoke fluctuating activity. We previously identified fluctuating activity patterns in the MF face patch (Caruso et al., 2018), but we did not separate face–face and face–object stimulus conditions. Here, we revisited that dataset to conduct that comparison. We also sought to compare the results in MF to those in face patch AL. Recent work has suggested that MF is ‘earlier’ in the face patch hierarchy than AL, and that AL shows properties such as view-invariance that are less evident in MF (Moeller et al., 2008; Freiwald and Tsao, 2010; Ebihara, 2015). Thus, any differences between the results in MF and AL could shed light on how fluctuating activity varies within the processing hierarchy at this late stage of visual processing.

Accordingly, we assessed MF and AL face patch responses with both face–face and face–object stimulus combinations. In face–face combinations, one face evoked the strongest response from among a stimulus set (preferred face), and the other face evoked the weakest response from the same set. The preferred face was presented at the center of the neuron’s receptive field, while the non-preferred face was presented at a different location (Figure 1c). Similarly, the object stimuli were also chosen to evoke weak responses and were presented at a different location than the preferred face. The stimulus conditions were randomly interleaved, and the data were collected while monkeys performed simple fixation tasks.

Both face patch regions and both types of stimulus combinations elicited fluctuating activity. Figure 4a–d shows the responses of four example neurons, each of which exhibited spike counts that fluctuated across trials between appearing to respond to the preferred vs. non-preferred face (Figure 4a, c) or to the face vs. the object (Figure 4b, d).

To evaluate these results at the population level, we used the same statistical comparison described above for the MT analysis. As shown in Figure 4e, we identified ‘mixtures’ in face patch MF for both face–face (dark brown bar) and face–object (mustard-colored bar) stimulus combinations. We also observed ‘intermediates’ and ‘singles’, but no ‘outsides’. Overall, ‘mixtures’ constituted about 32.5% of well-categorized conditions (Figure 4g, light brown bar). The proportion of ‘mixtures’ was higher for face–face stimulus combinations (43.6%, Figure 4g, dark brown bar) than for face–object stimulus combinations (22.7%, Figure 4g, mustard-colored bar). The difference in these proportions was just barely significant by chi-squared test (p = 0.043), but we note that when using stricter Fano factor inclusion criteria for the dataset (variance-to-mean ratio of single-stimulus responses less than 2 rather than 3, see Materials and methods: ‘Triplet’ definition and inclusion criteria), the reduction in the amount of included data increases this p value to p = 0.054 (see Figure 4—figure supplement 1). Accordingly, this aspect of the results should be viewed as of borderline significance.

‘Mixtures’ were even more prevalent in face patch AL (Figure 4f), constituting about 66% of the conditions for both face–face and face–object pairs (Figure 4g). The approximate doubling in the prevalence of ‘mixtures’ in AL vs. MF (65.9 vs. 32.5%) was statistically significant by chi-squared test (p = 0.00002). Like face patch MF, AL responses had no ‘outsides’, but ‘singles’ and ‘intermediates’ were still present. In MF, we previously showed that such ‘intermediates’ could show fluctuating activity within the timescale of individual trials (Caruso et al., 2018).