elifeeLifeeLife2050-084XeLife Sciences Publications, Ltd8983610.7554/eLife.8983610.7554/eLife.89836.3version of recordResearch ArticleNeuroscienceAttention modulates human visual responses to objects by tuning sharpeningDoostaniNargeshttps://orcid.org/0000-0001-5775-6595narges.doostani.d@gmail.com12Hossein-ZadehGholam-Ali13CichyRadoslaw Mhttps://orcid.org/0000-0003-4190-60712Vaziri-PashkamMaryamhttps://orcid.org/0000-0003-1830-2501mvaziri@udel.edu45https://ror.org/04xreqs31School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM)TehranIranhttps://ror.org/046ak2485Department of Education and Psychology, Freie Universität BerlinBerlinGermanyhttps://ror.org/05vf56z40School of Electrical and Computer Engineering, College of Engineering, University of TehranTehranIranhttps://ror.org/01sbq1a82Department of Psychological and Brain Sciences, University of DelawareNewarkUnited Stateshttps://ror.org/04xeg9z08Laboratory of Brain and Cognition, National Institute of Mental HealthBethesdaUnited StatesPeelenMarius VReviewing Editorhttps://ror.org/016xsfp80Radboud University NijmegenNetherlandsMooreTirinSenior Editorhttps://ror.org/006w34k90Stanford University, Howard Hughes Medical InstituteUnited States1612202412RP8983622062023This manuscript was published as a preprint.05062023This manuscript was published as a reviewed preprint.12092023The reviewed preprint was revised.14112024http://creativecommons.org/publicdomain/zero/1.0/This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Visual stimuli compete with each other for cortical processing and attention biases this competition in favor of the attended stimulus. How does the relationship between the stimuli affect the strength of this attentional bias? Here, we used functional MRI to explore the effect of target-distractor similarity in neural representation on attentional modulation in the human visual cortex using univariate and multivariate pattern analyses. Using stimuli from four object categories (human bodies, cats, cars, and houses), we investigated attentional effects in the primary visual area V1, the object-selective regions LO and pFs, the body-selective region EBA, and the scene-selective region PPA. We demonstrated that the strength of the attentional bias toward the target is not fixed but decreases with increasing target-distractor similarity. Simulations provided evidence that this result pattern is explained by tuning sharpening rather than an increase in gain. Our findings provide a mechanistic explanation for the behavioral effects of target-distractor similarity on attentional biases and suggest tuning sharpening as the underlying mechanism in object-based attention.
object-based attentiontuning sharpeningtarget-distractor similarityfMRIMVPAResearch organismHumanhttp://dx.doi.org/10.13039/100000002National Institutes of HealthZIA-MH002035Vaziri-PashkamMaryamThe funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.Author impact statementAttentional modulation is affected by target-distractor similarity, indicating tuning sharpening as the underlying mechanism for response enhancement during object-based attention.publishing-routeprcIntroduction
Everyday visual scenes typically contain a large number of stimuli. Since processing all the incoming information is impossible due to the brain’s limited neural resources, different stimuli compete for cortical representation and processing (Desimone and Duncan, 1995; Kastner et al., 1998; Reynolds et al., 1999; Beck and Kastner, 2005; Reddy et al., 2009; McMains and Kastner, 2011). This competition can be biased by the top-down signal of attention to enhance the parts of the input that are most relevant to the task at hand (Moran and Desimone, 1985; Desimone and Duncan, 1995; Reynolds et al., 1999). Evidence from electrophysiology and fMRI studies have demonstrated the role of attention in biasing the competition by enhancing the response related to the attended stimulus by approximately 30% compared to its response when unattended, in both electrophysiology studies of the monkey brain (Treue and Maunsell, 1996; Reynolds et al., 1999; Treue and Maunsell, 1999; Reynolds and Desimone, 2003; Fallah et al., 2007) and fMRI studies of the human brain (Reddy et al., 2009).
Competition and attentional bias likely depend on the nature of the visual scenes rather than being universally uniform. Behavioral studies indicate that the competition between stimuli is content-dependent (Cohen et al., 2014), with higher competition between stimuli that are located closer to each other (Franconeri et al., 2013), or between stimuli with more similar cortical representation patterns (Cohen et al., 2014; Cohen et al., 2017). This suggests that the attentional bias might also be affected by the relationship between the competing stimuli, such as the similarity of their cortical representation. Furthermore, behavioral studies on the effect of target-distractor similarity on performance have proposed that lower performance for more similar target-distractor pairs is due to the fact that the neural resources needed for detailed processing are shared to a greater extent (Cohen et al., 2014). However, a direct neuroscientific investigation of how target-distractor similarity affects visual representations, and a mechanistic explanation of how shared resources affect attentional biases are missing.
Here, we investigated the impact of similarity in cortical representation on attentional bias and the underlying mechanism with empirical and theoretical tools. First, using functional MRI and uni- as well as multivariate analysis, we investigated how the top-down effect of attention varies as target-distractor similarity changes for multiple presented objects. Specifically, we found that the strength of the attentional bias towards the target decreases with increasing target-distractor similarity in cortical representation.
Second, using simulations of neuronal populations we determined how this effect arises from attentional enhancement of neural responses. We considered two known mechanisms through which attention affects neural firing rate: response gain and tuning sharpening. The response gain model predicts a multiplicative scaling of responses through which neural responses are increased by a gain factor (McAdams and Maunsell, 1999; Reynolds and Chelazzi, 2004). The tuning sharpening model, instead, proposes that attentional enhancement depends on the neuronal tuning for the attended stimulus, leading to an increase in response for optimal stimuli, and little increase in response or even response suppression for non-optimal stimuli (Martinez-Trujillo and Treue, 2004). We find that the empirically-observed relationship between attentional enhancement and target-distractor similarity are predicted by the tuning sharpening model, but not the response gain model.
Together, our results show that attentional enhancement is dependent on the similarity between the target and the distractor in neural representation, and a more similar distractor causes the target to receive less attentional bias in the competition. Moreover, these results suggest tuning sharpening as the underlying mechanism of attentional enhancement during object-based attention.
Materials and methodsMain experimentParticipants
17 healthy human participants (nine females, age: mean ± s.d.=29.29 ± 4.5 y) with normal or corrected-to-normal vision took part in the study. We estimated the number of participants conservatively based on the smallest amount of attentional modulation observed in our previous study (Doostani et al., 2023). For a medium effect size of 0.3 and a power of 0.8, we needed a minimum number of 16 participants. Participants gave written consent and received payment for their participation in the experiment. Data collection was approved by the Ethics Committee of the Institute for Research in Fundamental Sciences, Tehran.
The behavioral data for two participants was not correctly saved during the scanning due to technical problems. While we used the fMRI data of these two participants, all behavioral reports include the performance of the 15 participants for whom the behavioral data was properly saved.
Stimulus set and experimental design
To determine the effect of target-distractor similarity on attentional modulation, we used object stimuli from four categories (human bodies, cars, houses, and cats). We included body and house categories because there are regions in the brain that are highly responsive and unresponsive to each of these categories, which provided us with a range of responsiveness in the visual cortex. We chose the two remaining categories based on previous behavioral results to include categories that provided us with a range of similarities (Xu and Vaziri-Pashkam, 2019). Thus, for each category there was a range of responsiveness in the brain and a range of similarity with the other categories.
We presented stimuli from each category in semi-transparent form, either in isolation (isolated conditions), or paired with stimuli from another category (paired conditions). Thus, the experiment consisted of 16 conditions: 4 isolated conditions in which isolated stimuli from one of the four categories were presented, and 12 paired conditions (six category pairs × two attentional targets for each pair) in which a target stimulus from the cued category was superimposed with a distractor stimulus from another category for all category combinations. Figure 1B depicts all stimulus conditions. We used isolated conditions to assess the similarity between different categories, and paired conditions to determine the effect of similarity in a category pair on attentional modulation.
Stimuli, paradigm, and regions of interest.
(A) Top images represent the four categories used in the main experiment: body, car, house, and cat. The stimulus set consisted of 10 exemplars from each category (here: cats), with exemplars differing in pose and 3D-orientation. (B) The experimental design comprised 16 task conditions (12 paired, four isolated). The 4×4 matrix on the left illustrates the 12 paired conditions, with the to-be-attended category (outlined in orange for illustration purposes, not present in the experiment) on the y-axis and the to-be-ignored category on the x-axis. The right column illustrates the four isolated conditions. (C) Experimental paradigm. A paired block is depicted with superimposed body and house stimuli. In this example block, house stimuli were cued as targets, and the participant responded on the repetition of the exact same house in two consecutive trials, as marked here by the arrow. (D) Regions of interest for an example participant; the primary visual cortex (V1), the object-selective regions lateral occipital cortex (LO) and posterior fusiform gyrus (pFs), the body-selective region extrastriate body area (EBA), and the scene-selective region parahippocampal place area (PPA).
Behavioral performance of the participants during the recording.
B, Cr, H, and Ct represent the Body, Car, House, and Cat categories, respectively. Each x-axis label represents the two conditions with stimuli from the same two categories but with different attentional targets, with the dark gray bar illustrating the results related to the condition in which the first category was attended, and the light gray bar representing the results in the condition in which the second category was attended. For instance, the B-Cr bars represent the results related to the BodyatCar and BodyCarat conditions, respectively. Error bars represent standard errors of the mean. N=15 human participants. (A) Average detection rate for each category pair. (B) Average reaction time for each category pair.
The stimulus set consisted of gray-scaled images from the four object categories of human bodies, cats, cars, and houses, similar to stimuli used in previous studies (Vaziri-Pashkam and Xu, 2017; Vaziri-Pashkam and Xu, 2019; Xu and Vaziri-Pashkam, 2019). Each category consisted of 10 exemplars all varying in identity, 3D-orientation (for houses and cars), and pose (for bodies and cats, see Figure 1A).
Images were presented at the center of the screen on a gray background, subtending 10.2o of visual angle. A red fixation point subtending 0.45o of visual angle was presented at the center of the screen throughout the run (Figure 1C).
Procedure
We used a blocked design for the main experiment. At the beginning of each block, participants were cued by a word to attend to either bodies, cars, houses, or cats. During the block, participants maintained attention on the images from the cued category, and performed a one-back repetition detection task on them by pressing the response button when the same stimulus from the attended category appeared in two consecutive trials. Repetition occurred two to three times at random times in each block. The experiment consisted of 16 block types, corresponding to the 16 task conditions (Figure 1B).
Each block lasted for 10 s, starting with the cue word presented for 1 s, followed by 1 s of fixation. Then, ten images from the cued category were presented in isolation or paired with ten images from another category. Each image was presented for 400 ms, followed by 400ms of fixation (Figure 1C). There were 8 s of fixation in between the blocks, and a final 8 s fixation after the last block.
We organized blocks in runs, each lasting 4 min 56 s. Each run started with 8 s of fixation followed by block presentations. The presentation order of the 16 task conditions was counterbalanced across each experimental run. 10 participants completed 16 runs and 7 participants completed 12 runs of the main experiment.
Localizer experiments
Considering that we used object categories, we investigated five different regions of interest (ROIs): the object-selective areas lateral occipital cortex (LO) and posterior fusiform gyrus (pFs) as general object-selective regions, the body-selective extrastriate body area (EBA) and the scene-selective parahippocampal place area (PPA) as regions that are highly selective for specific categories, and the primary visual cortex (V1) as a control region. We chose these regions because they could all be consistently defined in both hemispheres of all participants and included a large number of voxels. To define these ROIs, each participant completed four localizer runs described in detail below.
Early visual area localizer
We used meridian mapping to localize the primary visual cortex V1. Participants viewed a black-and-white checkerboard pattern through a 60-degree polar angle wedge aperture. The wedge was presented either horizontally or vertically. Participants were asked to detect luminance changes in the wedge in a blocked-design paradigm. Each run consisted of four horizontal and four vertical blocks, each lasting 16 s, with 16 s of fixation in between. A final 16 s fixation followed the last block. Each run lasted 272 s. The order of the blocks was counterbalanced within each run. Participants completed two runs of this localizer.
Category localizer
We used a category localizer to localize the cortical regions selective to scenes (PPA), bodies (EBA), and objects (LO, pFs). In a blocked-design paradigm, participants viewed stimuli from the five categories of faces, scenes, objects, bodies, and scrambled images. The stimuli differed from those used in the main experiment. Each localizer run contained two 16 s blocks of each category, with the presentation order counterbalanced within each run. An 8 s fixation period was presented at the beginning, in the middle, and at the end of the run. In each block, 20 stimuli from the same category were presented. Stimuli were presented for 750 ms followed by 50 ms of fixation on a gray background screen. Participants were asked to maintain their fixation on a red circle at the center of the screen throughout and press a key when they detected a slight jitter in the stimuli that happened two to three times per block. Each run lasted 344 s. Participants completed two runs of this localizer.
Stimulus presentation inside the scanner
We back-projected the stimuli onto a screen positioned at the rear of the magnet using an LCD projector with a refresh rate of 60 Hz and a spatial resolution of 768 × 1024. Participants observed the screen through a mirror attached to the head coil.
MRI data acquisition
We recorded the data of 10 participants using the Siemens 3T Tim Trio MRI system with a 32-channel head coil at the Institute for Research in Fundamental Sciences (IPM). We collected the data of seven additional participants on a Siemens Prisma MRI system using a 64-channel head coil at the National Brain-mapping Laboratory (NBML). For each participant, we performed a whole-brain anatomical scan using a T1-weighted MPRAGE sequence. For the functional scans, including the main experiment and the localizer experiments, we acquired 33 slices parallel to the AC-PC line using T2*-weighted gradient-echo echo-planar imaging (EPI) sequences covering the whole brain (TR=2 s, TE=30 ms, flip angle=90o, voxel size=3 × 3 × 3 mm3, matrix size = 64 × 64).
fMRI data preprocessing
We performed fMRI data analysis using FreeSurfer (https://surfer.nmr.mgh.harvard.edu), Freesurfer Functional Analysis Stream (Dale et al., 1999) and in-house MATLAB codes. fMRI data preprocessing steps included 3D motion correction, slice timing correction, and linear and quadratic trend removal. We performed no spatial smoothing on the data. We used a double gamma function to model the hemodynamic response function. We eliminated the first four volumes (8 s) of each run to avoid the initial magnetization transient. We compared the SNR values of the two groups of participants and observed no significant difference between these values (ps>0.34,ts<0.97).
fMRI data analysis
For the main experiment, we performed a general linear model (GLM) analysis for each participant to estimate voxel-wise regression coefficients in each of the 16 task conditions. The onset and duration of each block were convolved with a hemodynamic response function and were then entered into the GLM as regressors. We also included movement parameters and linear and quadratic nuisance regressors in the GLM. We did not enter the cue to the GLM as a predictor. The obtained voxel-wise coefficients for each condition are thus related to the cue and the stimuli presented in that condition. We used these voxel-wise coefficients from the five ROIs as the basis for all further analyses.
For the early visual area localizer experiment, we estimated voxel regression coefficients in each of the two conditions (i.e. vertical and horizontal wedge) using a separate GLM. After convolving with a hemodynamic response function, the onset and duration of each block were entered to the GLM as regressors of interest. We also included movement parameters and linear and quadratic nuisance regressors in the GLM. We used the obtained coefficients to define the V1 ROI.
For the category localizer, we used another GLM to estimate voxel-wise regression coefficients in the five task conditions (i.e. faces, scenes, objects, bodies, and scrambled images). The GLM procedure was similar to the other two experiments. We then used these estimated coefficients to define the LO, pFs, EBA, and PPA ROIs.
Definition of ROIs
We determined the V1 ROI using a contrast of horizontal versus vertical polar angle wedges that reveals the topographic maps in the occipital cortex (Sereno et al., 1995; Tootell et al., 1998). To define the object-selective areas LO in the lateral occipital cortex and pFs in the posterior fusiform gyrus (Malach et al., 1995; Grill-Spector et al., 1998), we used a contrast of objects versus scrambled images. We selected the active voxels in the lateral occipital and ventral occipitotemporal cortex as LO and pFS, respectively, following the procedure described by Kourtzi and Kanwisher, 2000. We used a contrast of scenes versus objects for defining the scene-selective area PPA in the parahippocampal gyrus (Epstein et al., 1999), and a contrast of bodies versus objects for defining the body-selective area EBA in the lateral occipitotemporal cortex (Downing et al., 2001). We thresholded the activation maps for both the early visual localizer and the category localizer at p<0.001, uncorrected. We selected this threshold to allow for the selection of a reasonable number of voxels in each hemisphere across all participants.
Univariate fMRI analysis
We first used a univariate analysis to determine the effect of attention for different category pairs. Using the voxel-wise coefficients of the isolated conditions associated with each category, we examined the relative response of each voxel to the two categories for each category pair. This relative response determined which of the two categories was more preferred by the voxel. Therefore, for each category pair and each voxel, the category that elicited a higher response in the isolated condition was assigned the relatively more preferred category (M) label and the other the relatively less preferred category (L) label.
Univariate distance based on the isolated conditions
We had six pairs of categories: Body-Car, Body-House, Body-Cat, Car-House, Car-Cat, and House-Cat. As a measure of the difference between the response evoked by each of the two categories in a pair, we defined a univariate distance. We calculated the univariate distance for each pair of categories simply as the difference in voxel responses of the two isolated conditions (Equation 1):Univariatedistance=RMat−RLat
Here, R denotes the average voxel response across runs, and the subscripts M and L denote the presence of the more preferred and the less preferred stimuli, respectively. The superscript at denotes the attended stimulus. Note that in the isolated conditions, the presented stimulus was always attended. Thus, RMat is the average response related to the isolated preferred stimulus, and RLat is the average response to the isolated less preferred stimulus. For example, the Body-Car univariate distance was assessed by RBodyat−RCarat for voxels more responsive to bodies than cars, and by RCarat−RBodyat for voxels more responsive to cars than bodies. Thus, according to this measure, two categories that elicited closer responses had less univariate distance, indicating more similarity in univariate response between the two categories.
Univariate effect of attention based on the paired conditions (Univariate shift)
For each of the six category pairs, we had two paired conditions, in which stimuli from both categories were presented, but with attention directed to either one or the other category (for example, BodyatCar and BodyCarat conditions for the Body-Car pair, with the superscript at denoting the to-be-attended stimulus). Since these paired conditions differed only in the attentional target and not in the stimuli, any difference observed in cortical response can be ascribed to the shift in attention (O’Craven et al., 1999; Ni et al., 2012; Vaziri-Pashkam and Xu, 2017; Doostani et al., 2023). We thus defined the univariate shift for each pair of categories as the change in response when attention shifted from the more preferred stimulus to the other:Univariateshift=RMatL−RMLat
Here, RMatL denotes the response related to the paired condition with attention directed to the more preferred category, while RMLat is the elicited response when attending to the less preferred category in the pair. For example, considering the Body-Car pair, we assessed the univariate shift by RBodyatCar−RBodyCarat for a voxel preferring bodies to cars, and by RBodyCarat−RBodyatCar for a voxel preferring cars to bodies.
Multivariate pattern analysis
To determine the effect of attention at the multivariate level, and to examine the attentional bias that the representation of the stimulus in each category pair receives, we used a multivariate pattern analysis. Here, rather than comparing the mean values of voxel-wise coefficients in each ROI, we instead considered the ROI response pattern in each condition as a response vector, with the voxel-wise coefficients as its elements. Therefore, we had 16 response vectors, one for each task condition, in each ROI. Similar to the univariate analysis, we used the responses in the isolated conditions to assess category distance, and the responses in the paired conditions to evaluate the effect of attention.
Figure 2A illustrates the response vectors for two stimulus categories (here termed x and y) in the isolated conditions, as well as the response vector to the paired condition with attention directed to x. The three vectors Vxat , Vyat , and Vxaty illustrate the response patterns of these three conditions, with V representing the response vector in an ROI, subscripts x and y denoting the presence of the x and y stimuli, respectively, and the superscript at denoting the attended stimulus. Therefore, Vxat represents the response vector related to the isolated x condition (in which x was automatically attended), and Vxaty represents the response vector related to the paired xy condition with attention directed to the x stimulus. The projection of the paired-condition vector Vxaty onto the plane defined by the two isolated responses Vxat and Vyat is illustrated as Pxaty. Using this projection vector, we calculate the weight of Vxat and Vyat in the paired response.
Response vectors related to <inline-formula><mml:math id="inf46"><mml:mstyle displaystyle="true" scriptlevel="0"><mml:mi>x</mml:mi></mml:mstyle></mml:math></inline-formula> and <inline-formula><mml:math id="inf47"><mml:mstyle displaystyle="true" scriptlevel="0"><mml:mi>y</mml:mi></mml:mstyle></mml:math></inline-formula> stimuli in isolated and paired conditions.
(A) Example illustration of the isolated and paired responses in three-dimensional space. Vxat and Vyat denote the response vectors related to isolated x and isolated y conditions and Vxaty illustrates the response in the paired condition with attention directed to stimulus x. The paired response is projected on the plane defined by the two isolated responses Vxat and Vyat. This projection is illustrated by Pxaty. (B) Two-dimensional illustration of the plane defined by the two isolated response vectors Vxat and Vyat, along with the paired response vectors Vxat and Vyat and Pxaty as the projection of the paired response, Vxaty, on the plane. We calculated the weight of the two isolated responses in the paired response using multiple regression, with the weights of Vxat and Vyat shown as a1 and a2, respectively.
Multivariate distance based on the isolated conditions
As illustrated in Figure 2B, the two isolated response vectors Vxat and Vyat have a certain distance because the response across the voxels varies for the two stimuli. For two stimuli that elicit more similar response patterns in an ROI, the isolated response vectors are closer to each other. Thus, we defined the multivariate distance between the two isolated response vectors Vxat and Vyat in each ROI using Pearson’s correlation, as shown in Equation 3:Multivariatedistance=1−ρ(Vxat,Vyat)
where Vxat and Vyat represent the response vectors related to the isolated x and y conditions and ρ denotes Pearson’s correlation coefficient between the two response vectors. For stimuli with more similar response patterns, the correlation between their response vectors will be higher, leading to lower multivariate distance.
Multivariate effect of attention based on the paired conditions (Attentional weight shift)
Similar to the isolated conditions, we considered the response pattern in the paired conditions as vectors, Vxaty and Vxyat. We first projected the paired vectors on the plane defined by the isolated vectors (Figure 2A) and then determined the weight of each isolated vector in the projected vector (Figure 2B). Thus, the response vectors in the paired conditions can be written as the linear combination of the response vectors in the isolated conditions, with an error term denoting the deviation of the paired-condition responses from the plane defined by the isolated-condition responses (Reddy et al., 2009), as shown in Equation 4a:Vxaty=a1⋅Vxat+a2⋅Vyat+ϵ1Vxyat=b1⋅Vxat+b2⋅Vyat+ϵ2
Here, parameters a1 and a2 are the weights of the isolated x and y responses, respectively, when x is attended, and parameters b1 and b2 are the respective weights of isolated x and y responses when y is attended. ϵ1 and ϵ2 denote the error terms related to the deviation of the Vxaty and Vxyat from the Vxat−Vyat plane, respectively. While this model has been previously called weighted average (Reddy et al., 2009), we chose the more general term linear combination because we did not impose any limits on the estimated weights of the two isolated responses in the paired response.
A higher a1 compared to a2 indicates that the paired response pattern is more similar to Vxat than to Vyat, and vice versa. For instance, after calculating the weights of the Body and Car stimuli in the paired response related to the simultaneous presentation of both stimuli in the LO ROI, we obtain: VBodyatCar=0.79VBody+0.31VCar, VBodyCarat=0.43VBody+0.68VCar (See Appendix 1—table 4 for the average weights of the two stimuli for all pairs in all ROIs). Note that these weights are averaged across participants. As can be observed, in the presence of both body and car stimuli, the weight of each stimulus is higher when attended compared to the case when it is unattended. In other words, when attention shifts from body to car stimuli, the weight of the isolated body response (VBodyat) decreases in the paired response. We can, therefore, observe in this instance that the response in the paired condition is more similar to the isolated body response pattern when body stimuli are attended and more similar to the isolated car response pattern when car stimuli are attended.
In the presence of two stimuli, if attention could completely remove the effect of the unattended stimulus, the paired response would be the same as the response to the isolated attended stimulus. However, the information related to the unattended stimulus is not fully removed and attention has been shown to increase the weight of the response related to the attended stimulus in the paired response without completely removing the effect of the unattended stimulus (Reddy et al., 2009), as observed in the above example of the Body-Car pair. As shown here, even when body stimuli were attended, the effect of the unattended car stimuli was still present in the response, shown in the weight of the isolated car response (0.31). This weight increased when attention shifted towards car stimuli to a value of less than 1 (0.68 in the attended case), showing the effect of attention on the response. To examine whether this increase in the weight of the attended stimulus is constant or if it depends on the similarity of the two stimuli in cortical representation, we defined the weight shift as the multivariate effect of attention:Δw=a1a1+a2−b1b1+b2
Here, a1, a2, b1, and b2 are the weights of the isolated responses, estimated using Equation 4a. We calculate the weight of the isolated x response once when attention is directed towards x (a1), and a second time when attention is directed towards y (b1). In each case, we calculate the relative weight of the isolated x in the paired response by dividing the weight of the isolated x by the sum of weights of x and y (a1+a2 when attention is directed towards x, and b1+b2 when attention is directed towards y). We then define the weight shift, Δw, as the change in the relative weight of the isolated x response in the paired response when attention shifts from x to y. A higher Δw for a category pair indicates that attention is more efficient in removing the effect of the unattended stimulus in the pair. We used relative weights as a normalized measure to compensate for the difference in the sum of weights for different category pairs. Thus, using the normalized measure, we calculated the share of each stimulus in the paired response. For instance, considering the Body-Car pair, the share of the body stimulus in the paired response was equal to 0.72 and 0.38, when body stimuli were attended and unattended, respectively. We then calculated the change in the share of each stimulus caused by the shift in attention using a simple subtraction (Equation 5: Δw=0.34 for the above example of the Body-Car pair in LO) and used this measure to compare between different pairs.
Simulations
We investigated the mechanisms underlying the observed effect of stimulus similarity on attentional enhancement using simulations. To examine which attentional mechanism leads to the effects observed in the empirical data, we generated the neural response to unattended object stimuli as a baseline response in the absence of attention, using the data reported by neural studies of object recognition in the visual cortex (Ni et al., 2012; Bao and Tsao, 2018). Then, using an attention parameter for each neuron and different attentional mechanisms, we simulated the response of each neuron to the different task conditions in our experiment. Finally, we assessed the population response by averaging neural responses. We considered two models for attentional enhancement: a response gain model (McAdams and Maunsell, 1999; Reynolds and Chelazzi, 2004) and a tuning sharpening model (Martinez-Trujillo and Treue, 2004; Ling et al., 2009).
According to the response gain model, attention to an object multiplicatively increases neural responses to that object (Figure 3A). For instance, for a body-selective neuron, this mechanism can be implemented using Equation 6a:RBodyat=β⋅RBodyRCarat=β⋅RCar
Attentional modulation by the response gain and tuning sharpening models.
We illustrate the models here for the example of a neuron with high selectivity for cat stimuli. Solid curves denote the response to unattended stimuli and dashed curves denote the response to attended stimuli. (A) According to the response gain model, the response of the neuron to attended stimuli is scaled by a constant attention factor. Therefore, the response of the cat-selective neuron to an attended stimulus is enhanced to the same degree for all stimuli. (B) According to the tuning sharpening model, the response modulation by attention depends on the neuron’s tuning for the attended stimulus. Therefore, for optimal and near-optimal stimuli such as cat and body stimuli the response is highly increased, while for non-optimal stimuli such as houses, the response is suppressed.
Here, RBody is the neuron’s response to an ignored body stimulus, and RBodyat is the response of the neuron to the attended body stimulus, which is enhanced by the attention factor, β and RCarat in Equation 6b denote the response of the same body-selective neuron to an ignored and an attended car stimulus, respectively. The response gain model posits that attention to either stimulus enhances the response of the neuron by the same attention factor. This multiplicative scaling preserves the shapes of the neurons’ tuning curves (Treue and Martínez Trujillo, 1999; McAdams and Maunsell, 1999).
In contrast, according to the tuning sharpening model, attention to an object increases neural responses relative to their responsiveness to that object (Figure 3C). Therefore, while the response of a neuron is substantially enhanced when an optimal stimulus is attended, its response to an attended non-optimal stimulus is increased to a lesser degree or even decreased. The tuning sharpening model thus predicts a sharpening of the neurons’ tuning curve with attention (Ling et al., 2009).
We implemented this mechanism using Equation 7a:RBodyat=β⋅s1⋅RBodyRCarat=β⋅s2⋅RCars1=RBodyRmaxs2=RCarRmax
In the above equations, RBody, RCar, RBodyat , and RCarat denote the neuron’s response to the unattended body, unattended car, attended body, and attended car stimuli, respectively. Parameters s1 and s2 denote the degree of the neuron’s selectivity to body and car stimuli, respectively. Parameter β is the attention factor. Rmax is the response of the neuron to its optimal stimulus.
We simulated the action of the response gain model and the tuning sharpening model using numerical simulations. We composed a neural population of 4×105 neurons in equal proportions body-, car-, cat- or house-selective. Each neuron also responded to object categories other than its preferred category, but to a lesser degree and with variation. We chose neural responses to each stimulus from a normal distribution with the mean of 30 spikes/s and a standard deviation of 10 and each neuron was randomly assigned an attention factor in the range between 1 and 10 using a uniform distribution. These values are comparable with the values reported in neural studies of attention and object recognition in the ventral visual cortex (Ni et al., 2012; Bao and Tsao, 2018). We also added Poisson noise to the response of each neuron (Britten et al., 1993), assigned randomly for each condition of each neuron.
Attention was implemented according to the above equations. Using Equations 6a and 7a, we calculated the response of each neuron to the same 16 conditions as our main fMRI experiment. Then, we randomly chose 1000 neurons with similar selectivity from the population, and averaged their responses to make up a voxel.
We modeled two neural populations: a general object-selective population in which each voxel shows preference to a particular category and voxels with different preferences are mixed in with each other (similar to LO and pFS), and a category-selective population in which all voxels have a similar preference for a particular category (similar to EBA and PPA). Finally, we performed the same univariate and multivariate analyses as those used for the fMRI data to compare the predictions of each model with the observed data.
ResultsBehavioral results
Participants performed a one-back repetition detection task to maintain attention toward the cued stimuli. Detection rate in each experimental run was checked during the scan to ensure that participants followed the instructions. Participants had an average detection rate of 90.49% across all runs, confirming effective attention towards the cued stimuli (Figure 1—figure supplement 1). As expected, the average detection rate in the isolated conditions (94.82% ± 0.046) was significantly higher than in the paired conditions (89% ± 0.07, with t(14)=7.2 and p<0.0001), since detecting a repetition in the superimposed case was more difficult.
The effect of attention varies dependent on the target-distractor difference in response
We considered the effect of attention in five ROIs: the primary visual cortex V1, the object-selective regions LO and pFs, the body-selective region EBA, and the scene-selective region PPA. We obtained the voxel-wise responses through a GLM in those ROIs for all task conditions, consisting of four isolated conditions (blocks with isolated stimuli from one category) and 12 paired conditions (blocks with superimposed stimuli from two categories, see Figure 1B). There were six combinations of category pairs: Body-Car, Body-House, Body-Cat, Car-House, Car-Cat and House-Cat. For each voxel, we determined its relative preference for the two categories of each category pair, based on its response to the two categories in isolation. Thus, for each pair, one category was labeled as the more preferred category (M), and the other as the less preferred category (L). Considering the isolated and paired conditions related to each category pair, we hereafter refer to the conditions related to each category pair as Mat, MatL, MLat, and Lat, with M and L denoting the more preferred and the less preferred categories for each voxel, and the superscript at denoting the attended stimulus.
For instance, for the Body-Car pair, for a voxel that showed a higher response to body stimuli than to car stimuli, the four associated conditions related to the pair were referred to as Mat (attended body stimuli), MatL (attended body stimuli paired with ignored car stimuli), MLat (attended car stimuli paired with ignored body stimuli), and Lat (attended car stimuli). If the same voxel was more responsive to cats than bodies, then the four conditions related to the Body-Cat pair would be referred to as: Mat (attended cat stimuli), MatL (attended cat stimuli paired with ignored body stimuli), MLat (attended body stimuli paired with ignored cat stimuli), and Lat (attended body stimuli).
Note that the response in paired conditions can be higher or lower than the response to the isolated more preferred stimulus (condition Mat), depending on the voxel response to the two presented stimuli (see Figure 4C–D), as previously reported (Doostani et al., 2023). This is consistent with previous studies reporting the response to multiple stimuli to be higher than the average, but lower than the sum of the response to isolated stimuli (Reddy et al., 2009).
Average voxel response in the extastriate body area (EBA) for each pair of stimulus categories.
The x-axis labels represent the four conditions related to each category pair, Mat, MatL, MLat, Lat, with M and L denoting the presence of the more preferred and the less preferred category and the superscript at denoting the attended category. For instance, Mat refers to the condition in which the more preferred stimulus was presented in isolation (and automatically attended), and MLat refers to the paired condition in which the less preferred stimulus was attended to. Red arrows in each panel illustrate the observed change in response (univariate shift) caused by the shift of attention from the more preferred to the less preferred stimulus. Green arrows in panels B and C illustrate the difference in the response to isolated stimuli. Error bars represent standard errors of the mean. N=17 human participants.
Average voxel response in the primary visual cortex (V1) for each pair of stimulus categories.
The x-axis labels represent the four conditions related to each category pair, Mat, MatL, MLat, Lat, with M, and L denoting the presence of the more preferred and the less preferred category and the superscript at denoting the attended category. For instance, Mat refers to the condition in which the more preferred stimulus was presented in isolation (and automatically attended), and MLat refers to the paired condition in which the less preferred stimulus was attended to.
Average voxel response in the lateral occipital cortex (LO) for each pair of stimulus categories.
The x-axis labels represent the four conditions related to each category pair, Mat, MatL, MLat, Lat, with M, and L denoting the presence of the more preferred and the less preferred category and the superscript at denoting the attended category.
Average voxel response in the posterior fusiform gyrus (pFs) for each pair of stimulus categories.
The x-axis labels represent the four conditions related to each category pair, Mat, MatL, MLat, Lat, with M, and L denoting the presence of the more preferred and the less preferred category and the superscript at denoting the attended category.
Average voxel response in the parahippocampal place area (PPA) for each pair of stimulus categories.
The x-axis labels represent the four conditions related to each category pair, Mat, MatL, MLat, Lat, with M, and L denoting the presence of the more preferred and the less preferred category and the superscript at denoting the attended category.
We next determined the amount of univariate shift for each category pair using the voxel-wise coefficients related to the two paired conditions, MatL and MLat. As illustrated in Figure 4, we defined univariate shift for each category pair as the reduction in response when attention shifted from the M category to the L category in the presence of both stimuli (O’Craven et al., 1999; Ni et al., 2012; Vaziri-Pashkam and Xu, 2017; Doostani et al., 2023).
We observed a significant univariate shift when attention shifted from the M stimulus to the L stimulus for all pairs in the higher-level ROIs (ts>3,ps<0.04,corrected) except for the Body-Car, Body-Cat, and Car-Cat pairs in PPA (ts<2,ps>0.3,corrected) and the Car-House pair in EBA (t(16)=1,p=0.9,corrected). In V1, we observed no significant univariate shift for any pairs (ts<2.5,ps>0.1,corrected) except for the Body-Car pair (t(16)=3.8,p<0.01,corrected). Thus, the observed effect was limited to higher-level visual areas. Since the presented stimuli were the same in both conditions, this effect is caused by the shift in attention. It is important to note that since the cue was not separately modeled in the GLM, the signals related to the cue and the stimuli were mixed. However, given that the cues were brief and presented in the form of words, they are unlikely to have an effect on the responses observed in the higher-level ROIs.
Closer comparison of the results suggests that for pairs with significant univariate shift, the shift is not uniform. Instead, it is greater for pairs in which the M and L stimuli elicited more different responses compared to pairs with M and L stimuli eliciting closer responses. For example, we observed a larger univariate shift for the Body-House pair (Figure 4B) compared to the Body-Cat pair (Figure 4C) in all ROIs (ts>4,ps<0.001, Figure 4B–C, compare the size of the red arrows) except for V1 (t(16)=0.65,p=0.5). Comparing the isolated responses for these two pairs, we observed that the difference between the response of the isolated Body and isolated House conditions was generally higher than the difference between the isolated Body and isolated Cat conditions in all ROIs (ts>4,ps<0.001, Figure 4B–C, compare the size of the green arrows).
To examine this relationship quantitatively for all category pairs, we used two approaches. First, in a univariate analysis using average voxel responses, we determined the relationship between the observed univariate shift and the difference in isolated responses. Next, in a multivariate pattern analysis, we considered the response patterns in each ROI and looked for the underlying basis of this effect of attention at the multivariate level. This analysis enabled us to determine whether the bias of attention on the representation of the attended stimulus differed for different category pairs.
The univariate effect of attention decreases for target-distractor pairs that elicit closer responses
We first used a univariate analysis to determine the relationship between the univariate shift and category distance across pairings and in different ROIs. We split the fMRI data into two halves. Using the first half, we determined the voxel-wise M and L categories for each category pair. We then calculated the difference in the isolated response elicited by the two categories (univariate category distance) using the two isolated conditions Mat and Lat (Equation 1).
Then, using the second left-out part of the data, we assessed the univariate shift related to the pair as the amount of the reduction in response when attention shifted from the M stimulus to the L stimulus in the paired presentation of both stimuli (Equation 2). For instance, for the Body-Car pair and a voxel more responsive to bodies than cars, univariate category distance was calculated by RBodyat−RCarat, and univariate shift was calculated by RBodyatCar−RBodyCarat.
We observed a significantly positive correlation between univariate shift and category distance in all ROIs (ts>2.5,ps<0.02) except V1 (t(16)=0.56,p=0.58, see Figure 5). These results demonstrate that for stimuli that elicit more different responses, attention causes a greater response modulation, while the shift of attention between stimuli with more similar responses causes little response change. This indicates that the amount of univariate shift is related to the response difference between the two presented stimuli.
Univariate shift versus category distance in each region of interest (ROI).
(A–E) The value of univariate shift versus category distance. MatL and MLat denote the two paired conditions with attention directed to the more preferred (M) or less preferred (L) stimulus, respectively. Mat and Lat represent the isolated conditions, respectively, with the more preferred or the less preferred stimulus presented in isolation. The blue, red, yellow, purple, green, and sky blue circles in each panel represent the values related to the Body-Car, Body-House, Body-Cat, Car-House, Car-Cat, and House-Cat pairs, respectively. The correlation coefficients in each ROI were calculated for single participants and the lines in the average plots are only shown for illustration purposes. (F) Correlation coefficient for the correlation between the univariate shift and category distance in each ROI. Asterisks indicate that the correlation coefficients are significantly positive (p<0.05). Error bars represent standard errors of the mean. N=17 human participants.
The multivariate effect of attention decreases for more similar target-distractor pairs
The univariate analysis above considers average response only and thus cannot capture other aspects of response variance. For example, in an object-selective region with diverse selectivity for different objects, the average response to body and house stimuli is close, but the response pattern may be very different since voxels highly responsive to bodies do not show high responses to houses, and vice versa. Thus, we had to consider voxel preferences in the univariate analysis to observe the difference in response between the two categories. Furthermore, although the paired responses can be greater than responses to both isolated conditions (Figure 4C–D), there is still the possibility that the univariate shift is limited by the amount of the difference between isolated-condition responses for each category pair.
We complement the univariate approach with a multivariate pattern analysis to assess the relationship between the effect of attention and category distance at the multivariate level. By considering the whole response pattern in an ROI to each stimulus, we can compare the responses to each stimulus without considering voxel preferences. Moreover, using this method we can determine the weight of the response to each isolated stimulus in the total response, and determine the attentional bias related to each category pair.
The multivariate representation of two simultaneously-presented stimuli can be modeled as the linear combination of the representations of the two stimuli presented in isolation (Reddy et al., 2009): When one stimulus is attended, the weight of the response to that stimulus increases in the multivariate representation.
Taking this approach, for each category pair (e.g. Body-Car), we considered the multivariate representation of the two paired conditions (VBodyatCar and VBodyCarat, with V denoting the multivariate response pattern of each condition), and determined the weight of each of the isolated-stimulus responses (VBodyat and VCarat) in the paired response (Figure 2B). We then calculated the difference between the weight of each stimulus when it was the target and when it was the distractor (e.g. for the Body-Car pair, the difference between the weight of VBodyat in VBodyatCar and VBodyCarat).
If attention could perfectly remove the effect of the distractor, the weight of the attended stimulus would equal one and the representation of the pair would be identical to the representation of the isolated target. In this case, the difference between the weight of the stimulus representation when attended and ignored would be a maximal value of one. However, if the distractor is not completely removed, this leads to a weight shift value smaller than one. Thus, the magnitude of the weight shift is an indicator of the efficiency of attention, with greater values indicating a higher efficiency of attention in removing the distractor.
To compare the efficiency of attention across category pairs, we calculated the weight shift for each category pair (Equation 5). Similar to the univariate analysis, we took a cross-validation approach and used one-half of the data to calculate the weight shift. Then, to determine whether this multivariate effect of attention was dependent on the similarity between the target and the distractor in their cortical representation, we calculated the multivariate category distance for each category pair using the second left-out half of the data (Equation 3).
As illustrated in Figure 6A–E, we observed that the attentional weight shift was not constant for different category pairs, and that weight shift and category distance correlated positively in LO, pFs, and EBA (ts>4.4,ps<5×10−3), marginally significantly in PPA (t(16)=1.8,p=0.09), and not in V1 (t(16)=0.42,p=0.68). Less significant results in PPA might arise from the fact that this region shows no response to body and cat stimuli and little response to car stimuli (see Appendix 1—table 2). Therefore, it is not possible to observe the effect of attention for all category pairs in PPA.
Weight shift versus category distance in each region of interest (ROI).
(A–E) Attentional weight shift versus category distance. The blue, red, yellow, purple, green, and sky blue circles in each panel represent the values related to the Body-Car, Body-House, Body-Cat, Car-House, Car-Cat, and House-Cat pairs, respectively. We calculated the correlation coefficients for single participants and the lines in these average plots are only for illustration purposes. (F) Correlation coefficient for the correlation between attentional weight shift and category distance in each ROI. Asterisks indicate that the correlations are significantly positive (p<0.05). Error bars represent standard errors of the mean. N=17 human participants.
Sum of weights in the multivariate analysis for each category pair in each region of interest (ROI), averaged across conditions with attention directed to each of the two categories of a pair and across participants.
For instance, the blue Body-Car bars represent the average results related to the BodyatCar and BodyCarat conditions, averaged across participants. Error bars represent standard errors of the mean. N=17 human participants.
We performed the analysis including only voxels that had a significantly positive GLM coefficient across the runs and observed the same results. Moreover, to check whether the effect is robust over more selective thresholds for ROI definition, we redefined the left EBA region with p<0.0001 and p<0.00001 criteria. We observed a similar weight shift effect for both criteria. We also calculated category distance based on the euclidean distance between response patterns of category pairs and observed a similarly positive correlation between the weight shift and the euclidean category distance in all ROIs (ps<0.01,ts>2.9) except V1 (p=0.5,t=0.66). These results are in agreement with our main multivariate results, indicating that the attentional bias towards a stimulus in a pair decreases as the similarity between the two stimuli in neural representation increases.
Tuning sharpening predicts the dependence of attentional modulation on target-distractor similarity
We observed empirically that attentional enhancement is not constant and content-independent, but rather depends on the response similarity between the target and the distractor. We next asked whether a gain increase or tuning changes predict the observed effect of target-distractor similarity on the attentional bias.
Based on the response gain model, attention increases neural responses by scaling the responses by a constant attention factor (McAdams and Maunsell, 1999; Reynolds and Chelazzi, 2004). Therefore, the response gain model predicts that attention scales the neurons’ tuning function without affecting its shape (Figure 3A).
In contrast, the tuning sharpening model proposes that attention enhances the response of each neuron based on its preference for the attended stimulus (Martinez-Trujillo and Treue, 2004; Ling et al., 2009). Therefore, this model predicts that attention causes a sharpening of the neurons’ tuning function, with a sharp increase in the response to optimal stimuli, and no increase in the response to the non-optimal stimuli (Figure 3B).
To examine which of these mechanisms could account for the observed results, we simulated the responses of a neural population to isolated and paired stimuli from the four categories of bodies, cars, houses, and cats. Equivalent to the fMRI experiment, we determined neuronal responses to stimuli presented either in isolation or paired with stimuli from another category (Figure 1B). We implemented attentional enhancement of the neural responses either using the response gain model (Equation 6a), or the tuning sharpening model (Equation 7a). We then used the univariate and multivariate analyses equivalent to those used for the fMRI data to determine which model predicts the empirical data.
We created two neural populations: (i) a population with varying selectivity across neurons, representing object-selective regions, in which neurons show different selectivities (similar to LO and pFs), and (ii) a population with similar selectivity across all neurons to represent a region with a strong preference for a specific object category, in which neurons generally show high response to stimuli from that category (similar to EBA and PPA). Then we assessed the univariate shift using the reduction in response when attention shifted from the stronger to the weaker stimulus in a pair (Equation 2), and examined its relationship with univariate category distance (Equation 1).
We found that the response gain model predicted no relationship between univariate shift and category distance in either population (Figure 7A–B). In contrast, the tuning sharpening model predicted a positive correlation between univariate shift and category distance in both neural populations (Figure 7C–D). Thus, the tuning sharpening model provides a better prediction of the empirical data compared to the response gain model.
Univariate shift as a function of category distance, as predicted by the two attentional mechanisms.
MatL and MLat denote the two paired conditions with attention directed to the more preferred (M) or the less preferred (L) stimulus, respectively. Mat and Lat represent the isolated conditions, respectively with the M or the L stimulus presented in isolation. Top panels represent predictions in a region with a strong preference for a specific category, and bottom panels illustrate predictions in an object-selective region. Each circle represents a pair of categories. (A–B) Predicted univariate shift based on the response gain model in a region with a strong preference for a specific category (A) and in an object-selective region (B). (C–D) Predicted univariate shift based on the tuning model in a region with strong preference for a specific category (C) and in an object-selective region (D).
Next, for the multivariate analysis, we assessed the attentional weight shift for each category pair as attention shifted from one stimulus to the other (Equation 5), and examined its relationship with the multivariate category distance (Equation 3). Here, too, we find that the response gain model predicted no relationship between attentional weight shift and category distance (Figure 8A–B). In contrast, the tuning sharpening model predicted a positive relationship between weight shift and category distance for both neural populations (Figure 8C–D), providing further evidence for tuning sharpening as the underlying mechanism for attentional enhancement.
Predicted weight shift as a function of category distance.
Weight shift for each pair is calculated using Equation 5. Category distance represents the difference in multi-voxel representation between responses to the two isolated stimuli, calculated by Equation 3. Top panels are related to predictions in a region with a strong preference for a specific category and the bottom panels illustrate predictions in an object-selective region. (A–B) Weight shift predicted by the response gain model in a region with a strong preference for a specific category (A) and in an object-selective region (B). (C–D) Weight shift predicted by the tuning model in a region with strong preference for a specific category (C) and in an object-selective region (D).
We also tested a third model based on a labeled line mechanism for attentional enhancement (see Appendix 1). The labeled line model posits that attention to a stimulus enhances the neural response only when the attended stimulus is the neuron’s preferred stimulus. Therefore, this model is a special case of change in the neurons’ tuning curve (Appendix 1—figure 1). Although the labeled line model could predict the positive correlation between the univariate shift and category distance in a region with high selectivity for a certain category, it failed to predict the results in other cases (Appendix 1—figure 2).
In sum, the tuning model predicts the empirically-observed effect of target-distractor similarity on attentional effects both at the univariate and at the multivariate level, while the response gain model does not.
Discussion
Visual stimuli compete for resources in the brain. The biased competition model posits that attention to a stimulus biases this competition in favor of the attended stimulus (Moran and Desimone, 1985; Desimone and Duncan, 1995; Reynolds et al., 1999). Here, we examined the change in this attentional bias by systematically varying the target and distractors. Using fMRI, we showed that rather than being a constant top-down bias, attentional enhancement depends on the similarity between the target and the distractor in their cortical representation, both at the univariate level and at the multivariate level. Using simulations, we arbitrated between the response gain model and the tuning sharpening model as mechanisms of attention for the observed effect, and showed that the empirical results were explained by the latter and not the former.
Effect of target-distractor similarity on the attentional bias
Using stimuli from four object categories, our study reveals the neural basis of the attentional effect graded by target-distractor similarity in the human brain both at the univariate level and at the multivariate level. This finding has two important implications:
First, our results show that in the competition between multiple stimuli, the attentional bias is not constant. Previous studies have shown attentional modulation in the human brain as an average value without considering its variance for different pairings of targets and distractors (Reddy et al., 2009). These previous accounts of attention cannot explain the variance in performance for the same number of stimuli from different categories. Assessing the role of stimulus content in the bias caused by attention, we confirm that attention enhances the response related to the target. We refine our understanding by showing that however, the attentional bias offers less advantage for a more similar target-distractor pair.
Second, this finding provides direct neural evidence for the adverse effects of target-distractor similarity on performance, as previously reported in behavioral studies (Cohen et al., 2014; Cohen et al., 2017). While behavioral data have suggested that this effect is due to limitations in processing, no investigation has been made to determine the underlying reason or find a mechanistic explanation. Our results demonstrate that this reduction in performance is because the representation of the target (relative to the distractor) is less effectively enhanced by attention when the target becomes more similar to the distractor.
We observed a significant univariate shift in higher-level regions of the occipito-temporal cortex, but not in V1. Evidence on the effect of attention on V1 responses is divergent, with some previous neuroimaging studies showing a significant effect of attention on neural responses (Somers et al., 1999; Gandhi et al., 1999), while others report no significant effect of attention (Corbetta et al., 1990; Thorat and Peelen, 2022; Doostani et al., 2023). We believe that this apparent discrepancy results from the form of attention under study. Here, we study object-based attention with a superimposed design that excludes response modulation by space-based attention. Previous reports of significant attentional modulation in V1 include studies of space-based attention with stimuli presented at different locations (Somers et al., 1999; Gandhi et al., 1999). Considering the high reliance of V1 responses to location, the effect of attention is less pronounced when the two stimuli are presented at the same location, as is the case in the present study.
Although examples of superimposed cluttered stimuli are not very common in everyday life, they still do occur in certain situations, for example, reading text on the cellphone screen in the presence of reflection and glare on the screen or looking at the street through a patterned window. Such instances recruit object-based attention which was the aim of this study, whereas in more common cases in which attended and unattended objects occupy different locations in space, both space-based and object-based attention may work together to resolve the competition between different stimuli. Here, we chose to move away from usual everyday scenarios to study the effect of object-based attention in isolation. Future studies can reveal the effect of target-distractor similarity, i.e., proximity in space, on space-based attention and how the effects caused by object-based and space-based attention interact.
Please note that we used a blocked design in which the target and distractor categories could be predicted across each block. While it is possible that the current design has led to an enhancement of the observed effect, previous behavioral data Cohen et al., 2014; Xu and Vaziri-Pashkam, 2019 have reported the same effect in experiments in which the distractor was not predictable. To study the effect of predictability on fMRI responses, however, an event-related design is more appropriate, an interesting venue for future fMRI studies.
A model for object-based attentional enhancement
Using a simulation approach, we provide a mechanistic explanation for the observed graded attentional effect. Our modeling results have two implications:
First, we demonstrate that tuning sharpening, but not response gain, predicts the observed reduction in the effect of attention for more similar target-distractor pairs both at the univariate and at the multivariate level. Previous research has shown that a change in the tuning function improves attentional selection at high external noise levels (Ling et al., 2009). Our results indicate that a change in tuning function could also lead to behavioral disadvantage in an environment where the target is not very different from the surrounding items. When attention is directed towards the target, the response to non-target objects that are more similar to the target is also enhanced, albeit to a lesser amount, leading to an overall weaker effect of attention for a more similar target-distractor pair.
Second, providing evidence from the human brain in favor of tuning sharpening, we suggest tuning sharpening as the underlying mechanism in the domain of object-based attention. By comparing the response gain model and the tuning sharpening model directly in a single study, we provide strong evidence that arbitrates between the theories. The effects of attention have generally been explained by attention acting through increasing the contrast or response gain, especially for space-based attention (McAdams and Maunsell, 1999; Reynolds and Chelazzi, 2004; Fox et al., 2023). However, a simple increase in gain cannot explain all reported effects of attention, and a change in the shape of the tuning curves has been observed during visual search (Çukur et al., 2013), and feature-based attention (Martinez-Trujillo and Treue, 2004; David et al., 2008; Ling et al., 2009).
While tuning curves are commonly used for feature dimensions such as stimulus orientation or motion direction, here, we used the term to describe the variation in a neuron’s response to different object stimuli. With a finite set of object categories, as in the current study, the neural response in object space is discrete, rather than a continuous curve illustrated for features such as stimulus orientation. The neuron might have to tune for a particular feature such as curvature or spikiness (Bao et al., 2020) that is present to different degrees in our object stimuli in a continuous way, but we are not measuring this directly. Nevertheless, since more preferred and less preferred features (objects in this case) can still be defined, we illustrate the neural response using a hypothetical curve in object space. As such, here, tuning sharpening refers to the fact that the response to the more preferred object categories has been enhanced while the response to the less preferred stimulus categories is suppressed.
It is important to note that our speculation on the role of tuning sharpening in object-based attention is based on simulations and not neural data. To ascertain tuning sharpening as the underlying mechanism for object-based attention, intracranial recordings from the human brain are needed.
Conclusion
In sum, our results unravel the cortical basis by which target-distractor similarity affects attentional modulation, and indicate tuning sharpening as the underlying mechanism for response enhancement during object-based attention.
Additional informationCompeting interests
No competing interests declared
Author contributions
Conceptualization, Data curation, Formal analysis, Investigation, Visualization, Methodology, Writing - original draft, Writing – review and editing
Resources, Supervision, Validation, Methodology, Writing – review and editing
Supervision, Validation, Writing – review and editing
Participants gave written consent and received payment for their participation in the experiment. Data collection was approved by the Ethics Committee of the Institute for Research in Fundamental Sciences, Tehran. Reference number for the ethical approval: 98/60/2184.
Additional filesData availability
fMRI data have been deposited in OSF under https://doi.org/10.17605/OSF.IO/2QTF6.
We thank Sajad Aghapour for the helpful discussions. We thank Kiarash Farahmandrad for his help with the graphical illustration of the vector plot. Maryam Vaziri-Pashkam was supported by NIH Intramural Research Program ZIA-MH002035.
ReferencesBaoPTsaoDY2018Representation of multiple objects in macaque category-selective areasNature Communications9177410.1038/s41467-018-04126-729720645BaoPSheLMcGillMTsaoDY2020A map of object space in primate inferotemporal cortexNature58310310810.1038/s41586-020-2350-532494012BeckDMKastnerS2005Stimulus context modulates competition in human extrastriate cortexNature Neuroscience81110111610.1038/nn1501BrittenKHShadlenMNNewsomeWTMovshonJA1993Responses of neurons in macaque MT to stochastic motion signalsVisual Neuroscience101157116910.1017/S0952523800010269CohenMAKonkleTRheeJYNakayamaKAlvarezGA2014Processing multiple visual objects is limited by overlap in neural channelsPNAS1118955896010.1073/pnas.131786011124889618CohenMAAlvarezGANakayamaKKonkleT2017Visual search for object categories is predicted by the representational architecture of high-level visual cortexJournal of Neurophysiology11738840210.1152/jn.00569.201627832600CorbettaMMiezinFMDobmeyerSShulmanGLPetersenSE1990Attentional modulation of neural processing of shape, color, and velocity in humansScience2481556155910.1126/science.2360050ÇukurTNishimotoSHuthAGGallantJL2013Attention during natural vision warps semantic representation across the human brainNature Neuroscience1676377010.1038/nn.338123603707DaleAMFischlBSerenoMI1999Cortical Surface-Based AnalysisNeuroImage917919410.1006/nimg.1998.0395DavidSVHaydenBYMazerJAGallantJL2008Attention to stimulus features shifts spectral tuning of V4 neurons during natural visionNeuron5950952110.1016/j.neuron.2008.07.00118701075DesimoneRDuncanJ1995Neural mechanisms of selective visual attentionAnnual Review of Neuroscience1819322210.1146/annurev.ne.18.030195.0012057605061DoostaniNHossein-ZadehG-AVaziri-PashkamM2023The normalization model predicts responses in the human visual cortex during object-based attentioneLife12e7572610.7554/eLife.75726DowningPEJiangYShumanMKanwisherN2001A cortical area selective for visual processing of the human bodyScience2932470247310.1126/science.1063414EpsteinRHarrisAStanleyDKanwisherN1999The parahippocampal place area: recognition, navigation, or encoding?Neuron2311512510.1016/s0896-6273(00)80758-810402198FallahMStonerGRReynoldsJH2007Stimulus-specific competitive selection in macaque extrastriate visual area V4PNAS1044165416910.1073/pnas.061172210417360494FoxKJBirmanDGardnerJL2023Gain, not concomitant changes in spatial receptive field properties, improves task performance in a neural network attention modeleLife12e7839210.7554/eLife.7839237184221FranconeriSLAlvarezGACavanaghP2013Flexible cognitive resources: competitive content maps for attention and memoryTrends in Cognitive Sciences1713414110.1016/j.tics.2013.01.01023428935GandhiSPHeegerDJBoyntonGM1999Spatial attention affects brain activity in human primary visual cortexPNAS963314331910.1073/pnas.96.6.331410077681Grill-SpectorKKushnirTHendlerTEdelmanSItzchakYMalachR1998A sequence of object-processing stages revealed by fMRI in the human occipital lobeHuman Brain Mapping631632810.1002/(SICI)1097-0193(1998)6:4<316::AID-HBM9>3.0.CO;2-6KastnerSDe WeerdPDesimoneRUngerleiderLG1998Mechanisms of directed attention in the human extrastriate cortex as revealed by functional MRIScience28210811110.1126/science.282.5386.1089756472KourtziZKanwisherN2000Cortical regions involved in perceiving object shapeThe Journal of Neuroscience203310331810.1523/JNEUROSCI.20-09-03310.200010777794LingSLiuTCarrascoM2009How spatial and feature-based attention affect the gain and tuning of population responsesVision Research491194120410.1016/j.visres.2008.05.02518590754MalachRReppasJBBensonRRKwongKKJiangHKennedyWALeddenPJBradyTJRosenBRTootellRB1995Object-related activity revealed by functional magnetic resonance imaging in human occipital cortexPNAS928135813910.1073/pnas.92.18.81357667258Martinez-TrujilloJCTreueS2004Feature-based attention increases the selectivity of population responses in primate visual cortexCurrent Biology1474475110.1016/j.cub.2004.04.028McAdamsCJMaunsellJH1999Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4The Journal of Neuroscience1943144110.1523/JNEUROSCI.19-01-00431.19999870971McMainsSKastnerS2011Interactions of top-down and bottom-up mechanisms in human visual cortexThe Journal of Neuroscience3158759710.1523/JNEUROSCI.3766-10.201121228167MoranJDesimoneR1985Selective attention gates visual processing in the extrastriate cortexScience22978278410.1126/science.40237134023713NiAMRaySMaunsellJHR2012Tuned normalization explains the size of attention modulationsNeuron7380381310.1016/j.neuron.2012.01.00622365552O’CravenKMDowningPEKanwisherN1999fMRI evidence for objects as the units of attentional selectionNature40158458710.1038/44134ReddyLKanwisherNGVanRullenR2009Attention and biased competition in multi-voxel object representationsPNAS106214472145210.1073/pnas.090733010619955434ReynoldsJ HChelazziLDesimoneR1999Competitive mechanisms subserve attention in macaque areas V2 and V4The Journal of Neuroscience191736175310.1523/JNEUROSCI.19-05-01736.199910024360ReynoldsJHDesimoneR2003Interacting roles of attention and visual salience in V4Neuron3785386310.1016/s0896-6273(03)00097-712628175ReynoldsJHChelazziL2004Attentional modulation of visual processingAnnual Review of Neuroscience2761164710.1146/annurev.neuro.26.041002.13103915217345SerenoMIDaleAMReppasJBKwongKKBelliveauJWBradyTJRosenBRTootellRBH1995Borders of multiple visual areas in humans revealed by functional magnetic resonance imagingScience26888989310.1126/science.7754376SomersDCDaleAMSeiffertAETootellRB1999Functional MRI reveals spatially specific attentional modulation in human primary visual cortexPNAS961663166810.1073/pnas.96.4.16639990081ThoratSPeelenMV2022Body shape as a visual feature: Evidence from spatially-global attentional modulation in human visual cortexNeuroImage25511920710.1016/j.neuroimage.2022.11920735427768TootellRBHadjikhaniNKVanduffelWLiuAKMendolaJDSerenoMIDaleAM1998Functional analysis of primary visual cortex (V1) in humansPNAS9581181710.1073/pnas.95.3.8119448245TreueSMaunsellJHR1996Attentional modulation of visual motion processing in cortical areas MT and MSTNature38253954110.1038/382539a08700227TreueSMartínez TrujilloJC1999Feature-based attention influences motion processing gain in macaque visual cortexNature39957557910.1038/2117610376597TreueSMaunsellJH1999Effects of attention on the processing of motion in macaque middle temporal and medial superior temporal visual cortical areasThe Journal of Neuroscience197591760210.1523/JNEUROSCI.19-17-07591.199910460265Vaziri-PashkamMXuY2017Goal-directed visual processing differentially impacts human ventral and dorsal visual representationsThe Journal of Neuroscience378767878210.1523/JNEUROSCI.3392-16.201728821655Vaziri-PashkamMXuY2019An Information-driven 2-pathway characterization of occipitotemporal and posterior parietal visual object representationsCerebral Cortex292034205010.1093/cercor/bhy08029659730XuYVaziri-PashkamM2019Task modulation of the 2-pathway characterization of occipitotemporal and posterior parietal visual object representationsNeuropsychologia13210714010.1016/j.neuropsychologia.2019.10714031301350Appendix 1The labeled line model
We also simulated a special case of change in the tuning curve called the labeled line model. Based on this model, attention to a certain stimulus enhances the neural response only if the neuron is specifically labeled for that stimulus (Appendix 1—figure 1). For instance, attention to body stimuli causes an enhancement in the response of body neurons, but no enhancement in the response of car neurons which might respond to body stimuli to a lesser level. We implemented this mechanism using Equation 8a: for a body-selective neuron:RBodyat=β⋅RBodyRCarat=RCar
For a car-selective neuron:RBodyat=RBodyRCarat=β⋅RCar
For a house-selective neuron:RBodyat=RBodyRCarat=RCar
Then, using the labeled line mechanism for attentional enhancement, we simulated the response of two neural populations in the 16 task conditions (see Materials and methods and Results). Performing the univariate analysis on the simulated responses, we assessed the univariate shift for all category pairs. The labeled line model predicted a positive correlation between the univariate shift and category distance in the population with a strong preference for a certain category, while it predicted no relationship between univariate shift and category distance in the object-selective population (Appendix 1—figure 2A, B).
In the multivariate analysis, the labeled line model predicted no relationship for the neural population with a strong preference for a certain category (Appendix 1—figure 2C), while it predicted a negative correlation between weight shift and category distance in the object-selective population (Appendix 1—figure 2D).
Based on the labeled line model, attention enhances the response of a neuron only when the attended stimulus is the neuron’s preferred stimulus.
We illustrate the models here for the example of a neuron with high selectivity for cat stimuli. Solid curves denote the response to unattended stimuli and dashed curves denote the response to attended stimuli.
Simulation results for the labeled line model.
(A–B) Predicted univariate shift based on the labeled line model in a region with a strong preference for a specific category (A) and in an object-selective region (B). (C–D) Weight shift predicted by the labeled line model in a region with a strong preference for a specific category (C) and in an object-selective region (D).
Percentage of voxels in each regions of interest (ROI) most responsive to each of the four stimulus categories, averaged across participants.
Preferred Category
V1
LO
pFs
EBA
PPA
Body
21%
31%
21%
70%
10%
Car
19%
18%
17%
4%
12%
House
33%
32%
40%
3%
68%
Cat
27%
19%
22%
23%
10%
Average voxel response (general linear model, GLM coefficients) to each category in each region of interest (ROI), averaged across participants.
Stimulus Category
V1
LO
pFs
EBA
PPA
Body
0.28
1.59
0.88
1.62
0.001
Car
0.28
1.43
0.88
0.60
0.12
House
0.47
1.51
1.03
0.48
0.55
Cat
0.35
1.49
0.89
1.32
–0.02
Single-subject correlation coefficients for the correlation between the univariate shift and category distance in all regions of interest (ROIs).
V1
LO
pFs
EBA
PPA
–0.60
0.67
–0.073
0.72
0.18
0.16
0.38
0.015
0.64
0.46
–0.53
0.52
0.89
0.60
0.42
–0.062
0.78
0.93
0.87
0.69
–0.098
0.072
1.0
0.72
0.96
0.63
0.93
0.88
0.94
0.58
–0.31
0.52
–0.11
0.88
0.89
0.29
0.18
0.62
0.67
0.63
–0.49
0.094
0.63
0.81
0.80
0.21
0.57
0.31
0.81
0.87
–0.015
–0.57
0.79
0.35
0.69
0.35
0.79
0.90
0.90
0.88
–0.47
0.058
–0.050
0.93
–0.24
–0.047
–0.21
0.91
0.28
0.86
0.66
0.70
0.91
0.64
0.45
–0.74
0.54
0.24
0.60
–0.31
–0.095
0.73
0.89
0.64
0.95
Weights of each stimulus for each category pair in each region of interest (ROI), averaged across participants.
Pair
Attended Stimulus
V1
LO
pFs
EBA
PPA
Body-Car
Body
0.54×RBat+0.53×RCrat
0.79×RBat+0.31×RCrat
0.68×RBat+0.42×RCrat
0.93×RBat+0.15×RCrat
0.41×RBat+0.42×RCrat
Body-Car
Car
0.40×RBat+0.70×RCrat
0.43×RBat+0.68×RCrat
0.39×RBat+0.70×RCrat
0.53×RBat+0.46×RCrat
0.35×RBat+0.54×RCrat
Body-House
Body
0.43×RBat+0.65×RHat
0.65×RBat+0.45×RHat
0.65×RBat+0.47×RHat
0.84×RBat+0.21×RHat
0.42×RBat+0.61×RHat
Body-House
House
0.27×RBat+0.78×RHat
0.3×RBat+0.74×RHat
0.27×RBat+0.81×RHat
0.37×RBat+0.57×RHat
0.18×RBat+0.86×RHat
Body-Cat
Body
0.63×RBat+0.55×RCtat
0.86×RBat+0.35×RCtat
0.81×RBat+0.45×RCtat
0.90×RBat+0.30×RCtat
0.61×RBat+0.32×RCtat
Body-Cat
Cat
0.59×RBat+0.55×RCtat
0.69×RBat+0.48×RCtat
0.67×RBat+0.55×RCtat
0.67×RBat+0.48×RCtat
0.55×RBat+0.37×RCtat
Car-House
Car
0.42×RCrat+0.65×RHat
0.44×RCrat+0.70×RHat
0.55×RCrat+0.60×RHat
0.56×RCrat+0.59×RHat
0.35×RCrat+0.72×RHat
Car-House
House
0.31×RCrat+0.73×RHat
0.25×RCrat+0.85×RHat
0.36×RCrat+0.78×RHat
0.46×RCrat+0.65×RHat
0.27×RCrat+0.93×RHat
Car-Cat
Car
0.59×RCrat+0.52×RCtat
0.68×RCrat+0.44×RCtat
0.74×RCrat+0.40×RCtat
0.47×RCrat+0.54×RCtat
0.53×RCrat+0.40×RCtat
Car-Cat
Cat
0.45×RCrat+0.70×RCtat
0.41×RCrat+0.75×RCtat
0.55×RCrat+0.60×RCtat
0.23×RCrat+0.92×RCtat
0.46×RCrat+0.46×RCtat
House-Cat
House
0.76×RHat+0.32×RCtat
0.87×RHat+0.24×RCtat
0.91×RHat+0.23×RCtat
0.56×RHat+0.44×RCtat
0.99×RHat+0.11×RCtat
House-Cat
Cat
0.61×RHat+0.47×RCtat
0.47×RHat+0.65×RCtat
0.53×RHat+0.62×RCtat
0.17×RHat+0.90×RCtat
0.57×RHat+0.38×RCtat
Single-subject correlation coefficients for the correlation between weight shift and category distance in all regions of interests (ROIs).
V1
LO
pFs
EBA
PPA
–0.36
0.93
–0.25
0.46
0.06
0.31
–0.06
0.16
0.82
–0.25
–0.29
0.35
0.80
0.48
–0.42
–0.038
0.70
0.75
0.57
0.38
0.72
0.60
0.90
0.58
–0.77
0.53
0.69
0.76
0.51
0.59
–0.32
0.90
–0.25
0.29
0.065
0.33
0.22
–0.36
0.24
–0.48
0.52
0.31
0.63
0.54
0.91
0.019
0.96
0.80
0.61
0.71
–0.11
0.87
0.89
0.69
0.51
–0.27
0.94
0.81
0.63
0.50
–0.28
0.28
–0.11
0.49
0.59
0.42
0.11
0.61
0.42
0.59
0.67
0.77
0.78
0.36
0.36
–0.96
0.78
0.48
0.43
–0.24
–0.12
0.69
0.82
0.63
0.56
10.7554/eLife.89836.3.sa0eLife AssessmentPeelenMarius VReviewing EditorRadboud University NijmegenNetherlandsConvincingValuable
This valuable study has the potential to shed mechanistic light on how attention mechanisms that influence competition between multiple visual stimuli are modulated by the relative neural similarity of these stimuli. The study provides convincing data that will also be used for future modeling efforts. The study will be of interest to researchers working on the neural basis of visual attention.
The authors report an fMRI investigation of the neural mechanisms by which selective attention allows capacity-limited perceptual systems to preferentially represent task-relevant visual stimuli. Specifically, they examine competitive interactions between two simultaneously-presented items from different categories, to reveal how task-directed attention to one of them modulates the activity of brain regions that respond to both. The specific hypothesis is that attention will bias responses to be more like those elicited by the relevant object presented on its own, and further that this modulation will be stronger for more dissimilar stimulus pairs. This pattern was confirmed in univariate analyses that measured the mass response of a priori regions of interest, as well as multivariate analyses that considered the patterns of evoked activity within the same regions. The authors follow these neuroimaging results with a simulation study that favours a "tuning" mechanism of attention (enhanced responses to highly effective stimuli, and suppression for ineffective stimuli) to explain this pattern.
Strengths:
The manuscript clearly articulates a core issue in the cognitive neuroscience of attention, namely the need to understand how limited perceptual systems cope with complex environments in the service of the observer's goals. The use of a priori regions of interest (and a control region), and the inclusion of both univariate and multivariate analyses as well as a simple model, are further strengths. The authors carefully derive clear indices of attentional effects (for both univariate and multivariate analyses) which makes explication of their findings easy to follow.
Weaknesses:
Direct estimation of baseline responses may have improved the validity of the modelling. The presentation of transparently overlapping items has some methodological advantages, but somewhat limits the ecological validity of connections to real-world visual "clutter".
10.7554/eLife.89836.3.sa2Author responseDoostaniNargesAuthorInstitute for Research in Fundamental SciencesTehranIranHossein-ZadehGholam-AliAuthorUniversity of TehranTehranIranCichyRadoslaw MAuthorFreie Universität BerlinBerlinGermanyVaziri-PashkamMaryamAuthorNational Institute of Mental HealthBethesdaUnited States
The following is the authors’ response to the original reviews.
Public Reviews:
Reviewer #1 (Public Review):
Summary:
The authors report an fMRI investigation of the neural mechanisms by which selective attention allows capacity-limited perceptual systems to preferentially represent task-relevant visual stimuli. Specifically, they examine competitive interactions between two simultaneously-presented items from different categories, to reveal how task-directed attention to one of them modulates the activity of brain regions that respond to both. The specific hypothesis is that attention will bias responses to be more like those elicited by the relevant object presented on its own, and further that this modulation will be stronger for more dissimilar stimulus pairs. This pattern was confirmed in univariate analyses that measured the mass response of a priori regions of interest, as well as multivariate analyses that considered the patterns of evoked activity within the same regions. The authors follow these neuroimaging results with a simulation study that favours a "tuning" mechanism of attention (enhanced responses to highly effective stimuli, and suppression for ineffective stimuli) to explain this pattern.
Strengths:
The manuscript clearly articulates a core issue in the cognitive neuroscience of attention, namely the need to understand how limited perceptual systems cope with complex environments in the service of the observer's goals. The use of a priori regions of interest, and the inclusion of both univariate and multivariate analyses as well as a simple model, are further strengths. The authors carefully derive clear indices of attentional effects (for both univariate and multivariate analyses) which makes explication of their findings easy to follow.
Weaknesses:
There are some relatively minor weaknesses in presentation, where the motivation behind some of the procedural decisions could be clearer. There are some apparently paradoxical findings reported -- namely, cases in which the univariate response to pairs of stimuli is greater than to the preferred stimulus alone -- that are not addressed. It is possible that some of the main findings may be attributable to range effects: notwithstanding the paradox just noted, it seems that a floor effect should minimise the range of possible attentional modulation of the responses to two highly similar stimuli. One possible limitation of the modelled results is that they do not reveal any attentional modulation at all under the assumptions of the gain model, for any pair of conditions, implying that as implemented the model may not be correctly capturing the assumptions of that hypothesis.
We thank the reviewer for the constructive comments. In response, in the current version of the manuscript we have improved the presentation. We further discuss how the response in paired conditions is in some cases higher than the response to the preferred stimulus in this letter. For this, we provide a vector illustration, and a supplementary figure of the sum of weights to show that the weights of isolated-stimulus responses for each category pair are not bound to the similarity of the two isolated responses.
Regarding the simulation results, we have clarified that the univariate effect of attention is not the attentional modulation itself, but the change in the amount of attentional modulation in the two paired conditions. We provide an explanation for this in this letter below, and have changed the term “attentional modulation” to “univariate shift” in the manuscript to avoid the confusion.
Reviewer #2 (Public Review):
Summary:
In an fMRI study requiring participants to attend to one or another object category, either when the object was presented in isolation or with another object superimposed, the authors compared measured univariate and multivariate activation from object-selective and early visual cortex to predictions derived from response gain and tuning sharpening models. They observed a consistent result across higher-level visual cortex that more-divergent responses to isolated stimuli from category pairs predicted a greater modulation by attention when attending to a single stimulus from the category pair presented simultaneously, and argue via simulations that this must be explained by tuning sharpening for object categories.
Strengths:
- Interesting experiment design & approach - testing how category similarity impacts neural modulations induced by attention is an important question, and the experimental approach is principled and clever.
- Examination of both univariate and multivariate signals is an important analysis strategy.
- The acquired dataset will be useful for future modeling studies.
Weaknesses:
- The experimental design does not allow for a neutral 'baseline' estimate of neural responses to stimulus categories absent attention (e.g., attend fixation), nor of the combination of the stimulus categories. This seems critical for interpreting results (e.g., how should readers understand univariate results like that plotted in Fig. 4C-D, where the univariate response is greater for 2 stimuli than one, but the analyses are based on a shift between each extreme activation level?).
We are happy to clarify our research rationale. We aimed to compare responses in paired conditions when the stimuli were kept constant while varying the attentional target. After we showed that the change in the attentional target resulted in a response change , we compared the amount of this response change to different stimulus category pairs to investigate the effect of representation similarity between the target and the distractor on the response modulation caused by attentional shift. While an estimate of the neural responses in the absence of attention might be useful for other modeling studies, it would not provide us with more information than the current data to answer the question of this study.
Regarding the univariate results in Fig. 4C-D (and other equivalent ROI results in the revised version) and our analyses, we did not impose any limit on the estimated weights of the two isolated responses in the paired response and thus the sum of the two weights could be any number. We however see that the naming of “weighted average”, which implies a sum of weights being capped at one, has been misleading . We have now changed the name of this model to “linear combination” to avoid confusion
Previous studies (Reddy et al., 2009, Doostani et al., 2023) using a similar approach have shown a related results pattern: the response to multiple stimuli is higher than the average, but lower than the sum of the isolated responses, which is exactly what our results suggest. We have added discussion on this topic in the Results section in lines 409-413 for clarification:
“Note that the response in paired conditions can be higher or lower than the response to the isolated more preferred stimulus (condition Mat), depending on the voxel response to the two presented stimuli, as previously reported (Doostani et al. 2023). This is consistent with previous studies reporting the response to multiple stimuli to be higher than the average, but lower than the sum of the response to isolated stimuli (Reddy et al. 2009).”
We are not sure what the reviewer means by “each extreme activation level”. Our analyses are based on all four conditions. The two isolated conditions are used to calculate the distance measures and the two paired conditions are used for calculating the shift index. Please note that either the isolated or the paired conditions could show the highest response and we seeboth cases in our data. For example, as shown in Figure 4A in EBA, the isolated Body condition and the paired BodyatCar condition show the highest activation levels for the Body-Car pair, whereas in Figure 4C, the two paired conditions (BodyatCat and BodyCatat) elicit the highest response.
- Related, simulations assume there exists some non-attended baseline state of each individual object representation, yet this isn't measured, and the way it's inferred to drive the simulations isn't clearly described.
We agree that the simulations assume a non-attended baseline state, and that we did not measure that state empirically. We needed this non-attended response in the simulations to test which attention mechanism led to the observed results. Thus, we generated the non-attended response using the data reported in previous neural studies of object recognition and attention in the visual cortex (Ni et al., 2012, Bao and Tsao, 2018). Note that the simulations are checking for the profile of the modulations based on category distance. Thus, they do not need to exactly match the real isolated responses in order to show the effect of gain and tuning shift on the results. We include the clarification and the range of neural responses and attention parameters used in the simulations in the revised manuscript in lines 327-333:
“To examine which attentional mechanism leads to the effects observed in the empirical data, we generated the neural response to unattended object stimuli as a baseline response in the absence of attention, using the data reported by neural studies of object recognition in the visual cortex (Ni et al., 2012, Bao and Tsao, 2018). Then, using an attention parameter for each neuron and different attentional mechanisms, we simulated the response of each neuron to the different task conditions in our experiment. Finally, we assessed the population response by averaging neural responses.”
- Some of the simulation results seem to be algebraic (univariate; Fig. 7; multivariate, gain model; Fig. 8)
This is correct. We have used algebraic equations for the effect of attention on neural responses in the simulations. In fact, thinking about the two models of gain and tuning shift leads to the algebraic equations, which in turn logically leads to the observed results, if no noise is added to the data. The simulations are helpful for visualizing these logical conclusions. Also, after assigning different noise levels to each condition for each neuron, the results are not algebraic anymore which is shown in updated Figure 7 and Figure 8.
- Cross-validation does not seem to be employed - strong/weak categories seem to be assigned based on the same data used for computing DVs of interest - to minimize the potential for circularity in analyses, it would be better to define preferred categories using separate data from that used to quantify - perhaps using a cross-validation scheme? This appears to be implemented in Reddy et al. (2009), a paper implementing a similar multivariate method and cited by the authors (their ref 6).
Thank you for pointing out the missing details about how we used cross-validation. In the univariate analysis, we did use cross validation, defining preferred categories and calculating category distance on one half of the data and calculating the univariate shift on the other half of the data. Similarly, we employed cross-validation for the multivariate analysis by using one half of the data to calculate the multivariate distance between category pairs, and the other half of the data to calculate the weight shift for each category pair. We have now added this methodological information in the revised manuscript.
- Multivariate distance metric - why is correlation/cosine similarity used instead of something like Euclidean or Mahalanobis distance? Correlation/cosine similarity is scale-invariant, so changes in the magnitude of the vector would not change distance, despite this likely being an important data attribute to consider.
Since we are considering response patterns as vectors in each ROI, there is no major difference between the two measures for similarity. Using euclidean distance as a measure of distance (i.e. inverse of similarity) we observed the same relationship between weight shift and category euclidean distance. There was a positive correlation between weight shift and the euclidean category distance in all ROIs (ps < 0.01, ts > 2.9) except for V1 (p = 0.5, t = 0.66). We include this information in the revised manuscript in the Results section lines 513-515:
“We also calculated category distance based on the euclidean distance between response patterns of category pairs and observed a similarly positive correlation between the weight shift and the euclidean category distance in all ROIs (ps < 0.01, ts >2.9) except V1 (p = 0.5, t = 0.66).”
- Details about simulations implemented (and their algebraic results in some cases) make it challenging to interpret or understand these results. E.g., the noise properties of the simulated data aren't disclosed, nor are precise (or approximate) values used for simulating attentional modulations.
We clarify that the average response to each category was based on previous neurophysiology studies (Ni et al., 2012, Bao and Tsao, 2018). The attentional parameter was also chosen based on previous neurophysiology (Ni et al., 2012) and human fMRI (Doostani et al., 2023) studies of visual attention by randomly assigning a value in the range from 1 to 10. We have included the details in the Methods section in lines 357-366:
“We simulated the action of the response gain model and the tuning sharpening model using numerical simulations. We composed a neural population of 4⨯105 neurons in equal proportions body-, car-, cat- or house-selective. Each neuron also responded to object categories other than its preferred category, but to a lesser degree and with variation. We chose neural responses to each stimulus from a normal distribution with the mean of 30 spikes/s and standard deviation of 10 and each neuron was randomly assigned an attention factor in the range between 1 and 10 using a uniform distribution. These values are comparable with the values reported in neural studies of attention and object recognition in the ventral visual cortex (Ni et al. 2012, Bao and Tsao 2018). We also added poisson noise to the response of each neuron (Britten et al. 1993), assigned randomly for each condition of each neuron.”
- Eye movements do not seem to be controlled nor measured. Could it be possible that some stimulus pairs result in more discriminable patterns of eye movements? Could this be ruled out by some aspect of the results?
Subjects were instructed to direct their gaze towards the fixation point. Given the variation in the pose and orientation of the stimuli, it is unlikely that eye movements would help with the task. Eye movements have been controlled in previous experiments with individual stimulus presentation (Xu and Vaziri-Pashkam, 2019) and across attentional tasks in which colored dots were superimposed on the stimuli (Vaziri-Pashkam and Xu, 2017) and no significant difference for eye movement across categories or conditions was observed. As such, we do not think that eye movements would play a role in the results we are observing here.
- A central, and untested/verified, assumption is that the multivariate activation pattern associated with 2 overlapping stimuli (with one attended) can be modeled as a weighted combination of the activation pattern associated with the individual stimuli. There are hints in the univariate data (e.g., Fig. 4C; 4D) that this might not be justified, which somewhat calls into question the interpretability of the multivariate results.
If the reviewer is referring to the higher response in the paired compared to the isolated conditions, as explained above, we have not forced any limit on the sum of the estimated weights to equal 1 or 2. Therefore, our model is an estimation of a linear combination of the two multivariate patterns in the isolated conditions. In fact, Leila Reddy et al. (reference 6) reported that while the combination is closer to a weighted average than to a weighted sum, the sum of the weights are on average larger than 1. In Figure 4C and 4D the responses in the paired conditions are higher than either of the isolated-condition responses. This suggests that the weights for the linear combination of isolated responses in the multivariate analysis should add up to larger than one. This is what we find in our results. We have added a supplementary figure to Figure 6, depicting the sum of weights for different category pairs in all ROIs. The figure illustrates that in each ROI, the sum of weights are greater than 1 for some category pairs. It is however noteworthy that we normalized the weights in each condition by the sum of weights to calculate the weight shift in our analysis. The amount of the weight shift was therefore not affected by the absolute value of the weights.
- Throughout the manuscript, the authors consistently refer to "tuning sharpening", an idea that's almost always used to reference changes in the width of tuning curves for specific feature dimensions (e.g., motion direction; hue; orientation; spatial position). Here, the authors are assaying tuning to the category (across exemplars of the category). The link between these concepts could be strengthened to improve the clarity of the manuscript.
The reviewer brings up an excellent point. Whereas tuning curves have been extensively used for feature dimensions such as stimulus orientation or motion direction, here, we used the term to describe the variation in a neuron’s response to different object stimuli.
With a finite set of object categories, as is the case in the current study, the neural response in object space is discrete, rather than a continuous curve illustrated for features such as stimulus orientation. However, since more preferred and less preferred features (objects in this case) can still be defined, we illustrated the neural response using a hypothetical curve in object space in Figure 3 to show how it relates with other stimulus features. Therefore, here, tuning sharpening refers to the fact that the response to the more preferred object categories has been enhanced while the response to the less preferred stimulus categories is suppressed.
We clarify this point in the revised manuscript in the Discussion section lines 649-659:
“While tuning curves are commonly used for feature dimensions such as stimulus orientation or motion direction, here, we used the term to describe the variation in a neuron’s response to different object stimuli. With a finite set of object categories, as is the case in the current study, the neural response in object space is discrete, rather than a continuous curve illustrated for features such as stimulus orientation. The neuron might have tuning for a particular feature such as curvature or spikiness (Bao et al., 2020) that is present to different degrees in our object stimuli in a continuous way, but we are not measuring this directly. Nevertheless, since more preferred and less preferred features (objects in this case) can still be defined, we illustrate the neural response using a hypothetical curve in object space. As such, here, tuning sharpening refers to the fact that the response to the more preferred object categories has been enhanced while the response to the less preferred stimulus categories is suppressed.”
Recommendations for the authors:
Reviewer #1 (Recommendations For The Authors):
a. The authors should address the apparent paradox noted above (and report whether it is seen in other regions of interest as well). On what model would the response to any pair of stimuli exceed that of the response to the preferred stimulus alone? This implies some kind of Gestalt interaction whereby the combined pair generates a percept that is even more effective for the voxels in question than the "most preferred" one?
The response to a pair of stimuli can exceed the response to each of the stimuli presented in isolation if the voxel is responsive to both stimuli and as long as the voxel has not reached its saturation level. This phenomenon has been reported in many previous studies (Zoccolan et al., 2005, Reddy et al., 2009, Ni et al., 2012, Doostani et al., 2023) and can be modeled using a linear combination model which does not limit the weights of the isolated responses to equal 1 (Doostani et al., 2023). Note that the “most preferred” stimulus does not necessarily saturate the voxel response, thus the response to two stimuli could be more effective based on voxel responsiveness to the second stimulus.
As for the current study, the labels “more preferred” and “less preferred” are only relatively defined (as explained in the Methods section), meaning that the more preferred stimulus is not necessarily the most preferred stimulus for the voxels. Furthermore, the presented stimuli are semi-transparent and presented with low-contrast, which moves the responses further away from the saturation level. Based on reported evidence for multiple-stimulus responses, responses to single stimuli are in many cases sublinearly added to yield the multiple-stimulus response (Zoccolan et al., 2005, Reddy et al., 2009, Doostani et al., 2023). This means that the multiple-stimulus response is lower than the sum of the isolated responses and not lower than each of the isolated responses. Therefore, it is not paradoxical to observe higher responses in paired conditions compared to the isolated conditions. We observe similar results in other ROIs, which we provide as supplementary figures to Figure 4 in the revised manuscript.
We address this observation and similar reports in previous studies in the Results section of the revised manuscript in lines 409-413:
“Note that the response in paired conditions can be higher or lower than the response to the isolated more preferred stimulus (condition Mat), depending on the voxel preference for the two presented stimuli, as previously reported (Doostani et al., 2023). This is consistent with previous studies reporting the response to multiple stimuli to be higher than the average, but lower than the sum of the response to isolated stimuli (Reddy et al., 2009).”
b. Paradox aside, I wondered to what extent the results are in part explained by range limits. Take two categories that evoke a highly similar response (either mean over a full ROI, or in the multivariate sense). That imposes a range limit such that attentional modulation, if it works the way we think it does, could only move responses within that narrow range. In contrast, the starting point for two highly dissimilar categories leaves room in principle for more modulation.
We do not believe that the results can be explained by range limits because responses in paired conditions are not limited by the isolated responses, as can be observed in Figure 4. However, to rule out the possibility of the similarity between responses in isolated conditions affecting the range within which responses in paired conditions can change, we turned to the multivariate analysis. We used the weight shift measure as the change in the weight of each stimulus with the change in the attentional target. In this method, no matter how close the two isolated vectors are, the response to the pair could still have a whole range of different weights of the isolated responses. We have plotted an example illustration of two-dimensional vectors for better clarification. Here, the vectors Vxat and Vyat denote the responses to the isolated x and y stimuli, respectively, and the vector Pxaty denotes the response to the paired condition in which stimulus x is attended. The weights a1 and a2 are illustrated in the figure, which are equal to regression coefficients if we solve the equation Pxaty = [a1 a2] [x y]’. While the weight values depend on the amplitude of and the angle between the three vectors, they are not limited by a lower angle between Vxat and Vyat.
We have updated Figure 2 in the manuscript to avoid the confusion. We have also added a figure including the sum of weights for different category pairs in different regions, showing that the sum of weights are not dependent on the similarity between the two stimuli. The conclusions based on the weight shift are therefore not confounded by the similarity between the two stimuli.
c. Finally, related to the previous point, while including V1 is a good control, I wonder if it is getting a "fair" test here, because the range of responses to the four categories in this region, in terms of (dis)similarity, seems compressed relative to the other categories.
We believe that V1 is getting a fair test because the single-subject range of category distance in V1 is similar to LO, as can be observed Author response image 1_:_
Range of category distance in each ROI averaged across participants.
The reason that V1 is showing a more compressed distance range on the average plot is that the category distance in V1 is not consistent among participants. Although the average plots are shown in Figure 5 and Figure 6, we tested statistical significance in each ROI based on single-subject correlation coefficients.
Please also note that a more compressed range of dissimilarity does not necessarily lead to a less strong effect of category distance on the effect of attention. For instance, while LO shows a more compressed dissimilarity range for the presented categories compared to the other object selective regions, it shows the highest correlation between weight shift and category distance. Furthermore, as illustrated in Figure 5, no significant correlation is observed between univariate shift and category distance in V1, even though the range of the univariate distance in V1 is similar to LO and pFs, where we observed a significant correlation between category distance and univariate shift.
d. In general, the manuscript does a very good job explaining the methods of the study in a way that would allow replication. In some places, the authors could be clearer about the reasoning behind those methodological choices. For example: - How was the sample size determined?
Estimating conservatively based on the smallest amount of attentional modulation we observed in a previous study (Doostani et al., 2023), we chose a medium effect size (0.3). For a power of 0.8, the minimum number of participants should be 16. We have added the explanation to the Methods section in lines 78-81:
“We estimated the number of participants conservatively based on the smallest amount of attentional modulation observed in our previous study (Doostani et al., 2023). For a medium effect size of 0.3 and a power of 0.8, we needed a minimum number of 16 participants.”
- Why did the authors choose those four categories? What was the evidence that would suggest these would span the range of similarities needed here?
We chose these four categories based on a previous behavioral study reporting the average reaction time of participants when detecting a target from one category among distractors from another category (Xu and Vaziri-Pashkam, 2019). Ideally the experiment should include as many object categories as possible. However, since we were limited by the duration of the experiment, the number of conditions had to be controlled, leading to a maximum of 4 object categories. We chose two animate and two inanimate object categories to include categories that are more similar and more different based on previous behavioral results (Xu and Vaziri-Pashkam, 2019). We included body and house categories because they are both among the categories to which highly responsive regions exist in the cortex. We chose the two remaining categories based on their similarity to body and house stimuli. In this way, for each category there was another category that elicited similar cortical responses, and two categories that elicited different responses. While we acknowledge that the chosen categories do not fully span the range of similarities, they provide an observable variety of similarities in different ROIs which we find acceptable for the purposes of our study.
We include this information in the Methods section of the revised manuscript in lines 89-94:
“We included body and house categories because there are regions in the brain that are highly responsive and unresponsive to each of these categories, which provided us with a range of responsiveness in the visual cortex. We chose the two remaining categories based on previous behavioral results to include categories that provided us with a range of similarities (Xu and Vaziri-Pashkam, 2019). Thus, for each category there was a range of responsiveness in the brain and a range of similarity with the other categories.”
- Why did the authors present the stimuli at the same location? This procedure has been adopted in previous studies, but of course, it does also move the stimulus situation away from the real-world examples of cluttered scenes that motivate the Introduction.
We presented the stimuli at the same location because we aimed to study the mechanism of object-based attention and this experimental design helped us isolate it from spatial attention. We do not think that our design moves the stimulus situation away from real-world examples in such a way that our results are not generalizable. We include real-world instances, as well as a discussion on this point, in the Discussion section of the revised manuscript, in lines 611-620:
“Although examples of superimposed cluttered stimuli are not very common in everyday life, they still do occur in certain situations, for example reading text on the cellphone screen in the presence of reflection and glare on the screen or looking at the street through a patterned window. Such instances recruit object-based attention which was the aim of this study, whereas in more common cases in which attended and unattended objects occupy different locations in space, both space-based and object-based attention may work together to resolve the competition between different stimuli. Here we chose to move away from usual everyday scenarios to study the effect of object-based attention in isolation. Future studies can reveal the effect of target-distractor similarity, i.e. proximity in space, on space-based attention and how the effects caused by object-based and space-based attention interact.”
- While I'm not concerned about this (all relevant comparisons were within-participants) was there an initial attempt to compare data quality from the two different scanners?
We compared the SNR values of the two groups of participants and observed no significant difference between these values (ps > 0.34, ts < 0.97). We have added this information to the Methods section.
Regarding the observed effect, we performed a t-test between the results of the participants from the two scanners. For the univariate results, the observed correlation between univariate attentional modulation and category distance was not significantly different for participants of the two scanners in any ROIs (ps > 0.07 , ts < 1.9). For the multivariate results, the observed correlation between the weight shift and multivariate category distance was not significantly different in any ROIs (ps > 0.48 , ts < 0.71) except for V1 (p-value = 0.015 , t-value = 2.75).
We include a sentence about the comparison of the SNR values in the preprocessing section in the revised manuscript.
e. There are a couple of analysis steps that could be applied to the existing data that might strengthen the findings. For one, the authors have adopted a liberal criterion of p < 0.001 uncorrected to include voxels within each ROI. Why, and to what extent is the general pattern of findings robust over more selective thresholds? Also, there are additional regions that are selective for bodies (fusiform body area) and scenes (occipital place area and retrosplenial cortex). Including these areas might provide more diversity of selectivity patterns (e.g. different responses to non-preferred categories) that would provide further tests of the hypothesis.
We selected this threshold to allow for selection of a reasonable number of voxels in each hemisphere across all participants. To check whether the effect is robust over more selective thresholds, we exemplarily redefined the left EBA region using p < 0.0001 and p < 0.00001 and observed that the weight shift effect remained equivalent. We have made a note of this analysis in the Results section. As for the additional regions suggested by the reviewer, we chose not to include them because they could not be consistently defined in both hemispheres of all participants. Please note that the current ROIs also show different responses to non-preferred categories (e.g. in LO and pFs). We include this information in the Methods section in lines 206-207:
“We selected this threshold to allow for selection of a reasonable number of voxels in each hemisphere across all participants.”
And in the Results section in lines 509-512:
“We performed the analysis including only voxels that had a significantly positive GLM coefficient across the runs and observed the same results. Moreover, to check whether the effect is robust over more selective thresholds for ROI definition, we redefined the left EBA region with p < 0.0001 and p < 0.00001 criteria. We observed a similar weight shift effect for both criteria.”
f. One point the authors might address is the potential effect of blocking the paired conditions. If I understood right, the irrelevant item in each paired display was from the same category throughout a block. To what extent might this knowledge shape the way participants attend to the task-relevant item (e.g. by highlighting to them certain spatial frequencies or contours that might be useful in making that particular pairwise distinction)? In other words, are there theoretical reasons to expect different effects if the irrelevant category is not predictable?
We believe that the participants’ knowledge about the distractor does not significantly affect our results because our results are in agreement with previous behavioral data (Cohen et al., 2014, Xu and Vaziri-Pashkam, 2019), in which the distractor could not be predicted. These reports suggest there is a theoretical reason to expect similar effects if the participants could not predict the distractor. To directly test this, one would need to perform an fMRI experiment using an event-related design, an interesting venue for future research.
We have made a note of this point in the Discussion section of the revised manuscript in lines 621-626:
“Please note that we used a blocked design in which the target and distractor categories could be predicted across each block. While it is possible that the current design has led to an enhancement of the observed effect, previous behavioral data (Cohen et al., 2014, Xu and Vaziri-Pashkam, 2019) have reported the same effect in experiments in which the distractor was not predictable. To study the effect of predictability on fMRI responses, however, an event-related design is more appropriate, an interesting venue for future fMRI studies.”
g. The authors could provide behavioural data as a function of the specific category pairs. There is a clear prediction here about which pairs should be more or less difficult.
We provide the behavioral data as a supplementary figure to Figure 1 in the revised manuscript. We however do not see differences in behavior for the different category paris. This is so because our fMRI task was designed in a way to make sure the participants could properly attend to the target for all conditions. The task was rather easy across all conditions and due to the ceiling effect, there was no significant difference between behavioral performance for different category pairs. However, the effect of category pair on behavior has been previously tested and reported in a visual search paradigm with the same categories (Xu and Vaziri-Pashkam, 2019), which was in fact the basis for our choice of categories in this study (as explained in response to point “d” above).
h. Figure 4 shows data for EBA in detail; it would be helpful to have a similar presentation of the data for the other ROIs as well.
We provide data for all ROIs as figure supplements 1-4 to Figure 4 in the revised manuscript.
i. For the pFs and LOC ROIs, it would be helpful to have an indication of what proportion of voxels was most/least responsive to each of the four categories. Was this a relatively even balance, or generally favouring one of the categories?
In LO, the proportion of voxels most responsive to each of the four categories was relatively even for Body (31%) and House (32%) stimuli, which was higher than the proportion of Car- and Cat-preferring voxels (18% and 19%, respectively). In pFs, 40% of the voxels were house-selective, while the proportion was relatively even for voxels most responsive to bodies, cars, and houses with 21%, 17%, and 22% of the voxels, respectively. We include the percentage of voxels most responsive to each of the four categories in each ROI as Appendix 1-table 1.
j. Were the stimuli in the localisers the same as in the main experiment?
No, we used different sets of stimuli for the localizers and the main experiment. We have added the information in line 146 of the Methods section.
Reviewer #2 (Recommendations For The Authors):
(1) Why are specific ROIs chosen? Perhaps some discussion motivating these choices, and addressing the possible overlap between these and retinotopic regions (based on other studies, or atlases - Wang et al, 2015) would be useful.
Considering that we used object categories, we decided to look at general object-selective regions (LO, pFS) as well as regions that are highly selective for specific categories (EBA, PPA). We also looked at the primary visual cortex as a control region. We have added this clarification in the Methods section lines 128-133:
“Considering that we used object categories, we investigated five different regions of interest (ROIs): the object-selective areas lateral occipital cortex (LO) and posterior fusiform (pFs) as general object-selective regions, the body-selective extrastriate body area (EBA) and the scene-selective parahippocampal place area (PPA) as regions that are highly selective for specific categories, and the primary visual cortex (V1) as a control region. We chose these regions because they could all be consistently defined in both hemispheres of all participants and included a large number of voxels.”
(2) The authors should consider including data on the relative prevalence of voxels preferring each category for each ROI (and/or the mean activation level across voxels for each category for each ROI). If some ROIs have very few voxels preferring some categories, there's a chance the observed results are a bit noisy when sorting based on those categories (e.g., if a ROI has essentially no response to a given pair of categories, then there's not likely to be much attentional modulation detectable, because the ROI isn't driven by those categories to begin with).
We thank the reviewer for the insightful comment.
We include the percentage of voxels most responsive to each of the four categories in each ROI in the Appendix (Appendix 1-table 1, please see the answer to point “i” of the first reviewer).
We also provide a table of average activity across voxels for each category in all ROIs as Appendix 1-table 2.
As shown in the table, voxels show positive activity for all categories in all ROIs except for PPA, where voxels show no response to body and cat stimuli. This might explain why we observed a marginally significant correlation between weight shift and category distance in PPA only. As the reviewer mentions, since this region does not respond to body and cat stimuli, we do not observe a significant change in response due to the shift in attention for some pairs. We include the table in the Appendix and add the explanation to the Results section of the revised manuscript in lines 506-508:
_“_Less significant results in PPA might arise from the fact that PPA shows no response to body and cat stimuli and little response to car stimuli (Appendix 1-table 2). Therefore, it is not possible to observe the effect of attention for all category pairs.”
a. Related - would it make sense to screen voxels for inclusion in analysis based on above-basely activation for one or both of the categories? [could, for example, imagine you're accidentally measuring from the motor cortex - you'd be able to perform this analysis, but it would be largely nonsensical because there's no established response to the stimuli in either isolated or combined states].
We performed all the analyses including only voxels that had a significantly positive GLM coefficient across the runs and the results remained the same. We have added the explanation in the Results section in line 509-510.
(3) Behavioral performance is compared against chance level, but it doesn't seem that 50% is chance for the detection task. The authors write on page 4 that the 1-back repetition occurred between 2-3 times per block, so it doesn't seem to be the case that each stimulus had a 50% chance of being a repetition of the previous one.
We apologize for the mistake in our report. We have reported the detection rate for the target-present trials (2-3 per block), not the behavioral performance across all trials. We have modified the sentence in the Results section.
(4) Authors mention that the stimuli are identical for 2-stimulus trials where each category is attended (for a given pair) - but the cue is different, and the cue appears as a centrally-fixated word for 1 s. Is this incorporated into the GLM? I can't imagine this would have much impact, but the strict statement that the goals of the participant are the only thing differentiating trials with otherwise-identical stimuli isn't quite true.
The word cue was not incorporated as a separate predictor into the GLM. As the reviewer notes, the signals related to the cue and stimuli are mixed. But given that the cues are brief and in the form of words rather than images, they are unlikely to have an effect on the response in the regions of interest.
To be more accurate, we have included the clarification in the Methods section in lines 181-182:
“We did not enter the cue to the GLM as a predictor. The obtained voxel-wise coefficients for each condition are thus related to the cue and the stimuli presented in that condition.”
And in the Results section in lines 425-428 :
“It is important to note that since the cue was not separately modeled in the GLM, the signals related to the cue and the stimuli were mixed. However, given that the cues were brief and presented in the form of words, they are unlikely to have an effect on the responses observed in the higher-level ROIs.”
(5) Eq 5: I expected there to be some comparison of a and b directly as ratios (e.g., a_1 > b_1, as shown in Fig. 2). The equations used here should be walked through more carefully - it's very hard to understand what this analysis is actually accomplishing. I'm not sure I follow the explanation of relative weights given by the authors, nor how that maps onto the delta_W quantity in Equation 5.
We provide a direct comparison of a and b, as well as a more thorough clarification of the analysis, in the Methods section in lines 274-276:
“We first projected the paired vector on the plane defined by the isolated vectors (Figure 2A) and then determined the weight of each isolated vector in the projected vector (Figure 2B).”
And in lines 286-297:
“A higher a1 compared to a2 indicates that the paired response pattern is more similar to Vxat compared to Vyat, and vice versa. For instance, if we calculate the weights of the Body and Car stimuli in the paired response related to the simultaneous presentation of both stimuli, we can write in the LO region: VBodyatCar = 0.81 VBody + 0.31 VCar, VBodyCarat = 0.43 VBody + 0.68 VCar. Note that these weights are averaged across participants. As can be observed, in the presence of both body and car stimuli, the weight of each stimulus is higher when attended compared to the case when it is unattended. In other words, when attention shifts from body to car stimuli, the weight of the isolated body response (VBody) decreases in the paired response. We can therefore observe that the response in the paired condition is more similar to the isolated body response pattern when body stimuli are attended and more similar to the isolated car response pattern when car stimuli are attended.”
And lines 303-306:
“As shown here, even when body stimuli are attended, the effect of the unattended car stimuli is still present in the response, shown in the weight of the isolated car response (0.31). However, this weight increases when attention shifts towards car stimuli (0.68 in the attended case).”
We also provide more detailed clarification for the 𝛥w and the relative weights in lines 309-324:
“To examine whether this increase in the weight of the attended stimulus was constant or depended on the similarity of the two stimuli in cortical representation, we defined the weight shift as the multivariate effect of attention:
𝛥w = a1/(a1+a2) – b1/(b1+b2) (5)
Here, a1, a2, b1,and b2 are the weights of the isolated responses, estimated using Equation 4. We calculate the weight of the isolated x response once when attention is directed towards x (a1), and a second time when attention is directed towards y (b1). In each case, we calculate the relative weight of the isolated x in the paired response by dividing the weight of the isolated x by the sum of weights of x and y (a1+a2 when attention is directed towards x, and b1+b2 when attention is directed towards y). We then define the weight shift, Δw, as the change in the relative weight of the isolated x response in the paired response when attention shifts from x to y. A higher Δw for a category pair indicates that attention is more efficient in removing the effect of the unattended stimulus in the pair. We used relative weights as a normalized measure to compensate for the difference in the sum of weights for different category pairs. Thus, using the normalized measure, we calculated the share of each stimulus in the paired response. For instance, considering the Body-Car pair, the share of the body stimulus in the paired response was equal to 0.72 and 0.38, when body stimuli were attended and unattended, respectively. We then calculated the change in the share of each stimulus caused by the shift in attention using a simple subtraction (Equation 5: Δw=0.34 for the above example of the Body-Car pair in LO) and used this measure to compare between different pairs.”
We hope that this clarification makes it easier to understand the multivariate analysis and the weight shift calculation in Equation 5.
We additionally provide the values of the weights (a1, b1, a2, and b2) for each category pair averaged across participants as Appendix 1 -table 4.
(6) For multivariate analyses (Fig. 6A-E), x axis is normalized (pattern distance based on Pearson correlation), while the delta_W does not seem to be similarly normalized.
We calculated ΔW by dividing the weights in each condition by the sum of weights in that condition. Thus, we use relative weights which are always in the range of 0 to 1, and ΔW is thus always in the range of -1 to 1. This means that both axes are normalized. Note that even if one axis were not normalized, the relationship between the independent and the dependent variables would remain the same despite the change in the range of the axis.
(7) Simulating additional scenarios like attention to both categories just increasing the mean response would be helpful - is this how one would capture results like those shown in some panels of Fig. 4?
We did not have a condition in which participants were asked to attend to both categories. Therefore it was not useful for our simulations to include such a scenario. Please also note that the goal of our simulations is not to capture the exact amount of attentional modulation, but to investigate the effect of target-distractor similarity on the change in attentional modulation (univariate shift and weight shift).
As for the results in some panels of Figure 4, we have explained the reason underlying higher responses in paired conditions compared to isolated conditions in response to the “weaknesses” section of the second reviewer. We hope that these points satisfy the reviewer’s concern regarding the results in Figure 4 and our simulations.
(8) Lines 271-276 - the "latter" and "former" are backwards here I think.
We believe that the sentence was correct, but confusing.. We have rephrased the sentence to avoid the confusion in lines 371-376 of the revised manuscript:
“We modeled two neural populations: a general object-selective population in which each voxel shows preference to a particular category and voxels with different preferences are mixed in with each other (similar to LO and pFS), and a category-selective population in which all voxels have a similar preference for a particular category (similar to EBA and PPA).”
(9) Line 314 - "body-car" pair is mentioned twice in describing the non-significant result in PPA ROI.
Thank you for catching the typo. We have changed the second Body-Car to Body-Cat.
(10) Fig. 5 and Fig. 6 - I was expecting to see a plot that demonstrated variability across subjects rather than across category pairs. Would it be possible to show the distribution of each pair's datapoints across subjects, perhaps by coloring all (e.g.) body-car datapoints one color, all body-cat datapoints another, etc? This would also help readers better understand how category preferences (which differ across ROIs) impact the results.
We demonstrated variability across category pairs rather than subjects because we aimed to investigate how the variation in the similarity between categories (i.e. category distance) affected the univariate and multivariate effects of attention. The variability across subjects is reflected in the error bars in the bar plots of Figure 5 and Figure 6.
Here we show the distribution of each category pair’s data points across subjects by using a different color for each pair:
Univariate shift versus category distance including single-subject data points in all ROIs.
Weight shift versus category distance including single-subject data points in all ROIs.
As can be observed in the figures, category preference has little impact on the results. Rather, the similarity in the preference (in the univariate case) or the response pattern (in the multivariate case) to the two presented categories is what impacts the amount of the univariate shift and the weight shift, respectively. For instance, in EBA we observe a low amount of attentional shift both for the Body-Cat pair, with two stimuli for which the ROI is highly selective, and the Car-House pair, including stimuli to which the region shows little response. A similar pattern is observed in the object-selective regions LO and pFs which show high responses to all stimulus categories.
We believe that the figures including the data points related to all subjects are not strongly informative. However, we agree that using different colors for each category pair helps the readers better understand that category preference has little impact on the results in different ROIs. We therefore present the colored version of Figure 5 and Figure 6 in the revised manuscript, with a different color for each category pair.
(11) Fig. 5 and Fig. 6 use R^2 as a dependent variable across participants to conclude a positive relationship. While the positive relationship is clear in the scatterplots, which depict averages across participants for each category pair, it could still be the case that there are a substantial number of participants with negative (but predictive, thus high positive R^2) slopes. For completeness and transparency, the authors should illustrate the average slope or regression coefficient for each of these analyses.
We concluded the positive relationship and calculated the significance in Figure 5 and Figure 6 using the correlation r rather than r.^2 This is why the result was not significantly positive in V1. We acknowledge that the use of r-squared in the bar plot leads to confusion. We have therefore changed the bar plots to show the correlation coefficient instead of the r-squared. Furthermore, we have added a table of the correlation coefficient for all participants in all ROIs for the univariate and weight shift analyses supplemental to Figure 5 and Figure 6, respectively.
(12) No statement about data or analysis code availability is provided
Thanks for pointing this out. The fMRI data is available on OSF. We have added a statement about it in the Data Availability section of the revised manuscript in line 669.