One of the main use cases of our tools is the volumetric analysis of different cerebral regions of interest (ROIs) from dissection photographs, without requiring cadaveric or ex vivo MRI. We used our tools to analyze 21 post mortem confirmed Alzheimer’s disease (AD) cases from the Massachusetts Alzheimer’s Disease Research Center (MADRC), as well as 12 age-matched controls (N = 33 in total). We note that these cases comprise thick (∼10 mm) slabs, sliced by hand, without cutting guides – as cutting on anatomic landmarks was prioritized over consistent slice thickness. Therefore, this dataset is representative of a challenging, real-world scenario.

Examples of the inputs and outputs of the pipeline can be found in Figure 1 and Video 1 in the supplementary material (also available at https://youtu.be/wo5meYRaGUY). The 3D surface scan (Figure 1a) reveals the coarse shape of the specimen, in this case a left hemisphere. Our preprocessing tools correct for the perspective and pixel size of routine dissection photographs (b). Then, the 3D reconstruction tool uses information from the surface scan to produce a 3D reconstruction of the photographs into an imaging volume (c). This volume consists of highly anisotropic voxels, since the slice thickness is much larger than the pixel size of the photograph. The 3D reconstruction is fed to our machine learning segmentation method (Photo-SynthSeg), which produces a high-resolution, isotropic segmentation (d, e), independently of the slice thickness. The segmentations are then used to compare the volumes of brain ROIs between the two groups (e.g., the hippocampus, as in Figure 1f).

Table 1 shows the area under the receiver operating characteristic curve (AUROC) and the p-value for non-parametric Wilcoxon rank sum tests (Mann and Whitney, 1947) comparing the ROI volumes of AD vs controls; we leave the accumbens area and ventral diencephalon out of the analysis as their segmentations are not reliable due to poor contrast (Fischl et al., 2002). We note that the AUROC is the non-parametric equivalent of the effect size; a value of 0.5 represents chance, while 1.0 represents perfect separation. Age and gender were corrected with a general linear model, whereas volumes of contralateral ROIs were averaged when full brains (rather than hemispheres) were available. Our method successfully captures well-known atrophy patterns of AD, such as hippocampal atrophy and ventricle enlargement.

While the AD experiment illustrates the ability of our method to detect differences in real-world data, it is crucial to specifically assess the accuracy of our segmentation and 3D reconstruction methods. To evaluate Photo-SynthSeg, we 3D reconstructed the brain volumes from the photographs of 24 cases from the AD Research Center at the University of Washington (UW-ADRC), which were cut into uniform 4 mm thick slices. Crucially, isotropic FLAIR MRI scans were acquired for these specimens ex vivo prior to dissection, which enables the use of MRI as gold standard.

We compare Photo-SynthSeg against ‘SAMSEG’, which is (to the best of our knowledge) the only available competing method. SAMSEG is a segmentation algorithm which we originally conceived for brain MRI (Puonti et al., 2016), and which employs Bayesian techniques to segment MRI scans irrespective of their contrast and pulse sequence. As we showed in Tregidgo et al., 2020, this technique can be adapted to 3D reconstructed photographs. Figure 2 shows the segmentation of an UW-ADRC case using both methods. As opposed to SAMSEG, Photo-SynthSeg effectively ‘interpolates’ the segmentation in between slices, independently of their thickness, and is more robust against uneven intensities than the Bayesian method. Photo-SynthSeg also includes a volumetric parcellation of the cortex based on the original SynthSeg pipeline (Billot et al., 2023a), which relies on the Desikan–Killiany atlas (Desikan et al., 2006).

To evaluate Photo-SynthSeg and SAMSEG quantitatively, we computed Dice scores against manual segmentations made on a single selected slice per subject. This slice is visually chosen to be close to the mid-coronal plane, while maximizing visibility of subcortical structures; an example of such slice, along with the manual and automated segmentations, is shown in Appendix 1—figure 3. The Dice scores are displayed in Figure 3; as in the previous analysis, the accumbens area and ventral diencephalon are left out of the analysis. The figure also shows two ablations: using a probabilistic atlas instead of the case-specific reference; and using a version of Photo-SynthSeg dedicated to 4 mm slice thickness (i.e., the thickness of the UW-ADRC dataset, rather than using the general, thickness-agnostic model). The former assesses the impact of having to rely on a generic atlas when surface scans are not available, whereas the latter evaluates the ability of our neural network to adapt to thicknesses that are not known a priori (thanks to our domain randomization approach). The results show that Photo-SynthSeg generally outperforms SAMSEG, producing Dice scores over 0.8 for all structures except the amygdala – even though SAMSEG is better at segmenting the ventricles, thanks to their large size and strong contrast. The plots also show that using the probabilistic atlas produces realistic enough reconstructions, such that the two-dimensional (2D) Dice scores remain high. Finally, the results also show the superiority of the thickness randomization strategy over the fixed spacing strategy, even if the latter had access to the ground truth spacing of the photographs. Even though the reader may initially find this result counterintuitive, it is consistent with our previous findings in MRI segmentation, where domain randomization strategies also outperformed targeted simulations (Billot et al., 2020).

This evaluation with Dice scores above is direct, but: (1) is based on a relatively small number of slices, and (2) disregards the ultimate purpose of segmentation, which is downstream analysis in 3D (e.g., volumetry). For this purpose, we also indirectly evaluated the methods by analyzing the volumes of brain ROIs derived from the segmentations of the whole stack. Specifically, we correlated these volumes with silver standard values derived from the isotropic FLAIR MRI scans using our ‘standard’ SynthSeg for MRI (Billot et al., 2023a). Table 2 shows the correlations and p-values for Steiger tests comparing the correlation coefficients achieved by SAMSEG and Photo-SynthSeg – while considering their dependency due to the common ground truth sample. The results show once more that Photo-SynthSeg outperforms SAMSEG for nearly every structure – and in the few cases in which it does not, the Steiger test does not yield statistically significant differences. We note that the correlations are above 0.8 for most brain structures, indicating that the 3D reconstructed photographs yield usable volumes in volumetric analysis. We also note that the correlations are slightly but consistently lower for the reconstructions with the probabilistic atlas, yielding correlations close to 0.8 for most brain regions.

In addition to the segmentation, it is also desirable to evaluate the 3D reconstruction algorithm with the registration error (in mm). Measuring such error with real data would require manual annotation of pairs of matching landmarks, which is labor-intensive, error-prone, and not reproducible. Instead, we use simulated (digitally sliced) data created from MRI scans. While errors estimated this way may be optimistic compared with real sliced tissue, this approach enables us to analyze the error as a continuous function of slice thickness without manual annotation effort.

For this purpose, we used 500 subjects from the Human Connectome Project (HCP) dataset, which includes T1- and T2-weighted MRI scans acquired at 0.7 mm isotropic resolution. After skull stripping with FreeSurfer (Fischl, 2012), we simulated dissection photographs and matching surface scans by: (1) digitally slicing the T2 scans and (2) using the surface of the T1 as a 3D reference to reconstruct the T2 slices. Specifically, we simulated slices with S×0.7 mm thickness (S=2,4,8,16), including random affine transforms and illumination fields for every simulated slice (see example in Appendix 1—figure 1). While we could use a nonlinear model, the results would depend heavily on the strength of the simulated deformation. Instead, we keep the warps linear as we believe that the value of this experiment lies in the trends that the errors reflect, that is, their relative rather than absolute value.

After digitally distorting the images, we used our method to 3D reconstruct the slices into their original shape. The registration error was calculated as the mean voxel displacement in mm between the reconstructed and ground truth T2 slices. Additionally, to test the robustness of the reconstruction algorithm to uneven slice spacing during dissection, we also analyze the error when a random variation of the nominal thickness (thickness jitter) is introduced for every slice.

Figure 4 shows the box plot for the mean reconstruction error as a function of the slice spacing and thickness jitter. The results show that the reconstruction error is reasonably robust to increased slice spacing. On the other hand, thickness jitter yields greater increases in reconstruction error, particularly at larger slice spacings. This result highlights the importance of keeping the slice thickness as constant as possible during acquisition.

Surface scanning (also known as ‘profilometry’) is a technology that is becoming increasingly inexpensive (a few thousand dollars), mainly via structured light technology (Salvi et al., 2004). The preferred version of our proposed pipeline relies on a surface scan of the specimen (Figure 1a) acquired before slicing. This surface scan, represented by a triangular mesh, is used as an external reference to guide the 3D reconstruction of the photographs (details below). While there are no specific technical requirements for the surface scan (e.g., minimum resolution), minimizing geometric distortion (i.e., deformation) between scanning and subsequent slicing is crucial. The surface scan may be acquired with a handheld scanner with the specimen placed on a table, or with the scanner on a tripod and the sample on a turntable. As mentioned earlier, our pipeline does not strictly require the surface scan, as it can be replaced with a probabilistic atlas with a slight loss of accuracy (as shown in the Results section).

In preparation for the downstream 3D reconstruction, some requirements exist for the dissection and photography of the brain slabs. Specifically, our pipeline assumes that approximately parallel slices are cut in coronal direction – which is the standard practice in most brain banks. We further assume that the thickness of these slices is approximately constant, as ‘thickness jitter’ has a detrimental effect on the results (Figure 4).

Moreover, we require the presence of fiducials to enable pixel size calibration and perspective correction. Ideally, four fiducials are placed on the corners of a rectangle of known dimensions (Figure 5a). In the absence of such fiducials, we require the presence of at least a ruler or two orthogonal rulers; the former enables pixel size calibration via image scaling, whereas the latter enables approximate perspective correction by fitting an affine transform.

Our pipeline allows for multiple slices to be present in one photograph but requires that all photographs are of the same side of the slices (either anterior or posterior) and that the inferior–superior direction of the anatomy is approximately aligned with the vertical axis of the image (as in Figure 5a). Ideally, the slices should be photographed on a flat board with a solid background color that stands out from the brain tissue, as stronger contrast between tissue and background greatly facilitates image segmentation when preprocessing the photographs (further details below).

Image preprocessing starts with geometric correction, that is, pixel size calibration and perspective correction. In the ideal scenario with four fiducials, the widespread widespread scale-invariant feature transform (SIFT, Lowe, 1999) is used to detect such fiducials (Figure 5b) and compute a spatial transform that corrects for perspective distortion and calibrates the pixel size – see Figure 5c, where the pixel size calibration enables superimposition of a digital ruler.

Our software also supports a manual mode where the user clicks on two, three, or four landmarks: two points with a known distance in between (enables approximate pixel size calibration); three points at the ends of two rulers plus their intersection (enables approximate perspective correction with an affine transform); or four points on the corners of a rectangle of known dimensions (enables full perspective correction). This manual mode is useful when no fiducials are present, but the user can still identify features with known dimensions in the photograph, for example, on an imprinted grid pattern, or along rulers.

After geometric correction, our methods require a binary segmentation of the brain tissue, separating it from the background. While our tools do not have any specific requisites in terms of background, using a solid background with a distinct color (e.g., a ‘green screen’) can greatly facilitate segmentation; otherwise, more extensive manual intervention may be needed. In our experiments, using a flat black background enabled us to automatically segment the tissue with a combination of thresholding and morphological operations, keeping manual edits to a minimum (a couple of minutes per photograph, mostly to erase bits of cortical surface that are sometimes visible around the edge of the face of the slice).

Given this binary mask, the final preprocessing step requires the user to specify the order of the slices within the photograph – which may be anterior to posterior or vice versa, but must be consistent within and across photographs of the same case. Moreover, the user also needs to specify whether two connected components belong to the same slice, which often happens around the temporal pole. This can be quickly accomplished with a dedicated graphical user interface that we distribute with our tools (Figure 5d).

Recovering a 3D volume from a stack of 2D images entails a consistent 3D reconstruction of the stack via joint image alignment – known as ‘registration’ (Maintz and Viergever, 1998; Pluim et al., 2003; Zitová and Flusser, 2003; Sotiras et al., 2013). We pose 3D reconstruction as a joint optimization problem (Pichat et al., 2018; Mancini et al., 2019) and use a 3D reference volume for the registration. This volume is ideally a binary 3D mask obtained by rasterizing (filling) the triangular mesh from the surface scan. If a surface scan is not available, one can instead use a probabilistic atlas (Shattuck et al., 2008) of brain shape, which provides a rough reference for the reconstruction. However, this reference cannot drive the reconstruction toward the actual shape of the specimen nor correct deviations from the user-provided slice thickness.

Given N photographed slices and their corresponding masks, the goal of this optimization framework is to simultaneously identify: (1) a set of N 2D affine geometric transforms {ϕn}n=1,…,N (rotation, translation, shear, and scaling for each slice); (2) a scaling factor s in the anterior–posterior direction shared by all slices; and (3) a rigid 3D transform Ψ for the reference (rotation and translation). These transforms seek to align the slices with each other and with the reference volume.

We note that affine transforms (rather than rigid) are required for the photographs due to imperfections in the image preprocessing of the previous section; nevertheless, we expect the shear and scaling of these transforms to be small. Furthermore, we use affine rather than nonlinear transforms because the latter compromise the robustness of the registration, as they introduce huge ambiguity in the space of solutions (e.g., one could add an identical small nonlinear deformation to every slice almost without changing the feature of the 3D reconstruction). This affine model makes it particularly important to place connected components of the same slice in a correct relative position when there is more than one, for example, in the temporal pole. We further note that the scaling s in the anterior–posterior direction is required to correct deviations from the slice thickness specified by the user, which in practice is never completely exact.

The optimal set of transforms is obtained by maximizing the objective function F in Equation 1. This objective encodes four desired attributes of the reconstructed data, with relative weights α, β, γ, and υ:

(1)F(Ψ,{Φn})=αD(M[x;{Φn},s],R[x;Ψ])+β1N−1∑n=1N−1C(Sn[x;Φn,s],Sn+1[x;Φn+1,s])+γ1N−1∑n=1N−1D(Mn[x;Φn,s],Mn+1[x;Φn+1,s])−ν1N∑n=1Nf(Φn)

Mathematically, overlap in Equation 1 is measured with the soft Dice coefficient D (Dice, 1945; Sorensen, 1948; Milletari et al., 2016); image similarity is measured with the normalized cross-correlation C; and the scaling and shearing are measured with the absolute value of the logarithm of the determinant of the affine matrices, that is, f(Φn) is the absolute log-determinant of the 3 × 3 matrix encoding Φn. The relative weights α,β,γ,ν are set via visual inspection of the output on a small pilot dataset. For the surface reference, we used: α=.95,β=γ=.025,ν=γ/100. For the probabilistic atlas, we trust the reference less and also use more regularization to prevent excessive deformation of slices: α=.8,β=γ=ν=.1. Either way, the 3D reconstruction is in practice not very sensitive to the exact value of these parameters. We also note that, as opposed to the preprocessing described in the previous section, SIFT is not a good candidate for matching consecutive slices: while it is resilient against changes in pose (e.g., object rotation), perspective, and lightning, it is not robust against changes in the object itself – such as changes between one slice to the next.

The objective function F is minimized with standard numerical methods – specifically the limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) algorithm (Fletcher, 1987). The LBFGS optimizer is initialized by stacking the photographs with their centers of gravity on coordinate (0,0), and then matching the center of gravity of the whole stack with the center of gravity of the 3D reference (as illustrated in Video 1 in the supplement, see also https://youtu.be/wo5meYRaGUY).

If a probabilistic atlas is used instead of the surface scan, the same objective function is used but with a slightly different search space. Since we can no longer trust the external reference to correct for fine shape correction: (1) we keep the scaling factor of the anterior–posterior direction fixed to s=1 (i.e., we trust the slice thickness specified by the user); (2) we use rigid rather than affine transforms {ϕn} for the slices (which also has the effect of making the regularizer equal to zero); and (3) we use a full 3D affine transform Ψ for the reference. Sample reconstructions of a case with a 3D surface and a probabilistic atlas are shown in Figure 5e–g. The probabilistic atlas produces a plausible reconstruction, which is however far from the real shape of the specimen given by the surface (Figure 5g). An additional example from the MADRC dataset is shown in Appendix 1—figure 2. We note that 3D reconstruction is implemented in PyTorch, and runs efficiently on a GPU – less than 5 min on a Nvidia Quadro RTX 6000.

The 3D reconstructed photographs have various applications (e.g., volumetry, computationally guided dissection) that require image segmentation, that is, assigning neuroanatomical labels to every spatial location (Pham et al., 2000; Despotović et al., 2015; Akkus et al., 2017). There are two main challenges when segmenting the 3D reconstructed photographs. First, the appearance of the images varies widely across cases due to differences in camera hardware (sensor, lens), camera settings, illumination conditions, and tissue preparation (e.g., fixed vs fresh). And second, accurate volumetry requires estimating the segmentation not only on the planes of the photographs but also in between slices.

In our previous work (Tregidgo et al., 2020), we adopted a Bayesian segmentation strategy that handled the first issue but not the second. Here, we extend a machine learning approach based on domain randomization (Tobin et al., 2017) that we have successfully applied to segment clinical brain MRI scans with large slice spacing (Billot et al., 2020; Billot et al., 2023a; Billot et al., 2023b). Specifically, our newly proposed approach ‘Photo-SynthSeg’ trains a convolutional neural network for image segmentation (a 3D U-Net, Ronneberger et al., 2015; Çiçek et al., 2019) with synthetic data as follows.

‘Photo-SynthSeg’ starts from a training dataset (the T1-39 dataset) that comprises a pool of 3D segmentations of brain images at isotropic 3D resolution (Figure 6a). At every iteration during training, one of these 3D segmentations is randomly selected and geometrically deformed with a random nonlinear transform, which simulates imperfect 3D reconstruction by including a ‘deformation jitter’ in the coronal direction, that is, small but abrupt deformations from one coronal slice to the next (Figure 6b). This deformed isotropic segmentation is used to generate a synthetic 3D image (also at isotropic resolution) using a Gaussian mixture model conditioned on the segmentation (Figure 6c). Crucially, we randomize the Gaussian parameters (means and variances) to make the neural network robust against variations in image appearance. This synthetic isotropic image is then ‘digitally sliced’ into a 3D stack of synthetic coronal photographs, each of which is independently corrupted by a random 2D smooth illumination field that simulates inhomogeneities in the illumination of the photographs (Figure 6d). Importantly, we also randomize the thickness of the simulated slices in every iteration and introduce small stochastic variations in thickness around the central value within each simulation. This strategy makes the network agnostic to the slice thickness of the reconstructed 3D stack at test time. Finally, the simulated stack, which has anisotropic resolution (e.g., 1 mm in-plane and several mm in slice thickness), is resampled to the original isotropic resolution of the (deformed) 3D segmentation (Figure 6e) – typically 1 mm isotropic.

This generative process mimics the image formation process in the real world and has two crucial aspects. First, the super-resolution component: Photo-SynthSeg is a U-Net that will produce a high-resolution (1 mm) isotropic 3D segmentation for every input at test time, independently of the slice spacing of the 3D reconstructed stack of photographs (Figure 2). And second, domain randomization: sampling different Gaussian parameters, illumination fields, and slice thicknesses at every iteration beyond realistic limits (as in Figure 6, where even contralateral regions have different appearance) forces the U-Net to learn features that are independent of the intensity profiles of the photographs, as well as of the spacing between slices. This process makes the U-Net robust against changes in the acquisition. We note that the Gaussian distributions are univariate rather than trivariate, that is, they model grayscale rather than red–green–blue triplets; we tried training a U-Net with the latter, but the performance on color images was worse than when converting them to grayscale and using the U-Net trained with univariate Gaussians.

During training, resampled stacks and corresponding segmentations (Figure 6b–e) are fed to a U-Net. The U-Net architecture is the same as in our previous works with synthetic scans (Billot et al., 2020; Billot et al., 2023a): it consists of five levels, each separated by a batch normalization layer (Ioffe and Szegedy, 2015) along with a max-pooling (contracting path) or an upsampling operation (expanding path). All levels comprise two convolution layers with 3 × 3 × 3 kernels. Every convolutional layer is associated with an Exponential Linear Unit activation (Clevert et al., 2016), except for the last one, which uses a softmax. While the first layer comprises 24 feature maps, this number is doubled after each max-pooling, and halved after each upsampling. Following the original U-Net architecture, skip connections are used across the contracting and expanding paths. The network is trained with a soft Dice loss (Milletari et al., 2016) and the Adam optimizer (Kingma and Ba, 2014). The deep learning model is implemented in Keras (Chollet, 2015) with a Tensorflow backend (Abadi et al., 2016) and runs on a few seconds on a Nvidia Quadro RTX 6000 GPU. Training takes around 7 days on the same GPU.

Finally, we note that there are two different versions of Photo-SynthSeg: one for full cerebra and one for single hemispheres. The later is trained with left hemispheres and flipped right hemispheres, and can thus be used as is for left hemispheres. To process a right hemisphere, we simply left–right flip it, segment it, and flip the results back. While the default Photo-SynthSeg pipeline segments the cerebral cortex as a whole (i.e., as in Figure 6a), our tool also offers the option of subdividing it into parcels as defined by the Desikan–Killiany atlas (Desikan et al., 2009), like in Figure 2. This is achieved with the cortical parcellation module (Segmenter S3) of our tool ‘SynthSeg’ (Billot et al., 2023a), which is also distributed with FreeSurfer.