The collected data was analyzed to evaluate the prevalence of the use of SDEs in preclinical cardiovascular research between 2011 and 2021. Categorical variables are reported as a number (%). RStudio and Microsoft Excel software were used to perform statistical significance comparisons. A median-based linear model analysis was performed to analyze changes in SDE inclusion over time, and nonparametric Kruskal-Wallis tests were performed to analyze differences in SDE reporting across journals. Chi-squared tests were performed for other analyses to assess differences in SDE reporting across experimental models, between animals vs humans within the same study, and across different animal models. A threshold of p < 0.05 was considered statistically significant, however, we used the Holm-Bonferroni correction to correct for multiple comparisons.

This study was pre-registered in the Open Science Framework (OSF) registry. The preregistration can be accessed via the following link: https://doi.org/10.17605/OSF.IO/F4NH9 (Patel et al., 2022). We adhered to the methodology detailed in the pre registration for this analysis.

The data and analytical methods used for this study are available at https://osf.io/52Q6W/ (Zahiri et al., 2022.)

Table III shows the proportion of studies that included each of the four SDEs, stratified by animal-only studies, as well as animal and human substudies in human/animal studies. In animal-only studies, both sexes were used in 15.2% (N=28) studies, single-sex was used in 48.4% (N=89) studies, and there was no reporting on the sex of study subjects used in 36.4% (N=67) studies. In animal substudies in human/animal studies, both sexes were used in 17.5% (N=20) studies, single-sex was used in 53.5% (N=61) studies, and there was no reporting on the sex of study subjects used in 29.0% (N=33) studies. In human substudies in human/animal studies, both sexes were used in 36% (N=41) of studies, single-sex was used in 8.8% (N=10) of studies, and there was no reporting on the sex of study subjects used in 55.2% (N=63) studies.

In terms of randomization, 30.4% (N=56) of animal-only studies, 36% (N=41) of animal substudies in human/animal studies, and 6% (N=7) of human substudies in human/animal studies used randomization to any degree in their experiments. In terms of blinding, 32.1% (N=59) of animal-only studies, 24.5% (N=28) of animal substudies in human/animal studies, and 11.4% (N=13) of human substudies in human/animal studies used blinding to any degree in their experiments.

In terms of sample size estimations for animal-only studies, 8.2% (N=15) used statistical analysis to determine the sample size for experiments, 4.9% (N=9) provided other justification for the sample size they selected, 2.7% (N=5) indicated that no sample size estimation was done, and 84.2% (N=155) did not report sample size justification. Examples of ‘other justification for sample size selection’ include estimating sample size based on pilot studies or determining sample size based on previous standards in the field. In terms of sample size estimations for animal substudies in human/animal studies, 7% (N=8) used statistical analysis to determine the sample size for experiments, 2.6% (N=3) provided other justification for the sample size they selected, 5.3% (N=6) indicated that no sample size estimation was done, and 85.1% (N=97) did not report any sample size justification. Lastly, in terms of sample size estimations for human substudies in human/animal studies, 1.8% (N=2) used statistical analysis to determine the sample size for experiments, 12.3% (N=14) provided other justification for the sample size they selected, 2.6% (N=3) indicated that no sample size estimation was done, and 83.3% (N=95) did not report any sample size justification.

A median-based linear analysis of SDE inclusion between 2011 and 2021 revealed no statistically significant difference in the proportion of studies including animals of both biological sexes between 2011 and 2021 (corrected p=1.476) (Figure 2). Of studies screened from 2011, 10% (N=3 of 29) included animals of both biological sexes. Of studies screened from 2017, 27% (N=9 of 33) included animals of both biological sexes. This proportion ultimately decreases to 10% (N=2 of 21) of studies in 2021 (Figure 2). Although there appears to be an overall general increase in the proportion of studies including animals of both biological sexes across the decade of interest, no statistically significant difference was found (corrected p>0.05).

Similarly, a median-based linear analysis of randomization implementation revealed no statistically significant difference in the proportion of studies implementing randomization between 2011 and 2021 (corrected p=1.617). Approximately half of preclinical cardiovascular studies implemented randomization in 2011, as 48% (N=14 of 29) of 2011 studies mentioned randomization in their methods. This proportion declined to 29% (N=6 of 21) of studies in 2021 (Figure 2). Although there appears to be an overall general decrease in the proportion of studies including randomization across the decade of interest, no statistically significant decrease was found (corrected p>0.05)

Interestingly, a median-based linear analysis of blinding implementation also showed no statistically significant difference in the proportion of studies including blinding as part of the study design between 2011 and 2021 (corrected p=0.335) (Figure 2). Of studies screened from 2011, 24% (N=7 of 29) included blinding. This proportion decreased to 19% (N=4 of 21) in 2021 (Figure 2). Likewise, comparable results were found for sample size justification (corrected p=0.390) (Figure 2). Of studies screened from 2011, 14% (N=4 of 29) justified their sample size using size estimation (justification or statistical estimation). This proportion dropped to 0% (N=0 of 22) studies in 2018 but ultimately increased back up to 14% (N=3 of 21) in 2021 (Figure 2). Again, although there appears to be an overall general decrease in the proportion of studies including blinding and sample size justification as part of the study design, a median-based linear model ultimately revealed no statistically significant decrease across the decade of interest (corrected p>0.05).

Of all 298 journal articles screened in this study, the four journals with the highest numbers of cardiovascular preclinical studies were: Proceedings of the National Academy of Sciences of the United States of America (Proc Natl Acad Sci) 45.3% (N=135), European Heart Journal (Eur Heart J) 20.5% (N=61), Journal of the American College of Cardiology (J Am Coll Cardiol) 14.1% (N=42), and Nature 10.4% (N=31). A general difference in the reporting of the four SDEs was observed across all four journals, but differences were not determined to be statistically significant by Kruskal-Wallis tests: both sexes: H(10)=6.625, corrected p=2.281; randomization: H(10)=6.597, corrected p=1.526; blinding: H(10)=3.498, corrected p=0.967, sample size estimation: H(8)=8.344, corrected p=2.804) (Figure 3).

Articles screened across the four most prevalent journals in this study varied in frequency across the ten years (2011-2021). Among studies screened from Eur Heart J, (N=1 of 11) 11% included animals of both biological sexes in 2011 while (N=2 of 8) 25% included both biological sexes in 2019. Across studies screened from the J Am Coll Cardiol, (N=1 of 6) 17% included animals from both biological sexes in 2011 while 50% (N=2 of 4) included both biological sexes in 2019 (Figure 3). A difference of +14% in the Eur Heart J and +33% in the J Am Coll Cardiol between the journals was not found to be statistically significant (corrected p=2.804).

In studies screened from Eur Heart J, 56% (N=5 of 9) reported using randomization in 2011 and 43% (N=3 of 7) reported using randomization in 2021. Meanwhile, in studies screened from the Proc Natl Acad Sci, 44% (N=4 of 9) reported using randomization in 2011 and 22% (N=2 of 9) reported using randomization in 2021 (Figure 3). Similarly, although a difference of -13% over time was observed in Eur Heart J versus a difference of -22% in the Proc Natl Acad Sci, no statistically significant difference between these decreasing rates was found across the ten years (corrected p=1.526).

With regards to reporting of blinding, in studies screened from the Proc Natl Acad Sci, 22% (N=2 of 9) reported implementing blinding in 2011 and 33% (N=3 of 9) reported implementing blinding in 2021. In studies screened from the J Am Coll Cardiol, 33% (N=2 of 6) used blinding in 2011 while 100% (N=1 of 1) used blinding in 2021 (Figure 3). A change of +11% overtime was observed across studies screened from the Proc Natl Acad Sci versus a change of +77% across studies screened from the J Am Coll Cardiol, but no statistically significant difference was found in comparing these differences as well (corrected p=0.967).

Finally, in assessing differences in reporting statistical estimations for study sample size, a difference of +3% (N=2 of 9 to N=1 of 4) was found across studies screened from the Eur Heart J between 2011 and 2020 (Figure 3). A difference of -10% (N=3 of 18 to N=1 of 15) was observed between studies screened from Proc Natl Acad Sci from 2012-2017 (Figure 3). No studies before 2012 nor beyond 2017 from the Proc Natl Acad Sci reported using sample size estimations. Observed differences were not determined to be statistically significant (corrected p=2.804).

There were slight differences in SDE reporting for animal substudies in human/animal studies vs animal-only studies (Figure 4). In human/animal studies, 18% (N=20) used subjects of both sexes, 53% (N=61) used only one sex of study subjects, and 29% (N= 33) did not report the sex of study subjects used, as opposed to 16% (N=28), 47% (N=89), and 37% (N=67), respectively, for animal only studies. Randomization was only used in 36% (N=41) of human/animal studies and 30% (N=56) of animal-only studies. Blinding was only used in 25% (N=28) of human/animal studies and 32% (N=59) of animal-only studies. In terms of sample size estimations for human/animal studies, 7% (N=8) performed sample size estimations, 3% (N=3) justified not using sample size estimations, 5% (N=6) did not perform sample size estimation, and 85% (N=97) did not report any information on sample size estimation. For animal-only studies, these values were 8% (N=15), 5% (N=9), 3% (N=5), and 84% (N=155), respectively.

However, there were no statistically significant differences in SDE reporting in human/animal or animal-only studies in terms of the sex of study subjects used (X²=1.775, df=2, corrected p=2.470), randomization (X²=0.7448, df=1, corrected p=3.104), blinding (X²=1.5715, df=1, corrected p=1.89), or sample size estimation (X²=2.2518, df=3, corrected p=2.609).

Within human/animal studies, there were variations in SDE reporting for human vs animal substudies (Figure 5). For human substudies in human/animal studies, 36% (N=41) used subjects of both sexes, 8.7% (N=10) used only one sex of study subjects, and 55.3% (N= 63) did not report the sex of study subjects used, as opposed to 17.5% (N=20), 53.5% (N=61), and 29% (N=33), respectively, for animal only studies. Randomization was used in 6% (N=7) of human substudies and 36% (N=41) of animal substudies, and blinding was used in 11% (N=13) of human substudies and 25% (N=28) of animal substudies in human/animal studies. In terms of sample size estimations for human substudies, 2% (N=2) performed sample size estimations, 12% (N=14) justified not using sample size estimations, 3% (N=3) did not perform sample size estimation, and 83% (N=95) did not report any information on sample size estimation. For animal substudies, these values were 7% (N=8), 3% (N=3), 5% (N=6), and 85% (N=97), respectively.

There was a statistically significant difference in the use of randomization (X²=24.083, df=1, corrected p=1.85e-05) and sample size estimation inclusion (X²=38.911, df=3, corrected p=3.81e-07) of animal vs human substudies in human/animal studies. However, there was no statistically significant difference for blinding (X²=5.4878, df=1, corrected p=0.326) and sex of study subjects used in animal vs human substudies in human/animal studies (X²=3.123, df=2, corrected p=2.098).

Reporting of biological sex SDE is greater in large animal studies whether including both sexes or stating only one sex was used when compared to small animal studies (Figure 6). Note that the number of small animal studies was far greater than the number of large animal studies that were used for this analysis. For small animals, both sexes were used in 16% (N=44) studies, a single-sex was used in 50% (N=135) studies, and sex was not reported for 34% (N=93) studies. Large animal studies exhibited a similar trend in that both sexes were used in 15% (N=4) studies, a single-sex was used in 58% (N=15), and sex was not reported in 26% (N=7) studies. There was no significant difference in the proportions of studies reporting sex between the small and large animals (chi-square test of interdependence, X²=0.689, df=2, corrected p=2.836). In small animal studies, randomization was used in 30% (N=82) whereas for large animal studies, 58% (N=15) studies used randomization. There was no significant difference in the proportions of studies using randomization between the large and small animals (chi-square test of interdependence, X²=6.995, df=1, corrected p=0.152). Small animal studies had 27% (N=74) of studies that used blinding whereas large animals had 50% (N=13) of studies that used blinding. There was no significant difference in the proportion of studies using blinding between the two variables (chi-square test of interdependence, X²=4.913, df=1, corrected p=0.390). Lastly, 86% (N=234) of small animal studies did not report any information regarding sample size estimation. For large animal studies, 69% (N=18) did not report any information regarding sample size estimation. There was no significant difference in the proportion of studies using sample size estimation between the two variables (chi-square test of interdependence, X²= 7.7154, df=3, corrected p=0.676). This data is summarized in Figure 6.

The proportion of single-sex studies was found to be greater for other animal models in comparison to rodents (Figure 7). Of the studies in which rodents were the primary animal model, 48.1% (N=128) were single-sex studies. This is less than other animal model studies where 68.8% (N=22) were single-sex studies. There is a statistically significant difference between single-sex inclusion in rodent studies and other animal studies (X²=4.073, df=1, corrected p=0.610).

The low reproducibility of preclinical studies may partially be attributed to the pressure to publish in high-impact journals, which is more easily achieved if a study has positive results (Mlinaric et al., 2017). Researchers are also incentivized to share positive results to secure funding or highly competitive academic jobs. Studies with limited SDE incorporation also make it harder to perform systematic reviews (O’Connor, 2018). Though the reproducibility crisis has been acknowledged by the NIH (Collins & Tabak, 2014) and mitigative efforts have been employed, the progress and success of these efforts continue to warrant assessment. In addition to Ramirez et al. (2017) finding that preclinical cardiovascular research studies had low SDE inclusion, Williams et al. (2022) determined that preclinical research articles do not adequately adhere to many ARRIVE guidelines, including having sufficient power for t-tests, use of randomization, and blinding. Our results support the findings of both of these previous studies.

Although some SDEs can be relevant only to specific domains of a study, (Provencher, 2018), several SDEs can broadly be applied across preclinical research. The use of single-sex in a study compromises generalizability, therefore, the omission of sex as a variable may hinder later applications or the reproducibility of results in translational or clinical settings (Ramirez et al., 2017). Animal studies show a significant lack of incorporation of both sexes. In addition, there is a significant difference between the reporting of both sexes among rodent and non-rodent studies indicating that the type of animal model may be a factor in reporting (Figure 7).

Failure to apply randomization can exaggerate results, leading to unreliable findings (Hirst et al., 2014). This SDE is critical to preventing selection bias and is also the assumption upon which many statistical tests are based. In our study, the proportion of studies reporting the usage of randomization was low for both large and small animals.

The use of blinding in research also advantageously limits selection and procedural bias that could influence experimental results. Articles were screened for any reporting of blinding used whether it was single, double, or triple blinding, however, no significant differences were found.

Sample size estimation enables researchers to identify the necessary number of subjects for conclusions to be drawn and applied to the general population. This consists of predetermining a set number of animals needed for random group assignment. Sample size estimation was the lowest reported SDE among the four SDEs observed - only 8% of animal-only studies provided any justification, statistical or otherwise, for their sample size.

Although we hypothesized that the level of SDE inclusion would increase over time, the general prevalence of SDEs, whether they be applied to animal or human substudies, was remarkably low across the 298 articles screened (Figure 2). This contrasts with Ramirez et al. (2017), who found a positive trend in reporting for the journals they observed. It is important to note that their study screened studies published between 2006 and 2016, and we evaluated those published between 2011 and 2021. Another positive trend in temporal SDE inclusion in elements such as randomization and blinding has more recently been reported, however, sample size estimation and inclusion of both sexes remained low (Jung et al., 2021). In contrast, our findings indicate decreasing SDE inclusion rates for blinding and sample size estimation.

We evaluated SDE usage in animal-only studies compared to human/animal studies and SDE usage in animal substudies vs. human substudies within the same overall study. SDE incorporation in animal only and animal with human substudies were comparable though overall SDE reporting was low for both study types. Sample size estimation remained the lowest reported SDE for both study types.

Evaluating differences between SDE usage for animal or human subjects within the same study showed differences in the inclusion of randomization and sample size estimation. For these studies, a larger proportion of studies exhibit greater reporting of randomization and sample size estimation in animal substudies compared to human substudies.

Although all articles were subjected to randomized double screenings, we cannot be completely confident that an article did not use an SDE. In some instances, certain SDEs cannot be incorporated due to the conditions of the study, however, studies where SDE inclusion was not possible were still considered as lacking the SDE. Also, although the SDEs evaluated in this study are critical to conducting rigorous experimentation, other SDEs discipline-specific may ultimately be more relevant to the reproducibility of a paper.

Future studies should expand the scope of SDEs reviewed in preclinical cardiovascular publications to include more domain-specific SDEs such as the use of comparison groups, units of concern, or other forms of subject allocation (O’Connor, 2018). A more detailed investigation of reasons publications do not include sample size estimation would also be extremely valuable, given the complexity of justification for including or not including the SDE. A further extension of this work would be to survey the authors of these studies to determine the underlying reason for either the omission or inclusion of different SDEs. This would allow both verification of this study’s screening, as well as expand upon the reasons why investigators do or do not use SDEs in their studies.

The inclusion of SDEs improves reproducibility, which is important for translating preclinical findings into clinical outcomes. Contrary to our hypothesis, we found that over the past decade, there has been no significant increase in SDE incorporation in the preclinical cardiovascular publications we studied. Sample size estimation remains the least reported study design element. These trends all indicate the need for further efforts to increase the incorporation of rigorous study design elements in research projects to the point that they become routine. Future research efforts should evaluate a wider range of SDEs and investigate the reasons why SDEs have such low incorporation in preclinical cardiovascular research.