To determine the prevalence of accessible and reproducible research practices within cardiovascular literature, data was gathered from a random selection of all published studies in 2019 in the following three leading cardiology journals (Opthof, 2019): Circulation, European Heart Journal, and the Journal of the American College of Cardiology (JACC). To limit the scope of our analysis, we only included articles published in the year of 2019. This specific year was selected because it was the most recent full year not influenced by changes in reporting practices due to the COVID-19 pandemic.

The following PubMed search string was used to obtain the full list of articles screened: ((‘Circulation’[Journal] OR ‘J Am Coll Cardiol’[Journal] OR ‘Eur Heart J’[Journal]) AND 2019/01/01:2019/12/31[Date – Publication]) NOT (Published Erratum[Publication Type]). This search string retrieved 2786 articles. This list was randomized and the first 639 were screened for our study, and researchers screened the listed articles sequentially. A PRISMA flow diagram (Appendix 1—figure 1) of the inclusion and exclusion information is provided in the supplementary materials.

No blinding was involved in this study and each article was screened at least twice by separate individuals. Some researchers screened a higher proportion of articles than other authors, but each author screened at least 180 articles once. Any ambiguities identified during screening were resolved either through additional review, or through discussion among participating researchers to achieve consensus.

We initially screened 400 randomly selected articles, however of those articles only 153 were empirical research articles for which we could calculate accessibility measures. To increase the number of empirical research articles in our study, we therefore conducted a second round of screening. All aspects remained the same during additional screening other than generating the following new PubMed search string: (‘Circulation’[Journal] OR ‘J Am Coll Cardiol’[Journal] OR ‘Eur Heart J’[Journal]) AND 2019/01/01:2019/12/31 [Date – Publication] NOT ((review[Publication Type]) OR (systematic review[Publication Type]) OR (editorial[Publication Type]) OR (comment[Publication Type]) OR (‘case reports’[Publication Type]) OR (‘Introductory Journal Article’[Publication Type]) OR (‘Historical Article’[Publication Type]) OR (‘Practice Guideline’[Publication Type])). This search string retrieved 1461 primarily empirical articles. This list was also randomized and the first 239 articles that had not already been screened were added to the first set of 400 screened articles. Researchers screened the new articles sequentially and any remaining non-empirical articles were removed from the list and replaced with an empirical research article.

Following a modified version of a previously established coding protocol for reproducible and accessible research practices (Iqbal et al., 2016), the following criteria were screened during the evaluation process: type of reported study, pre-registration status, protocol availability, materials availability, data availability, analysis script availability, conflict of interest (COI) status, and open access of the article. An article’s ‘accessibility score’ was calculated as the fraction of several accessibility criteria results with the specific criteria satisfied out of the total possible to be satisfied for that specific study type. See Table 1 for a list of screening criteria that contributed to the calculation of the accessibility score. Note that for papers without the ability to share Materials (e.g. Meta-analyses), the materials criterion was omitted from the accessibility score calculation. Our article coding form is derived from Iqbal et al., 2016 and can be found as Qualtrics.qsf and Word files in the pre-registration (Campos-Melendez et al., 2021).

The four criteria defining repeatability – methods (protocol or pre-registration), materials, data, and analysis script availability – are each considered an accessibility category. Each individual criterion in these four categories can be either satisfied or not. For a study to be considered fully ‘replicable’ all three of the following categories must be available: methods, data, and analysis scripts. For a study to be considered fully ‘reproducible’, all three of the following categories must be available: methods, data, and analysis scripts. If an article is either partially replicable or partially reproducible it is considered ‘partially repeatable’. Definitions of replicability and reproducibility were set based on those introduced by the National Academies of Sciences, Engineering and Medicine (NASEM) (2019). We defined the ability to replicate a study as the attempt to obtain the same results as a study by collecting new data. Reproducing a study describes the attempt to obtain the same results as a study by re-analyzing its data.

Only articles that were fully screened and for which all ambiguities had been resolved were included in the final analysis. Papers written in a language other than English were also excluded, as well as any articles for which we could not access the full text. Articles without empirical data (e.g. review articles, commentaries, and editorials) were evaluated for author location, language, COI, funding statements, and public access of the article itself, but were otherwise not screened for accessibility criteria because these study types by their nature cannot share, for example, materials, data, or analysis code. We also screened whether studies could share materials or not. For example, meta-analyses are empirical research studies, but do not typically have any shareable materials. One such example is a meta-analysis (Russo et al., 2019), which was screened as an empirical paper but could not have materials due to the nature of the research conducted. Our analysis does assume, however, that all empirical research articles involve a protocol, the collection or re-use of data, and analysis of that data, and therefore methods, data, and analysis code or scripts should always be available.

To determine how frequently published cardiovascular research reports provide sufficient resources to replicate or reproduce their studies, we quantified the fraction of papers that are either partially replicable or partially reproducible – defined as partially repeatable. For all accessibility criteria, we calculated the proportion of papers in our dataset that exhibit each possible outcome. For example, for the accessibility criterion ‘How does the statement indicate the materials are available’, we calculated the proportion of papers that state that materials are available through an online third party repository, a personal or institutional webpage, supplemental information hosted by a journal, or upon request from authors. We also calculated the distribution of accessibility scores across our full dataset and further collected data regarding the location of the corresponding author. We use standard deviations to show the variability of the data.

To test for inconsistent, or variable, levels of accessibility, we examined our data using multi-way ANOVAs. Our independent variables included ‘study type’ and ‘journal’ and our dependent variable was the accessibility score. For the hypotheses addressing differences between the types of studies, journals, and accessibility scores using multi-way ANOVA, p-values <0.05 were considered to be statistically significant.

To evaluate the relationship between accessibility criteria (Materials, Methods, Data, and Analysis Code) and study types, we performed several sequential chi-squared tests of proportions. In cases with 0 samples in a group, we instead performed Fisher’s exact tests. Tests were performed using R. We compared the proportion of papers that did and did not allow access to those four categories of information and resources, across all four categories. We also used chi-squared tests to further compare the proportion of papers that provided access to each category of information and which study types shared more or less of a specific type of information compared to other study types. We considered p-values ≤0.05 to indicate significant differences in proportions. We used Bonferroni correction to adjust the p-value for the number of tests performed. Note that this analysis differs slightly from that proposed in our pre-registration: Chi square Automatic Interaction Detection (CHAID).

This study was pre-registered through the Open Science Foundation and can be accessed at https://doi.org/10.17605/OSF.IO/QFSTH (Campos-Melendez et al., 2021).

After obtaining our data, not all exploratory analyses documented in the pre-registration were performed. We did not test the relationship between the articles described as accessible and those actually accessible due to the limited number of articles that exhibited large enough accessibility scores, nor did we perform textual analysis on the COI, funding statements, or the way resources were being shared. We also deviated from evaluating the relationship between corresponding author location and accessibility score given the low sample size for most countries. We also did not perform additional analyses with the accessibility criteria organized as a hierarchy, because of the subjectivity in determining a ranking of screening criteria. We also deviated from the pre-registration by using sequential chi-squared tests instead of the CHAID. CHAID analysis builds a predictive model using a classification tree with multiple levels. In our case, we only compare data across two levels and there is no hierarchical basis for a decision tree. CHAID was therefore an inappropriate method for our study. Lastly, we extended our data collection beyond what we initially proposed in our pre-registration so that we could have more empirical research articles in our dataset.

Looking only at articles that included empirical research (393 out of 639 total articles), the majority had had no pre-registration statement (286) or no linked and accessible protocol (383; Figure 2A). Of the papers that had a pre-registration statement, nearly all of them had accessible and openable pre-registrations (106 out of 107; Figure 2A). Among articles that had an openable and accessible pre-registration or protocol, almost all articles included methods (106 out of 106, 9 out of 10), approximately half included hypotheses (44 out of 106, 7 out of 10), and less than half included analysis plans (27 out of 106, 5 out of 10; Figure 2A). We define a publication to have all pre-registration aspects if hypotheses, methods, and analysis are pre-registered. Of the 393 screened publications that have empirical research, 53 had 1 out of 3 of the aspects, 35 had 2 out of 4 of the aspects, and 18 studies had all three aspects.

Only study types considered empirical research in which materials are theoretically available are summarized. Only 14% of articles made their materials available (56 out of 393), 31.8% of articles made their data available (125 out of 393), and 10.9% made their analysis scripts available (43 out of 393; Figure 2B). Across the articles that stated materials, data, and analysis scripts were available, it was most common for them only to be made available upon request from the authors (Figure 2B). Across the articles that stated materials, data, and analysis scripts were available, very few studies made the materials and data readily available for downloading or opening (1, 13) and only one study’s analysis scripts were readily downloadable (1 out of 43; Figure 2B). However, the majority of data that was downloadable or openable was also clearly documented (9 out of 13; Figure 2B).

The majority of cardiovascular publications are not readily replicable or reproducible. We predicted that the majority of publications will not provide sufficient resources to replicate or reproduce their studies. We tested both (1) that the simple majority, more than half of screened publications, will be lacking one or more specific replicability or reproducibility criteria, and (2) that the vast majority, over 90% of screened publications, will be lacking one or more specific reproducibility or replicability criteria. We found that 49.6% (195 out of 393) of empirical research studies were not partially reproducible and 49.4% (194 out of 393) were not partially replicable. 2% (7 out of 393) empirical research papers were fully reproducible, but only 5 were fully replicable (Figure 3). Therefore, the simple majority of empirical research articles were partially reproducible or replicable, but the vast majority of empirical research studies were neither fully reproducible nor fully replicable.

With the exception of one paper, the entire dataset exhibited low accessibility with scores lower than 0.6; the majority of articles (144 out of 393) had accessibility scores within the 0.20–0.29 range and 18 articles had scores below 0.1. Only one article exhibited an accessibility score of 0.6 or greater (Figure 4A).

We also predicted that the level of accessibility will be inconsistent across several dimensions. Specifically, we predicted that average accessibility score would vary across study types and potentially also across journals, depending on the publisher’s requirements. For example, if clinical trials must be registered and open access, clinical trial publications will satisfy more of the article coding form criteria than other studies. We tested this hypothesis by calculating an accessibility score for every article: the sum of the satisfied screening criteria divided by the total possible satisfiable criteria. We then calculated the average and standard deviation in accessibility scores across study type (Figure 4C). Overall accessibility scores were low across all study types, and standard deviations were large, indicating high variability across individual publications. We found that clinical trials had the highest average accessibility score (0.39), and clinical case studies or series had the lowest accessibility scores (0.08). Average accessibility scores were significantly different across study types (ANOVA, d.f. 7, F=36.37, p=5.500E−5).

We also predicted a significant difference in the number of accessibility criteria satisfied across journals due to differences in reporting policies. The largest proportion of articles screened were obtained from Circulation with a total number of 226 articles, and 74% of those were empirical research papers (168 out of 226). 63% of JACC articles (138 out of 220) and 45% of Circulation articles (87 out of 193) were empirical research for which we could calculate an accessibility score. After computing the average accessibility score for all three journals, Circulation had the highest average accessibility score (0.34), followed by the Journals of the American College of Cardiology (0.24), then the European Heart Journal (0.19) (Figure 4B). Accessibility scores vary significantly across journals (ANOVA, d.f. 2, F = 80.07, p = 1.230E−2). There was not a significant interaction between study type and journal (p = 0.284).

We also predicted some categories of resources will be accessible significantly more frequently than other categories (e.g. materials availability vs analysis script availability). We found that there was a significant difference in the proportion of papers that shared specific categories of resources (X² = 55.2, p = 6.17e−12). We then further identified whether different study types exhibited differences in the proportions of shared resources, for example, whether there was a difference in the proportion of papers that have accessible data depending on study type. Note that we did not include Clinical Studies with Interventions, Meta-analysis or Systematic Reviews, Non-clinical Secondary Data Analysis, or Laboratory Cell or Molecular Studies in this analysis due to their small sample size. For all four shareable resources (Materials, Methods, Data, and Analysis), we found a significant difference across study types (Table 2).

To determine specifically how study types differ in which resources they share, we performed multiple chi-squared tests with Bonferroni correction, comparing each study type with every other study type for a given resource (Figure 5; Table 3). We found that a significantly higher proportion of laboratory animal studies stated that they would share materials than clinical observational studies or clinical trials. We also, as expected, found a significantly higher proportion of clinical trials shared their methods compared to any other study type. With regard to data sharing, we found that a significantly higher proportion of laboratory animal studies shared their data than any other study type. Lastly, a significantly higher proportion of laboratory animal studies shared their analysis code compared to clinical observational studies and clinical trials.

In collecting data on the fraction of papers that were publicly accessible, we found that only 34 out of 639 articles (5%) were not publicly accessible (Figure 1F). Although this is a fairly low fraction of inaccessible articles, a study’s publication is in many ways the most tangible output of the research, and having publicly accessible publication should be a minimum standard for research. The NIH’s public access policy has been instrumental in ensuring public access to published research reports, but reports funded purely through private sources are not currently under the same reporting mandate.

It is notable that over half of the 400 papers we initially randomly selected for screening were non-empirical research, e.g. reviews, editorials, and commentaries. Although these emissions may be effective ways to communicate quickly and effectively with larger audiences, they may not go through the same rigor of review process. We also occasionally screened articles that were simplified summaries of the original research study, geared toward a lay audience. This type of article is valuable in that it makes research more accessible to a broader audience; however, because methods were absent and results were condensed, it was ambiguous whether the studies being reported in these articles had undergone full scientific review. Readers run the risk of assuming that these articles describe a full scientific story as opposed to a news highlight.

In addition, publications of replication studies, or studies that included a replication study, were virtually absent from our dataset (4 out of 393 empirical research publications) – suggesting the field of cardiovascular research puts very little emphasis on replication work. Of concern, several studies also stated outright that their data was not available for replication, stating for example ‘The investigators will not make the data, methods used in the analysis, and materials used to conduct the research available to any researcher for purposes of reproducing the results or replicating the procedure.’ Although there can be legitimate reasons why data sharing is not possible, these statements are typically not accompanied by any justification.

Although data was shared more frequently than materials or analysis scripts, there were still only a total of 13 out of 393 empirical research papers for which data was readily openable to a reader. We acknowledge that authors will often have legitimate reasons for not being able to share resources, including patient privacy. However, it has also been shown that patients are in general very willing to make their data available to further research (Seltzer et al., 2019; Kim et al., 2019). We advocate for research studies actively seeking consent from human subject participants to make their data available and if that consent cannot be obtained to specify that justification for not sharing data in their report.

Although basic research studies more frequently shared materials, data, and analysis scripts than clinical trials, that sharing was frequently ‘upon request from the authors’, which previous studies have shown to be hit-or-miss in terms of yield (Hardwicke et al., 2018). Clinical trials had by far the most consistent availability of a resource category: methods, in the form of pre-registrations. Because pre-registrations are mandated by the FDA to conduct a clinical trial, our results suggest that resource-sharing requirements by funding or approval agencies are effective means of changing practices. It should be noted however, that a recent study by Goldacre et al., 2019 showed that in trials with pre-registrations, only 76% of pre-specified trial outcomes were correctly reported. Therefore, even when trials are pre-registered variable switching is common when reporting results.

Publications are a critical form of communicating research, and the content and results of publications are often prioritized to the point that COI and funding statements are easy to overlook. As a community, we are often more interested in the results described in publications than the process behind getting them. Personal interests and finances are undeniably a part of experimental integrity and can impact experimental design and therefore COI and funding statements should be given as much attention as other components of the publication. Although it was beyond the scope of this study to perform a rigorous analysis of COI and funding statements, our screening process did reveal numerous cases of ambiguous COI and funding declarations. In Table 4, we capture both problematic and positive examples of COI and funding statements. For example, many COI and funding statements were vague in that they lacked details on how different funders of interests specifically influenced the study, which is important for interpretation of the results. As another example, we also identified ‘Disclosures: None’ as a problematic statement, because it could be interpreted as the authors having no disclosures or that the authors declined to list their disclosures. The relationship between funders and COIs is also ambiguous. Authors frequently listed funders and declared ‘no conflict of interest’; however, funders frequently do have an interest in and impact on the study and therefore represent a COI. If nothing else, it is in the interest of researchers to produce compelling results to maintain good relations and receive future funding from agencies, even if they were funded solely through public organizations.

To avoid these issues, we advocate that journals require more complete and standardized COI and funding information that recognizes their overlap, including: a clear description on how each funding entity supported the work including whether they influenced the experiments, analysis, or dissemination of the work. We recommend that journals require authors to complete a consistent and transparent funding and COI questionnaire or decision tree that is clearly and prominently associated with the article.

Although we did not have sufficient articles from different countries of corresponding authors to determine whether there was a significant difference in accessibility practices across countries, we did see a wide range in accessibility score values. In terms of reporting, it was in general very straightforward to identify the corresponding author of an article. However, sometimes corresponding authorship was ambiguous in that no contact information was given, or was listed as, ‘Published on behalf of […]. All rights reserved. The Author(s) 2019. For permissions, please email: […].’ In these circumstances, it is unclear who the corresponding author is and who is ultimately responsible for questions regarding the research.

Although we attempted to achieve completeness and consistency in screening by ensuring each article was screened by two separate individuals, we acknowledge we may have missed or misinterpreted statements in screened articles. For example, in capturing the type of funding for a study as public, private, or a combination of both, we may have misidentified the funding type for some organizations, especially for foreign funding bodies. Another example would be identifying whether the study type was clearly stated versus inferred with further reading of the article. These determinations are subjective and may vary with individuals’ experience with different study types and interpretation of language used in articles. Note that regular readers of publications will face the same challenges.

Furthermore, the accessibility score we used could be further developed to better represent the diversity of publications. For example, there may be types of studies that we identified as not providing materials, because no materials statement was given, but for which all materials may already be adequately described in the text and therefore actually readily available. In these cases, our accessibility score is not accurate. Case studies and case series are the study type most likely to be affected by this limitation due to their limited data and analysis. Only 21 out of 393 empirical research studies (5%) were case studies or case series. Similarly, our current screening criteria only gives credit to papers that state they share a protocol and that protocol is actually linked. Our screening protocol does not capture articles that have a protocol availability statement but do not provide a link.

In the future, the accessibility score could be improved to be more specific to different article types. For example, case studies typically represent patient cases where replicating or reproducing the study would not necessarily be expected. Lastly, this study also only used data from some of the highest ranking cardiovascular research journals. These journals are likely to have higher reporting standards than other journals in the field, so our results are likely to overestimate the reporting and sharing practices of publications in cardiovascular research in general.

The data collected from 639 screened papers provides numerous future directions for not only exploratory analysis on the existing dataset, but also for new projects assessing accessibility and reproducibility of scientific literature. The accessibility scores we calculated are a rough, quantitative estimate of an article’s actual accessibility and further work is required to more fully describe how accessible and reproducible cardiovascular literature is. For example, future work could identify which criteria are the biggest needs for the field, and then evaluate the quality of an article’s accessibility by weighting those criteria or organizing the criteria into a hierarchy of importance. Future studies could also investigate text excerpts describing how resources are or are not being made available to determine the causes that promote or undermine accessible research practices.

Our study shows that there is a high degree of variability in the resources cardiovascular research publications make available – across study types and journals. Universally, however, publications almost never provide sufficient materials, protocol information, data, or analysis scripts for another group to fully replicate or reproduce their work. When federal policies mandate the open research practices, such as ensuring publications are publicly accessible or clinical trials are pre-registered, these practices are adopted. To ensure the highest caliber of research in the future, we urge journals and funding agencies to require higher standards in their materials, protocol, data, and analysis script sharing – regardless of the study type or the funding source.