In a recent study published in JAMA Network Open, researchers compared and determined the concordance between the study design, results, and inferences of clinical and observational studies and meta-analyses that were first posted as medRxiv preprints and were later peer-reviewed and published in journals.

Background

Preprints posted on servers such as medRxiv can be a double-edged sword. While access to non-peer-reviewed clinical research findings can result in the rapid dissemination of information and allow the evaluation of the study by the scientific community at large, open access to such results without peer review could also propagate misinterpretations, erroneous results, and faulty research methods that can have consequences on health practices.

The few studies that have compared the study design, results, and the major inferences across preprints and their subsequently published versions have largely focused on those preprints published in peer-reviewed journals with high impact factors (greater than 10). It can be inferred that the preprints that do eventually get published in journals with high impact factors are already of high quality and do not require major alterations after peer review.

To determine the reliability of clinical study results posted in preprints, it is important to compare preprints and their subsequent published versions in peer-reviewed journals independent of the impact factor.

About the study

In the present study, the researchers used the medRxiv preprint server’s application programming interface to obtain all manuscripts first submitted to the server in September 2020. For the preprints updated with a newer version after September 2020, the most recent versions were obtained.

The study design of each preprint was manually characterized into one of many categories, including meta-analysis with or without a systematic review, clinical, observational, modeling studies, or other. The preprints were then matched with their corresponding peer-reviewed journal publications.

To ensure that the changes made in the updated versions of the preprint were not a result of the peer-review process, the updated versions posted after the journal acceptance date were excluded. The time between the date of posting in medRxiv and the electronic publication date was also determined. The impact factor of the journals was obtained from the InCites Journal Citation Reports.

For each preprint-journal article pair, information on the sample size, primary measured outcomes, results for each outcome, and inferences drawn from the results was compared. The sample size for clinical and observational studies was defined as the number of participants in the cohorts or database, while for meta-analyses, the sample sizes were based on the number of studies included in the analysis. The primary endpoints included ascertainment time and measurement scale, which included factors such as mortality rates, odd ratio, etc.

Concordance in sample size was determined based on numerical similarity, while concordance in primary outcomes or endpoints was concluded if the publication did not report any additional primary outcomes. Discordances in sample size and primary endpoints were investigated further to determine the type and reasons for the discordance. The interpretations were concluded to be concordant if similar statements were made about the results in the preprint and the publication.

Results

The results reported that as of September 2022, 1077 of the 1399 preprints posted to medRxiv during September 2020 had been published in peer-reviewed journals. Of these, 547 described meta-analyses, clinical trials, or observational studies, and 53.6% (293) of these were related to coronavirus disease 2019 (COVID-19). The preprint-journal article pairs that reported sample sizes were 535, of which 86.4% (462) were concordant, and 43 out of the 73 discordant pairs reported a larger sample size in the publication.

Out of the 547 pairs, 97.6% (534) were concordant in their reports of the primary endpoints, and only 2.4% were discordant. For pairs with numerical results, 81.1% (434 out of 535) were concordant, while 101 had discordant results for the primary outcomes. Of these 101, the effect estimates of 65.3% (66) were statistically similar. The interpretations of the results were concordant for 96.2% (526) of the 547 pairs.

Conclusions

Overall, the results suggested that despite the absence of peer review in preprints posted to servers such as medRxiv, the study design, measured primary outcomes, results, and inferences drawn from the results were consistent between preprints and their subsequently peer-reviewed and published versions.

More than three-quarters of the preprint and journal article pairs were concordant in characteristics such as primary outcomes, results, and interpretations. Furthermore, the discordant pairs in the results were still similar in their interpretations.