Files and links (1)
Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews_ Stage 2 of the INSPECT-SR project829.23 kB
Published VersionCC BY V4.0, Open Access
Journal article
Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project
Journal of Clinical Epidemiology, Vol.184, pp.1-13
2025
PMID: 40349737
Abstract
Background and Objectives
The aim of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project is to develop a tool to identify problematic RCTs in systematic reviews. In stage 1 of the project, a list of potential trustworthiness checks was created. The checks on this list must be evaluated to determine which should be included in the INSPECT-SR tool.
Methods
We attempted to apply 72 trustworthiness checks to randomized controlled trials (RCTs) in 50 Cochrane reviews. For each, we recorded whether the check was passed, failed, or possibly failed or whether it was not feasible to complete the check. Following application of the checks, we recorded whether we had concerns about the authenticity of each RCT. We repeated each meta-analysis after removing RCTs flagged by each check and again after removing RCTs where we had concerns about authenticity to estimate the impact of trustworthiness assessment. Trustworthiness assessments were compared to Risk of Bias and Grading of Recommendations Assessment, Development and Evaluation (GRADE) assessments in the reviews.
Results
Ninety-five RCTs were assessed. Following application of the checks, assessors had some or serious concerns about the authenticity of 25% and 6% of the RCTs, respectively. Removing RCTs with either some or serious concerns resulted in 22% of meta-analyses having no remaining RCTs. However, many checks proved difficult to understand or implement, which may have led to unwarranted skepticism in some instances. Furthermore, we restricted assessment to meta-analyses with no more than five RCTs (54% contained only 1 RCT), which will distort the impact on results. No relationship was identified between trustworthiness assessment and Risk of Bias or GRADE.
Conclusion
This study supports the case for routine trustworthiness assessment in systematic reviews, as problematic studies do not appear to be flagged by Risk of Bias assessment. The study produced evidence on the feasibility and impact of trustworthiness checks. These results will be used, in conjunction with those from a subsequent Delphi process, to determine which checks should be included in the INSPECT-SR tool.
Plain Language Summary
Systematic reviews collate evidence from randomized controlled trials (RCTs) to find out whether health interventions are safe and effective. However, it is now recognized that the findings of some RCTs are not genuine, and some of these studies appear to have been fabricated. Various checks for these “problematic” RCTs have been proposed, but it is necessary to evaluate these checks to find out which are useful and which are feasible. We applied a comprehensive list of “trustworthiness checks” to 95 RCTs in 50 systematic reviews to learn more about them and to see how often performing the checks would lead us to classify RCTs as being potentially inauthentic. We found that applying the checks led to concerns about the authenticity of around 1 in three RCTs. However, we found that many of the checks were difficult to perform and could have been misinterpreted. This might have led us to be overly skeptical in some cases. The findings from this study will be used, alongside other evidence, to decide which of these checks should be performed routinely to try to identify problematic RCTs, to stop them from being mistaken for genuine studies and potentially being used to inform health care decisions.
Details
- Title
- Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project
- Authors
- Jack Wilkinson - Manchester Academic Health Science CentreCalvin Heal (Corresponding Author) - Manchester Academic Health Science CentreGeorgios A. Antoniou - Manchester University NHS Foundation TrustElla Flemyng - CochraneLove Ahnström - Karolinska InstitutetAlessandra Alteri - Istituti di Ricovero e Cura a Carattere ScientificoAlison Avenell - University of AberdeenTimothy Hugh Barker - The University of AdelaideDavid N. Borg - Australian Sports CommissionNicholas J.L. Brown - Linnaeus UniversityRob Buhmann - University of the Sunshine Coast, Queensland, School of Health - Sports & Exercise ScienceJose A. Calvache - University of CaucaRickard Carlsson - Linnaeus UniversityLesley-Anne Carter - University of ManchesterAidan G. Cashin - Neuroscience Research AustraliaSarah Cotterill - Manchester Academic Health Science CentreKenneth Färnqvist - Karolinska InstitutetMichael C. Ferraro - Neuroscience Research AustraliaSteph Grohmann - CochraneJill A. Hayden - Dalhousie UniversityLyle C. Gurrin - The University of MelbourneKylie E. Hunter - The University of SydneyNatalie Hyltse - Linnaeus UniversityLukas Jung - Independent Researcher, Heidelberg, GermanyAshma Krishan - Manchester Academic Health Science CentreSilvy Laporte - Centre Hospitalier Universitaire de Saint-ÉtienneToby J. Lasserson - CochraneDavid R.T. Laursen - Odense University HospitalSarah Lensen - Royal Women's HospitalWentao Li - National Centre for Clinical Research on Emerging DrugsTianjing Li - University of Colorado Anschutz Medical CampusJianping Liu - Beijing University of Chinese MedicineClara Locher - InsermZewen Lu - Manchester Academic Health Science CentreAndreas Lundh - University of Southern DenmarkAntonia Marsden - Manchester Academic Health Science CentreGideon Meyerowitz-Katz - University of WollongongBen W. Mol - Royal Women's HospitalZachary Munn - National Centre for Clinical Research on Emerging DrugsFlorian Naudet - InsermDavid Nunan - University of OxfordNeil E. O'Connell - Brunel University of LondonNatasha Olsson - Karolinska InstitutetLisa Parker - The University of SydneyEleftheria Patetsini - University of YorkBarbara Redman - New York UniversitySarah Rhodes - Manchester Academic Health Science CentreRachel Richardson - CochraneMartin Ringsten - Skåne University HospitalEwelina Rogozińska - MRC Clinical Trials Unit at UCLAnna Lene Seidler - University of RostockKyle Sheldrick - UNSW SydneyKatie Stocking - Manchester Academic Health Science CentreEmma Sydenham - CochraneHugh ThomasSofia Tsokani - Aristotle University of ThessalonikiConstant Vinatier - InsermColby J. Vorland - Indiana University BloomingtonRui Wang - The University of SydneyBassel H. Al Wattar - Anglia Ruskin UniversityFlorencia Weber - Universitätsklinikum WürzburgStephanie Weibel - Universitätsklinikum WürzburgMadelon van Wely - Amsterdam University Medical CentersChang Xu - Eastern Hepatobiliary Surgery HospitalLisa Bero - University of Colorado Anschutz Medical CampusJamie J. Kirkham - Manchester Academic Health Science Centre
- Publication details
- Journal of Clinical Epidemiology, Vol.184, pp.1-13
- Publisher
- Elsevier Inc.
- Date published
- 2025
- DOI
- 10.1016/j.jclinepi.2025.111824
- ISSN
- 1878-5921; 0895-4356
- PMID
- 40349737
- Copyright note
- (c) 2025 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
- Data Availability
- Link to data and code is included in main text.
- Grant note
- NIHR203568 / NIHR Research for Patient Benefit program (https://doi.org/10.13039/501100009128)
- Organisation Unit
- School of Health - Sports & Exercise Science
- Language
- English
- Record Identifier
- 991144109502621
- Output Type
- Journal article
Metrics
17 Record Views
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Web Of Science research areas
- Health Care Sciences & Services
- Public, Environmental & Occupational Health
UN Sustainable Development Goals (SDGs)
This output has contributed to the advancement of the following goals:
Source: InCites