Skip to Main Content

Introduction to systematic reviews

Assessing quality

The next step in the systematic review process is to undertake a quality assessment of the results retrieved. Critically appraising a piece of research combines analysis of the design of the study, the validity of the findings in relation to the design of the study, the likelihood of bias, and the relevance of the overall results to other current research.  These are the key steps as outlined in Greenhalgh, J. and Brown, T. Quality assessment: where do I begin? In: A. Boland, M.G. Cherry and R. Dickson (eds.) Doing a systematic review: a student's guide, 2nd edn. London: SAGE, 107-130.

Step 1: Note the design(s) of the studies to be included in your review

Step 2: Identify the type(s) of quality assessment tool(s) to suit your review

Step 3: Source appropriate quality assessment tool(s)

Step 4: Carry out quality assessment using the appropriate tool(s)

Step 5: Tabulate and summarize the results of your quality assessment

Step 6: Think about how the quality assessment results might impact on the recommendations and conclusions of your systematic review

Critical appraisal tools: specific study types

CASP

The Critical Appraisal Skills Programme has produced a set of eight checklists which can be used when appraising studies for inclusion/exclusion in the systematic review. The checklists provide a series of study-specific questions to use when reading research to enable the critical appraisal of each item.

  • Randomised Controlled Trial
  • Systematic Review
  • Qualitative Studies
  • Cohort Study
  • Diagnostic Study
  • Case Control Study
  • Economic Evaluation
  • Clinical Prediction Rule

Available from https://casp-uk.net/

 

BestBETS

BestBETs provide a range of Critial Appraisal (CA) Worksheets for a range of study types, including:

  • Case-control (including harm)
  • Cohort
  • Decision Rule
  • Diagnosis
  • Economic
  • Educational Intervention
  • Guideline
  • Prognosis
  • Qualitative
  • Randomised Control Trial
  • Review or Meta-analysis
  • Screening
  • Survey (including pre-test probabilities)

Available from https://bestbets.org/

 

SIGN (Scottish Intercollegiate Guidelines Network)

SIGN, part of the Evidence Directorate of Healthcare Improvement Scotland, has produced a number of checklists to help critically evaluate systematic reviews, randomised controlled trials, etc.

These are available at: https://www.sign.ac.uk/what-we-do/methodology/checklists/

 

Centre for Evidence-Based Medicine (CEBM)

The Centre for Evidence-Based Medicine (CEBM), based at the University of Oxford, has developed tools and worksheets to help with the critical appraisal of different types of medical evidence.

These are available at: https://www.cebm.ox.ac.uk/resources/ebm-tools/critical-appraisal-tools

 

JBI: Joanna Briggs Institute Critical Appraisal Tools

JBI's Critical Appraisal Tools can assist with assessing the trustworthiness, relevance and validity of research papers. Includes:

  • Analytical Cross Sectional Studies
  • Case Control Studies
  • Case Reports
  • Case Series
  • Diagnostic Test Accuracy Studies
  • Economic Evaluations
  • Prevalence Studies
  • Qualitative Research
  • Quasi-Experimental Studies
  • Randomised Controlled Trials
  • Systematic Reviews
  • Text and Opinion

Available from https://jbi.global/critical-appraisal-tools

 

Mixed Methods Appraisal Tool (MMAT)

The Mixed Methods Appraisal Tool (MMAT) is intended to be used as a checklist for appraising and/or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative and mixed methods studies).

Available from http://mixedmethodsappraisaltoolpublic.pbworks.com/

A PDF user guide for MMAT is available for download

Not sure what study type you are looking at?

The Centre for Evidence-Based Medicine (CEBM)'s brief guide on Study designs can help you spot the difference and to identify the advantages/disadvantages of different study types.  The guide is available from https://www.cebm.ox.ac.uk/resources/ebm-tools/study-designs

Critical Appraisal Tools: Appraising for Anti-racism

Appraising for Anti-Racism

Created by Ramona Naicker from Northern Care Alliance NHS Group

Available from 

https://www.northerncarealliance.nhs.uk/application/files/1916/4871/3337/Supplementary_Checklist_Critically_Appraising_for_Antiracism_March_2022.pdf

 

Critical appraisal of an RCT

Barts Health NHS Trust's Knowledge and Library Services have produced a series of short videos taking you through how to critically appraise a Randomised Controlled Trial (RCT) using the CASP checklist. The video below is exploring how to answer question 1: Did the study address a clear focused question?

Critical appraisal of a systematic review

This video is a hands on introduction to using the CASP checklist to appraise systematic reviews facilitated by Dr Becky Gould, hosted by Cochrane UK in September 2021.