Literature reviews - a guide how to carry them out

The next step in the review process is to undertake a quality assessment of the results retrieved. Critically appraising a piece of research combines analysis of the design of the study, the validity of the findings in relation to the design of the study, the likelihood of bias, and the relevance of the overall results to other current research.  These are the key steps as outlined in Greenhalgh, J. and Brown, T. Quality assessment: where do I begin? In: A. Boland, M.G. Cherry & R. Dickson (eds.) Doing a systematic review: a student's guide (2nd ed.). SAGE, 107-130.

Step 1: Note the design(s) of the studies to be included in your review

Step 2: Identify the type(s) of quality assessment tool(s) to suit your review (critical appraisal tool)

Step 3: Source appropriate quality assessment tool(s)

Step 4: Carry out quality assessment using the appropriate tool(s)

Step 5: Tabulate and summarize the results of your quality assessment

Step 6: Think about how the quality assessment results might impact on the recommendations and conclusions of your review

There are a variety of critical appraisal tools you could use:

• PICO. Available from https://casp-uk.net/pico-framework/
• The Critical Appraisal Skills Programme (CASP) has produced a set of eight checklists which can be used when appraising studies for inclusion/exclusion in a review. The checklists provide a series of study-specific questions to use when reading research to enable the critical appraisal of each item. Available from https://casp-uk.net/
• JBI's Critical Appraisal Tools (Joanna Briggs Institute) can assist with assessing the trustworthiness, relevance and validity of research papers. Available from https://jbi.global/critical-appraisal-tools

• The Mixed Methods Appraisal Tool (MMAT) is intended to be used as a checklist for appraising and/or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative and mixed methods studies). Available from http://mixedmethodsappraisaltoolpublic.pbworks.com/

When you are evaluating your sample, you will need to address any bias that affects the credibility of your study. You may like to use the ROBIS framework to do this. Information about ROBIS can be found here. Although developed with healthcare in mind, you can use the modified documents below to help you.

• ROBIS worksheet 1

• ROBIS worksheet 2

When you have your final sample of material, you are ready to start extracting data from the material you have selected for your review. The hallmark of a good systematised review is the consistency in which data is extracted from sources

The way you do this will be dependent on the formulation of your question.

Jesson, Matheson and Lacey (2012, 123) give the following example.

a) Bibliographic details

b) Discipline

c) Focus of the paper

d) Method details (sample selection, size, method design, location date etc.)

e) Theory or model used