How To Guides (OPH): How to do a systematic review

Step 1 - Assemble a team

A systematic review should be undertaken by more than one person so that it decreases the likelihood of errors.
It is recommended that tasks such as the selection of studies for eligibility and data extraction be performed by at least two people. Consulting a librarian will helpful when searching. Having someone with experience in quantitative methods may also be required.

References:
• IOM (Institute of Medicine). Standards for initiating a systematic review

Step 2 - Formulate a question

The review question needs to be clear and well defined.
 

1. Check that a review has not already been done on your research topic

Where to search:

• Cochrane Database of Systematic Reviews
• PROSPERO (International Prospective Register of Systematic Reviews)
• Joanna Briggs Institute (for Nursing & Allied Health)
• PubMed - Clinical Queries

Subject specific resources

• OTSeeker (Occupational therapy)
• PEDro (Physiotherapy)
• PsycBITE (Psychological medicine)
• SpeechBITE (Speech pathology)

2. Use question frameworks

These will help you develop an answerable and focused question. Developing concepts will help during the searching process at step 4.

For clinical questions, consider using this framework

Example:
For patients 70 years or older, how effective is the use of the influenza vaccine at preventing flu as compared to patients who have not received the vaccine?

For qualitative questions, consider using this framework

Example:
What are the lived experiences of patients with adolescent idiopathic scoliosis of having scoliosis and wearing a brace?

3. It may also be worth considering the following elements:

• Type of question
  Is your question about therapy, diagnosis/screening, prognosis, aetiology//harm, prevention, experiences?
   
• Type of study
  What study design would best answer my question?
  Randomised controlled trials, cohort studies, case control studies, case series or case reports, qualitative studies

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 5: Defining the review question and developing criteria for including studies
• Cumpston M. Defining a review question (Cochrane slidecast)
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 2: Qualitative protocol and title development or Chapter 5: Quantitative protocol and title development
• OvidSP. Structured searching PICO worksheet
• UWA Medical and Dental Library. PICO guide

For question frameworks:
• Primary Health Care Research & Information Service. Getting started guide: formulating a research question
• Kloda LA, Bartlett JC. Formulating answerable questions: question negotiation in evidence-based practice. J Can Health Libr Assoc. 2013 Aug 1;34(2):55-60

Step 3 - Write a protocol

A protocol is a document that presents an explicit plan for a systematic review. It details the rationale and a priori methodological and analytical approach of the review.

References for protocol preparation:

• A 17 item PRISMA-P checklist is available from PRISMA for systematic review protocols (PRISMA-P)
   
• PROSPERO - International Prospective Register of Systematic Reviews
  See Guidance notes for registering a systematic review protocol with PROSPERO
  
  An example of a PROSPERO protocol:
  Bloom D, Fonseca E, Johnson S, Stawasz A. The full benefits of adult pneumococcal vaccination: a systematic review. PROSPERO 2016:CRD42016038335
   
• For Cochrane systematic review protocols, interested authors should contact the Cochrane Review Group that covers their interest area for information about the Cochrane process.
  Writing a protocol, this Cochrane slidecast introduces the process.
  
  An example of a Cochrane protocol:
  Zegelbone PM, Reljic T, Wilson D, Mhaskar R, Miladinovic B, Kumar A, Djulbegovic B. Chemoprevention agents for prostate cancer (Protocol). Cochrane Database of Systematic Reviews 2016, Issue 6. Art. No.: CD012228
   
• Joanna Briggs Institute (JBI)
  See Joanna Briggs Institute Reviewers’ Manual: 2014 edition for details on protocol requirements.
  
  An example of a JBI protocol:
  Effectiveness of pressure ulcer prevention strategies for adult patients in intensive care units: a systematic review protocol. JBI Database of Systematic Reviews and Implementation Reports. 2016 1;14(3):35-44.

References:
• Centre for Reviews and Dissemination, University of York. Guidance notes for registering a systematic review protocol with PROSPERO
• Cochrane Handbook for Systematic Reviews of Interventions. Part 1, Chapter 4: Guide to the contents of a Cochrane protocol and review
• Cumpston M. Writing a protocol (Cochrane slidecast)
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 2: Qualitative protocol and title development or Chapter 5: Quantitative protocol and title development
• Livingstone N, Sambunjak D, Watts C. Common errors and best practice when writing a review protocol (90 min. Cochrane webinar)
• Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1

Step 4 - Conduct a literature search

The literature searching process identifies published and unpublished articles and other information that addresses the research question.

If you are unfamiliar with searching databases, getting help from a librarian is recommended.
 

1. Where to search

♦ Major clinical databases

The databases you search will depending on your research area. More than one database should be searched to ensure you are capturing relevant literature.

See How to Search Databases tab to get more detailed instructions on constructing a search strategy.

♦ Clinical trial registers

• Australian Cancer Trials
• Australian Clinical Trials (NHMRC)
• Australian New Zealand Clinical Trials Registry (ANZCTR)
• ClinicalTrials.gov
• Cochrane Central Register of Controlled Trials (CENTRAL)
• International Clinical Trials Registry Platform (ICTRP)
• ISRCTN Registry
• Western Australia Cancer Clinical Trials Registry
   

♦ Grey literature

This is literature that has not been formally published.

It may include reports, conference proceedings, theses or dissertations, clinical trials, government documents, census data, standards, practice guidelines and more.

A selection of grey literature resources:

• Australian Research Online (via Trove)
• Google       Tip: use a selection of the main keywords. Limit to site:org and/or site:gov domains and by filetype:pdf
• Google Scholar
• Open Grey
• Open Access - Theses & Dissertations

Reference:
• University of Western Australia. Information for systematic reviewers: Grey literature

♦ Handsearching

This is an extended search technique that of involves manually looking through resources that may contain useful references.

Some suggestions:

• Look at the references listed in included articles
• Look at the references listed in similar reviews
• Contact authors or experts in the subject area for advice
   

2. Consider using search filters

Search filters, or hedges, are sets of search terms designed by specialists to retrieve certain kinds of results. They are useful for narrowing to particular age groups or particular study designs such as randomised controlled trials.

The Cochrane Handbook for Systematic Reviews of Interventions (Chapter 6.4.11) lists randomised controlled trial filters for Ovid Medline and PubMed. 

Some pre-tested search filters:

• Centre for Reviews and Dissemination ISSG Search Filters Resource 
• McMaster University Hedges
• SIGN (Scottish Intercollegiate Guidelines Network) Search Filters

3. Document of the search strategy

The search strategy for a systematic review needs to be saved and documented so that it can be reported in the review. The search strategy is recorded so that it can be reproduced if needed. This information is often found in the appendices, abstract, methods or the results sections.

The following should be recorded:

• all databases searched, together with the platform searched (eg. OVID Medline, Ovid Embase)
• date each search were conducted
• the full search strategy (including subject headings and keywords)
• the total number of hits retrieved for each search strategy
• number of duplicates identified
• numbers of articles included and excluded

The PRISMA Flow Diagram is highly recommended as the flow diagram maps the number of records identified, included and excluded, and the reasons for exclusions.

Example of search strategy documentation:

Zhou H, Della PR, Roberts P, Goh L, Dhaliwal SS. Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open. 2016; 6:6:e011060
(See the methods, results and supplementary appendix)

Tanner J, Dumville JC, Norman G, Fortnam M. Surgical hand antisepsis to reduce surgical site infection. Cochrane Database of Systematic Reviews 2016, Issue 1. Art. No.: CD004288
(See the methods and appendices 1-4)

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.4: Designing search strategies
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.6: Documenting and reporting the search process
• IOM (Institute of Medicine). Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 13: An introduction to searching
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 7 & 8)
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (flow diagram)
• Yale University Library. Systematic reviews (11 short videos covering the fundamental concepts of systematic searching of the literature)

Step 5 - Select relevant studies

This step involves reading the title and abstract of articles to identify those that fit your inclusion criteria. This should be carried out by more than one researcher. The full text of relevant articles will be needed.

Citation management software

This software can help you import references from multiple databases, remove duplicates, organise and manage the references, format bibliographies and include in-text citations within manuscripts.

• EndNote is available to OPH staff. For home installation, see the EndNote@Home section. Log a job with the Health Information Network to have EndNote installed on your work PC.
• Mendeley is a free reference manager. Available for Windows, Mac, Linux, iPad, iPhone and Android.

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 7: Selecting studies and collecting data
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 9 & 17)

Step 6 - Appraise the quality of studies

This step involves assessing the methodological quality of each included article. This should be carried out by more than one researcher.
Some points to consider:

• internal validity - the degree the study design, conduct, analysis and presentation has minimised or avoided bias. Main types of bias include selection, performance, detection, attrition and reporting.
• external validity - the extent findings are generalisable to other populations

Qualitative studies are often judged on the basis of authenticity and trustworthiness rather than by validity or reliability

Some critical appraisal tools to consider:

• Critical Appraisal Skills Programme (CASP)
• Joanna Briggs Institute. Critical appraisal tools
• GRADE Working Group. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. 2013
• McMaster University Occupational Therapy Evidence-based Practice Group. Quantitative and qualitative research review forms
• Scottish Intercollegiate Guidelines Network. Critical appraisal: notes and checklists
   

References:
• Agency for Healthcare Research and Quality (AHRQ). Assessing the risk of bias of individual studies in systematic reviews of health care interventions
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 8: Assessing risk of bias in included studies
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 12, 17, 19 & 22)

Step 7 - Extract data from studies

This step involves extracting the data from each article that is needed to answer the research question. It's usual for more than one researcher to be involved in this process. To standardise the process, it's crucial that a data extraction form consists of elements dependent on the research question.

Some data extraction tools:

• Abstrackr
• Covidence
• DistillerSR
• JBI-CReMS (for Joanna Briggs Institute)
• OpenMeta[Analyst]
• RevMan (for Cochrane Reviews)
• Systematic Review Data Repository (SRDR)
   

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 7.3: What data to collect
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist item 10, 11)

Step 8 - Analyse and synthesise studies

This step involves analysing and summarising the data to answer the research question. In a quantitative review, if the results are similar, then a meta-analysis may be possible. Consider presenting your findings in a tabulated or chart form.
 

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 9: Analysing data and undertaking meta-analyses
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 11: Presenting results and ‘Summary of findings’ tables
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Step 9 - Report what is known and not known

This step involves writing up and publishing your results. Discuss the strengths and weaknesses of the included studies as well as any related biases. Outline the strength of the evidence found and make conclusions based on the best evidence.
Include all sections according to the PRISMA checklist.

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 12: Interpreting results and drawing conclusions
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses

How to search databases

Databases have their own particular interface so how you perform a search may vary in each one. Searching for one concept at a time and then using the Boolean operators (OR, AND) to combine concepts, is the principle applicable to searching any database.
Most databases use their own controlled vocabulary so search strategies may use different terms in each database. Using a combination of subject headings and keywords when searching is recommended.

If you have used the question frameworks suggested in step 2 to identify your research question, beginning the searching process will be easier.
Not all of the concepts in the framework will need to be used when searching. Typically a search will include the population AND intervention, sometimes the comparison and the outcome may not included.

This example question will show the searching process:

Is laparoscopic treatment (key hole surgery) equivalent to open surgical treatment for treatment of people with gastric (stomach) cancer?

Broken into PICO (Population / Intervention / Comparison / Outcome) framework:

Search for each part of your question separately. The O (outcome) of the question will not be used.

Type of study that will best answer this - randomised controlled trials
 

♦ Population search - stomach cancer

Identify synonyms and related terms and always use a combination of subject headings and keywords

Example of some search terms:
• Subject headings – Abdominal Neoplasms, Intestinal Neoplasms, Stomach Neoplasms
• Keywords – (stomach or intest* or gastr* or gut or digest* or abdom*) adjacent to (cancer* or neoplasm* or tumo?r* or malig*)

Subject heading and keywords are combined with OR
 

♦ Intervention search – laparoscopic treatment

Identify synonyms and related terms and always use a combination of subject headings and keywords

Example of some search terms:
• Subject heading – Laparoscopy
• Keywords – laparoscop*

                                                                                        Subject heading and keywords are combined with OR
 

♦ Comparison search - open surgical treatment

Identify synonyms and related terms and always use a combination of subject headings and keywords

Example of some search terms:
• Subject heading – Gastrectomy
• Keywords – gastrectom*

                                                                                        Subject heading and keywords are combined with OR

These 3 searches are then combined with the Boolean operator AND

Population search AND Intervention search AND Comparison search

The search results will contain these 3 concepts.
 

♦ Using a search filter

Search filters are sets of search terms designed by specialists to retrieve certain kinds of results. They are useful for narrowing to particular age groups or particular study designs such as randomised controlled trials.

The Cochrane Library Handbook (section 6.4.11) lists randomised controlled trial filters for Ovid Medline and PubMed. 

Some pre-tested search filters:

• Centre for Reviews and Dissemination ISSG Search Filters Resource 
• McMaster University Hedges
• SIGN (Scottish Intercollegiate Guidelines Network) Search Filters

♦ The final search strategy

 This is what the search strategy might look like in the Ovid Medline database.

Other databases tips

• Truncation - Use this to ensure retrieval of all possible variations of a word. Each database uses a different truncation symbol so check the database help.
  Example: laparoscop* will find laparoscopy or laparoscopic
   
• Wild cards – these stand for zero or one character within a word or at the end of a word. Each database uses a different wild card symbol so check the database help.
  # (one character)    Example: wom#n       This will look for women or woman
  ? (zero or one character)   Example: tumo?r        This will look for tumor or tumour
  Used when looking for differences with American and British spellings.
   
• Proximity - or adjacency searching allows you to specify the proximity of words to each other. Each database uses different proximity wording so check the database help.
  Example: abdom* adj2 (cancer* or neoplasm*)     This will look for the words variations of abdominal, abdomen next to the word variations of cancer, cancers, neoplasm, neoplasms
  The number with adj specifies the proximity of words from another. Adj is used in Ovid MEDLINE and Embase.

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.4:  Designing search strategies
• Cumpston M. Defining a review question (Cochrane slidecast)
• Kloda LA, Bartlett JC. Formulating answerable questions: question negotiation in evidence-based practice. J Can Health Libr Assoc. 2013 Aug 1;34(2):55-60
• Primary Health Care Research & Information Service. Getting started guide: formulating a research question
• UWA Medical and Dental Library. PICO guide
• Yale University Library. Systematic reviews (11 short videos covering the fundamental concepts of systematic searching of the literature)

From: Bettany-Saltikov J. Learning how to undertake a systematic review: part 1. Nurs Stand. 2010; 24(50):47-55
 

Examples of Reviews

♦ Cochrane Systematic Review

• Sodium-glucose cotransporter (SGLT) 2 inhibitors for prevention or delay of type 2 diabetes mellitus and its associated complications in people at risk for the development of type 2 diabetes mellitus

♦ Systematic Review

• Use of non-pharmaceutical interventions to reduce the transmission of influenza in adults: a systematic review

• The effects of surgical hand scrubbing protocols on skin integrity and surgical site infection rates: a systematic review

♦ Literature Review

• Pharmacological management of bronchorrhea in malignant disease: a systematic literature review

• Time to analgesia and pain score documentation best practice standards for the Emergency Department – a literature review

♦ Joanna Briggs Institute Systematic Review

• The influence of workplace culture on nurses’ learning experiences: a systematic review of qualitative evidence

For further information on review types:
• Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009 Jun;26(2):91-108