Skip to main content

How To Guides (OPH): How to do a systematic review

What is a Systematic Review

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question.


The key characteristics of a systematic review are:
•  a clearly stated set of objectives with pre-defined eligibility criteria for studies
•  an explicit, reproducible methodology
•  a systematic search that attempts to identify all studies that meet the eligibility criteria
•  an assessment of the validity of findings of the included studies
•  a systematic presentation & synthesis of the characteristics & findings of the included studies

Systematic reviews can be either quantitative or qualitative.

References:

Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions, 2011

• Image: Centre for Health Communication and Participation, La Trobe University. 'The concept of a systematic review' 2011

Select a Tab

Useful resources

• Agency for Healthcare Research and Quality (AHRQ). Methods guide for effectiveness and comparative effectiveness reviews
Cochrane handbook for systematic reviews of interventions
EQUATOR Network - Enhancing the QUAlity and Transparency Of health Research
• IOM (Institute of Medicine). Finding what works in health care: standards for systematic reviews
Joanna Briggs Institute reviewers’ manual: 2014 edition
PRISMA - Preferred Reporting Items for Systematic Reviews and Meta-Analyses
PRISMA for Systematic Review Protocols (PRISMA-P)

 

References on the systematic review process

• Bettany-Saltikov J. Learning how to undertake a systematic review: part 1. Nurs Stand. 2010; 24(50):47-55
• Bettany-Saltikov J. Learning how to undertake a systematic review: part 2. Nurs Stand. 2010; 24(51):47-56
• Wright RW, Brand RA, Dunn W, Spindler KP. How to write a systematic review. Clin Orthop Relat Res. 2007; 455:23-9
• Umscheid CA. A primer on performing systematic reviews and meta-analyses. Clin Infect Dis. 2013 Sep 1;57(5):725-34

Step 1 - Assemble a team

A systematic review should be undertaken by more than one person so that it decreases the likelihood of errors.
It is recommended that tasks such as the selection of studies for eligibility and data extraction be performed by at least two people. Consulting a librarian will helpful when searching. Having someone with experience in quantitative methods may also be required.

References:
• IOM (Institute of Medicine). Standards for initiating a systematic review


Step 2 - Formulate a question

The review question needs to be clear and well defined.
 

1. Check that a review has not already been done on your research topic

Where to search:

Subject specific resources


2. Use question frameworks

These will help you develop an answerable and focused question. Developing concepts will help during the searching process at step 4.

For clinical questions, consider using this framework

Example:
For patients 70 years or older, how effective is the use of the influenza vaccine at preventing flu as compared to patients who have not received the vaccine?

For qualitative questions, consider using this framework

Example:
What are the lived experiences of patients with adolescent idiopathic scoliosis of having scoliosis and wearing a brace?


3. It may also be worth considering the following elements:

  • Type of question
    Is your question about therapy, diagnosis/screening, prognosis, aetiology//harm, prevention, experiences?
     
  • Type of study
    What study design would best answer my question?
    Randomised controlled trials, cohort studies, case control studies, case series or case reports, qualitative studies


References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 5: Defining the review question and developing criteria for including studies
• Cumpston M. Defining a review question (Cochrane slidecast)
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 2: Qualitative protocol and title development or Chapter 5: Quantitative protocol and title development
• OvidSP. Structured searching PICO worksheet
• UWA Medical and Dental Library. PICO guide

 

For question frameworks:
• Primary Health Care Research & Information Service. Getting started guide: formulating a research question
• Kloda LA, Bartlett JC. Formulating answerable questions: question negotiation in evidence-based practice. J Can Health Libr Assoc. 2013 Aug 1;34(2):55-60

Step 3 - Write a protocol

A protocol is a document that presents an explicit plan for a systematic review. It details the rationale and a priori methodological and analytical approach of the review.

References for protocol preparation:


References:
• Centre for Reviews and Dissemination, University of York. Guidance notes for registering a systematic review protocol with PROSPERO
• Cochrane Handbook for Systematic Reviews of Interventions. Part 1, Chapter 4: Guide to the contents of a Cochrane protocol and review
• Cumpston M. Writing a protocol (Cochrane slidecast)
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 2: Qualitative protocol and title development or Chapter 5: Quantitative protocol and title development
• Livingstone N, Sambunjak D, Watts C. Common errors and best practice when writing a review protocol (90 min. Cochrane webinar)
• Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1

Step 4 - Conduct a literature search

The literature searching process identifies published and unpublished articles and other information that addresses the research question.

If you are unfamiliar with searching databases, getting help from a librarian is recommended.
 

1. Where to search

Major clinical databases

The databases you search will depending on your research area. More than one database should be searched to ensure you are capturing relevant literature.

Database

Subject Coverage and Indexing

CINAHL            Subscription database
Cumulative Index to Nursing & Allied Health Literature
Coverage: 1937- present     Platform: EBSCO

Nursing, Biomedicine, Alternative/Complementary Medicine, Consumer Health, Allied Health

Uses controlled vocabulary thesaurus - CINAHL Subject Headings

Embase            Subscription database
Coverage: 1974 – present
Publisher: Elsevier     Platform: Ovid

Biomedicine, Pharmacology, Psychiatry, Mental Health, Dentistry, Biotechnology, Health Policy and Management, Public, Occupational and Environmental Health, Nursing

Includes MEDLINE citations & conference proceedings

Uses controlled vocabulary thesaurus - Emtree

Ovid MEDLINE            Subscription database
Coverage: 1946 – present
Publisher: U.S. National Library of Medicine
Platform: Ovid

Biomedicine, Nursing, Psychiatry, Allied Health

Use Ovid MEDLINE for complex searches as it allows for better search strategy manipulation than PubMed.

If you have already searched PubMed, you don't need to search Ovid Medline.

Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH)

PubMed            Free database
Coverage: 1946 – present
Publisher: U.S. National Library of Medicine

Use the link provided by SCGH Library to access library subscribed content

Biomedicine, Nursing, Psychiatry, Allied Health

Provides free access to MEDLINE.

Excellent for quick searches. Advanced searching is more difficult.
Use Ovid MEDLINE for complex searches as it allows for better search strategy manipulation.

If you have already searched Ovid Medline, you don't need to search PubMed.

Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH)

PsycINFO            Subscription database 
Coverage: 1806 – present
Publisher: American Psychological Association     Platform: Ovid

Psychology, Psychiatry, Social Work, Speech Language & Hearing

Uses controlled vocabulary thesaurus - Thesaurus of Psychological Index Terms

Cochrane Library             Free access in Australia

Publisher: John Wiley & Sons

Systematic Reviews, Clinical Trials

Includes the Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), Health Technology Assessment Database

Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH) in the Advanced Search

JBI EBP Database             Subscription database

Coverage: 1998 – present
Publisher Joanna Briggs Institute     Platform: Ovid

Nursing and Allied Health Systematic Reviews

Includes the JBI Library of Systematic Reviews, Best Practice Information sheets, Evidence Summaries, Evidence Based Recommended Practices

Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH)


See How to Search Databases tab to get more detailed instructions on constructing a search strategy.

Clinical trial registers

♦ Grey literature

This is literature that has not been formally published.

It may include reports, conference proceedings, theses or dissertations, clinical trials, government documents, census data, standards, practice guidelines and more.

A selection of grey literature resources:

Reference:
• University of Western Australia. Information for systematic reviewers: Grey literature


♦ Handsearching

This is an extended search technique that of involves manually looking through resources that may contain useful references.

Some suggestions:

  • Look at the references listed in included articles
  • Look at the references listed in similar reviews
  • Contact authors or experts in the subject area for advice
     

2. Consider using search filters

Search filters, or hedges, are sets of search terms designed by specialists to retrieve certain kinds of results. They are useful for narrowing to particular age groups or particular study designs such as randomised controlled trials.

The Cochrane Handbook for Systematic Reviews of Interventions (Chapter 6.4.11) lists randomised controlled trial filters for Ovid Medline and PubMed. 

Some pre-tested search filters:


3. Document of the search strategy

The search strategy for a systematic review needs to be saved and documented so that it can be reported in the review. The search strategy is recorded so that it can be reproduced if needed. This information is often found in the appendices, abstract, methods or the results sections.

The following should be recorded:

  • all databases searched, together with the platform searched (eg. OVID Medline, Ovid Embase)
  • date each search were conducted
  • the full search strategy (including subject headings and keywords)
  • the total number of hits retrieved for each search strategy
  • number of duplicates identified
  • numbers of articles included and excluded

The PRISMA Flow Diagram is highly recommended as the flow diagram maps the number of records identified, included and excluded, and the reasons for exclusions.

Example of search strategy documentation:

Zhou H, Della PR, Roberts P, Goh L, Dhaliwal SS. Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open. 2016; 6:6:e011060
(See the methods, results and supplementary appendix)

Tanner J, Dumville JC, Norman G, Fortnam M. Surgical hand antisepsis to reduce surgical site infection. Cochrane Database of Systematic Reviews 2016, Issue 1. Art. No.: CD004288
(See the methods and appendices 1-4)


References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.4: Designing search strategies
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.6: Documenting and reporting the search process
• IOM (Institute of Medicine). Standards for finding and assessing individual studies
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 13: An introduction to searching
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 7 & 8)
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (flow diagram)
• Yale University Library. Systematic reviews (11 short videos covering the fundamental concepts of systematic searching of the literature)

Step 5 - Select relevant studies

This step involves reading the title and abstract of articles to identify those that fit your inclusion criteria. This should be carried out by more than one researcher. The full text of relevant articles will be needed.

Citation management software

This software can help you import references from multiple databases, remove duplicates, organise and manage the references, format bibliographies and include in-text citations within manuscripts.


References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 7: Selecting studies and collecting data
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 9 & 17)


Step 6 - Appraise the quality of studies

This step involves assessing the methodological quality of each included article. This should be carried out by more than one researcher.
Some points to consider:

  • internal validity - the degree the study design, conduct, analysis and presentation has minimised or avoided bias. Main types of bias include selection, performance, detection, attrition and reporting.
  • external validity - the extent findings are generalisable to other populations

Qualitative studies are often judged on the basis of authenticity and trustworthiness rather than by validity or reliability

Some critical appraisal tools to consider:

References:
• Agency for Healthcare Research and Quality (AHRQ). Assessing the risk of bias of individual studies in systematic reviews of health care interventions
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 8: Assessing risk of bias in included studies
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 12, 17, 19 & 22)


Step 7 - Extract data from studies

This step involves extracting the data from each article that is needed to answer the research question. It's usual for more than one researcher to be involved in this process. To standardise the process, it's crucial that a data extraction form consists of elements dependent on the research question.

Some data extraction tools:

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 7.3: What data to collect
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist item 10, 11)

Step 8 - Analyse and synthesise studies

This step involves analysing and summarising the data to answer the research question. In a quantitative review, if the results are similar, then a meta-analysis may be possible. Consider presenting your findings in a tabulated or chart form.
 

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 9: Analysing data and undertaking meta-analyses
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 11: Presenting results and ‘Summary of findings’ tables
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses


Step 9 - Report what is known and not known

This step involves writing up and publishing your results. Discuss the strengths and weaknesses of the included studies as well as any related biases. Outline the strength of the evidence found and make conclusions based on the best evidence.
Include all sections according to the PRISMA checklist.

 

References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 12: Interpreting results and drawing conclusions
Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses

How to search databases

Databases have their own particular interface so how you perform a search may vary in each one. Searching for one concept at a time and then using the Boolean operators (OR, AND) to combine concepts, is the principle applicable to searching any database.
Most databases use their own controlled vocabulary so search strategies may use different terms in each database. Using a combination of subject headings and keywords when searching is recommended.

If you have used the question frameworks suggested in step 2 to identify your research question, beginning the searching process will be easier.
Not all of the concepts in the framework will need to be used when searching. Typically a search will include the population AND intervention, sometimes the comparison and the outcome may not included.

This example question will show the searching process:

Is laparoscopic treatment (key hole surgery) equivalent to open surgical treatment for treatment of people with gastric (stomach) cancer?

Broken into PICO (Population / Intervention / Comparison / Outcome) framework:

Search for each part of your question separately. The O (outcome) of the question will not be used.

Type of study that will best answer this - randomised controlled trials
 

♦ Population search - stomach cancer

Identify synonyms and related terms and always use a combination of subject headings and keywords

Example of some search terms:
• Subject headings – Abdominal Neoplasms, Intestinal Neoplasms, Stomach Neoplasms
• Keywords – (stomach or intest* or gastr* or gut or digest* or abdom*) adjacent to (cancer* or neoplasm* or tumo?r* or malig*)

Subject heading and keywords are combined with OR
 

 

♦ Intervention search – laparoscopic treatment

Identify synonyms and related terms and always use a combination of subject headings and keywords

Example of some search terms:
• Subject heading – Laparoscopy
• Keywords – laparoscop*

                                                                                        Subject heading and keywords are combined with OR
 

 

♦ Comparison search - open surgical treatment

Identify synonyms and related terms and always use a combination of subject headings and keywords

Example of some search terms:
• Subject heading – Gastrectomy
• Keywords – gastrectom*

                                                                                        Subject heading and keywords are combined with OR

 

These 3 searches are then combined with the Boolean operator AND

Population search AND Intervention search AND Comparison search

The search results will contain these 3 concepts.
 

 


♦ Using a search filter

Search filters are sets of search terms designed by specialists to retrieve certain kinds of results. They are useful for narrowing to particular age groups or particular study designs such as randomised controlled trials.

The Cochrane Library Handbook (section 6.4.11) lists randomised controlled trial filters for Ovid Medline and PubMed. 

Some pre-tested search filters:


♦ The final search strategy

 This is what the search strategy might look like in the Ovid Medline database.


 

Other databases tips

  • Truncation - Use this to ensure retrieval of all possible variations of a word. Each database uses a different truncation symbol so check the database help.
    Example: laparoscop* will find laparoscopy or laparoscopic
     
  • Wild cards – these stand for zero or one character within a word or at the end of a word. Each database uses a different wild card symbol so check the database help.
    # (one character)    Example: wom#      This will look for women or woman
    ? (zero or one character)   Example: tumo?       This will look for tumor or tumour
    Used when looking for differences with American and British spellings.
     
  • Proximity - or adjacency searching allows you to specify the proximity of words to each other. Each database uses different proximity wording so check the database help.
    Example: abdom* adj2 (cancer* or neoplasm*)     This will look for the words variations of abdominal, abdomen next to the word variations of cancer, cancers, neoplasm, neoplasms
    The number with adj specifies the proximity of words from another. Adj is used in Ovid MEDLINE and Embase.


References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.4:  Designing search strategies
• Cumpston M. Defining a review question (Cochrane slidecast)
• Kloda LA, Bartlett JC. Formulating answerable questions: question negotiation in evidence-based practice. J Can Health Libr Assoc. 2013 Aug 1;34(2):55-60
• Primary Health Care Research & Information Service. Getting started guide: formulating a research question
• UWA Medical and Dental Library. PICO guide
• Yale University Library. Systematic reviews (11 short videos covering the fundamental concepts of systematic searching of the literature)

 

Systematic review

Literature review

Question

Focused on a clearly defined single question

Not necessarily focused on a single question, but may describe an overview

Protocol

A protocol or plan is included

No protocol is included

Background

Both provide summaries of the available literature on a topic

Objectives

Clear objectives are identified

Objectives may or may not be identified

Inclusion / Exclusion Criteria

Criteria stated before the review is conducted

Criteria not specified

Search Strategy

Comprehensive search conducted to locate all relevant research with strategy stated

May not attempt to locate all relevant research and strategy not explicitly stated

Article Selection

Usually clear and explicit

Not described

Article Evaluation

Comprehensive evaluation of study quality

Evaluation of study quality may or may not be included

Data Extraction

Usually clear and specific

Not clear and explicit

Results / Data Synthesis

Clear summaries of studies based on high quality evidence

Summary based on studies where the quality of the articles may not be specified. May also be influenced by the reviewer's theories, needs and beliefs

Discussion

Written by an expert or group of experts with a detailed and well grounded knowledge of the issues

From: Bettany-Saltikov J. Learning how to undertake a systematic review: part 1. Nurs Stand. 2010; 24(50):47-55
 

Examples of Reviews

♦ Cochrane Systematic Review

Sodium-glucose cotransporter (SGLT) 2 inhibitors for prevention or delay of type 2 diabetes mellitus and its associated complications in people at risk for the development of type 2 diabetes mellitus

♦ Systematic Review

Use of non-pharmaceutical interventions to reduce the transmission of influenza in adults: a systematic review

The effects of surgical hand scrubbing protocols on skin integrity and surgical site infection rates: a systematic review

♦ Literature Review

Pharmacological management of bronchorrhea in malignant disease: a systematic literature review

Time to analgesia and pain score documentation best practice standards for the Emergency Department – a literature review

♦ Joanna Briggs Institute Systematic Review

The influence of workplace culture on nurses’ learning experiences: a systematic review of qualitative evidence

 

For further information on review types:
• Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009 Jun;26(2):91-108