The key characteristics of a systematic review are:
• a clearly stated set of objectives with pre-defined eligibility criteria for studies
• an explicit, reproducible methodology
• a systematic search that attempts to identify all studies that meet the eligibility criteria
• an assessment of the validity of findings of the included studies
• a systematic presentation & synthesis of the characteristics & findings of the included studies
Systematic reviews can be either quantitative or qualitative.
References:
• Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions, 2011
• Image: Centre for Health Communication and Participation, La Trobe University. 'The concept of a systematic review' 2011
Useful resources
• Agency for Healthcare Research and Quality (AHRQ). Methods guide for effectiveness and comparative effectiveness reviews
• Cochrane handbook for systematic reviews of interventions
• EQUATOR Network - Enhancing the QUAlity and Transparency Of health Research
• IOM (Institute of Medicine). Finding what works in health care: standards for systematic reviews
• Joanna Briggs Institute reviewers’ manual: 2014 edition
• PRISMA - Preferred Reporting Items for Systematic Reviews and Meta-Analyses
• PRISMA for Systematic Review Protocols (PRISMA-P)
References on the systematic review process
• Bettany-Saltikov J. Learning how to undertake a systematic review: part 1. Nurs Stand. 2010; 24(50):47-55
• Bettany-Saltikov J. Learning how to undertake a systematic review: part 2. Nurs Stand. 2010; 24(51):47-56
• Wright RW, Brand RA, Dunn W, Spindler KP. How to write a systematic review. Clin Orthop Relat Res. 2007; 455:23-9
• Umscheid CA. A primer on performing systematic reviews and meta-analyses. Clin Infect Dis. 2013 Sep 1;57(5):725-34
Step 1 - Assemble a team
A systematic review should be undertaken by more than one person so that it decreases the likelihood of errors.
It is recommended that tasks such as the selection of studies for eligibility and data extraction be performed by at least two people. Consulting a librarian will helpful when searching. Having someone with experience in quantitative methods may also be required.
References:
• IOM (Institute of Medicine). Standards for initiating a systematic review
Step 2 - Formulate a question
The review question needs to be clear and well defined.
1. Check that a review has not already been done on your research topic
Where to search:
Subject specific resources
2. Use question frameworks
These will help you develop an answerable and focused question. Developing concepts will help during the searching process at step 4.
For clinical questions, consider using this framework
Example:
For patients 70 years or older, how effective is the use of the influenza vaccine at preventing flu as compared to patients who have not received the vaccine?
For qualitative questions, consider using this framework
Example:
What are the lived experiences of patients with adolescent idiopathic scoliosis of having scoliosis and wearing a brace?
3. It may also be worth considering the following elements:
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 5: Defining the review question and developing criteria for including studies
• Cumpston M. Defining a review question (Cochrane slidecast)
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 2: Qualitative protocol and title development or Chapter 5: Quantitative protocol and title development
• OvidSP. Structured searching PICO worksheet
• UWA Medical and Dental Library. PICO guide
For question frameworks:
• Primary Health Care Research & Information Service. Getting started guide: formulating a research question
• Kloda LA, Bartlett JC. Formulating answerable questions: question negotiation in evidence-based practice. J Can Health Libr Assoc. 2013 Aug 1;34(2):55-60
Step 3 - Write a protocol
A protocol is a document that presents an explicit plan for a systematic review. It details the rationale and a priori methodological and analytical approach of the review.
References for protocol preparation:
References:
• Centre for Reviews and Dissemination, University of York. Guidance notes for registering a systematic review protocol with PROSPERO
• Cochrane Handbook for Systematic Reviews of Interventions. Part 1, Chapter 4: Guide to the contents of a Cochrane protocol and review
• Cumpston M. Writing a protocol (Cochrane slidecast)
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 2: Qualitative protocol and title development or Chapter 5: Quantitative protocol and title development
• Livingstone N, Sambunjak D, Watts C. Common errors and best practice when writing a review protocol (90 min. Cochrane webinar)
• Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1
Step 4 - Conduct a literature search
The literature searching process identifies published and unpublished articles and other information that addresses the research question.
If you are unfamiliar with searching databases, getting help from a librarian is recommended.
1. Where to search
♦ Major clinical databases
The databases you search will depending on your research area. More than one database should be searched to ensure you are capturing relevant literature.
Database |
Subject Coverage and Indexing |
CINAHL Subscription database |
Nursing, Biomedicine, Alternative/Complementary Medicine, Consumer Health, Allied Health Uses controlled vocabulary thesaurus - CINAHL Subject Headings |
Embase Subscription database |
Biomedicine, Pharmacology, Psychiatry, Mental Health, Dentistry, Biotechnology, Health Policy and Management, Public, Occupational and Environmental Health, Nursing Includes MEDLINE citations & conference proceedings Uses controlled vocabulary thesaurus - Emtree |
Ovid MEDLINE Subscription database |
Biomedicine, Nursing, Psychiatry, Allied Health Use Ovid MEDLINE for complex searches as it allows for better search strategy manipulation than PubMed. If you have already searched PubMed, you don't need to search Ovid Medline. Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH) |
PubMed Free database |
Biomedicine, Nursing, Psychiatry, Allied Health Provides free access to MEDLINE. Excellent for quick searches. Advanced searching is more difficult. If you have already searched Ovid Medline, you don't need to search PubMed. Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH) |
PsycINFO Subscription database |
Psychology, Psychiatry, Social Work, Speech Language & Hearing Uses controlled vocabulary thesaurus - Thesaurus of Psychological Index Terms |
Cochrane Library Free access in Australia Publisher: John Wiley & Sons |
Systematic Reviews, Clinical Trials Includes the Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), Health Technology Assessment Database Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH) in the Advanced Search |
JBI EBP Database Subscription database Coverage: 1998 – present |
Nursing and Allied Health Systematic Reviews Includes the JBI Library of Systematic Reviews, Best Practice Information sheets, Evidence Summaries, Evidence Based Recommended Practices Uses controlled vocabulary thesaurus - Medical Subject Headings (MeSH) |
See How to Search Databases tab to get more detailed instructions on constructing a search strategy.
♦ Clinical trial registers
♦ Grey literature
This is literature that has not been formally published.
It may include reports, conference proceedings, theses or dissertations, clinical trials, government documents, census data, standards, practice guidelines and more.
A selection of grey literature resources:
Reference:
• University of Western Australia. Information for systematic reviewers: Grey literature
♦ Handsearching
This is an extended search technique that of involves manually looking through resources that may contain useful references.
Some suggestions:
2. Consider using search filters
Search filters, or hedges, are sets of search terms designed by specialists to retrieve certain kinds of results. They are useful for narrowing to particular age groups or particular study designs such as randomised controlled trials.
The Cochrane Handbook for Systematic Reviews of Interventions (Chapter 6.4.11) lists randomised controlled trial filters for Ovid Medline and PubMed.
Some pre-tested search filters:
3. Document of the search strategy
The search strategy for a systematic review needs to be saved and documented so that it can be reported in the review. The search strategy is recorded so that it can be reproduced if needed. This information is often found in the appendices, abstract, methods or the results sections.
The following should be recorded:
The PRISMA Flow Diagram is highly recommended as the flow diagram maps the number of records identified, included and excluded, and the reasons for exclusions.
Example of search strategy documentation:
Zhou H, Della PR, Roberts P, Goh L, Dhaliwal SS. Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open. 2016; 6:6:e011060
(See the methods, results and supplementary appendix)
Tanner J, Dumville JC, Norman G, Fortnam M. Surgical hand antisepsis to reduce surgical site infection. Cochrane Database of Systematic Reviews 2016, Issue 1. Art. No.: CD004288
(See the methods and appendices 1-4)
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.4: Designing search strategies
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.6: Documenting and reporting the search process
• IOM (Institute of Medicine). Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 13: An introduction to searching
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 7 & 8)
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (flow diagram)
• Yale University Library. Systematic reviews (11 short videos covering the fundamental concepts of systematic searching of the literature)
Step 5 - Select relevant studies
This step involves reading the title and abstract of articles to identify those that fit your inclusion criteria. This should be carried out by more than one researcher. The full text of relevant articles will be needed.
Citation management software
This software can help you import references from multiple databases, remove duplicates, organise and manage the references, format bibliographies and include in-text citations within manuscripts.
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 7: Selecting studies and collecting data
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 9 & 17)
Step 6 - Appraise the quality of studies
This step involves assessing the methodological quality of each included article. This should be carried out by more than one researcher.
Some points to consider:
Qualitative studies are often judged on the basis of authenticity and trustworthiness rather than by validity or reliability
Some critical appraisal tools to consider:
References:
• Agency for Healthcare Research and Quality (AHRQ). Assessing the risk of bias of individual studies in systematic reviews of health care interventions
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 8: Assessing risk of bias in included studies
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist items 12, 17, 19 & 22)
Step 7 - Extract data from studies
This step involves extracting the data from each article that is needed to answer the research question. It's usual for more than one researcher to be involved in this process. To standardise the process, it's crucial that a data extraction form consists of elements dependent on the research question.
Some data extraction tools:
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 7.3: What data to collect
• IOM (Institute of Medicine. Chapter 3: Standards for finding and assessing individual studies
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (checklist item 10, 11)
Step 8 - Analyse and synthesise studies
This step involves analysing and summarising the data to answer the research question. In a quantitative review, if the results are similar, then a meta-analysis may be possible. Consider presenting your findings in a tabulated or chart form.
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 9: Analysing data and undertaking meta-analyses
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 11: Presenting results and ‘Summary of findings’ tables
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Step 9 - Report what is known and not known
This step involves writing up and publishing your results. Discuss the strengths and weaknesses of the included studies as well as any related biases. Outline the strength of the evidence found and make conclusions based on the best evidence.
Include all sections according to the PRISMA checklist.
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 12: Interpreting results and drawing conclusions
• Joanna Briggs Institute reviewers’ manual: 2014 edition. See Chapter 3: The systematic review and synthesis of qualitative data or Chapter 6: The systematic review and synthesis of quantitative data
• The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses
How to search databases
Databases have their own particular interface so how you perform a search may vary in each one. Searching for one concept at a time and then using the Boolean operators (OR, AND) to combine concepts, is the principle applicable to searching any database.
Most databases use their own controlled vocabulary so search strategies may use different terms in each database. Using a combination of subject headings and keywords when searching is recommended.
If you have used the question frameworks suggested in step 2 to identify your research question, beginning the searching process will be easier.
Not all of the concepts in the framework will need to be used when searching. Typically a search will include the population AND intervention, sometimes the comparison and the outcome may not included.
This example question will show the searching process:
Is laparoscopic treatment (key hole surgery) equivalent to open surgical treatment for treatment of people with gastric (stomach) cancer?
Broken into PICO (Population / Intervention / Comparison / Outcome) framework:
Search for each part of your question separately. The O (outcome) of the question will not be used.
Type of study that will best answer this - randomised controlled trials
♦ Population search - stomach cancer
Identify synonyms and related terms and always use a combination of subject headings and keywords
Example of some search terms:
• Subject headings – Abdominal Neoplasms, Intestinal Neoplasms, Stomach Neoplasms
• Keywords – (stomach or intest* or gastr* or gut or digest* or abdom*) adjacent to (cancer* or neoplasm* or tumo?r* or malig*)
Subject heading and keywords are combined with OR
♦ Intervention search – laparoscopic treatment
Identify synonyms and related terms and always use a combination of subject headings and keywords
Example of some search terms:
• Subject heading – Laparoscopy
• Keywords – laparoscop*
Subject heading and keywords are combined with OR
♦ Comparison search - open surgical treatment
Identify synonyms and related terms and always use a combination of subject headings and keywords
Example of some search terms:
• Subject heading – Gastrectomy
• Keywords – gastrectom*
Subject heading and keywords are combined with OR
These 3 searches are then combined with the Boolean operator AND
Population search AND Intervention search AND Comparison search
The search results will contain these 3 concepts.
♦ Using a search filter
Search filters are sets of search terms designed by specialists to retrieve certain kinds of results. They are useful for narrowing to particular age groups or particular study designs such as randomised controlled trials.
The Cochrane Library Handbook (section 6.4.11) lists randomised controlled trial filters for Ovid Medline and PubMed.
Some pre-tested search filters:
♦ The final search strategy
This is what the search strategy might look like in the Ovid Medline database.
Other databases tips
References:
• Cochrane Handbook for Systematic Reviews of Interventions. Part 2, Chapter 6.4: Designing search strategies
• Cumpston M. Defining a review question (Cochrane slidecast)
• Kloda LA, Bartlett JC. Formulating answerable questions: question negotiation in evidence-based practice. J Can Health Libr Assoc. 2013 Aug 1;34(2):55-60
• Primary Health Care Research & Information Service. Getting started guide: formulating a research question
• UWA Medical and Dental Library. PICO guide
• Yale University Library. Systematic reviews (11 short videos covering the fundamental concepts of systematic searching of the literature)
Systematic review |
Literature review |
|
Question |
Focused on a clearly defined single question |
Not necessarily focused on a single question, but may describe an overview |
Protocol |
A protocol or plan is included |
No protocol is included |
Background |
Both provide summaries of the available literature on a topic |
|
Objectives |
Clear objectives are identified |
Objectives may or may not be identified |
Inclusion / Exclusion Criteria |
Criteria stated before the review is conducted |
Criteria not specified |
Search Strategy |
Comprehensive search conducted to locate all relevant research with strategy stated |
May not attempt to locate all relevant research and strategy not explicitly stated |
Article Selection |
Usually clear and explicit |
Not described |
Article Evaluation |
Comprehensive evaluation of study quality |
Evaluation of study quality may or may not be included |
Data Extraction |
Usually clear and specific |
Not clear and explicit |
Results / Data Synthesis |
Clear summaries of studies based on high quality evidence |
Summary based on studies where the quality of the articles may not be specified. May also be influenced by the reviewer's theories, needs and beliefs |
Discussion |
Written by an expert or group of experts with a detailed and well grounded knowledge of the issues |
From: Bettany-Saltikov J. Learning how to undertake a systematic review: part 1. Nurs Stand. 2010; 24(50):47-55
Examples of Reviews
♦ Cochrane Systematic Review
♦ Systematic Review
♦ Literature Review
• Pharmacological management of bronchorrhea in malignant disease: a systematic literature review
♦ Joanna Briggs Institute Systematic Review
For further information on review types:
• Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009 Jun;26(2):91-108