Systematic Literature Review

Systematic reviews and meta-analysis studies sit at the top of the evidence-based research pyramid, which means that they provide the strongest form of evidence available. These types of studies fill an essential role in the research literature, as they provide researchers and practitioners with a summary and assessment of the state of current knowledge on specific topics, all within a single research article. Systematic reviews are particularly important for topics that have been heavily researched, as they help to distill the findings of multiple studies down to much more succinct sets of results or findings.

Such distillation of the results of many separate studies can be important for a number of reasons. For example, practitioners in health and therapeutic fields rely on systematic reviews and meta-analysis research for keeping up to date on effectiveness of treatments and therapies for various health conditions. Researchers also draw upon the condensed results of systematic reviews and meta-analysis studies to identify areas of conflict or research gaps in their fields of study, both of which are helpful in developing directions for future research.

With Precision’s expertise in both quantitative and qualitative methods and analysis, we can assist with all stages of systematic review and meta-analysis from initial research question formulation through collection of relevant studies for the sample and analysis of pooled data. The key to quality in a systematic review and meta-analysis is a rigorous, transparent process from start to finish. The gold standard is the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which refers to the evidence-based minimum requirements for reporting in systematic reviews and meta-analyses.

Our assistance to clients with systematic reviews always follows the PRISMA checklist, which specifies information required in all major sections: title, abstract, introduction, methods, results, discussion, and funding. Reflecting the importance of a rigorous process to quality in a systematic review, the PRISMA checklist contains the highest number of required items in the methods (12 required items) and results (7 required items) sections.

In addition to systematic reviews and meta-analysis, we provide assistance on both integrative and narrative reviews. Do note, however, that because of a less systematic and transparent process, narrative reviews may contain bias or be viewed as containing bias. For our clients who are choosing to conduct review studies, we wholeheartedly encourage use of a systematic review process, as the narrative review process yields findings that are considered less trustworthy. However, there is a place in the literature for narrative reviews, and we are happy to assist with any form of review you prefer.

We can also assist with developing the manuscript to adhere to various formatting requirements. When working with established researchers who are seeking to publish their studies, we can assist with shaping the manuscript to comply with submission criteria for the selected peer-reviewed journal. Systematic reviews and meta-analysis can also be ideal for graduate research—usually at the master’s thesis level—and we can assist with all required phases of such research projects in accordance with any university guidelines.

Systematic Review

The overall goal of the systematic review is to locate and critically appraise all studies on a given topic, providing a synthesis of the study’s findings to answer one or more guiding research questions. The systematic review approach was first developed in the biomedical research field but has since been adopted by researchers of a variety of disciplines.

The hallmark of the systematic review is a transparent process that is aimed at reducing bias at every stage of the study. Toward that end, processes are established a priori for sample study search criteria, inclusion and exclusion criteria, quality evaluation, and data extraction and analysis. To meet the minimum requirements in these areas, a systematic review should be designed to conform with the PRISMA guidelines.

Systematic review and meta-analysis are often considered synonymous, but this is not actually the case. A systematic review often but not always involves a meta-analysis element, but this is only appropriate when studies chosen for the sample are sufficiently similar in terms of population, intervention, and outcome measurement. Studies with populations and/or variables that are too different cannot be combined for meta-analysis purposes but are instead summarized in a table, with comparisons of outcomes discussed in a qualitative sense.

A systematic review’s scope includes all studies on the topic relevant to the research question(s), which reduces the potential for selection bias. Because of the comprehensive scope, it is not uncommon for researchers to spend anywhere from 18 to 24 months working on a systematic review. Owing to their transparent, systematic method, systematic reviews are considered to be the basis for evidence-based treatment and practice and are therefore used by practitioners to guide decision making in clinical practice.
Meta-Analysis

As noted above, meta-analysis is often viewed as synonymous with systematic review, but in truth, meta-analysis is simply a subset of systematic review. Although not all systematic reviews include meta-analysis, all meta-analysis studies should be conducted within a systematic review context. This is because “systematic review” refers to the larger, transparent process detailed in the PRISMA guidelines, which are the elements that confer the high level of quality for which systematic reviews are known.

Meta-analysis refers to statistical techniques that are used to analyze the pooled results of multiple quantitative studies that have the same topic and focus (i.e., same population, independent variables, dependent variables). This involves transforming the results of multiple studies to allow for determination of overall effect size, confidence intervals, and assessment of heterogeneity among the studies included in the sample. Forest plots are also typically presented to visualize the results.
Integrative Review

Integrative review refers to a systematic review that addresses its research question(s) using studies with both quantitative and qualitative findings. Although you are likely to come across such studies that are titled as systematic reviews–in accordance with PRISMA guidelines–integrative review is another term that is used to refer specifically to a rigorous, transparent mixed methods review study.

All procedures described under systematic review (i.e., the PRISMA checklist) are followed in an integrative review, and meta-analysis may also be completed given sufficiently similar quantitative studies in the review sample. In addition, however, qualitative findings for similar research questions across two or more studies may be combined, allowing for a content analysis of the pooled findings.
Narrative Review

Narrative literature reviews are similar to systematic reviews in that they provide up-to-date knowledge about a specific topic or theme. However, narrative reviews are primarily descriptive and lack the systematic, transparent processes that make systematic reviews so trustworthy. Additionally, narrative reviews are not designed to conform to PRISMA standards.

This type of review study may or may not address specific research questions, but regardless, narrative reviews cannot definitively answer any particular research question because they lack a comprehensive source search and selection process. Instead, narrative review samples may be compiled using a convenience method (i.e., what is available to the researcher at the time) and may also be compiled based on particular viewpoints.

These factors together create a bias that researchers must acknowledge when writing up a narrative review. Regardless, narrative reviews can make informative and useful contributions to the research literature as they summarize knowledge on a topic, describe the historical progression of knowledge on the topic, present areas of conflict in the literature, and highlight research gaps.

HOW PRECISION HELPS RESEARCHERS WITH SYSTEMATIC LITERATURE REVIEWS

Biomedical researchers frequently seek our assistance with systematic reviews and meta-analyses, often when they wish to synthesize the results of randomized control trials on treatments in their specific fields. However, we also assist researchers–including graduate students–working in many other fields. We work with each scholar to determine the best research question(s) and to tailor the various steps of the systematic review process to help them achieve their expressed goals.

Regardless of the focus or type of review study you are interested in conducting, we progress through a series of steps to generate a viable idea for a study and to bring the study to fruition. Note that although we can absolutely develop the study from the ground up, we are also happy to step in at any stage to assist exactly as you need. For example, if you specifically need assistance with the often highly complex data analysis required for the meta-analysis stage of a systematic review, we are happy to step in at this tough stage to help you out.

As a firm, we do a large amount of literature-based research for our academic consulting clients, and following an organized, step-by-step process ensures a thorough final product. For systematic reviews, we generally follow an eight-step approach:

Phase 1: Develop Review Question(s)

In this first step, we work with you to specify the area of research you would like to address in your review study. Review studies are commonly guided by one or more focused research questions, which should be clear, unambiguous, and follow a MECE structure (i.e., mutually exclusive and collectively exhaustive). To determine the state of knowledge on your focal topic, it can also be helpful at this stage to search for other review studies on the same research topic or questions.

Systematic reviews and meta-analyses in health treatment fields often address research questions that target a specific population, intervention, comparison, outcomes, and study design (PICOS). Specifying the PICOS dimensions as you develop research questions ensures that your review study remains focused, and that you can make valid comparisons between the outcomes of the studies you ultimately include in your sample.

Phase 2: Create Inclusion and Exclusion Criteria

The next step is to develop clear and specific criteria for inclusion and exclusion of studies in your review. As the studies you include in your sample must be useful in answering your research question(s), it is essential to use your PICOS elements to define inclusion and exclusion criteria. This means that we will develop your inclusion and exclusion criteria around the specific population, intervention, comparison, outcomes, and study design particulars that are specified in your research question(s).

Phase 3: Create Search Strategy and Locate Studies

In this step, we create a search strategy and then delve into the literature in search of relevant studies for your review. As with all other steps in developing a systematic review, we use the PRISMA method to complete this stage of the study. A thorough search strategy is essential to the quality of a systematic review, as it ensures that the literature search retrieves all studies that are pertinent to your research questions.

Therefore, the search strategy must include an exhaustive list of keywords and index terms to ensure that no relevant studies are missed. When compiling your list of search terms, we ensure that all possible terms that may relate to your PICOS dimensions are included. Search terms may be drawn from known studies on your topic, and informationists may also be helpful partners in compiling a list of search terms.

The search strategy may also specify Boolean operators to broaden or narrow your search as needed to collect the necessary data. For electronic searches, we will specify the research databases to be searched. Additionally, the search strategy may include a hand-search of particularly relevant journals as well as review of reference lists of studies that are especially relevant to your review. Following the search strategy to the letter, we then complete the search, documenting in exact detail the outcomes (i.e., number of sources retrieved) at each step of the search. The search strategy must be specific enough that our results can be reproduced by anyone else who follows the strategy.

Phase 4: Select Studies and Extract Data

In this phase, we apply the previously developed inclusion and exclusion criteria to determine which studies will be included in your review and which will be excluded. As with the search process, the outcomes of the selection process are documented at each step, with clear reasons given for exclusion of studies. As recommended per PRISMA, this step-by-step process is documented in a flow diagram, which starts with the total number of studies retrieved and concludes with the number of studies that will ultimately form the sample of your review.

Once we have the final set of studies to be included in your review, we can move forward with data extraction. This involves development of a table for tracking key information from each study, including authors, publication year, details regarding participants, the study’s design, and findings or outcomes. Data from the sampled studies may also be coded (to prepare for statistical analysis) on the data extraction form, which requires that a coding schema be developed and pilot tested before the full data extraction process is completed. For exceptionally large datasets–such as often encountered for dissertation projects–we may also use electronic tools to extract data for the review.

To ensure reliable data extraction, it is recommended that two individuals complete the process and that a third individual is consulted to resolve any extraction or coding discrepancies. Through the use of tools such as Cohen’s Kappa or an intraclass correlation coefficient (ICC), we are able to confirm the interrater reliability and scientifically prove that data extraction has been successful. However, this process should not be confused with the actual meta-analysis that is used to test hypotheses for each of your research questions.
Phase 5: Critical Appraisal of Study Quality

In addition to assessing the studies in relation to the inclusion and exclusion criteria for your review, we must also evaluate the included studies in terms of their overall quality. Following PRISMA guidelines, we assess risk of bias within individual studies, tracking this assessment on the table referenced above in the data extraction section. We also assess any potential bias that may affect the overall quality of the evidence, such as publication bias or selective reporting.

To complete this quality assessment, we follow established protocols or checklists to evaluate each study on various aspects of their study design, methods, and procedures. For example, the Cochrane risk of bias tool is well regarded by researchers and is often used to guide our quality evaluations of sample studies for systematic reviews. By answering a series of questions about each study, we can arrive at a risk classification for each.

Use of such detailed quality assessments allows us to reasonably present the strength of the evidence and to make recommendations for future research. It also supports examination of heterogeneity across studies and informs decisions regarding suitability of studies for meta-analysis. Note that any exclusions of studies on the basis of bias risk must be done in accordance with a priori inclusion/exclusion criteria. For this reason, aspects of study quality (e.g., randomization to conditions) are always considered when we draw up criteria for inclusion or exclusion.
Phase 6: Data Synthesis

Using the data extracted previously, we now move forward to synthesis of the data. If two or more studies retrieved for the sample are sufficiently similar, this will include a quantitative synthesis (meta-analysis), which involves statistical analysis of pooled quantitative results from multiple studies. If an overall meta-analysis cannot be done, subgroup analysis may be feasible.

The qualitative synthesis portion of this step involves tabulation of study characteristics and outcomes for all included studies, which is then discussed in a narrative manner to compare and contrast study characteristics and outcomes, to summarize the studies’ cumulative findings with regard to the research question(s), to recognize any risks of bias and how these affect interpretation, etc. If meta-analysis was possible, then the qualitative synthesis also contains a narrative description of those results.

When the scope of the systematic review includes studies with qualitative findings, we will also conduct qualitative analysis of these studies’ pooled findings. This is where our expertise with qualitative research and analysis comes into play, as we utilize coding software (e.g., NVivo) to complete a thematic or content analysis of the pooled findings. Although we are primarily concerned with systematic literature reviews here, you can find a general review of our strengths with qualitative analysis and coding software here.
Phase 7: Presenting Results

In this phase, we write a full report that documents the above results within the context of the full study, following the PRISMA guidelines. An introduction will provide the rationale and objectives of the study, which is followed by a highly detailed description of the methods and procedures as executed (e.g., search criteria, search results, data extraction). Results of the data synthesis described above are reported in narrative form, in tables, and via graphs and diagrams.

The research report also includes a description of the quality appraisal process and outcomes, with discussion of how any risk of bias identified might create limitations on inferences that can be made based on the results. We also provide a discussion of the study’s findings relative to the context of current research on the topic, which includes implications for practice and recommendations for future research to strengthen the related evidence.
Phase 8: Continued Updates

Once a systematic literature review is completed, it may be used as a resource in future studies. The author may work together with other scholars to build on previous work. Whether and when to update a review is determined by agencies responsible for systematic review portfolios, qualified journal editors, or teams of authors seeking further study. Decision makers determine the validity of the review, and determine whether continued research will bring value. We assist clients that have completed a systematic review to continue their research by connecting them with relevant decision makers.

Once the decision is made to update, search strategies are refined and may include new questions or data search parameters. Once you have completed your systematic literature review, we will ensure that the research continues through more in-depth iterations based on your original research question and data parameters.

More about the PRISMA method

Conducting a systematic review can be a very complex and tedious process. We use the PRISMA method in a variety of ways including the following:

• PRISMA for Abstracts
• PRISMA Equity
• PRISMA Harms
• PRISMA Individual Patient Data
• PRISMA for Network Meta-Analyses (PRISMA-NMA)
• PRISMA for Protocols
• PRISMA for Diagnostic Test Accuracy
• PRISMA for Scoping Reviews
• PRISMA for Acupuncture
• PRISMA Extensions in Development