Using librarians to conduct systematic reviews

Librarians can provide valuable expertise in uncovering available information and research on a topic, whether fulfilling the role of a specialist search expert or acting as the project lead. Challenges can include a lack of time or financial support, issues with the method used, as well as poor group dynamics and communication.

Margaret Foster identifies a six stage process to ensure that an effective systematic review is carried out:

1. Plan the review
   (a) Establish the main objective(s) of the review
   (b) Establish eligibility criteria for information to include
   (c) Consider project management issues
   (d) Conduct a scoping exercise to determine novelty of the topic and number of available publications
   (e) Estimate the time required to complete the review and confirm it fits with the projected timelines
2. Conduct the search
   (a) Consider the needs of the search, and how it may be different from other searches
   (b) Adopt a search methodology that matches the desired goals of the review
   (c) Selecting resources (eg databases of journals) to search
   (d) Developing and documenting search criteria for each research
   (e) Evaluating results, preferably with peer review
3. Screen and select results
   (a) Set up your panel of reviewers, with at least two independent screening and selection processes
   (b) Create questions to establish reasons for inclusion or exclusion
   (c) Screen articles using these questions, documenting answers as part of decision justification
4. Assess risk of bias
   (a) Identify flaws in selected studies
   (b) Determine impact of flaws on findings
5. Code final result set
   (a) Select characteristics of importance
   (b) Pilot coding using a subset of search results with at least two people
   (c) Check agreement on coding and refine coding approach if necessary
   (d) Complete coding
6. Write the report
   (a) Ensure that all authors agree to be accountable for the whole of the finished product
   (b) Document methods used in the report
   (c) Describe level of confidence in how complete the search results were
   (d) Publish final report and source data in a useful repository

There are three areas that warrant further discussion.

Firstly, pre-planning project management issues has been demonstrated to be particularly important for achieving a successful outcome. Considerations include the expected role of the librarian, team size and funding available, method for capturing and communicating search results, projected timeline for completion, and technology support tools to be used (such as citation management and collaboration software) can all have a significant impact on outcomes.

Secondly, a number of groups have designed methodologies that can be followed to ensure that high-quality, systematic literature reviews are carried out. Foster identifies the PRISMA statement as a good option. The statement includes a flow diagram and checklist to ensure a fully-documented approach where all collected articles are labeled, a screening for ‘true’ positive matches for items of interest is undertaken, and bias is minimized.

Thirdly, the approach used to minimise bias is of critical importance. When reviewing information provided by a literature review, the number one question that must be answered is “how much should I believe what the article or book is telling me?” There are many tools available to increase objectivity in evaluation of bias, including dozens of scales and checklists. A particularly strong option is the Cochrane handbook approach of domain-based evaluation.

The Cochrane approach minimises the risk of bias by evaluating literature search results against six domains:

• sequence generation
• allocation concealment
• blinding
• incomplete outcome data
• selective outcome reporting
• ‘other issues’

Each domain is evaluated by a specific Yes/No question, with a “No” indicating a high risk of bias for that particular publication.

For people outside of the academic world, this may all seem like too much trouble. But it’s worth considering the alternative. If you were making an important decision worth millions or even hundreds of millions of dollars to your organisation — why should you make a decision based on bogus information?

Wouldn’t it be better to take the time to properly research what others who are trying to solve the same problem are doing, and to get the answer that is most likely to be true regardless of our preconceptions?

Source: Journal of European Association for Health Information and Libraries,