Canadian Health Services Research Foundation

Searching for Evidence

[N]OT ALL RESEARCH IS CREATED EQUAL, AND IT REQUIRES SOME SKILL TO SORT THE WHEAT FROM THE CHAFF BEFORE MAKING THE "SUMMARY LOAF" FROM THE BEST INGREDIENTS.¹²

Searching for evidence involves many challenges. First, there is the challenge of determining appropriate search terms and research databases to find relevant references. Second — and even more challenging — is the task of evaluating the body of evidence to assess which pieces are the most appropriate to include in the final summary. Since a summary can only ever reference the best available evidence, these tasks are of utmost importance.

This section provides an overview of the "what is evidence?" debate and proposes that, where possible, summaries should rely on conclusions drawn from systematic reviews or syntheses. At the same time, all kinds of evidence have a role to play in informing policy- and decision-making, and the focus of the summary will often dictate what types of evidence are best to include.

Proposed objective

Students should demonstrate their understanding of searching for and appraising different forms of research evidence as well as provide insights into the robustness of the research they have selected to dispel a widely held health services myth.

Understanding hierarchies of research evidence

Not all research evidence is created equal. For example, CHSRF has proposed that medical effectiveness research is "context-free scientific evidence," while social science-oriented research is "context-sensitive." These views of evidence are not incompatible. In fact, along with the expertise, views and realities of stakeholders (sometimes called "colloquial evidence"), these forms of evidence have an important role to play in producing evidence-informed guidance for the health system.¹¹

As a follow-up to the report Conceptualizing and Combining Evidence for Health System Guidance, CHSRF adopted the following definition of evidence:

Evidence is information that comes closest to the facts of a matter. The form it takes depends on context. The findings of high-quality, methodologically appropriate research are the most accurate evidence. Because research is often incomplete and sometimes contradictory or unavailable, other kinds of information are necessary supplements to or stand-ins for research. The evidence base for a decision is the multiple forms of evidence combined to balance rigour with expedience — while privileging the former over the latter.¹¹

When it comes to evidence, it is generally agreed that a well-designed systematic review, such as those undertaken by the Cochrane Collaboration,¹⁷ the Campbell Collaboration,¹⁸ the Centre for Reviews and Dissemination¹⁹ or the Joanna Briggs Institute,²⁰ is the most authoritative source of evidence because the researchers have compiled an overview of primary studies that contains an explicit statement of objectives, materials and methods and has been conducted according to explicit and reproducible methodology.³⁵ Many systematic reviews are also regularly updated to include new evidence.³⁶ These updates are necessary given that systematic reviews and meta-analyses are less powerful tools for informing clinical decisions once they have begun to "show their age."³⁷

Where possible, summaries should rely on conclusions drawn from systematic reviews or syntheses. While single study findings are generally insufficient for informing policy- and decision-making, systematic reviews summarize large amounts of information, identifying gaps in medical research as well as beneficial or harmful interventions. Systematic reviews (including meta-analyses, which are mathematical syntheses of the results of two or more primary studies that addressed the same hypothesis in the same way)³⁵ ultimately reduce large quantities of information into palatable pieces for digestion.

In the absence of a systematic review, there are other levels of evidence to consider. And there are systems and strategies for evaluating which of these forms of evidence is of the highest quality. These are two of the systems in place to assess quality of the available evidence:

• The U.S. Preventive Services Task Force³⁸ has developed a system for ranking the quality of evidence about the effectiveness of treatments or screening.
• The Centre for Evidence-Based Medicine³⁹ at Oxford suggests levels of evidence according to the study designs and offers critical appraisal of prevention, diagnosis, prognosis, therapy and harm studies.

Even with these systems, it is important to note that the "best" evidence depends on what myth one hopes to debunk. Randomized controlled trials, for example, may not provide the best evidence to debunk certain myths. Even when they do, randomized controlled trials will not always be available to counter a widely held myth. In any case, it will be important to critically appraise each reference in a research summary to ensure the evidence is robust enough.

In Is your evidence robust enough? Questions for policy makers and practitioners, Shaxson⁴⁰ identifies five components of robustness and proposes a series of questions to address them. These components are:

• credibility — credible evidence has a clear line of argument and is the result of tested analytical methods in both its collection and analysis;
• generalizability — also called transferability, this refers to the extent to which evidence collected for a specific purpose can be used in a different context or to answer a different question;
• reliability — in evidence-based policy-making, reliability is the ability to use evidence for monitoring, evaluation or impact assessment;
• objectivity — objective evidence is not influenced or restricted by assumptions or values; and
• rootedness — also defined as authenticity, rootedness is about understanding nuance, exploring assumptions, encouraging others to question the status quo, and thinking about who uses what evidence for what purpose.

Conducting a literature search

There are a number of databases available to students who wish to conduct a literature search. To identify the databases best-suited to a given query, it is wise to book an appointment with a subject or reference librarian to narrow down the most appropriate range of resources. For example, a pharmacological query might be best suited to the database EMBASE,⁴¹ while a nursing query would bring back more relevant results in Cinahl.⁴² Different databases are designed to target specific audiences with specific information. Therefore, ensuring the appropriate database selections goes a long way to promoting a higher relevancy rate in one's search results. With this in mind, it's easy to see why the tendency to default to a broader PubMed⁴³ search can be limiting.

Nonetheless, PubMed is a particularly popular database that includes more than 17 million citations from MEDLINE and other life science journals for biomedical articles. One of the special search tools within PubMed is preset with an interface and search algorithm that takes a specifically health services research perspective.⁴⁴ When a literature search is carried out with a targeted "hedge" like this, the resulting scholarship will tend to inherently include research that is pertinent to the audience. In this case, searches using this tool are expected to yield healthcare quality and costs literature, which is relevant to an audience of health system decision makers.

Whichever database one decides to use, it is worthwhile to take the time to figure out how it works. Not all databases operate in the same way. One should particularly never make the mistake of thinking that all search engines and databases operate like Google, a popular, publicly available search interface. Just knowing a database's basic functionality can lead to more relevant and accurate hits. Fortunately, students have a number of online tutorials available to them to help reduce the guesswork of conducting literature searches. A couple of examples are provided. However, similar tutorials are likely available throughout Canadian universities.

• The University of Waterloo⁴⁵ offers an introduction and three learning modules through its educational web site, TILT@UW (Texas Information Literacy Tutorial). This tutorial highlights fundamental concepts of information literacy that are useful for any type of literature search.
• The University of Saskatchewan offers helpful library guides to introduce and navigate a number of databases, such as Cinahl,⁴⁶ Cochrane⁴⁷ and PubMed.⁴⁸ Similarly, it offers library guides for evaluating information.^49,50

Using "snowball" methods

Finding the kind of evidence for establishing the existence of and rebuttal to a widely held myth can sometimes require pursuing what has been called "snowball" methods. This technique involves "pursuing references of references."⁵¹

Of course, to ensure your search does not unfairly bias the final summary, formal search techniques are also necessary. It may also be wise to talk to researchers who are already doing work in the given area. These experts can guide you to (or confirm that you have found) the best available research evidence. Unlike a synthesis, a summary does not allow the luxury of showcasing all of the available evidence. Instead, summaries must only include the cream of the research crop.

Fine-tuning the search

Literature searches often yield broad findings at first. Finding references that get to the point will entail adapting and fine-tuning the literature search. In journalism, all stories are said to be about something and have a point.⁵² But about and point are not the same thing. What the story is about involves the context (the background, facts and people involved); the point of the story is the main theme, the thread that connects each part of the story, or the "so what" factor.⁵² Initial literature searches will inevitably lead you to find references that are about your topic. Finding references that get straight to the point will involve more groundwork.

It will also entail work to present the range of information policy makers and managers want about a given topic. Often times this range will be unavailable. For example, "while there may be extensive research on the effectiveness of health-care interventions, there is often less evidence on their cost-effectiveness, implementation, cultural appropriateness and effects on health inequalities, all of which are important considerations for policy making."²⁵ When fine-tuning the search, it is important to try to keep the wants and needs of policy makers and managers in mind. Doing so will help guide the literature search and avoid adding pieces of information to the summary as an afterthought.

Remembering to avoid interjecting personal bias

Section five of this resource is dedicated to expert review as a final, fact-checking process to ensure one's Mythbuster is firmly planted in the best available research evidence. However, it is advisable to reflect on one's personal biases throughout the Mythbusters exercise. This includes monitoring one's personal biases during the research and writing stages. At the research stage, it is important to ask: "Have I considered all of the different perspectives or views that exist about the myth I have selected?" In the writing stage, it is important to continue this personal reflection, also asking: "Am I missing or misrepresenting any of the research evidence in an attempt to make it more accessible?"