Reliable review requires thorough literature search

In order to obtain a correct idea of the benefit of an intervention, such as a treatment method, it is important to first correctly search for all of the research reports that have been published about the method in question. This challenge requires a literature search that is appropriately broad – neither so narrow that important articles are overlooked, nor so broad that they become impossible to sort out. At SBU, this balance is achieved jointly by information specialists, experts in the field and people trained in research methodology.

Within the framework of systematic reviews, SBU conducts literature searches in multiple steps. The point of departure for the search is to formulate structured questions that specify and define the subject. A typical review question includes what intervention should have been tested, in what population, compared with what interventions and how the outcome should have been measured. These elements are summarised by the acronym PICO – population, intervention, comparison (or control) and outcome.

Once the review authors have clarified their intention – and what criteria should be applied regarding study design, age of participants and language and year of publication of the articles – the next step is to select relevant databases and appropriate search terms. Interdisciplinary collaboration across professions and specialties is often required. Experienced researchers and practitioners know what search terms, synonyms and expressions they are likely to encounter in their respective fields, while information specialists are experts in various databases and what search strategy and vocabulary may be appropriate.

‘One good method to begin looking for useful keywords is to start with existing, well-designed systematic reviews, since their search strategy should always be described,’ says Klas Moberg, information specialist at SBU, with ten years of experience at the Karolinska Institutet University Library near Stockholm.

Another trick is to use known articles considered to be particularly relevant in the field as a point of departure.

‘The databases can be searched for information on how the articles were categorised and labelled, in other words indexed, using controlled keywords – index words. You can also find what other words and phrases, free text words, that researchers use in abstracts and titles.

The index words are obtained from the hierarchically arranged and controlled glossary – thesaurus – that is part of every large international database,’ says Klas Moberg. The thesaurus in the Medline medical database (and therefore also in PubMed) is known as Medical Subject Headings (MeSH). The thesaurus in the PsycINFO psychology database is known as the Thesaurus of Psychological Index Terms, while Sociological Abstracts uses the Sociological Thesaurus.

‘One advantage of using index words is that synonyms and conjugations need not be taken into account – a belabouring process which must be considered when using free text words.

However, index words also have their limitations.

‘For instance, entry of new publications into the databases takes time. The very latest articles might not yet have been indexed,’ says Klas Moberg.

Different databases may also use different index words for the same referent, and sometimes it is difficult to find a sufficiently precise word – such as in relation to a highly specific version of an intervention or therapy.

‘That’s why we usually combine index words with free text words using alternative spellings and conjugations. It is often possible to search using only the root of the word, the part of the word to which endings are added, followed by a truncation symbol, usually an asterisk.’

It is a good idea, according to Klas Moberg, to carry out sample test searches using index and free text words to see whether the articles in the hits are relevant to what you want to know.

‘If the hits prove to be irrelevant we adjust the search strategy.’
Professional test searches also provide a rough idea of the scope of the literature in the field and how much time and effort might be needed for sorting and evaluating hits from the main search.

The next step of is to create search blocks, one for each element or concept to be searched. The search block includes the selected index and free text words with possible synonyms and phrases.

‘Each block is first searched separately. Then the block searches are combined. Boolean operators such as OR, AND, and NOT are used to provide the database with specific commands.’

For example, the operator OR is used between synonyms and related terms within the same search block to indicate that it is sufficient for any one of the keywords to appear in a reference to generate a hit. The command OR makes the search broader and results in more hits.

The operator AND is used to combine different search blocks. This last command instructs that at least one word from each block must occur for a reference to be a hit. As a result the search becomes more specific and the hits fewer.

The Boolean operator NOT can also be used to narrow the search.
‘However, the command NOT is often avoided in systematic reviews since it may exclude relevant references,’ says Klas Moberg. ‘In systematic review projects, having a few too many hits is preferable to missing something relevant.’

The commands that can be used may differ from one database to another. In addition to OR, AND and NOT, many international databases also provide the option to use ‘proximity operators’, which control in what order and how close the search terms must be to provide hits.

‘The exception is PubMed,’ he says. ‘It does not accept proximity operators.’
Literature searches pertaining to systematic reviews must be structured to use an exhaustive approach. Consequently, the search should identify as many research results as possible from everything that has been published about the PICO questions relating to the review. The challenge lies in finding as many relevant articles as possible, without obtaining too many irrelevant hits in the process.

‘In the field, we say that a search has high recall when we can demonstrate that it successfully identified a large proportion of all conceivably relevant published articles. When a high proportion of the hits are relevant, the search is said to have high precision.’

An ideal search should have both. But in practice you must often decide which is more important in meeting the purpose, Klas Moberg explains.
‘In systematic reviews, high recall is considered to be especially important. This calls for a more inclusive, broader search instead of a narrower one, which is associated with greater risk of missing something essential. Often, we simply convert “population” and “intervention” in PICO into search blocks, to ensure that the search does not become too narrow.

‘The price of choosing the broad search strategy is that it results in more hits that are irrelevant to the PICO framework. The result is more time spent sorting out irrelevant hits than would have been necessary with a narrower search.’
The job of the SBU information specialist is not just to know what databases are relevant to a particular field, but also how they work.

SBU is rarely satisfied with a single database and generally covers at least three. Since these may require different approaches, the search strategy may need to be adjusted. For healthcare-related questions, Ovid MEDLINE (or PubMed), Embase and Cochrane are often used. For questions related to multidisciplinary issues and social work, PsycINFO, SocINDEX and Sociological Abstracts/Social Services Abstracts, plus Ovid MEDLINE (or PubMed) are used. The citation database Scopus may also be useful for the above searches.

‘Sometimes the databases have default search filters or hedges that may be useful. These search tools can be combined with search blocks to find a certain category of studies, such as a particular study design.

‘However, they don’t always work perfectly, not even those that have been tested scientifically. And they may become obsolete when new index words are introduced.’

Since such searches use an exhaustive approach, it is also important to remember that essential information may have been reported outside of scientific journals – for instance, in academic theses, guideline documents and research reports from authorities, organisations and businesses. Such sources, sometimes referred to as grey literature, may occasionally contain important and reliable information that may be difficult to access.

Transparency is required in systematic reviews, as well as in other scientific contexts. The authors of the overview must describe the details of their approach. The literature search must also be meticulously documented and presented so that it can be repeated with the same results, and to ensure that expert readers can evaluate its quality.

‘Literature searches for systematic reviews require both specialist knowledge and time,’ says Klas Moberg, ‘especially in areas with extensive literature from several research disciplines.’

However, well-designed searches are absolutely crucial in order for the overall results of the reviews of existing studies to be reliable.
‘If the search misses or eliminates essential findings, the composite picture may turn out completely wrong.’ • RL

Further reading

• SBU’s Assessment of methods in health care and social services, see https://www.sbu.se/en/method/
• Cooper C, et al. Defining the process to literature searching in systematic reviews ... BMC Med Res Methodol 2018;8:85, https://doi.org/10.1186/s12874-018-0545-3
• Liberati A, et al. The PRISMA statement for reporting systematic reviews ... PLoS Medicine, 2009. https://doi.org/10.1371/journal.pmed.1000100
• Higgins JPT, et al. Cochrane handbook for systematic reviews of interventions. Version 6.0, 2019. See https://training.cochrane.org/handbook/current
• Karolinska Institutet library. https://kib.ki.se/soka-vardera/systematiska-oversikter