Skip to Main Content

Using Grey Literature in your Research: Assessing Quality

A guide to searching and using grey literature

Introduction

Once you have located relevant grey literature for your review, it is important to assess its quality, just as you would for traditional literature. In a systematic review, quality assessment of each included study (grey literature or not) is a required step [1].

If you are combining evidence from grey and traditional literature in a review or meta-analysis, it can help increase the credibility of your review to explicitly indicate how grey literature contributes to your results [1] and/or to include an index of study quality in your analysis [2].

General factors to consider

There are several tools and checklists to help you evaluate study quality (see box on Tools), which focus on different types of literature. Broadly speaking, here are some important things to consider [3]:

  • Fit: Are the study methods appropriate for the purpose of the study?
  • Methodological Quality: Is the methodology technically sound?
  • Reporting Quality: Is the research shared in with enough detail and clarity to be critically appraised and used by others?
  • Relevance to policy and practice. Does the study make a significant impact on or addition to your review?

Tools

The AACODS Checklist was designed to help with appraisal of a broad range of grey literature sources [4]. This tool is particularly useful if your grey literature is heterogeneous and/or can’t be asessed using the same standards as a traditional study.

If your grey literature consists of unpublished randomized controlled trials (RCTs) or other studies, you can use the same quality assessment procedure as you would for a similar study from the traditional literature.

The Systematic Review Toolbox contains links to numerous checklists and tools designed for appraisal of various types of studies (e.g., RCT/experimental, observational, qualitative, mixed-method). Note: toolbox is temporarily unavailable.

The Critical Appraisal Skills Programme (CASP) and Joanna Briggs Institute (JBI) have each created separate checklists to help with appraisal of different study types.

1. Adams RJ, Smart P, Huff AS. Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies. Int J Manag Rev. 2016;19(4):432-54. [link] 

2. Johnson BT, Low RE, MacDonald HV. Panning for the gold in health research: incorporating studies' methodological quality in meta-analysis. Psychol Health. 2015;30(1):135-52. [link]*

3. Boaz A, Ashby D. Fit for purpose? Assessing research quality for evidence based policy and practice. ESRC UK Centre for Evidence Based Policy and Practice, Working Paper 11. [link]*

4. Tyndall J. How low can you go? Toward a hierarchy of grey literature. Australian Library and Information Association Biennial Conference; 2008; Alice Springs, NT, Australia. [link]

* In order to access starred articles, you must be logged in to your Radboud University account via EduVPN