Transparency of evidence: spot check

Transparency of evidence: a spot check of government policy proposals July 2016 to July 2017.
How transparent is government in sharing evidence behind policies?

Published: 31 January, 2018

Download Transparency of evidence: spot check pdf

Can someone outside government tell what the government is proposing to do, and why?

Transparency of evidence: a spot check of government policy proposals July 2016 to July 2017, scores 94 government policies produced by 12 departments, to assess how transparent they were about the evidence behind the policy.

Why spot check evidence transparency? Because without clarity about what the government has looked at, it is very difficult for citizens to understand the motivations for policy, decide whether they agree with it, participate or consider whether it is working.

This follows on from our assessment of good and bad practice across government – the first ever review of whether the UK government is transparent about its use of evidence when developing policies – published in November 2016. That report found that the public and researchers would struggle to follow the government’s reasoning, with standards of transparency varying widely between and within departments.

Policies in this spot check were scored against a framework looking at the Diagnosis, Proposal, Implementation and Testing & Evaluation plans of policy proposals and scoring them from 0 to 3. This report analyses what has changed over the past year, and where challenges remain.

Findings

Transparency is a prerequisite to assessing quality. This review does not assess the quality of the evidence used or the merits of the policy. A well-founded policy and a poorly founded policy may both score well for transparency; a transparent evidence base enables a better conversation to determine which is which.  

This report shows a general improvement since we highlighted good and bad practice in 2016’s report. But there were also large discrepancies and some frustrating cases where an otherwise transparent policy lost points on one section of the framework.

The findings include:

  • The scores reveal considerable variation in practice between departments, but more consistency within departments.
  • There were fewer examples of departments failing to publish evidence. One of the most alarming findings of the 2016 report was the discovery that many departments held valuable evidence but did not publish it. However, there were still some significant omissions, including on areas likely to be of considerable public interest.
  • Policies with Impact Assessments scored more highly, and were rarely awarded 0s.
  • Testing & Evaluation – evidence about whether the policy works, or plans to gather it – showed the most disappointing results and gave little indication of improvement since 2016.
  • The highest scoring policy was Department for Transport’s ‘Cutting down on noise from night flights’, which scored 3 against every section of the framework. It was commended by scorers for a range of innovative citizen-centred ways of presenting the government’s thinking and how the department had arrived at conclusions.
  • The lowest scoring policy was Department for Education’s ‘Modern foreign languages A and AS level content’, to remove the speaking assessments from some examinations. This was given 0 for every section of the framework. Scorers could form no idea of what it was based on.

Transparency score tables first page

Transparency score tables second page

Comments

Tracey Brown, director, Sense about Science and main author of the report: “Without clarity about what the government has looked at, it is very difficult for citizens to understand the motivations for policy, decide whether they agree with it, participate or consider whether it is working. Researchers and specialist contributors can’t see what they could add; and government is less able to build on its own past work, let alone determine whether initiatives such as the What Works centres are improving the evidence base for policy. A transparent chain of reasoning is vital to all of this.”

David Halpern, What Works national advisor: “If governments want to be better at listening and learning, a good place to start is to be open about setting out their thinking and evidence behind their actions and policies. Without this basic form of transparency, it’s hard for others – whether expert or lay – to assess whether the policy or proposal is based on strong foundations, or to add helpful additional material. This report marks where departments are on transparency and shows how they can improve further, not least by learning from each other.”

Catherine Haddon, senior fellow, Institute for Government: “When policy is made well, it means better use of public money and better outcomes for the public. The Institute for Government and Sense about Science developed the evidence transparency framework because clear and easy to follow evidence helps improve the policy process and allows parliament and the public to more effectively hold government to account. This report shows that the framework is working.”

Partners

Transparency of evidence: a spot check of government policy proposals July 2016 to July 2017 is published by Sense about Science, from research conducted in partnership with the Institute for Government and the Alliance for Useful Evidence. The Nuffield Foundation is an endowed charitable trust that aims to improve social well-being in the widest sense. It funds research and innovation in education and social policy and also works to build capacity in education, science and social science research. The Nuffield Foundation has funded this project, but the views expressed are those of the authors and not necessarily those of the Foundation. More information is available at www.nuffieldfoundation.org