The presentation from the 'Evaluation: Facing the Tricky Questions' session at the 2011 Science Communication Conference

2. Evaluation: Facing the Tricky Questions Mark Dyball, Director, People Science and Policy Ltd Diane Warburton, Evaluation manager, Sciencewise-ERC Laura Grant, Director, Laura Grant Associates Ltd

3. Evaluation: facing the tricky questionsAssessing Impact on Policy Mark Dyball, Director People Science & Policy Ltd

4. Basic Questions Can science communication have an impact on policy-making? What sort of science communications projects might impact on policy-making? Is it feasible to make an assessment of the impact of a science communication project on policy-making? Is it appropriate to devote resources to find out whether there has been an impact on policy?

5. Policy-makers … are not just politicians and civil servants include managers in Universities Charities Businesses are people that have the power to make decisions and set policies within their organisations

6. What influences policy-makers? Evidence Dogma/belief

7. Can science communication have an impact on policy making? Yes, but science communication is a broad term; different projects have different goals; and may influence different policy-makers. Was there an intention to influence policy-makers? If so, which ones?

8. What sort of projects might impact on policy-making? Research-based dialogue Understanding stakeholder attitudes and values Bringing stakeholders and policy-makers together e.g. HSE public dialogue on train protection Often commissioned by the relevant policy-maker

9. What sort of projects might impact on policy-making? Communication of science Human cloning Stem cells

10. What sort of projects might impact on policy-making? Communication and evaluation Fosters understanding of Audiences Activities that “work” Who are the effective communicators Evaluation can be used to influence policies of: Programme managers (funders) Project managers (communicators)

11. Resources? It is worth investing time and money in evaluating impact on policy-makers if It was a key goal of the project e.g. Beacons there is clear evidence that influence on one or more types of policy-maker was an unintended consequence In many cases it will not be worth this investment

12. How do you know if you had an influence? Interview policy-makers Follow document trails Observe relevant events The co-operation of the relevant policy-makers makes life easier for the evaluator e.g. EPSRC nanotechnology in healthcare

13. EPSRCNanotechnology in healthcare Large scale public dialogue (BMRB) Ongoing evaluation The quality of the project Observation Questionnaires interview The impact of the project Documentation Interviews

14. A Final Thought Which policy-makers would I most like to see influenced? Those who have commissioned the project/evaluation If you have paid for intelligence, then use it. This includes you!

15. Evaluation: facing the tricky questions Assessing value for money Diane Warburton, Evaluation Manager Sciencewise-ERC www.sciencewise-erc.org.uk diane@sharedpractice.org.uk

16. Question: When is a public engagement project good value for money?

17. Why we have assessed value Sciencewise has developed ways to measure and demonstrate the value of public dialogue because: • reduced budgets and increased scrutiny • need to maintain support as well as funding • useful to review costs of design decisions This became a priority for Sciencewise project evaluations over the last couple of years

18. Practical and other problems • "Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted" Albert Einstein • A cynic is "A man who knows the price of everything and the value of nothing" Oscar Wilde • Measuring can be expensive in time as well as money • It is very hard to get detailed accurate cost data We decided that measuring value needs to be relevant and proportionate. And it is not always appropriate.

19. Looked at all sorts of models • Classic VFM: economy, efficiency, effectiveness • SROI: Cabinet Office approach to social value • Classic cost benefit analysis Found were not appropriate because, as models: • too complex • too detailed • made too many assumptions • too much focus on ‘monetising value’ Learnt from these but took a different approach

20. Assessing impacts Rather than looking at benefits, we focused our evaluations on assessing four types of impacts: • impacts on policy and policy making • impacts on policy makers • impacts on public participants • impacts on scientists and others involved This approach allowed us to identify short term as well as long term impacts, on people and policy

21. Six questions on costs 1 What was the basic budget? e.g. Nanodialogues £240,000; Drugsfutures £300,000 2 What were public participants' and stakeholders' perceptions of whether it was 'money well spent'? e.g. Drugsfutures public participant: "yes, if our views are listened to" 3 Could costs have been reduced without losing quality? e.g. geoengineering dialogue: public access events did not add enough value to match costs, and very detailed evaluation

22. Six questions on costs, continued Could a small additional investment have achieved significant extra benefits? e.g. Synthetic biology dialogue: commissioning follow on work to maximise impacts of project reports; Big Energy Shift: additional public participants at workshop with policy makers What costs could be saved later by having had good public engagement? no examples yet

23. Six questions on costs, continued 6 What are the costs of engagement compared to overall programme budgets? e.g. • Geoengineering: cost £155,000; fed into EPSRC / NERC sandpit which alone allocated £2.5 million • Synthetic biology: cost £360,000; budget for synthetic biology research in UK 2005-10 £18 - £33 million • Nanodialogues: cost £240,000; value of nano research in 2007 estimated at $12 billion • Stem cell dialogue: cost £300,000; industry valued at more than £500 million per year "If you think dialogue is expensive, try conflict" Andrew Acland

24. Assessing value of engagement • Costs are only part of the story, but an important part and usually missed out • Overall balance of costs and benefits almost always depends on understanding longer term impacts, especially on policy • Different audiences perceive value differently. We have found: • policy makers value robust evidence and advice from public, raising public awareness, or testing ideas to manage risk • public participants value being listened to and having influence, or just the fun of it • experts and scientists value taking their work to new audiences, and learning new communications skills

25. Final thoughts We have found: • Evidence of value is vital. Numbers are always powerful, but hard to pin down • Different audiences want different evidence and so need different messages • Collaboration and sharing experience helps development, but … • Can it continue in the current climate?

26. Many thanks Diane Warburton, Evaluation Manager Sciencewise-ERC www.sciencewise-erc.org.uk diane@sharedpractice.org.uk

27. Facing the tricky questions:EVALUATING LONG TERM IMPACTS Dr Laura Grant laura@lauragrantassociates.co.uk

28. The problem(s) It’s hard to track people that have been involved in activities Impacts might diminish over time, making them harder to measure Other factors have more time to intervene, making attribution (more) difficult Who will do/fund this work?

29. Case study: Ingenious long term tracking study Ingenious is the Royal Academy of Engineering public engagement grant scheme, now in its 5th round Evaluation approach: Used Theory of Change to think about what success looked like (including long-term) Support grant holders in self-evaluation of short-term outcomes Long term tracking study to explore impacts after two years

30. Desired impacts (long-term) Changing practice – engineering PE is embedded in engineering practice and policy making To create a community of engineers that are able to communicate their work to many groups and understand how their work impacts on society For it to become standard practice to engage the public at an early enough stage to influence the development of technology For more engineers to be willing and able to work with the media Changing practice – public engagement A proportionate amount of engineering in the PE landscape (and for it to be labelled as such) For the ‘development’ funding model to become established Promote meaningful longer-term evaluation of programmes, such as the tracking study

31. The study Initial e-survey Round 1 n=21, 22% response rate Round 2 n=40, 63% response rate Potential to follow up with interviews May combine data from Round 1&2 Need to take into consideration the way the grant scheme has developed

32. What worked well Thinking through ‘what success looked like’ for the programme alongside the evaluation approach from the start The overall approach meaning the evaluator and grant holders meet at workshops, where the tracking study can be discussed People were interested enough to respond!

33. Challenges Collecting contact details Difficulty in benchmarking long-term impacts Very difficult to know if respondents could have achieved the impacts another way Attribution or contribution?

34. Final thoughts Lay the foundations for long-term work by defining success and evaluating short-term outcomes. Long-term evaluation can make a difference to programmes. Obviously this will not happen overnight. Think about who the impacts will be on and whether it’s possible for your evaluation to measure them. As a community, we have research questions around long-term impact that evaluations of individual projects/programmes are not best placed to answer. Who faces these questions?

36. Timekeeper

37. Reporter

38. Scribe

40. How did you deal with them? What did you learn?

41. What refined or new tricky question would you like to see future evaluations address?Note your comments on the sheets provided. Please be clear as we will be writing these up after the session. The reporter from each group will be asked to very briefly feed back one ‘tricky question’ at the end of the session.

42. Next steps Please hand the notes from your table in at the front of the room. We will write up the notes and questions and share the on the British Science Association website.