BOSS WP 2.7 Meeting Sheffield Hallam University 17 – 18th September 2018
Andreas Thomann (Technical University Munich) Barbara Eigenschenk (Technical University Munich)
Larissa Davies (Sports Industry Research Centre, Sheffield Hallam University)
Maxine Gregory (Sports Industry Research Centre, Sheffield Hallam University)
Professor Peter Taylor (Sports Industry Research Centre, Sheffield Hallam University)
Dr Girish Ramchandani (Sports Industry Research Centre, Sheffield Hallam University) Tim Goodspeed (SROI Network and external expert)
Antoine le Bellec (CREPS Rhone Alpes)
Mike McClure (Sport Northern Ireland) – 17th September only
Sheffield Hallam University hosted a working group meeting of key members of the partnership including the specialist researchers, academics, practitioners and policy makers for 2 days on 17-18th September 2018.
SHU team highlighted that the key outcome areas which they felt could be measured using primary data include:
- Perceived / self-reported physical health state.
- Mental health via a range of questions from a question bank. The potential to invite experts to input on potential questions was raised and will be followed up by SHU.
- Active citizenship.
- Crime reduction and anti-social behaviour.
- The observation was made that outdoor sport is often used as a tool to reduce crime and anti- social behaviour through engaging with young people at risk and the group were surprised that WP1 had not identified more evidence in this are The potential to include a case study in this area to improve the evidence base was suggested.
- Negative impacts also require consideration as part of the model, however WP1 did not identify any quantified studies in this ar
- Logic models
The production of a series of logic models as per A2.4 was discussed. This included consideration of defining the terms ‘outputs, outcomes, impacts’ and working towards establishing a common language for the project. It was agreed that a ‘glossary’ will be produced to provide explanations for a number of terms which are used within the BOSS project. Use of the term ‘logic models’ was also
discussed and it was agreed to use the term ‘Theory of Change’ for the quantified models for the commission (to comply with the application) and the term ‘Outcomes Model’ for the wider external audience, as the meaning of this may be clearer.
SHU presented the initial logic models that had been created using quantified evidence only. Whilst this was necessary for the development of the model, and illustrates how many studies focused on evidencing intermediate outcomes only, it was agreed that a broader set of models incorporating all identified outcomes was needed as these would be a useful output for the sector. The original models will be retained as the Commission may be interested in seeing this part of the process, however new models will be developed for an external audience.
SHU will consider whether an overall/overarching model can be created to summarise the impact of outdoor sport.
- Total value / added value
A basic principle of projects should be the ‘desire’ to capture any “unintended consequences”. It was noted that the inclusion of a ‘decision tree’ to show the pros and cons of the different pathways should be developed. It was also discussed that an International Social Value Certification process is available and further information will be made available on this.
It was agreed that to keep the process manageable for the case study implementation, limited options would be on offer in terms of outcomes and methods of assessing their scale and value, but that more options might be available to projects in the future (post-BOSS). It was noted that transparency in how things are done and values allocated is essential “transparency = credibility”
There was discussion on whether it should be a pre-requisite for participants to be not active at the time of starting the project was discussed for the physical health outcome and the valuation process. However, contrary to this, the deadweight adjustment in SROI does not take into account ‘preferences’. Whilst an individual could choose to participate in an activity elsewhere, they haven’t done so, they have chosen to take part in the project in question. There should be a recognition of economic preferences.
It was agreed that BOSS should present two values – one for the ‘total value’ (gross value of outdoor sport at case study level) and the other for the ‘added value’ (created by new participants involved in the project / case study and accounting for deadweight, and displacement). There needs to be a strong justification why this has been done and to reduce the risk of over-claiming. A set of questions to help identify causality were agreed:
1) What would you have done if you weren’t doing this?
2) Would you have done this anyway without this project?
It was noted that attribution should be referred to as ‘causality’ (deadweight/attribution/displacement).
- Framework v’s toolkit
There was discussion on the extent to which to provide flexibility and options, or a fixed and comparable tool?
Is the aim to produce and help people to use a prescribed toolkit (e.g. locked spreadsheet) or a framework (set of principles and options) which people takeaway and follow as guidance? To what extent is comparability needed across the case studies?
- Case studies
What will be the benefit / rationale / motivation for the case studies? What will they need to show? How will they want to demonstrate value? These are all fundamental questions that will help to shape the specification, recruitment and selection process.
The case studies need to incorporate small and large scale projects. Large scale national projects will have more challenging management arrangements but there is no reason to not include them if they can commit to the data collection requirements.
It was discussed whether there should be an open call for applicants. Information will be given including a list of data requirements, a project pro-forma and further information for applicants.
- Question bank
For mental health / wellbeing it was suggested to use the UK Office of National Statistics wellbeing questions with tested thresholds. It was agreed to get some further advice on this from partners involved in Sports psychology such as the EU GoGreenEX team that ENOS has been working with.
It was agreed that clear indicators to ensure clarity and transparency are needed.
It was noted that the history of the participant and the frequency of their participation will need to be considered.
Cultural differences between European countries were discussed. In particular the optimal length / timescale for questionnaire completion. There were clear difference in this and that there is dilemma between the quality of data received from each survey visa v numbers of surveys returned. Optimal length was anywhere between 5 – 20 minutes duration to complete from different perspectives.
The group discussed how surveys are usually validated to assess completion quality. SHU highlighted previous experience of what stage the data collected will be compromised in terms of quality and / or quantity as a result of ‘survey fatigue’ (respondents losing attention / interest).
It was noted that there would be benefits to using the same questionnaire across projects to enhance comparability, however each survey would include both bespoke and standardised elements. An initial short survey to identify outcomes which could then be drilled into further was suggested. This approach would incorporate a question bank enabling flexibility to take into account different focuses/outcomes.
The data collection would be a three layer process; initial stakeholder consultation, short survey (standardised elements), bespoke questions to ensure that relevant outcomes are measured.
Value system – it was suggested that personal values would be measured in a package/bundle/composite measure (rather than each one individually). It was highlighted that valuing different outcomes as a package would not satisfy the requirements of an SROI accreditation. A composite measure won’t enable the valuation of different outcomes so it wouldn’t be possible to tell which is the most important. We need to be able to value the most important outcomes. One way would be to identify the relative importance of different outcomes to individuals, by which means the composite value could be disaggregated.
The question was raised as to whether ‘willingness to pay’ is an appropriate concept for free activities but it was confirmed that it is often used in this context. The alternative to willingness to pay focuses on ‘compensation’ – if an activity benefit was withdrawn what would you want paid to compensate for this (e.g. how much in damages?) Contingent valuation needs to include the context so the valuation is ‘contingent’ on something.
It was noted that there are valuation validation questions but this would add even more time to the survey completion and may jeopardise the balance between achieving high quality data but keeping any surveys short and manageable.
- Identifying health costs
Health outcome costs have been collected for the core outcome areas for a range of European countries. This is a complex and time-consuming process because sometimes the number of cases and the cost of treating cases are available for different years. Furthermore, some of the costs contain different elements for the outcomes e.g. primary care, loss in productivity, etc, whereas others do not. Further work will be done on this area.
The group agreed a set of actions:
- A new set of logic models encompassing all outcome areas will be produced. This will be sent to Mike and Arne for design work to be undertaken.
- Work on a ‘decision tree’ with the SROI expert Tim Goodspeed
- Production of a ‘question bank’ for discussion and wider feedback on accessibility for different languages. This will include causality questions.
- Finalisation of the data requirements for case studies. It was noted that SHU don’t want to overstep the mark (go too far into WP3 territory) and will respect that the WP3 leads will want to coordinate things and have their say on this area too.
- Continued work on the spreadsheet process model as a template to guide case studies.