BOSS WP 2.1 Meeting Sheffield Hallam University 18 – 19th April 2018
Andreas Thomann (Technical University Munich)
Barbara Eigenschenk (Technical University Munich)
Larissa Davies (Sports Industry Research Centre, Sheffield Hallam University)
Maxine Gregory (Sports Industry Research Centre, Sheffield Hallam University)
Professor Peter Taylor (Sports Industry Research Centre, Sheffield Hallam University)
Dr Girish Ramchandani (Sports Industry Research Centre, Sheffield Hallam University)
Tim Goodspeed (SROI Network and external expert)
Mike McClure (Sport Northern Ireland) – 19th April only
Sheffield Hallam University hosted a working group meeting of key members of the partnership including the specialist researchers, academics, practitioners and policy makers for 2 days on 18-19th April 2018.
Re-classification of data
The SHU team highlighted how they had re-classified the data into core outcome areas as required for the process of developing a social value framework / model. The studies were ‘ranked’ as high / medium or low in terms of their relevance and evidence linking outdoor sport with social outcomes. It should be noted that the ranking of the studies was purely based on how useful they are deemed to be for WP2 processes, there is no judgement on the overall quality of each study from any other perspective.
This was fairly straightforward to do for health benefits but became more difficult when looking at other benefits such as education and life-long learning as it is difficult to identify the final outcomes other than simply better levels of academic attainment.
Examples of what are final and intermediate outcomes are as follows:
Final outcomes and intermediate outcomes
Using evidence in the social value process
There was discussion on how the evidence from WP1 will inform WP2. The quantifiable physical health papers focused both on specific diseases / conditions and on more holistic health benefits (e.g. all-cause mortality and life expectancy). Any social value process considering holistic measures will need to ensure that outcomes are not double counted. While the data that we have from the literature review does not allow us to directly value any outcomes (quantification is given in some cases but not the actual value of this), the data can form part of a model which other data will help to populate. How the data can be used will be context specific and will depend on the scale and scope of the case studies. It may be possible to ‘group’ outcomes and value them as a whole rather than attempting to value all identified outcomes individually.
The challenge of how can outcomes such as ‘life-long learning’ be valued when there is no final outcome was noted. There is a need for pragmatism and to establish a timeframe for the value. Standard SROI methodology would value the benefit over the lifetime. A mix of case studies would give a feel for a range of values and collection of primary data will be extremely valuable and should be central to the valuation process.
Discussion around the process of identifying change took place. It was noted that ‘looking at data and informing participants what changes for them’ is not the right ethical approach. The extent to which we can make assumptions about changes and the level of steer that participants need was also discussed e.g. participants can’t report something that they are unaware of (such as lower risk of health conditions). SHU explained that their standard practice is often to ask open questions first then to give some prompts (using the information from data). However, it was noted that different outcomes may require different approaches.
There was a discussion about how to deal with the gaps in evidence?
Often these created intermediate outcomes and were intervention specific and the final outcomes are difficult to quantify. It is necessary to avoid duplication the link between intermediate and final outcomes need to be carefully considered. It was also noted that there is often subjectivity between intermediate and final outcome classification – e.g. to some people an intermediate outcome might be viewed as a final outcome. It was noted that the ‘self’ outcomes (self-efficacy, self-esteem, self-confidence etc.) could all contribute to final outcomes that may be classified as physical health, mental health, mental wellbeing, education or active citizenship.
The group discussed ‘How to use the ‘underpinning story’?’ – this refers to the wide range of benefits that studies have described but that have not been quantified. It was agreed that these studies have significant value in shaping the primary data collection / case studies.
Ways to produce searchable outputs were also discussed, particularly with regards to WP1. It had been agreed to investigate options for developing an online searchable tool on the website that is an accessible resource for the sector. There could be a keyword search for benefits or within abstracts – or it could be something as simple as an excel spreadsheet with filters by main outcome areas or by sport.
Measuring the value
It was noted that health data can be incorporated into a social value model using CHD as the health variable that had been quantified. Discussion moved on to ‘how to value other benefits and what data should be excluded? It was noted that there is the option to ‘borrow’ other health variables from the wider sport model?
It was noted that there are different aspects for valuing data so for example they could be split into health and other (esteem) benefits and categorised by fiscal values and the value to participants / stakeholders. There are many layers to measurements and so an SROI approach must be transparent and conservative estimates are required. The group discussed what to do with the empirical evidence generated by case studies and how the data can be used most effectively? It was noted that the case studies will need expert support and it was suggested that saved travel costs from elsewhere in the project may be able to be used to enable face to face visits to support the case studies.
It was noted that there is no ‘objective’ way to value personal benefits (such as self-esteem) in the same way as wider fiscal benefits. It would be possible to present them separately e.g. ‘this is how government values a person being healthy and this is how an individual values it for themselves’.
It was noted that if only fiscal values get included to create a model it would be more scientifically robust but with the available evidence from WP1 this could be very limiting. Further, this would go against the spirit of SROI which is to value benefits to individuals and stakeholders who are affected, not just government cost savings. Whatever benefit values are generated can be set against the costs to provide an accounting overview.
- The two possibilities for a SROI assessment:
- Population model
- Intervention model
Population level model
To undertake an assessment for any country at a population level there needs to be quantifiable data identified through the literature research and correlating and associated costs available at a national level. Furthermore there is a need to have data on participation levels that achieve appropriate levels to be certain to gain the benefit and also the numbers suffering from the relevant condition. So for example:
It is known from “Blond, Rasmussen, Østergaard, and Grøntved (2016) that Danish cyclists had 11 to 18 percent fewer heart attacks” So in order to calculate the Social Return On Investment we need to know the number of people undertaking the activity (cycling), the number of people having heart attacks and the cost of dealing with a heart attack. The relevant value of the activity can be calculated taking into account the relevant percentages and costs.
Where there is no quantifiable data on the benefit but there is clear evidence of a benefit having been accrued, the team at SHU in the past have allocated a figure of 1% to that.
This overall process is complicated and detailed to do and requires a very significant level of data to be provided at a national or regional level. However, it is envisaged that such data will exist for many EU member states. Therefore it was agreed that there would be value in putting together a “how to do this” toolkit with explanations of what data the BOSS project has collated and what any organisation would need to do and what data would be required to do this in any of the partner countries (or even other member states).
Intervention level model
At an intervention level the data required is much less significant and the focus is on the participants.
The process is to undertake the activity and then pull together the participants in a focus group and ask initially open questions and if required some prompted questions in order to identify the significant factors that have changed for the participants.
A follow up questionnaire is done with the participants to identify at an individual level what has been the significant change that they can clearly attribute to the intervention. The participants are then asked what they would have been willing to pay to have gained that benefit. This can be done quite creatively by asking was the intervention worth more or less than other material things or experiences that must be purchased. The willingness to pay approach for the participants is a fairly straightforward model to be adopted but the key thing is to spend time ascertaining precisely what has changed and wording the surveys appropriately.
There was considerable discussion about taking into account other key stakeholders and their willingness to pay to have the relevant participants’ happier/healthier/better members of society or better levels of educational attainment etc. However, it is understood that the SROI model in the UK does not do this. There was a strong feeling from some team members that this should be a key element and in fact SHU had written this into the original application to identify the key stakeholders and members assumed that this is so that these stakeholders can also be interviewed to ascertain the value that they have placed on the activity.
Gaining information from relevant key statutory or otherwise stakeholders would be complicated and time consuming. It was noted that getting unequivocal information from stakeholders would add value but it was indicated that this is not done for SROI.
Further clarity is needed on what model we want to have as a partnership.
There was discussion on the potential additional testing of a model on a project in NI.
Mike highlighted a couple of projects that had the potential to be tested and it was agreed that the coastal rowing project might prove most beneficial.
It was agreed that Mike is to send through further details of this project which involved the creation of 9 new clubs in local rural villages around Strangford Lough with the aim of engaging local communities to the water and to create stronger community cohesion in the area. The project was funded by the Police Service community relations and the local authorities through the Strangford Lough Partnership. There are now approx. 500 members involved many of whom had no previous sporting interest.
For the main testing phase (WP3) it was highlighted that certain aspects of projects would be most useful (noted in green) and those from the application which may not be viable or useful (noted in red):
- Small scale local projects to larger scale national projects
- Projects involving a diversity of age ranges
- Projects involving a range of participants from a range of socio-economic backgrounds
- Short term intensive courses and longer-term regular but less intense courses
- Projects from at least 6 countries
- Projects from a range of environments including water, mountains and forests
- Projects delivered by voluntary, statutory or commercial organisations
- Projects involving a range of seasons including snow sports and summer water sports
- Projects involving coaching, instruction and leadership as well as those that are self-directed and independently undertaken
The group discussed how the outdoor sports model might differ from the general sports model. It was confirmed that the sports model did also contain outdoor sports. The group hypothesised that the overall cost of outdoor sports provision could be lower than indoor sports provision, and if so then the benefits (social value of outdoor sport) could potentially be greater. It was commented that the additional value of exercising outdoors (green exercise) would be a factor. The value of a person being more ‘environmentally responsible‘ might be small for an individual (is there a personal benefit? / do they care?) but that there might be a wider social value to this. The SHU team confirmed that they will be limited by what the evidence is – there is no population level data that shows an environmental value.
The logistics of developing the model were discussed: How could happiness be valued to the state? How much effort is it possible to put into the fiscal valuations? What data is available in the European partner countries? Primary data might be able to plug the gap – ways to access non-publically available statistical data might be possible. The closing note was that there is a risk of “comparing apples with badgers” – e.g. comparing things that are so different it is not even like the conventional saying of ‘trying to compare apples with pears’.