BOSS Meeting WP 3.3
11-13th March 2019
Paris- Union Nationale des Centres de Plein Air (UCPA)
Grant Agreement 2016-3225 / 001-001
A3.3 Training for the case study leaders
|Antoine Le Bellec (CREPS)
Maxine Gregory (SHU)
Barbara Eigenschenk (TUM)
Laure Dubos (UCPA)
Adrien Forestier (UCPA) (Day1 & 2)
Petar Iankov (NSA) (Day 1)
Mike McClure (SNI)
Larissa Davies (SHU)
Eduard Ingles Yuba (INEFC)
Joao Zamith (SCV)
Leonardo La Rocca (RL)
Arne Strate (EOG) (Day1 & 2)
Cecelia Do Teggour (FITE) (Day1 & 3)
Peter Taylor (SROI Expert
|Chris Kay (Battleback)
Jerôme Deschamps (En passant par la montagne)
Mariona Masdemont (Play and Train)
Sarah Broadfield (British Cycling MTB project)
Clare Brogan Community Rowing)
Claire Saunders (Park Walk)
Sabine Chapuis (Cheval et diversité)
Milena Kuleva (Course outdoor sports in NSA)
Sergio Barbosa Fernandes (Nautical in schools)
Marco Areias (Nautical in schools)
Friederike Lange (Crossing the Alps by foot)
Le Bellec welcomed everyone to the meeting and the meeting was opened. The group thanked the UCPA for hosting this meeting in its offices.
Laure Dubos who is the director of sport and education strategy at UCPA also welcomed everyone to UCPA and highlighted that they were delighted to be able to host the training.
Mike McClure provided an introduction to the overall BOSS project and each of the WP leaders in the project provided a short introduction to the 4 key areas of work:
Work Package 1 – examining the evidence of outdoor sports benefits – Barbara Eigenschenk
Work Package 2 – developing the model – Maxine Gregory
Work Package 3 – selecting and training the case studies – Petar Iankov & Adrien Forestiere
Work Package 4 – Dissemination and communication – Mike McClure
Each of the project leaders then provided a short presentation on each project:
- Cecelia Do Teggour (FITE) highlighted that the Cheval et diversité project provides quality opportunities for people with disabilities to access
- Chris Kay – Battleback. A very well established five day residential activity programme which is provided primarily for military personnel
- Andi Thomann (Crossing the Alpes) Participants who undertake a significant on foot journey across the Alps using an established trail
- Jerome – Mountain as springboard. A project that provides pathways for local young people in Chamonix to engage with and participate in mountain related activities that could lead to increased employment.
- Sarah Broadfield – British Cycling. An open programme with scope to focus in on different projects / target groups
- Claire Saunders – Parkwalk. A project to engage sedentary people with outdoor activity through walking in urban and/or rural parks on a weekly basis
- Mariona Masdemont – Play and train – Mountain and Watersports for people with disabilities
- Clare Brogan – Row the Erne. A project that seeks to provide community based rowing experiences on a large 10m traditional boat
- Sergio Barbosa Fernandes (Nautique in schools) A project that developed a partnership between schools, the local authority and clubs to increase participation in water sports
- Petar Iankov – NSA – A project to train students in the skills needed for employment in the outdoor industry
- Leonardo La Rocca – Step by step. A trekking project using donkeys that engages young people with mental health issues or learning difficulties
- Adriene Forestiere– Summer and winter camps for students on short term breaks
Training session 1
Maxine Gregory started by outlining the mechanisms of how the training will be structured. There were 2 flipcharts provided whereby questions and feedback could be placed. Maxine explained that they will aim to:
- Try to keep communications clear and simple
- Be flexible and try to change things if they are not working well
- Support case study leaders through this process, alongside the national lead / BOSS partner
- Agree a clear plan, next steps and timescales for all case studies at the end of the training
The current position is that:
SHU has done a lot of work in the UK on measuring social value, however this process on a pan-European basis for a new sector is very challenging. It was noted that the partners did not know what the ‘model’ for measuring social value would look like but have discussed, debated and investigated many options! The selected option of a ‘process model’ is necessary because of the limited amount of robust quantified evidence from WP1.
SHU have indicated that there is a need to continue to build this model and produce a ‘toolkit’ to share with the sector. This is the role of WP3 – rather than just testing the model, we are informing and further developing it. The model aims to provide something that is practical, as well as theoretically robust.
Work Package 2 is fundamentally connected to Work Package 3. There is a need to provide evidence on wider outcomes identified through the 12 case studies. The process model and guidance has been produced but this requires further development through the case studies.
The role of Work Package 3 goes beyond just testing the model created for Work Package 2 and is a vital part of the development of the outputs for the BOSS project. It will help to identify and understand outcome areas and to create a model that is ‘bottom up’ informed by participants and their experiences, rather than ‘top down’ based on generalised findings from other studies
In WP1 many of the reported benefits were not ‘quantified’. By this it is meant that the studies suggest that participants experience change but not ‘how much?’ change.
An example of quantified change:
- What has changed? Physical activity reduces risk of cancer
- Which cancers? A positive effect for 13 types including liver (-27%), lung (-26%), breast cancer (-10%). A negative effect for malignant melanoma and prostate cancer. No change for 9 cancer types
- How much change? A net reduction of 7% in risk in total cancers
- Sample size? 1.4million adults
Where this level of evidence exists, then this can be placed into the model. However, for 4 of the grouped benefits while there was clear evidence of the impact of outdoor sports there were no quantifiable benefits highlighted.
However, it will be possible to investigate further evidence from the case studies in relation to the following benefits:
The outcomes that were sufficiently evidenced to build a model from secondary data were purely from physical health. The wider list of outcomes for the other grouped outcomes required further examination through primary data collection from the case studies.
Understanding Social Value
Peter Taylor ran the next session and started by explaining that social value is all about the holistic positive results from an activity. It is a broad concept and is much more than financial revenues and economic impact.
Social value includes social, environmental and economic benefits from an activity – e.g. the value of health improvement, happiness, social cohesion, reduced crime and reduced pollution. It therefore goes beyond those costs which are tradeable (i.e. can be bought or sold). Conventional measures of measuring value are limited and don’t include issues such as happiness.
Understanding social value terminology
Social value can be expressed with alternative words:
- social benefits (positive effects)
- social costs (negative effects)
- social outcomes (positive and negative effects)
- social impacts (positive and negative effects)Other terms relevant to measuring social value are:Outputs (the quantity and quality of the activity created) – e.g. number of participants; duration or even intensity of their participationSocial value can be measured in monetary terms by using a variety of methods, including:
- Social value can be measured in absolute terms e.g. increased life expectancy, reduced repeat offending, increased density of pollutants. However, to estimate total social value from an activity there needs to be a common measure for different effects and that is typically monetary value (even when the effects are non-financial).
- Inputs (financial and non-financial costs incurred to create an activity) – e.g. equipment costs; volunteers’ time etc.
- However, ‘social value‘ is used because we aim to put monetary values on social outcomes/impacts.
- Secondary evidence from literature which provides reliable data on effects, e.g. the data from WP1 on physical health
- Direct measurement of effects
- Costs of generating effects by other means
- Costs of preventing or countering effects by other means
- Direct research with relevant stakeholders.
SOCIAL return on investment
Social value is only one half of the equation for the social return on investment of an activity. The other half is the costs and inputs / investment (which includes all inputs, financial and non-financial, to create an activity).
SROI, therefore, is the total social value of the impact divided by the total input costs
Peter closed by asking participants to decide whether they want to just calculate the overall value of their project or whether they want to work out the SROI for the project. The majority wanted to do the latter in the realisation that this will involve more effort.
Training session 2
The process for the model involves a 7 stage process as follows:
Stage 1 – Establishing Scope
In order to understand the scope of the project the following questions need to be considered:
- What is the purpose of the project in terms of social value?
- Who is the intended audience – internal (members / management committee) or external?
- What resources are available?
- Have you got help for the focus groups or the survey work?
- Do you want to undertake a study in social value or social return on investment?
- What timescale to cover this? This is more or less determined for the BOSS project?
- Also, in terms of activities – do we want to cover the range or focus?
At this stage a question was raised if it would be possible to survey participants who have previously been involved and the response was that there are advantages and disadvantages to this and no fixed answer.
Stage 2 – Identify stakeholders
Stakeholders are defined as people or organisations that experience change or affect the activity, whether positive or negative, as a result of the activity. One of the key issues is the “attribution” of the activity to the change.
It is important to recognise that participants are generally the most important stakeholders, as they are the people for whom the project is designed and who may experience beneficial outcomes from their participation in it.
It was highlighted that for stakeholders to be relevant they should either be affected by or effect the activity. For example if a project had a detrimental impact on the environment then an environmental agency would be a key stakeholder. Likewise if a project created significant community cohesion and inclusion benefits then a local authority may be a key stakeholder.
Stakeholders may also be organisations (or even individuals) that provided financial or non-financial support that enables the project to take place.
Based on knowledge of their projects, the project leaders were asked to fill in a pro-forma to:
- Identify the potential stakeholders
- Think about what, if anything, changes for them.
- Which of these stakeholders would be included/excluded and why?
What would each stakeholder invest/contribute?
Stage 3 – Identify inputs
In order to undertake an SROI effectively there needs to be thought given to what the costs are both in financial and non-financial terms. Key questions to ask are:
What do participants contribute in order to make the activity possible? (Membership fees or costs of gear etc.).
How much financial investment has gone into the project and where did it come from?
What value of goods and services in kind have been provided to enable the project? This could be equipment, hospitality etc.
What is the time required and cost of participation in an activity?
How much volunteer or paid instructional/guiding/coaching time was required to effectively deliver it?
A key issue is to beware of double-counting. So for example a state agency may give a local authority a set of funds to distribute appropriately. The local authority then provides a grant to your project. The grant must not be counted from the state agency AND the local authority as that would double its value!
Stage 4 Identify outcomes
Outcomes are the important changes that happen that that bring benefits to the individual and/or organisations. It is important to consider what outcomes there may be for your project but it is essential to engage with your stakeholders to help identify outcomes that you had not considered. Stakeholders should be asked to describe what changes for them and also if there are other stakeholders that can identify that you may not have thought of. It is important to work with your stakeholders to develop suitable outcome indicators.
Physical health and to a lesser extent mental health outcomes could be identified from the research in WP1. There is clear scientific evidence that participation in outdoor sport, and in sport and exercise generally, can contribute to improved physical health with a 7% reduced risk of cancer, a 30% reduced risk of Coronary Heart Disease and a 10% reduced risk of type 2 diabetes. It also showed that it can contribute to improved mental health with a 21% reduced risk of depression and a 30% reduced risk of dementia.
Outcomes for education and lifelong learning, active citizenship and crime reduction and anti-social behaviour will need to be developed through stakeholder engagement.
Stage 5 Evidence and value outcomes
Evidencing health outcomes.
It is essential to find out from your participants how much physical activity they gain from the project and so you need to collect data from participants about their physical activity levels in terms of:
- Frequency of activity sessions
- The duration of their sessions
- The intensity of the activity
The definition of inactive to active:
It is also important to collect data from participants about their age and overall health (so for example the health benefit of reduced risk of cancer does not apply to a participant who already has cancer (although it may reduce reoccurrence).
It is also essential to source data about prevalence rates of specific health conditions in your country, then using the evidence of the link between sport/exercise and health improvements it is possible to calculate the potential cases of disease prevented
Valuing health outcomes
In order to calculate the value of the health outcomes it is also important to know the cost of treatment of certain conditions and diseases. For example, SHU has accessed data on costs of treating diabetes in Europe which are highlighted below. This shows significant variations in costs across the European Countries.
It will be important for project leaders to try to get data from within their own countries for the main health improvements associated with outdoor sports of:
- Cancer treatment
- Coronary heart Disease
- Type 2 diabetes
Evidencing other outcomes
In order to evidence the other outcomes associated with outdoors sports there will need to primary research with the participants through engagement and surveys. These need to explore the extent to which they experience outcomes identified. It is important to consider two key issues:
- Deadweight is a measure of the amount of outcome that would have happened even if the activity had not taken place.
- Attribution is an assessment of how much of the outcome was caused by your organisation / project as opposed to that which has been the contribution of other organisations or people.
Valuing other outcomes
One of the simplest ways to value the other outcomes that stakeholders have identified is to ask stakeholders directly how they value things in terms of how much they would be willing to pay to have or avoid something. While it doesn’t matter whether or not people could afford to buy something – they can still place a value on it.
It is important to package the outcomes together when looking the value. For example if stakeholders identify that they have improved mental health, have developed a series of new skills and also have made significant friendships and feel less isolated – these must all be packaged together to assess value rather than each one separately.
Stage 6 Reporting
A template will be provided later in the year to help each project report on their findings but there are a number of principles that are important. First it will be possible to highlight the social value of the project
“The social value created by XYZ project each year is estimated to be approximately €10,000.”
If you decide to undertake a full Social return on Investment assessment then you can express this as follows:
“The project’s SROI ratio is X:1, which means that for every €1 invested, €X of social value is created in terms of potential healthcare savings and improved wellbeing of participants.”
However, it is important not to just focus on social value or the SROI ratio but rather such data needs to be presented alongside the story of how change is being created. Therefore in the report you should include participant case studies and quotes from stakeholders to illustrate particular findings.
Training session 3
Larissa Davies introduced the principle of logic modelling as a way of mapping changes created.
It was highlighted that a logic model is simply a term used in social value to describe a diagram which represents the theory of how an intervention or programme produces change. It was explained that changes can be intended (as per the programmes plan) but also can be unintended. While there is the hope that changes will be positive there has to be recognition that they can also be negative. A logic model shows the relationship between the inputs, activities (outputs) and outcomes.
SHU has developed a series of logic models for outdoor activities at a population level using ‘scientific evidence’ provided by WP1.
For physical health the quantifiable evidence was fairly strong and it has been able to create a logic model for certain diseases and conditions (see Table 1 below)
However, for the areas of mental health and well-being, education and life-long learning, active citizenship and crime and antisocial behaviour while there was evidence of benefits it was not quantifiable. These logic models demonstrate benefits at the ‘population’ level and need to be translated into your project in terms of the numbers (activities and outputs). For some outcomes the logic is clear and the evidence is good but for other outcomes there is less robust evidence and the logic is less clear. It is important to be cautious when applying such logic models to your project.
A logic model is key to understanding the change that occurs as a result of your case study. It is the ‘story’ of how your programme makes a difference to the world. Your stakeholders give you information to help you map the logic of your programme.
Peter highlighted the difference between intermediate outcomes and final outcomes. In physical health terms the final outcomes are things such as improved good health status, decreased healthcare costs, improved quality of life and reduced risk for mental illness. However, there a number stepping stones on the way and these are the intermediate outcomes and can be quite difficult to measure.
Maxine then talked through the process of mapping out a programme logic model and used a model that SHU had done to analyse the impact of “Run Birmingham” project.
The first column is the context of the project. This can be captured from the reasons why a project was created in the first place.
The logic (last) column captures the assumptions that are being made at the start of the project. This helps with understanding the logic and so a key question is to consider why the project has been developed. Is it to….?
- Improve the health of participants?
- Create personal benefits for participants?
- Create community benefits?
Inputs are what those stakeholders invest in the project both financial and time
Activities are what is being done in terms of the programme
Outputs are the numbers of participants, number of sessions, duration of sessions etc.
Intermediate outcomes are stepping stones to health, wellbeing, educative or societal benefits
Final outcomes are major factors that benefit the individual or society such as improved good health status, decreased healthcare costs, improved quality of life and reduced risk for mental illness.
Each of the project leaders were then tasked with developing their own logic model and to report back on their project.
Maxine highlighted that the logic models that have been created are from the perspective of the project leader as funders / organisers, however, the logic model for the programme should be developed with feedback from the stakeholders. This can be done in many ways e.g. focus groups; interviews; surveys etc.
Training Session 4
Consulting with Stakeholders
Maxine noted that in the example that they had supplied on Run Birmingham they had consulted their stakeholders through a series of focus groups. The project that Mike McClure outlined on coastal rowing also engaged with focus groups while the British Cycling project outlined by Sarah Broadfield used direct surveys. This highlighted that there a number of ways to engage and there is not a defined approach to it.
Participants on the training were then asked to consider how they will consult with their stakeholders to identify what changes because of their programme and how they will facilitate these discussions?
Example questions that can be used for stakeholders are as follows:
For the leaders / coaches volunteers:
- What changes have you noticed amongst the group? (personal, social, wellbeing)
- What has changed for you?
To the participants:
- What changes have you experienced?
- Have there been any unexpected impacts / changes?
- What has been the most important benefit you have experienced from attending the project?
Top tips for consulting stakeholders
Ideally it is best to try to get the stakeholders in one place and ask them the questions directly. It is useful to have a mechanism to record answers (through post it notes, flip charts or an independent person recording). However, it is important to give consideration to what time and resource you have available to do this and also what time and resource the stakeholders have. It is useful if you can arrange to meet the stakeholders when they are already gathering or meeting together anyway. To encourage responses it is worth considering if you can incentivise their contributions with some form of prize draw etc.
Training Session 5
Peter Taylor introduced the concepts of survey design which aimed to help delegates to get started in this process.
Ideally the survey will be designed after consultation with stakeholders in order that the survey fits with the main changes that they highlight and also after constructing the logic model. However, in order to facilitate the training session, it needs to be covered now. Peter highlighted that SHU will help delegates to decide on how to approach survey design and administration and provide support for the work on this.
Organising a survey
Surveys are extremely useful for management decisions, policy development and accessing funding. They can provide statistically valid and reliable results and can be accurate, unambiguous, unbiased and have low error margins (related to the size of the sample)
There are a couple of options as to how the survey will be administered:
- Contract a specialist research organisation to conduct the survey for you.
- This option will only apply if you have the financial resources to do this but is good if you do not have the people or expertise to do it yourself
- Carry out the survey yourself.
- If you have in house staff with the experience, training and support / supervision to undertake it then this option is good.
There are a number of stages to organising your survey (before you even decide on the questions to include within it):
- Who do you want to be surveyed, you should count or estimate normal numbers? The aim is to keep things as normal as possible (don’t carry out your survey or count numbers at a major event that is not part of your normal activities)
- What is the minimum number of responses required either as a number (for larger projects) or a percentage (for smaller projects)?
- What sampling method will you use – random or quota sampling?
- How do you deal with younger children and or people with a learning disability?
What time period do you have to undertake it and what locations will be used for surveying?
Think about proportionality – put in an appropriate amount of effort and energy for the scale and scope of your project. Small scale projects it is important to survey as many participants as possible to get a reflective view while a large-scale project it will probably be only possible to get a representative sample of participants. If unsure about this, you can liaise with SHU who will provide advice.
Think about the resources / capacity that you have in place to deliver the required steps within the timescale as the process needs to be undertaken between March and June. Who in your organisation can lead / contribute to this? What support do you need?
The graph below shows the relationship between sampling errors and the sample size. Clearly the larger the sample size so the errors diminish. However, it should be noted that once over 1000 responses the level of error does not reduce significantly but the effort to manage a survey of this scale can be significant.
Designing your survey
There are a range of options to how your survey is designed and administrated:
Designing the survey
- Self-completion / participant administered – this can be given out and collected back for example at the end of a session OR it could be done online.
- Interviewer administered – face to face interviews carried out (Consideration needs to be given to where and when these would take place and involve a lot of time for the interviewer.
- The type of survey
- Paper based approach – does it get delivered and collected by hand OR do you use a postal system (expensive). Either way you will need to input all the data by hand not a spreadsheet
- Online approach – done via email or website and the data can automatically be transferred to a spreadsheet.
- The format of the survey
- One off survey (after participation) – reporting the changes and clearly will be attributed to the project
- Repeat measures (a before and after participation approach) – provides baseline data and allows for follow up questions and again changes can be attributed to the project
- Stages of survey development
- Consult with stakeholders and identify the outcomes
- Add generic survey questions (regarding health etc.)
- Add specific and tailored questions relating to the outcomes they have identified
- Ask for and gain feedback from SHU team
- Pilot the survey with a few key people and redraft questions if necessary
- Carry out the survey with stakeholders
Support with survey development
Think about who will process, clean and analyse the results? Also it is important to consider how you want to present the data and who your report is aimed at:
- Will it be specifically for your own organisation for development and management purposes OR
- Will it be for publicity, accountability or funding purposes and therefore need to be very clear for an outside source to understand it
One important consideration in survey administration is that you should try to retain the ability to send reminders to potential respondents, in order to drive the sample numbers up. To do this you need to know who the individuals are that you are surveying.
The survey Questions
It is anticipated that all the projects will have a series of standard questions based on:
- Demographics of your respondents
- Health status
- Participation levels (for health purposes)
Then there will be a series of more bespoke questions relating to:
- The type of activity
- Outcome areas that have been identified through the stakeholder consultation
- Valuation questions
- Qualitative feedback
- Anything else that you need to inform the development of your project / evaluate your project objectives
Collating the data
SHU will provide a template excel spreadsheet for all the data produced by your survey, and so all the data will need to be inputted to this table. If you had a paper based survey this will prove to be very time consuming and so an online version is recommended as it will be much easier to transfer the data.
It will not be necessary to translate all your data into English, but please place English headings in the columns for the quantitative data. It would be useful to summarise qualitative data into English.
Training Session 6
CALCULATING SOCIAL VALUE AND SROI
Inputs are what the stakeholders / participants put into the project. They can be financial or non-financial
Consider all the income that has gone into the project from both external and internal sources
This can included funding from national, regional and local governments as well as government agencies or funding from charitable organisations. It can include sponsorship from private sector organisations and fund raising that the club may have done themselves. In terms of the government funding – be careful that there is no double counting! Internal financial costs are what the participants pay for membership fees, equipment, clothing, footwear, travel etc.
If a project involves some level of capital investment which will be in existence for a number of years then the cost of that investment will be recorded on a pro-rata per annum basis.
For example in a cycling project and a participant’s bike costs 1000€ but could be used for 10 years and the project is for just one year then only a proportion of the cost of the bike (10% – 100€) should be allocated to the project.
An important element of this includes volunteers’ time and is calculated by multiplying the number of hours volunteered x the average hourly earnings in that country. One of the challenges is to calculate the average earnings. This could be at a population level or for the sports sector. However, the latter becomes a problem as the rate of pay for a coach could be very different to an administrator. Whatever you use it is important to be transparent about where the data came from.
Non-financial inputs can also include sponsorship or gifts of goods and services (known as in kind contributions. This can include equipment, consumables, transport etc. (Calculate what it would really cost to provide these).
Example for SROI for Sport in England
Through accessing data on your health care costs it will be possible to create an estimate of the healthcare cost savings associated with potential cases of disease prevented for specific health conditions through engagement with outdoor sport interventions. This can be added to the subjective value placed by participants on any personal benefits experienced by them from engaging in outdoor sport interventions. Finally and where applicable, the value placed by other stakeholders on any positive outcomes identified by participants can be included.
The social return on investment (SROI) of outdoor sport interventions will provide the health and other benefits in relation to the associated costs.
* National leads will need to find out the costs of specific healthcare interventions and the prevalence of specific diseases in each country.