Increasingly, charities are being asked to demonstrate the impact of the programs and services they deliver. Beyond descriptions of the programs and services they offer, tabulations of the number of clients served, or stories about the organization’s impact on individuals, organizational stakeholders (funders, board members, policymakers, members of the media, and individual Canadians) are hungry for information to help them understand how the charities they support are affecting both the populations they serve and the broader social, economic, and environmental systems they are embedded in.
Repeatedly during Imagine’s work to help establish a new narrative for charities and the charitable sector, participants and informants told us that being able to demonstrate these types of impact is critical to successfully reshaping the charitable sector’s conversation with Canadians. However, during our day to day interactions with charity leaders, many (particularly those leading smaller charities) tell us they struggle to do this. This is not because they are not measuring and evaluating their work and it is certainly not because their work is not having an impact. Instead, they struggle with demonstrating impact as part of a broader set of challenges related to measurement and evaluation. Looking at the evaluation landscape, they see a diverse range of potential approaches and techniques they could apply, an even wider range of possible uses for the results produced, and an incredible diversity of stakeholders and audiences to engage with. Faced with a sometimes overwhelming range of potential options and demands, what charity leaders need is a common understanding with their stakeholders of what norms and expectations are reasonable for their organization and circumstances.
Charities use evaluation findings in many different ways.
The most common uses of evaluation findings are reporting (to Boards of Directors and funders or supporters) and tracking and better understanding program outputs and outcomes. Charities using results in these ways tend to focus on evaluating their outputs and outcomes using the most common methods and resources. Charities that use evaluation findings in less common ways, such as reporting to more specialized audiences or to support organizational decision making, tend to evaluate more involved aspects of their work and use more specialized measures.
96% of charities evaluate their work in some way.
Charities are most likely to evaluate the outputs, outcomes and quality of their work. When they evaluate their impact or the return on investment of their work, it is almost always in addition to evaluating one or more of these three more common aspects of their work.
External funding plays a significant role in evaluation.
Charities receiving external funding tend to allocate more of their budget to evaluation. While it is uncommon for external funding to include monies specifically for evaluation—just 20% of externally funded charities reported receiving dedicated evaluation funding— charities that receive this funding tend to evaluate more involved aspects of their work (such as their impact and return on investment) and to use more specialized evaluation methods. However, they do not seem to use evaluation findings differently.
Charities draw on a range of quantitative and qualitative methods and resources to evaluate their work.
Quantitative resources such as administrative data and statistical compilations of services delivered are most commonly used, particularly when charities are evaluating their outputs. Charities evaluating more involved aspects of their work, such as their outcomes or impact, are more likely to use rarer and more involved approaches, such as focus groups, logic models / theories of change, and case studies.
A number of key enablers and barriers affect evaluation capacity.
The most important enablers are support from organizational leaders and buy-in from staff and stakeholders. Charities experiencing these key enablers are more likely to evaluate their impact, to use a range of more specialized evaluation measures, and to use evaluation results to communicate with less common audiences. The most common barriers are lack of staff time and financial resources, but while these barriers are commonly reported they do not seem to be associated with significant differences in what charities evaluate, the evaluation measures they draw on, or how they use results.
25% of charities belong to some sort of formal or informal group, network or association related to evaluation.
Charities draw on these relationships for a range of supports, most commonly: measurement and evaluation tools, training, and being part of larger evaluation projects. Charities belonging to these networks tend to have greater evaluation capacity.
21% of charities report having at least one paid staff member primarily devoted to measurement and evaluation work.
For the remaining charities, evaluation is an additional responsibility that is frequently spread across multiple staff positions and most commonly falls to the Executive Director / CEO or program staff / volunteers. Unsurprisingly, charities with dedicated evaluation staff have significantly higher evaluation capacity.
Overall, charities’ opinions about evaluation are quite favourable.
Charities are moderately happy with their evaluation capacity, giving it an average score of 6.4 out of 10. Charities experiencing barriers are less satisfied with their evaluation capacity, as are organizations that place high priorities on fundraising and revenue generation, organizational governance, and communications and marketing at the expense of other areas. Solid majorities of charities see a need for evaluation to guide their work and believe the resources they devote to this end are well-spent. However, significant numbers highlight difficulty making full use of the data they collect.
22% of charities report having worked with an external consultant or organization in some capacity related to evaluation over the previous year.
While charities that work with external evaluators tend to be less satisfied with their evaluation capacity, they tend to actually demonstrate greater capacity— evaluating more involved aspects of their work and drawing on more sophisticated methods and resources. Overwhelmingly, charities that work with external evaluators are happy about their experiences with them.
Imagine Canada is the national charitable organization whose cause is Canada’s charities. Our three broad aims are to amplify the sector’s collective voice, create opportunities to connect and learn from each other, and build the sector’s capacity to succeed. For a full copy of this report, click here.
