Impact Reporting: Is A Theory Of Change Always Appropriate To Measure Impact?

What is impact reporting?

Impact reporting is communicating the change that your organisation or charity has made in the communities that it is committed to serve. As a charity, social enterprise or mission-driven organisation, your social impact report should address the need for the change you are trying to make; the activities through which you intend to make the change and the outcomes achieved supported with evidence to prove your findings.    

Impact reports can take several forms depending on the targeted reader, who can range from funders (commissioners, Trusts and major donors), stakeholders (beneficiaries and volunteers), alongside those in the organisation itself (trustees, management team and staff). Typically, they contain an executive summary, infographics, key statistics and quotes from key stakeholders and beneficiaries. Further design, colour and imagery can be added to reflect your organisation’s brand attributes and can even be used in marketing content to celebrate achievements and successes.  

Why are impact reports so important?

Impact reports give an opportunity for organisations to reflect on their mission, and measure how successfully they have achieved the goals they set. Planning the research carefully will allow you to identify measurable indicators of success and evaluate the most and least effective means through which outcomes and therefore impact is achieved. Depending on how the data is collected, it could also be used to benchmark and track against for future comparison, allowing for pattern and trend recognition over time.

Impact reporting can be used to guide internal decision-making and allows organisations to learn from the work and review the services or policies they have in place to make societal changes. It can help them to also find ways to improve or develop new initiatives to make a positive change in their targeted communities. Other organisational benefits include being able to motivate staff and trustees through celebrating achievements. It is also important to highlight areas where impact may be lacking, as this promotes a sense of accountability and identifies initiatives that need reviewing. Conducting such research builds trust and further credibility to organisations who can demonstrate achievements for communities in need of further attention and resources.  

Overall, impact reports can be used as a tool to connect organisations with funders, stakeholders and members of the organisation itself with full transparency. The report could reflect the differing views in which these groups believe the goal should be achieved, all while their intended goals remain consistent. Lastly, impact reporting can be used to share lessons with other, like-minded organisations to augment further, more positive change in society. 

Theory of Change: a tool for measuring impact

Researchers have developed a variety of tried and tested methods to measure the impact of an organisation’s service on society. A Theory of Change is a way to systematically map how desired goals are achieved, where preconditions are mapped working backwards from the ultimate and desired goal.

The first phase considers how the goal can be achieved with long-term outcomes, followed by intermediate and shorter-term conditions needed to enable them. These are stated with detail and specifics to graphically illustrate a causal framework, which makes the Theory of Change a unique method for describing initiatives. Assumptions underly all hypotheses, and as such the causal connections mapped require researchers to articulate those assumptions, which can also be tested and measured. This strengthens the methodology as assumptions that stakeholders use to explain the Theory of Change have to be evidenced and supported by research.  

While a Theory of Change provides clarity in the strategies, outcomes and goals of an organisation, it can also: align stakeholders by demonstrating how different activities contribute to the overall goal; promote accountability by identifying assumptions; and detail expected outcomes of a service or initiative. 

Is a Theory of Change always appropriate to measure impact?

A Theory of Change is one of the most used tools to measure impact, however it does have its limitations. It can be a complex process which requires comprehensive understanding of the social context, causal relationships and underlying assumptions. Some social factors that require further understanding can be those related to socioeconomic status (e.g. income, education, occupation) or social identity (e.g. gender, ethnicity, race, religion) for example, which adds a layer of complexity and nuance to the framework.

A Theory of Change also requires significant resources, which could make it difficult to manage from a costs and logistical perspective. There may also be some difficulty in measuring indirect impacts that result from factors beyond an organisation’s initiatives. Such limitations can render this methodology a “descriptor of activities”, rather than a Theory of Change. 

A feature about this methodology that requires more consideration is the overemphasis on linear causality. In reality, it is rare that outputs are affected by only one causal factor, which some Theory of Change frameworks imply and is therefore at risk of oversimplifying the dynamic nature of social change processes. Changes in one part of the system can cause a disproportionate ripple effect on other parts of the system in non-linear ways, which can lead to inaccuracies in predicting outcomes.

Such linear thinking may fail to account for unintended consequences of interventions or activities. Temporal change may also be unaccounted for as activities and policies take time to produce a result. Consequently, time delays caused by social factors may not be immediately apparent, leading to unrealistic expectations about the pace of change. Lastly, a Theory of Change may not account for emerging patterns which appear unexpectedly due to unpredictable behaviours of social factors. It is therefore important to consider that there is less predictability than might originally be mapped in a Theory of Change, and that the interconnectedness between social factors need to be recognised and anticipated from the outset.  

How to overcome the limitations of linear causality in a Theory of Change

To overcome the overemphasis on linear causality, a more nuanced and anticipatory approach is required, which acknowledges the complexity and interconnectedness of social factors. Firstly, a systems-oriented approach should be incorporated into the framework to recognise the interactions and feedback loops that exist between different components of the social system. This in turn warrants the development of non-linear modelling techniques to illustrate the dynamic links between activities, short- and long-term outcomes.

To help navigate complex systems, it may be worth looking out for emerging patterns or trends in society. Using a more strategic approach would break the problem down into manageable bites, allowing for a focus on shifting patterns rather than tackling the entire social issue at once.

A diverse group of stakeholders with lived-in experiences should be consulted to gather insight into the complexity of social systems, to create a more robust and contextually-relevant Theory of Change. This in turn calls for collaboration across sectors, to develop a more interdisciplinary and holistic understanding of the social change at hand.

Fostering such openness should lend to a framework that is more agile to change, and open to adopting new strategies in response to evidenced calls for change. 

Conclusion

Therefore, a Theory of Change doesn’t have to be the main method in which an organisation’s impact is evaluated. We believe an amalgamation of methods can be used to work towards change, as opposed to starting at the end point.

In the context of a complex social system, this approach allows you choose the direction to move forward in. The first step would be to develop intended goals with the activities or initiatives planned to be in place. The research should seek to find the “who, what, when, where and how”, to map the actual impact or outcomes on the organisation’s community. The wider economic and social implications the organisation may have with the service in place vs without, should also be considered.  

While developing social impact measures, effort should be made to enable their benchmarking and tracking over time, allowing for monitoring within the framework. Literature search in the form of secondary data collection can support your findings, or provide evidence for developing new or alternative initiatives. Lastly, using a storytelling approach can consolidate the impact achieved in reality, through case study generation from an individualistic and wider economic and social perspective. 

Overall, while a Theory of Change can graphically illustrate the processes required to bring about a specific social change, it should also anticipate the real, dynamic nature of social processes, where change is very much a constant. It should reflect the complexity of social systems and ultimately lead to a more effective measurement of the social impact in question. This would allow for a more accurate evaluation of services or policies, to ensure only the most meaningful are implemented in society.