According to the Canadian Government (2012), “Theory based evaluation is an approach to evaluation (i.e., a conceptual analytical model) and not a specific method or technique. It is a way of structuring and undertaking analysis in an evaluation.”
In layman’s terms, it is a particular way of thinking about and doing an evaluation.
This approach is effectively based on what is called deductive reasoning. You start with a theory and then test and refine it.
There are a number of such theories but they will be broken down here into two categories: Realist (or Realistic) evaluation and Theory of Change approach. The latter is drawn upon heavily in our proposed toolkit.
Realistic evaluations ask ‘what works in which circumstances and for whom?’ rather than merely ‘does it work? (Pawson and Tilley (1997), Realistic Evaluation. London: Sage).
They believe that an intervention works (or does not) because actors make (or do not make) particular decisions in response to the intervention. They consider the ‘reasoning’ of the actors in response to the resources or opportunities.
Realistic evaluations home in on generative mechanisms (the powers which influence behaviour) or key causal factors in specific contexts (Better Evaluation, 2018). They also assume that generative mechanisms can only work if the circumstances are right. They develop and then tests hypotheses about what outcomes are produced by what mechanisms in what contexts.
For example, a training programme on the use of contraception in an African town might generate different outcomes and behaviours on the part of men and women taking part in it. The generative mechanism causing this difference may be different social norms or beliefs relating to the parenting and family-related responsibilities of men and women.
Are there social or cultural practices that lead men to want to father large families? The context may also affect the ability or willingness to put what they have learned into practice. Is the country or region going through a period of crisis? Are there large numbers of refugees or internally displaced people? These questions need to be answered by such an evaluation.
A more detailed example of such an evaluation is given by the website Better Evaluation. It focuses on hospital management practices in Ghana.
A realist evaluation yields information that indicates how the intervention works (i.e., generative mechanism) and the conditions that are needed for a particular mechanism to work (i.e., specification of contexts) and, thus, it is likely to be more useful to policymakers than other types of evaluation. (Better Evaluation, 2018).
Realist Evaluations in Practice
Blamey and Mackenzie (2007, p. 444) also provide advice on how to do a realistic evaluation, focusing on a programme to help smokers quit. To sum up, they identify:
- Stage 1: The evaluator, through dialogue with programme implementers, attempts to understand the nature of the programme/ target population/ contexts in which it will operate as well as the theories about why smoking cessation services work in some circumstances.
- Stage 2: The evaluator sets out mini-theories that link the contexts of the programme to the mechanisms by which it might operate to produce different outcomes. For example, pregnant women may be more susceptible to information campaigns about the dangers to their babies while non-pregnant women might be more sensitive to the impact on their self-image.
- Stage 3: The evaluator undertakes an outcome inquiry in relation to these mini-theories. This involves developing a quantitative and qualitative picture of the program in action. It might ask how different underlying motivations and mechanisms have been triggered in particular smokers by specific aspects of the programme.
- Stage 4: The evaluator refines and develops tentative theories of what works for whom in what circumstances.
Theory of Change
The approach that will be used extensively in this toolkit relates to the Theory of Change (ToC). This approach involves developing a ToC, which shows how your project or programme ought to work (in theory) and which spells out the assumption behind this theory/ thinking.
In a nutshell, your ToC is basically your best guess, namely that X will cause Y to change. It should ideally be developed on the basis of a range of stakeholders’ views and information sources. You will then seek to check whether the theory is matched by the facts relating to your project/ programme.
Your ToC can provide the basis for arguing that your project is making a difference in some areas (and not others). It can provide a convincing case where you have carried out your planned activities, have found your assumptions to be correct, and where you can show that contextual factors have had a limited impact.
In other words. the ToC can help you to say that in all probability it was because of the activities of your project that desired outcomes and goals were achieved. Alternatively, it can suggest the need for a new ToC and new assumptions that you had not previously considered to be relevant.
The ToC approach is good for testing assumptions, for identifying unintended outcomes and for involving beneficiaries and stakeholders in thinking about a project, how it is working and how it could work better.
Theory-based approaches can quickly become over-complex if an exhaustive list of determining factors is compiled. It can also lead to disagreement between stakeholders over which are factors are most important. These theories are also at a relatively early stage of their evolution. So check with your funder what their attitude to such approaches is before committing to a full-scale ToC approach.