Randomised Trials
The advice below draws on excellent sources such as
Better Evaluation and Jeffrey Hammer. It has been kept as simple as possible .
What is a Randomised Controlled Trial?
Randomised Controlled Trials (RCTs) can be quite sophisticated. A simple definition is nonetheless offered by National Cancer Institute as:
“A study in which the participants are divided by chance into separate groups that compare different treatments or other interventions. Using chance to divide people into groups means that the groups will be similar and that the effects of the treatments they receive can be compared more fairly. At the time of the trial, it is not known which treatment is best.”
When to Use RCTs?
RCTs are perhaps most commonly used in the medical field where one group is given treatment and another, the so-called control group, is not (or may instead be given a placebo).
They can equally be used in other sectors including education, where one class is given a particular training course and another is not.
RCTs enable you to see the impact of your project and at the same time what would have happened if you had not intervened. As Better Evaluation put it, ‘the control mimics the counterfactual.’
The key to RCTs is that the groups are selected randomly. This might involve putting the names of individuals, groups, schools or clinics in a hat and picking two or more. You are not pre-selecting or focusing on a purposive sample of individuals who are more enthusiastic about an issue than their peers. Such an approach would build in bias. Rather you are selecting one class, say in their final year of primary school, and another class, say in the final year of a different primary school. Randomisation is often done at the level of clusters: villages, schools, or health clinics.
How to do a RCT
Using a metrics-based evaluation, you are likely to follow these key steps:
- Conduct a baseline survey of the entire target sample.
- Randomise the sample into distinct groups. The control group and the group you are treating should have key indicators in common (the same average income, the same average health level, the same stage in the education process).
- The intervention is implemented in the ‘treatment’ group.
- Monitor the intervention, checking progress and ensuring that no contamination is taking place.
- Undertake a follow-up survey that shares many of the features of the baseline survey.
- Compare the outcomes on the treatment and control groups to derive the impact estimate. Results are reported to the implementing partner.
Watch Out For:
- Invalid comparisons. According to Better Evaluation:
Many times, evaluations compare groups that are quite different to the group receiving the program. For example: if we compare the outcomes for women who take up microcredit to those that do not, it could be that women who choose not to take up microcredit were different in important ways that would affect the outcomes. For example, women who do not take up microcredit might be less motivated, or less aware of financial products.
- Bias in the sampling/ selection process.
- Sample size, particularly where your sample is too small, this can render your findings less meaningful and statistically insignificant. Significance levels above 90% – preferably at 95% – are sufficient.
- Timing: RCTs cannot determine impacts of currently existing projects, that is, of projects that have already launched. So, you have to plan in advance.
- ‘Contamination’ across your selected groups. This is where treated or trained individuals mix and exchange ideas and potentially even ‘share’ treatment with individuals in the control group. This would ‘contaminate’ your impact, and our control group would no longer be a good comparison. By randomising at the group rather than individual level, you can limit this danger. In effect, one village is treated and the other village (some 20 km away) is the control group. This would limit the risk of contamination.
- Take-up rates, which can be lower than expected, and non-compliance which can be higher than anticipated. These can reduce the effective size of your sample and your claims to statistical significance. So too can attrition, which occurs when parts of your sample are no longer available for follow-up, for example, because they’ve moved away.
- False premises. When using RCTs, we sometimes assume we know best. According to Jeffrey Hammer:
Now we assume we know how to be better at farming than the farmers do, that we can balance their multiple objectives in life (which we either don’t ask about or consider to be errors that are randomized away) better than they can. They raise too many cows (of course, we are also upset when they choose to sell those extra cows when the household head gets sick); they don’t use enough fertilizer (which we discover in other contexts might be due to the lack of insurance we didn’t know about); they don’t visit the doctor as much as they should (though they know exactly how useful those doctors are); they don’t buy insurance (when they’ve been burned by other such schemes in the past).
For useful examples developed by Better Evaluation, click on the following links:
Simple (one treatment, one control)
Where to next?
Click here to return to the top of the page and here to return to Step 2 (Data Collection)