Kukka Ilmanen
The correct way to educate has always been a point of debate and people’s opinion are mainly influenced by their own stories and experiences. Can we scientifically decide which one is the best? One lauded answer has been Randomized Controlled Trials (RCTs), which allow a scientific and mostly unbiased way of measuring and comparing the effectiveness of different education policy measures. How does it work, where is it the answer and where are its limitations?
Randomized Controlled Trials were originally developed in the field of medicine to test whether a promising medicine was successful and to which degree. The basic principle is to assign people into randomized test groups, where some receive the considered treatment and the others receive a placebo and the difference in their reaction is the measured outcome. Since then RCTs have been used to evaluate policies and since the 1990s interventions in development economics.
Small Measures Can Achieve Large Effects
They have found large success, as they can scientifically determine how effectively an implemented measure did improve conditions and find causalities affecting education and learning. Their main benefit is that they give credibility to theories how to improve the education system. For example, a study in the rural area of Udaipur tried to prove that one of the main causes for the problems in learning wasn’t the attendance of pupils, but the attendance of teachers, which was at that time about 56% of their supposed working hours. According to the World Absenteeism Survey in 2002 and 2003 in the whole of India teachers missed on average one out of five working days. (Duflo, 2011) An NGO, Seva Mandir, conducted an RCT in the region and gave half of the teachers a camera to monitor their attendance by taking a time-stamped picture with their class at the beginning and at the end of the day. The teachers were then financially incentivized to come to school by making their pay partly dependent on the number of days, on which they could prove their attendance by these pictures. In consequence, attendance increased to 79% in the camera group compared to the 58% attendance rate of the teachers in the control group. Even more importantly the children in the camera classes increased their test scores by 0.17 standard deviations as well as being 62% more likely to be admitted to regular government schools in comparison to the control group. The study wasn’t perfectly cost-effective, since the camera equipment needed to be paid additionally to the normal teacher’s salary and amounted to 40% of the latter. However, it did show how absence of teachers was hindering effective learning and that better monitoring and financial incentives were an option to combat this causality. (PovertyActionLab, 2006)
Image: Teachers proving their class attendance by camera (PovertyActionLab, 2006)
Where are RCTs successful?
Often RCTs show that a specific intervention caused negligible effects. That can obviously be helpful in deciding which education policy measures not to take but that does not really answer the main question on how to provide successful education?
However, there have been cases where results have been positive and lead to nationwide and even international scaling up of the tested policies: One of the earliest RCTs in education and one of the most successful interventions was the program PROGRESA introduced in 1994 in Mexico. It introduced conditional cash transfers (CCTs), which gave poor families money under the condition that their children were regularly attending school. It was assumed that parents lacked the willingness to push their children’s education, as this caused further costs while the benefit of school for the children wasn’t always a given. By incentivizing school visits by monetary means the parents were supposed to have a further reason to push for the school attendance of all their children including the girls. The program increased secondary school enrollment by girls by 8% to 75% and by boys by 4% upto 77% enrollment. Furthermore, the same effects were not observed in similar villages, which didn’t receive the CCT. In consequence, CCTs became a popular intervention in Latin America and led to a rise in RCTs for further measures in education. (Duflo, 2011)
RCTs haven’t been limited to education in developing countries but have also found their way into American and British education. As an example, the British charity Education Endowment Foundation, who researches and helps to implement new methods to improve teaching and schooling in the UK, has conducted 150 RCTs in the last year that involved more than half the schools of England. Their current estimation is that RCTs can be the proclaimed “gold standard” for research, if they are implemented well. (EEF, 2019)
RCTs are the “Gold Standard” of Research?
RCTs have been judged to meet the highest standard of research, as they reduce many biases normally present in studies by assigning people into randomized groups. Furthermore, the control groups allow an estimation of how far an observed effect was due to the intervention and how far it was caused by other factors. However, that doesn’t mean that they are without problems or applicable in any situation:
As a first important point, these trials aren’t cheap. The EEF spends £ 500’000 per trial to ensure their scientific validity. This leads to the second point that though these trials are costly, there is no guarantee that they lead to valuable information. For example, a study on the results of RCTs showed that 40% of the large scale RCTS concerning education in the US and UK didn’t produce any useful information on the effectiveness of a policy measure (Lortie-Forgue et al., 2019).
Even a so-called successful intervention doesn’t conclusively mean that the true causes of its effectiveness are understood. One stark example was the aforementioned provision of conditional cash transfers in Mexico. A repetition of the intervention in Malawi found that unconditional cash transfers were as successful as conditional ones (Duflo, 2011). The second study proved that the assumption, families needed to receive incentives to send their children to school, was probably wrong and the causes might maybe be more related to extreme poverty changing the priorities of a family. If that is the right reasoning is open to discussion and exactly the problem with RCTs. They can answer if one small intervention was effective under certain circumstances, but there is no guarantee that the questions asked, will give us the right answers we need to improve our overall education system. And even if a measure proves effective, the exact cause why it is effective may not be understood properly and therefore prove difficult to transfer to other schools and countries.
RCTs are useful in discovering in an unbiased manner whether a measure will actually show any impact and therefore avoid large-scale unsuccessful education measures. This is a definite improvement on the previous attempts to plan education according to untested theories and by simple trial and error. However, in the end randomized controlled trials will still lead to an education system, which has to be tested step by step via trials and there always remains the possibility for errors.
References:
Duflo, Esther. Poor Economics: A Radical Rethinking Of The Way To Fight Global Poverty. New York : PublicAffairs, 2011. Print.
Lortie-Forgues H., Inglis M. (2019): Rigorous Large-Scale Educational RCTs are Often Uninformative : Should We Be
Concerned? Educational Researcher. ISSN 0013-189X (In Press)
Media Attributions
- Udaipur Example