The Randomista movement radically transformed how economic research is broached and public welfare policies are conceived through the incorporation of randomized control trials. The result has been a rejuvenation of disciplines such as development economics and the implementation of myriads of policies that have shown to benefit underprivileged populations, leading to RCTs being hailed as the “gold standard” of economic research. However, such a fulsome perception of RCTs is in itself deleterious, as it overlooks many of the internal and external validity concerns associated with them while simultaneously obstructing the process of building a pragmatic body of knowledge.
By Pranav Mittal
In 2004, two economists Michael Kremer and Edward Miguel published a randomized control trial-based research conducted in Kenya, demonstrating the relationship between deworming and school attendance rates. The study became one of the earliest scholarly efforts of what would eventually become the ‘Randomista’ movement, which advocated for the adoption of RCTs for economic analysis, particularly within development economics. The movement has played a critical role in shaping contemporary economic research, as shown by its pioneering economists Abhijit Banerjee, Esther Duflo and Kremer himself, being recently lauded with the Sveriges Riksbank Prize in Economic Sciences in December 2019 for their work on reducing poverty using RCTs. To better understand the implications of the same, it is essential to first view the reasons that instigated the movement.
RCT is a research methodology that evaluates the effect of a given treatment by comparing the ‘treatment’ group with the ‘control’ group across randomized samples and observations. For the longest time, development economics was a stagnant subject area that remained hung on macroeconomic questions surrounding poverty, economic growth, productivity and other questions relevant to economic development. Research in the field was characterized by intervention proposals that often lacked corresponding empirical evidence regarding their effectiveness, resulting in the misallocation of millions of dollars to inefficient policies. By incorporating RCTs into the research methodology under development economics, the Randomistas completely restructured the field.
The addition of RCTs shifted the focus of the questions from solving abstractly large issues, such as hunger and poverty, to finding more concrete, small-scale interventions aiming to improve the welfare of poor populations. In other words, instead of macroeconomically solving, say, poverty, RCTs nudged economists to devise interventions to improve the microeconomic functioning of poor populations. In many cases, the adoption of such a methodology disproved the efficacy of conventional wisdom. For instance, in a 2009 research based in Kenya, Kremer highlighted how the provision of textbooks, contrary to preexisting beliefs, did not increase average test scores, since they were written in English and had a predilected focus on high-scoring students. Similarly, in a 2015 study analyzing the ‘miracle’ of microfinance institutions in India, Banerjee and Duflo exhibited how contrary to conventional discourse, microfinance didn’t contribute to women’s empowerment or particularly help fuel people’s escape from poverty, since the businesses in which the money was invested in had low profitability.
Beyond rebutting prevailing theory, the use of RCTs in itself has engendered a plethora of research offering unique policy insights for improving public welfare. For instance, Banerjee and Duflo published research in 2005 based in Mumbai and Vadodara, India, comparing the efficacy of two educational programs: remedial classes and computer-assisted learning programs. Instead of merely theorizing which program would be more effective, they were able to quantify their findings and demonstrate how the remedial classes were 7.5 times as cost-effective as the CAL program, allowing policymakers to make more empirically grounded decisions. Similarly, for the longest time, the treatment of malaria using insecticide-treated bed nets was subject to a contentious debate in Economics. Many economists believed that such a policy measure would be inefficient and lead to a waste of resources by inducing many people, who have no use for bed nets, to take them. However, in a 2008 Kenya-based study conducted by Pascaline Dupas and Jessica Cohen, the authors proved otherwise by empirically exhibiting how there was no difference in the usage of free and cost-shared bed nets, and how the latter, in fact, actualized much narrower coverage by dampening demand because of its cost.
Owing to such tangible, evidence-based solutions, RCTs have resulted in a paradigm shift surrounding how public welfare policies and research are conceived. In the past two decades, there has been a significant rise in institutions funding RCT-based research to help formulate evidence-informed policies. For instance, Banerjee and Duflo founded the Abdul Latif Jameel Poverty Action Program, or J-PAL, in 2003, which, has conducted over 1000 randomized trials with its 294 affiliated professors. Having recently received a joint $6 million grant from Jameel Community and Co-Impact, the organization aims to further expand its evidence-to-policy government partnerships, which is expected to improve the lives of up to 6 million people. Innovation for Poverty Action, another policy non-profit that conducts RCTs, known as IPA for short, saw an increase in revenue from $250,000 in 2003, when it was founded, to $47.8 million in 2021, and currently works in collaboration with over 22 countries. Further, in the non-profit sector, Randomista advocate and Gates Foundation official Rajiv Shah created a fund called the Development Innovation Ventures to “move development into a new realm” by funding RCT research. From governments to private ventures, there has been a consistent rise in support for randomized experiments, as the reputation of RCTs rose to become a “gold standard” for economic research due to their theoretical capability to eliminate all forms of bias.
However, the perception of randomized trials as the “gold standard” is an exaggeration, as it belies many challenges with respect to their external and internal validity. If we examine external validity first, a research design is said to be externally valid when its results can be generalized to settings with different characteristics. Theoretically, RCTs are able to isolate the unbiased effect of a treatment on the sample independent of the influence of covariates, i.e, other independent factors that might affect the outcome of interest. However, in no manner does their research design warrant the unbiasedness of these estimates in an environment with different characteristics, which causes their results to be highly localized to the studied sample. This makes it difficult to scale up RCT-based interventions, as well as map them from one region to another. The problem can be highlighted by a 2015 study conducted by Banerjee and Duflo involving 11,000 households across Ethiopia, Ghana, Honduras, India, Pakistan and Peru, which analyzed the effect of the provision of assets such as cash, food, livestock and healthcare on the welfare of poor populations. Though the research was able to find a positive relationship between the variables, it also observed variable rates of return in different countries, from 133% in Ghana to 433% in India, and actually ended up failing in Honduras since the livestock wasn’t compatible with its climate.
Even in instances when RCT-based research is externally valid, several characteristics of their research design diminish their authority and efficacy. Firstly, the high up-front cost of RCTs, which on average is $500,000, stymies the pace of progress by limiting the policy proposals that can be tested. Many scholars have also criticized the narrow spatial scope of randomized experiments, compatible with evaluating mainly small-scale interventions, for leaving macroeconomic programs outside scrutiny. Moreover, as posited by Nobel Laureate Angus Deaton, “Development is ultimately about politics,” which is often forgotten in the decontextualized framework of RCT-based research. Owing to their inherent proclivity towards microeconomic analysis, RCTs exclude the role of sociopolitical institutions governing the issue at hand. This ultimately makes the identification strategy and intervention, not the importance and relevance of the policy question, the basis for guiding development policies.
Most importantly, RCTs present significant internal validity concerns — the research design being consistent enough to provide logically robust outcomes — which substantially undermines the unbiasedness of their estimates in practice. As explicated by Deaton in his paper “Understanding and misunderstanding randomized control trials,” unbiasedness in RCTs does not warrant the result to be true; the conflation of the relationship between the two would lead to detrimental consequences. Theoretically, a randomized experiment model removes the need for accounting covariate factors by indefinitely randomizing the treatment across the sample space. However, in practice, randomized experiments randomize the treatment only once, making their conclusions highly susceptible to manipulation by other covariates that were initially unaccounted for. Deaton further argues that causality given to the treatment “might, in fact, be coming from an imbalance in some other cause in our particular trial; limiting this requires serious thought about possible covariates.” Thus, while a randomized experiment might provide an unbiased estimate of the treatment’s effect, it doesn’t imply that it’s necessarily true. And noting how randomized trials have significantly small samples in comparison to other methodologies such as observational studies, the problem is only exacerbated as their results are more amenable to be influenced by outliers and covariates.
Yet despite these internal validity concerns, results from randomized experiments are often regarded as objective truths, creating an environment where many Randomista economists have gone as far as to reject research not using RCTs. For instance, in one randomized control research based in India, Duflo showed that the dissemination of job advertisements contributed to an increase in women’s employment to support her conjecture that female employment had been increasing in India. However, as London School of Economics economist Naila Kabeer later exhibited, the claim was unfounded since women’s labor force participation in India had been steadily declining.
This ultimately raises the question: is the use of RCTs for economic analysis unjustified? Not at all. RCTs have been shown to provide interventions along margins where traditional theory has little guidance. For instance, a research by Laura Derken at the University of Toronto utilized randomized trials to rationalize why people in areas with high HIV rates might not take antiretroviral medication. The study finds that the fear of being seen as HIV positive is the reason people abstain from taking the medication, instead of concerns regarding their cost or efficacy, which was what the predominant traditional theory previously assumed. Furthermore, although not all, many of the small-scale interventions proposed by Banerjee, Duflo and Kremer have been successfully scaled up. Pratham, a non-profit organization, successfully scaled the aforementioned Banerjee and Duflo’s 2005 study up into what is now the Read India program, successfully providing remedial classes to 33 million students in collaboration with governments in several Indian states.
All of the above evidence corroborates the huge usage potential of RCTs. What the criticisms imply, however, is a need to change the perception of RCTs as the sole indicator of truth to instead one of the many research methodologies, whose use should be subject to the policy question and prior knowledge at hand on the issue being examined. As summarized by Deaton, “Randomized controlled trials… do not occupy any special place in some hierarchy of evidence, nor does it make sense to refer to them as ‘hard’ while other methods are ‘soft.’” Like any other research design, randomized trials offer their own benefits and drawbacks, and thus their use, similar to any other mode of analysis, should be determined by how these characteristics interact with the question at hand. To truly formulate policies informed by rational analysis, it is vital we contextualize randomized experiments and their information within pre-existing knowledge structures, understanding it as a tool for helping the advance of scientific knowledge and not dictating it. □
The Economics Review 
