Does intervention have to cost a lot?

Another general misconception when it comes to RCT design, it is often believed that RCTs are only suitable for expensive interventions. It is mainly because RCTs often require extensive resources and long time intervals to observe the outcomes. Thus, spending resources to investigate cheap interventions can be perceived as a waste of resources. However, depending on the causal question, there might be low-cost interventions that can be tested and implemented. A common example of cheap interventions is encouragement designs which are used to test the best way to encourage citizens to the take-up of an entitlement program by the government or another party.

In the rest of this Medium article, we will first discuss what kind of questions can be answered with cheaper interventions and how we can identify the right cheap intervention. Secondly, we will briefly mention how the intermediate variables can be used to estimate the effect of the intervention at a smaller time frame. Lastly, we will go through two examples; text message reminders for savings in Peru and increasing volunteer retention in West Java via WhatsApp messages.

There are several reasons why investigating cheaper interventions can be beneficial for policymaking. Firstly, if the interventions are cheaper, it will help the implementers to save up on the budget which can be repurposed to iterate on the intervention design, retest, and scale up based on the collected evidence. In other words, using cheaper interventions enables the government or the NGOs to test cheap first and then scale later. In addition, having faster and cheaper interventions can be useful in developing countries where there is more political instability and oftentimes the fund for different types of programs depends on the elected government. Therefore, having a cost-effective mechanism to create supporting policy evidence to continue or stop an intervention at a faster rate is valuable to work in politically or funding-wise unstable environments.

At the end of the day, some government interventions aim to create a behavioral change to enhance the citizens’ lifestyles. Oftentimes, cheaper interventions are overlooked because implementers often think that small interventions will not create a big behavioral shift. Even though it is true that an SMS message has a smaller impact compared to building schools in closer proximity in a child’s education, they can still create a change in parents’ behavior which can affect the education of the child. In addition, the societal value of small behavioral changes increases in multitude when we start to think of the effects of the outcome on the society level rather than individual level.

Unfortunately, cheap interventions are not always suitable to answer any kind of causal question. Oftentimes, cheaper interventions are used to nudge the citizens to perform the right behavior that will increase the life quality of themselves and the overall public. According to 42ideas, there are three main criteria to identify if a problem is suitable for a behavioral-based solution. Firstly if the problem under examination is repeated frequently or performed by a big number of people, a behavioral-based solution can create a significant impact over time or on the societal scale. Secondly, if the problem that we want to address is naturally against the best interest of the individual, the individual can be convinced to act in their best interest through nudging. Lastly, if the problem is not directly or indirectly caused by politics or a third party, changing the individual behavior can resolve the problem as a whole or partially.

After identifying if the problem can be resolved through a behavioral-based approach, the researchers should identify the action or the information that is lacking for the individual to reach the desired outcome. At this step, the researcher should complete a literature review on the given issue and behavioral sciences to be able to create the blueprint for the causal mechanism between the context and the observed behavior. Having a general understanding of the possible causal mechanisms will help the researcher to ask the right questions which can answer through cheap behavioral-based intervention.

Once the researcher has a good understanding of the targeted causal question and the intervention, they should start to think of possible intermediate outcome variables that can be used to assess the impact of the program in a shorter time frame. Intermediate variables or sometimes referred to as surrogate variables are the variables that are used to estimate the intermediate effect. Often, the researcher aims to evaluate the long-term effect of an intervention. However, to be able to make policy decisions within a given time frame, intermediate outcomes can be utilized to assess the performance of the program. For example, rather than evaluating the estimated effect of SMS encouragements for schooling on university attendance, the researchers can estimate the effect of the intervention on the average yearly attendance. Even though the effectiveness of intermediate variables for program evaluation is unsure, it is still a valuable alternative to be considered if there is a need for rapid results.

If the intervention is a type of knowledge exchange, according to Doran 1981 there are five general categories of intermediate behavior, which are new knowledge, enhanced skill, improved consensus, enhanced connectivity, and lastly new and improved actions. These categories can be examined in detail alongside examples on the website of South-South Experience.

Lastly, after testing and analyzing the first batch of the results, the intervention can be iterated and tested again to see the possible optimization to intervention design. Perfecting the intervention and having a better understanding of the needs of the target population is necessary before the program can be scaled up to the other population that is experiencing a similar behavioral problem.

Case studies

In this section, we will go through two case studies that used cheap interventions to nudge people into a more desired behavior.

The Impact of Text Message Reminders on Savings in Peru

This impact evaluation was part of a bigger evaluation which was conducted in Ghana, Peru, the Philippines, and the Dominican Republic and aimed to understand the causal relationship between the simple reminders on savings. A group of IPA-affiliated researchers collaborated with Federación Peruana de Cajas Municipales de Ahorro y Crédito (FEPCMAC) which is the regulatory system of the credit unions in Peru to affect or behaviorally target text messages on the saving behavior of the individuals. Everyone amongst 30,276 bank clients with a mobile banking account randomly received one of the five variants of the crafted text message. Each variant is defined as:

• “Simple reminders: Received a straightforward reminder to save, such as “Don’t forget to save.”
• “Reminders of potential savings gains: Received reminders about how much savings could accumulate over time.”
• “Deposit encouragements: Received reminders to make a deposit at an agency or ATM.”
• “Planned expense reminders: Received reminders to save for specific purposes, like rent.”
• “Reminders about unplanned expenses: Received reminders to save for an unforeseen setback or emergency.”

Additionally, the researchers also randomized the time of the day, weekday, frequency, duration, and personalization of the text messages to evaluate the effect of these variables on the outcome. They used increased account activity as an intermediate variable to estimate the effect of text messages on customer behavior. Initial analysis showed that people who received a text message started using their account more frequently but the researcher did not find any significant effect of the variant of the text message or any of the other variables on the customer behavior. In addition, they did not find any association between increased account activity and saving behavior.

Increasing volunteer retention in West Java

In this impact evaluation, the Behavioural Insights Team and the West Java Province Environmental Agency (Dinas Lingkungan Hidup, or DLH) collaborated to assess the ‘Eco Village’ volunteering program by DLH. The main policy objective was to keep the existing volunteers motivated to participate in different programmings by Eco Village. As WhatsApp is one of the main ways of communication in Indonesia, the researchers took advantage of it to reach out to current volunteers at the minimal cost of a smartphone and a wifi connection. The participants were assigned randomly to 2 different treatment groups. Over the course of three months, the first group received weekly behavioral insights messages which target personal challenges, eco-village identity, and planning support. The second group received monthly messages which talked about the environmental challenges. The two intermediate outcome variables were volunteer retention and the intensity of volunteer engagement. As a result of the study, they did not find any significant differences between the outcome variables of the two groups.

Lastly, it is a pitfall that this randomized experiment does not have a control group and the research does not explain why they excluded the control group in their report. However, this does not disqualify their study completely as their finding still gives insight on the outcome differences between the two different treatment groups.

Resources

Cojocaru, L, et al. “Official Action: A Roadmap for Using Behavioral Science in Public Administration Reform.” 42ideas, Dec. 2020, www.ideas42.org/wp-content/uploads/2020/12/I42-1112_DesigningBetterGovernance_paper.pdf

Doran, G. T. (1981). There’s a S.M.A.R.T. way to write management’s goals and objectives. Management Review, Volume 70, Issue 11(AMA FORUM), pp. 35–36.

Karlan, D, et al. “The Impact of Text Message Reminders on Savings in Peru.” Innovations for Poverty Action, 6 May 2020, www.poverty-action.org/study/impact-text-message-reminders-savings-peru.

Gandy, K, et al. The Behavioral Insight Team, 2019, Increasing Volunteer Retention in West Java, www.bi.team/wp-content/uploads/2019/11/Project-Report-Volunteer-Retention-English-221019.pdf.