What Can Development Economists Learn From Business?

What can development economists learn from business?

During the interviews that I conducted with the experts in the field, another theme that has been brought up several times by different individuals was the lack of project management skills in the research field (McManus, personal communication, 2020; Holloway, personal communication, 2020). To address these organizational needs, I will transfer some of the best practices from business when it comes to organizational management.

There is a big compartmentalization between economics and business, even though they use similar methodologies to investigate, research, organize and improve. Therefore, in this chapter, I will focus on the best practices from business that can be transferred into RCT research in development economics. I acknowledge the fact that development economics requires more rigorous results as it affects the public directly. However, it doesn’t mean that there would not be tools that can be repurposed from business for development economics. In this chapter, I will investigate how the development economist can use time management and collaboration tools to increase efficiency. Lastly, I will touch upon how business approaches A/B testing and what economists can learn from it with a case study example from Georgia.

Time management tools

Experts' time is another major expense while conducting randomized control trials (McManus, personal communication, 2020; Gibreath, personal communication, 2021). Expert time can be defined as the time that it takes researchers to design, implement and conduct the analysis with collected data. In the majority of the cases, the experts who are conducting the analysis are coming from developed countries where wages are higher than the countries where the impact evaluation is taking place (McManus, personal communication, 2020). Thus the wages of the experts is one of the main expenses. Therefore, if we can organize the expert times more efficiently, we can minimize the cost that we spend on human resources.

Specifically, product management employs various techniques to organize their process. Even though checklists are one of the primal tools to keep track of the process, they can be useful if the steps are well organized and combined with online interactive tools. Kanban boards is an online project management tool that helps you to visualize the work process to show the program’s roadmap. Employment of such a tool can help with transparency while ensuring that everyone who is involved in the project has a clear understanding of the progress of the project.

In addition to using online project management tools, specific organizations or groups can come up with their own version of a checklist to make sure that there is a coherent process for tracking. Alternatively, they can utilize the checklist created by other organizations such as the Checklist for launching a randomized evaluation in the United States by JPAL.

A/B testing

A/B testing is a simple RCT that is generally conducted in an online environment to assess how consumer behavior changes with regards to changes on the website, advertisements, sales, etc. If A/B testing can be used to enhance the shopping experience of customers, some lessons that we can learn from A/B testing can be used by governments and NGOs to enhance the living standards of individuals. Since A/B testing is generally conducted through an online platform, it is still an area where there aren’t many rigorous examples. In this section, I aim to discuss the benefits of A/B testing in business, how some aspects of A/B testing can be transferred to development economics, and lastly, I will share some example cases which use A/B testing to evaluate human behavior to government announcements.

In business, A/B testing has been seen as an iterative process rather than looking for one definitive answer. With the increase of technology, people exchange information at a faster rate. Therefore, it is expected that human behavior towards certain phenomena would change at a faster rate compared to the past. This rapidly changing environment brings the question of how logical it is to rely on the findings from a couple of decades ago. “Just remember to keep testing regularly, since the effectiveness of anything can change over time” (Patel, 2020 ). It is common advice given for A/B testing for e-commerce which can be applicable to development economics to a certain extent. An encouragement design that worked a couple of years ago might not be as effective due to various reasons. Therefore seeing evidence-based policymaking as an iterative process rather than looking for one definitive answer can enhance the effectiveness of some of the interventions. As more governments and NGOs start interacting with citizens through online platforms, it will become easier to conduct iterative testing via these online platforms whether social media or individual websites. For example, Facebook shows a reminder on top of the user feed to encourage them to complete the census in the user’s area. These reminder links to the government websites with specific information on how to take the census. Facebook filters for the eligible individuals by using location and age data that they already have (Facebook). This example confirms that web-based interactions with the citizens can be utilized to collect data and use this infrastructure to repeatedly test various encouragement designs in the future.

Oftentimes Randomized Control Trials are harder to replicate and iterate due to design challenges. A/B testing can be a design alternative that allows for faster and cheaper adjustment while choosing similar sample frames. Encouragement design is one of the cheaper types of RCTs as the cost of interventions tends to be significantly lower than other forms of RCTs. Social media is a unique tool to directly engage with the general public. There are four main advantages of using social media as part of a nudge design. It enables you to reach out to bigger audiences compared to a face-to-face survey, firstly because a significant percentage of the population is social media users. Currently, Facebook has 2.38 billion users worldwide. Secondly, A/B testing programs that are integrated into the social media platforms often help you to control some of the demographic characteristics. This gives a unique opportunity for stratified RCT design. Also, it helps you to mitigate the selection bias which can be introduced due to users' self-selecting to participate in social media. Thirdly, even though it is not free, it tends to have lower costs compared to sending out actually printed letters or flyers. Lastly, data collection is done through the platform thus it is not associated with an additional cost. This is highly important given that face-to-face surveys require the use of enumerators. Using enumerators is likely to be costly and time-consuming because you will need to train and give compensations to the enumerators.

Case study

Understanding how A/B tests can be used is gaining more importance as most of the population is engaging with social media accounts where governments share information or reminders. For example, during the COVID-19 pandemic, the NHS has been utilizing social media to share information.

A/B testing decreases the church attendance during Easter 2019 due to COVID in Georgia

The main aim of this study is to have a rapid response on how many people are intended to attend the church ceremony during Easter in Georgia and if they can be nudged not to attend by being exposed to an announcement from a famous scientific figure (CRRC Georgia, & Turmanidze, 2020). Even though this study does not have statistically significant results, its study design can be used as an inspiration on how social media platforms like Facebook can be utilized for randomized experiments. Since the whole cost of the experiment was around 500$, it makes it highly cost-effective compared to other types of encouragement design (Gibreath, personal communication, 2021). In addition, the whole experiment took 3 days to run thus in emergency situations such as COVID-19 where short-term outcomes are important, these types of study designs can be utilized to understand human behavior.

Limitation of A/B testing for development economics

Facebook A/B split test functionality helps the researchers to test what strategy is the most suitable to attract the attention of the targeted audience. However, there has been criticism on its limited dependent variables. Facebook offers a relatively big set of demographics and behavioral variables to choose for the target audience. However, the researchers can only manipulate the content of the treatment which can make the platform useless for some questions (Orazi, 2020).

Even though A/B testing can be seen as a cost-effective version of field experiments, we should also consider the external validity of the results. A/B testing can estimate the causal relationship based on the observed outcomes of the subpopulation, thus the results of our analysis would be internally valid.

A pushback that we need to acknowledge is that since there hasn’t been much A/B testing for policymaking, it is hard to estimate if this is a cost-effective methodology. Since users of social media accounts are bombarded with advertisements constantly, there is a higher chance that they are overwhelmed by it and may ignore the government announcements that are through social media ads (VWO, 2021).

Conclusion

All in all, A/B testing has its limitations and perks. We are not proposing that A/B testing should and could replace any type of encouragement design-based RCT. However, A/B testing is a powerful complementary tool that can be used to evaluate online engagements with civilians. In the following sections, we will share some resources that can be used if you want to implement A/B testing for research purposes.

Resources

CRRC Georgia, & Turmanidze, K. (2020, October 06). Analysis: Study suggests large numbers in Georgia to celebrate Easter in church. Retrieved March 21, 2021, from https://oc-media.org/features/analysis-study-suggests-large-numbers-in-georgia-to-celebrate-easter-in-church/

Facebook. (n.d.). Why am I seeing information about the census on FACEBOOK?: Facebook help center. Retrieved March 21, 2021, from https://www.facebook.com/help/297467305157104/?qid=1424085111276533

Patel, N. (2020, January 24). A Beginner’s Guide To AB Testing: An Introduction. Retrieved November 12, 2020, from https://neilpatel.com/blog/ab-testing-introduction

VWO. (2021, February 27). What is a/b testing? A practical guide with examples. Retrieved March 21, 2021, from https://vwo.com/ab-testing/

Orazi, D. C., & Johnston, A. C. (2020). Running field experiments using Facebook split test. Journal of Business Research, 118, 189–198. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7331542/#s0040

Annotated bibliography

Orazi, D. C., & Johnston, A. C. (2020). Running field experiments using Facebook split test. Journal of Business Research, 118, 189–198. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7331542/#s0040

• This research paper aims to evaluate Facebook A/B split Test by comparing it to field experiments and Amazon Mechanical Turk. Section 4 explains the basics of setting up an A/B test on Facebook. In addition, in section 6, the paper explains the key terminology of the data that is collected by A/B testing and how they can be interpreted. It can be a useful source if you would like to set up a research experiment by using A/B testing on Facebook.