This blog is written by Cibel Mascarenhas, Antarang Foundation

Background

Antarang Foundation is a non-profit organisation that works for building employability in youth aged 14 and up. We conduct career guidance sessions in government and low-income private schools for Class 9-12. Antarang also has a career chatbot that provides career resources to all youth so that they are continually engaged in their careers and have access to career opportunities.

Need for A/B Testing

A user’s journey on Antarang’s Career Chatbot begins with a simple registration process, asking for basic demographic information. The purpose of this is to send context-specific information to the user (such as send x message to all users who are in Mumbai/ y message to all working professionals), as well as for internal and external reporting. While getting more information is useful to us to enhance the user experience, it was also speculated that the number of questions in the registration process leads to users not completing it and therefore not visiting the chatbot again. Therefore, we decided to pilot the A/B Testing feature developed by Glific to see whether users prefer a 3-step or 5-step registration process. 

Hypothesis & Success Indicators: Higher Completion Rate

Each flow had labels for both incoming and outgoing messages for each question and its response.

Completion Rate = No. of responses for last question / No. of responses for the first question * 100

The hypothesis was that Flow A (3-step registration) would have a higher completion rate than Flow B (5-step registration).

Testing Process

The existing registration flow (Flow A) has 3 questions, and a similar flow was made with 5 questions (Flow B) instead for testing purposes. 

Questions in Flow A:

1. Select a language
2. Please enter your name
3. Please select a user type (Student/teacher/etc)

Questions in Flow B:

1. Select a language
2. Please enter your name
3. Which city do you live in?
4. Please select a user type
5. Please select your class*

*(IF “Student” is selected as the user type, which was true for about 88% of users that were part of this test. For other user types the fourth question was the last.)

Getting users to randomly enter either one of the flows was as simple as adding a Split By Chance node to our Registration Flow. This was kept active for 10 days. When active, each new user randomly entered either one of the flows. So far, we got a total of 586 responses – 317 in Flow A and 269 in Flow B. 

While the numbers of how many users access which flow/node is available on Glific, we relied on the label data generated on the backend for analysis, as recommended by the Glific team. Each of the two flows was labelled separately: for example, the label reg_type was added to the message asking for user type in Flow A. The corresponding message in Flow B was labelled reg_type_b. Similar nomenclature was used for all of the incoming and outgoing messages.

Findings

As hypothesised, the completion rate for Flow A was higher, at 86.12% as opposed to 75.46% for Flow B.

Therefore, we decided to keep the registration process limited to 3 questions and collect other demographic information in other flows.

Additional Learning

There was a drop of about 21 students at the second question itself (8% of the total number of students that received the question) for both flows, i.e., that is when the users are asked to enter a name. Evidently the drop is not due to the length of the registration process, and hence we are exploring this further to increase registration completion and hence access to the bot content.

Using the Split By Chance node has helped us smoothly gather data to test our hypothesis by allowing both groups to access the A and B flows during the same time period. Previously, such experiments could not have a very reliable control group to compare results with as having randomly split groups has made it easier than observing two different groups during different time periods.