About the project

Arogya World is running their Diabetes awareness program. It consists of a set of 58 messages about the various aspects of Diabetes, and 32 questions that checks the behaviours of the end-users. The messages will be sent over a period of 6 months. It is to be run with a cohort of 1600 end users wherein 800 users(control group) will receive only the 58 messages in a fixed order while the other 800 users(experiment group) will receive questions and messages in an order decided by an AI engine. This program will run on WhatsApp and it will test how nudges can impact behaviours and help people in the prevention of Diabetes.

The Arogya World team has designed the program with messages, questions, times when the messages should be sent, process of onboarding the users, managing the conversations during the program, analysing responses and program outcomes. A team from The University of Oregon is working on the AI Engine which will read the responses of the users and plan the next message and question to be sent to the end users every week. The Glific team is building the support for scheduling messages both for the control group and the dynamic messages for the experiment group. As program partners, Glific is also helping with any WhatsApp template related approvals, reports to see how many people are reading/not reading the messages, planning a pilot and troubleshooting any challenges along the way. Finally, Head Held High team is implementing the program on ground with Arogya. 

Program planning

After putting a high level structure for the program such as deciding to use WhatsApp for dissemination and Glific as partners, planning static and AI driven message streaming, and creating a process for onboarding users, we moved into action. Different team were managing different pieces:

1. The frontline workers were being trained how they will get the users on the chatbot. Over an in-person meeting they would ask the users to save the phone number and send a message to start the conversation and opt-in the process. This is one part of the onboarding process where Arogya collects further consent and more bio details. But for this article, I will stick to the chatbot details.
2. The AI team has set up a process to receive user responses from Glific and based on that generate the next set of messages and questions. We performed a few trials with sample documents to bring clarity for everyone on the format of exchanges between Glific and AI engine.
3. The Glific team had set up an integration between Google sheet and Glific to directly pull messages from the sheet and send it to the end users. We also started applying for HSM templates to get ready for the pilot and to be prepared for any cycles of rejection and resubmission.
4. The Arogya team prepared all the messages in different languages – supported languages are English, Hindi and Kannada. They coordinated the details and flow of the program between the different teams.

The pilot

This is a very crucial part of the WhatsApp chatbot rollout according to us. Especially if an NGO hasn’t used WhatsApp or similar tools before. And we especially wanted to test out the AI engine and how it synced up with Glific since it was new to us. There were many moving parts hence the pilot was critical. We wanted the pilot to be a real reflection of the program, but due to time constraints we scaled the pilot down to 2 weeks from 24 weeks. We scaled it such that 1hour=1day and 1day=1week. All the team members who were a part of the project became the pilot participants. So Arogya, University of Oregon and Head Held High teams were part of the AI experiment group of users and the Glific team formed the control group.

For the pilot, all the participants were asked to opt-in to the chatbot in the same way as an end user would. After this the messages started being pushed to them.

Some of the cases we had to prepare and plan ahead were (and these are common to most chatbot program):

• What if the end users don’t respond to the questions in time. Should there be nudges or not. If yes, how they should be timed.
• What if some end users want to talk in between the flows. In case they send a genuine message or question how will the team handle that.
• What if they try to change their language somewhere in between the program.
• What if they respond to a two weeks old question at present.
• How to know whether the program is doing well/failing. Are all the messages being delivered. If yes, are they being read. If read, are they responded to. Each stage needs to be planned accordingly.
• How the relay between the AI engine and Glific happens. How the teams will coordinate during the program.

All these questions will have different answers as per the program. For example, in this case, we’re not building nudges and notifications. Instead, leveraging Frontline workers to follow up or remind the end-users.

How Glific is being used

Glific records the users consent to be a part of the program. All the conversations and users’ responses are being captured. The chatbot collects users’ preference of language and the time they want to receive the messages. The messages are then scheduled as per their preference. Messages have also been planned ahead which will automatically be triggered over the course of the program. Reports will be generated that help Arogya team to measure and analyse how the program is running. Glific also interacts with external systems like the AI engine in this case, to enable sophisticated program designs.

Challenges

1. One of the biggest challenges in the entire WhatsApp chatbot ecosystem is to get Facebook Approval. To get the documentation right, and according to Facebook. It is a closed system where it’s not always easy to get things done on our timelines and hence it is suggested to plan early – even 1 to 2 months in advance. Check the link above for the careful attention to documentation required by FB.
2. Along with Facebook verification, having a phone number early on is crucial, since it may take some time to get access to it. 
3. Plan for conversations, languages, message translations and the storyboarding for program implementation.
4. Make room for Pilots before launch. Create timelines that accommodate testing, pilot and trials – not just with internal members but also actual users.