The data from the pilot and the overall learnings is made available thanks to the contribution of Snehal Kulkarni and Clipsy Banji of SNEHA team. 

The Pilot

Following was the premise of the pilot

1. Two target groups of mothers were identified (~300)
2. Weekly two messages with curated knowledge were sent separately for these groups for period of 5 weeks

3. There were questions posed to determine engagement and absorption of the knowledge content being shared

4. Answer a yes/no question then prompted the mothers to ask open ended questions could be asked by the mothers which would be generated by the AI bot (insights on the LLM interaction here)

Story of pilot in numbers

1. ~300 users reached out, two times per week

2. Overall trend of engagement rates to the messages with 32% as the overall average engagement rate

3. Messaging cost incurred to run the pilot ~50$

4. Following the feedback taken from a random group on the expectation of the bot, reason for continuing engagement and overall satisfaction

Learnings from SNEHA team

● Ensuring Content Relevance: The identification of pertinent topics and information, such as prenatal and postnatal care, nutrition, exclusive breastfeeding techniques, and immunization, remains a priority.  During both home visits and group meetings, pregnant women and mothers of young infants frequently inquire about antenatal and postnatal care. Misconceptions surrounding immunization and breastfeeding persist, compounded by home deliveries in Bhiwandi. In an initial pilot around October 2023, SNEHA introduced a WhatsApp-based chatbot to 400 mothers of children aged 0-2 years, receiving a positive response. Consequently, SNEHA expanded its scope by incorporating additional topics such as institutional delivery and exclusive breastfeeding for two distinct target groups: pregnant women and mothers of infants aged 0 – 6 months. Given the low literacy levels among the target audience and insights gleaned from the pilot, emphasis was placed on audio and video messages over text for enhanced user understanding and accessibility.  

● Enhancing User Engagement and Information Delivery: To minimize response delays to user queries, SNEHA integrated AI/ML into the WhatsApp-based chatbot. This integration facilitated immediate responses to user inquiries, thereby boosting user engagement. Prior to introducing AI/ML in the second pilot, the SNEHA team collaborated with Glific to develop 2-3 prototypes. A substantial amount of content on relevant topics was fed into the AI/ML as a knowledge base, enabling tailored responses to queries. AI-generated responses were continually assessed and refined until deemed satisfactory by the team. Leveraging their familiarity with the target audience’s language, preferred timings, and common queries, SNEHA strategically timed message deliveries to coincide with peak user activity. Additionally, male participation was encouraged by involving husbands, who handled the phone during the day, were requested to relay the messages to their spouses when they reached home and thus were involved in the learning process. 

● Updating Knowledge Base for Comprehensive Responses: Recognizing the interconnectedness of issues faced by pregnant women and lactating mothers, efforts were made to update the knowledge base to address overlapping queries. For instance, addressing inquiries related to modern contraceptive methods alongside pregnancy-related concerns. Failure to address such overlaps could lead to misinformation, as evidenced by instances where contraceptive methods were mistaken for child medication.

● Ground Mapping: Assessing the readiness and willingness of beneficiaries to engage with the AI-enabled chatbot involves providing adequate training and support to users.  Scaling efforts may involve training ASHAs, aanganwadi sevikas, and health volunteers to facilitate user training, ensuring effective chatbot utilization.

● Incremental Approach and Tailored User Engagement: The strategy of starting small allows for focused attention on a select group of users, ensuring their specific needs are addressed effectively. Building on the success of initial pilots, subsequent phases aim to expand reach while tailoring interactions to individual user levels of understanding and engagement. Efforts are made to simplify communication by utilizing local language and minimizing data consumption through small-sized audio and video content. 

● Soliciting User Feedback and Satisfaction: Collecting feedback from participants regarding their experience with the chatbot, including ease of use, satisfaction levels and suggestions for improvement in user friendliness. We have incorporated feedback questions in the weekly messages, where after every push message and AI response we ask the user for their feedback and level of satisfaction. This feedback will help us fine tune our product even more and launch a better version next time.

The Approach 

Which made this fast paced delivery of innovation and collaboration possible between SNEHA and Glific team

• Prototype and try
• Diligent review
• Reflect and repeat 

The first prototype of large language model (LLM) responding to user questions was put together in less than 15 mins, with only a fraction of the knowledge base. The SNEHA team had something very tangible to play with, without spending any time on tech. They dedicated themselves to asking questions, understanding how the bot responded and comparing it with the idea they had, and what they knew was bound to fail with their beneficiaries, 

Within next week, they could communicate what would not work, and what are the things that ought to be there in the bot. The second iteration of the prototype took 1.5 hours to put together. With a wider knowledge base, and way more refined prompt with a few ideal responses for the LLM to mimic. 

The testing for the second iteration went on for 10 days or so, with a larger number of team members asking questions, and SNEHA team as rating the responses generated, and recording the feedback from the observations where things were going off. This helped to drastically understand the behavior of LLM and make tweaks in the prompt to achieve the desired output.

Read more on the insights from the LLM part of the interactions here.