This is a consolidation of data, tech and programmatic learnings from a chatbot assisted community engagement pilot undertaken by SNEHA with 320 women in early motherhood from Bhiwandi district near Mumbai

Within 2 months of embracing the LLM (Large Language Model) driven project, SNEHA team has shown incredible aptitude and hands on effort to go in depth with testing and acute observation. This is a testament to the fact that one does not need to be a techie in order to drive an excellent tech enabled intervention. Now more than ever nonprofit leaders can bring tremendous change through diligent observation, curiosity, being rooted to the mission and the experience of the user of the technology. 

Context

Pilot plan involved sending bi-weekly broadcast messages to young mothers of two categories 

1. Mothers of 0-6 months old babies
2. Expectant mothers

The broadcasts followed the following format: informative message + voice note or video. After going through which, the mothers could ask open ended questions to the bot, so that they get relevant information and help they need. One of the main ideas was to ensure the mothers could use this as a voice bot. 

Features of the AI flow

1. Mothers could ask questions in Hindi, Hinglish (text) as well as Hindi, Hinglish (voice)
2. The response from the bot had to be in the same language as the question asked (text or voice)

Inputs provided for AI flow

1. The knowledge base covered the training materials available for training of ASHA workers (see here)
2. The prompt written looked something like this (pic below)
3. Model used: (GPT4 for text generation from relevant chunks, and bhashini for interpreting and generating voice notes in languages)

The quick experimentation was made possible by the combination of Glific + Jugalbandi. This combination enables NGOs to be the custodians of the subject matter. To prototype and test within matter of days, and focus their efforts on finding the right tone and knowledge to use and best way to make the LLMs work for their beneficiaries.  With the worry about the technology involved, time and effort on creation and maintenance of the tech out of the way, NGO teams can focus on the thing that really matters VALUE CREATION for the end user.

Given the tough contexts most Indian women face and live in (not to mention the ones subject to life in low income urban settlements with a skewed gender ratio), shared below is a snapshot of numbers that hopefully capture a real value an AI ally for young Indian mothers created in this pilot. One capable of sharing reliable and compassionate answers to questions one may shudder to open up on.

Story of the pilot in numbers

1. Users and questions
   • 269 distinct contacts interacted with the LLM chatbot over the period of 1 month.
   • 1106 questions asked by the contacts over the period of pilot
   • of the 1106 questions asked, 355 (32%) were asked in voice
   • On average 4 questions asked per contact
   • A maximum of 38 questions asked by a single contact
2. Activity through the pilot

The spike in usage (after the initial activity on 20th) of questions reaching 45-60 per day on every 3rd day is caused due to messages broadcast as a notification to the contacts. Presenting a good indicator on benefits of broadcasting to re-engage the contacts. 

4. Rating of the responses generated by the LLM by experts from SNEHA
   • Of the 1106 questions and answer pair, SNEHA team reviewed 700 responses (63% of the responses) 
   • Rating of 1 (5%) indicates “inaccurate response” i.e. the response was not accurate.
   • Rating of 2 (7%) indicates “relevant but not the expected response” i.e. response was relevant to some of the words in the question but not the response that a subject matter expert would provide
   • Rating of 3 (54%) indicates “accurate response” i.e. exact response as expected to be produced for the contact.
   • Rating of 99 (26%) indicates that the question was not a question but some sort of acknowledgement like “Ok”, “Thank you” etc. 
   • Rating of 98 (6%) indicates the cases where there was downtime of the service and question was not answered. 

5. Satisfaction as rated by the users.

Out of the 965 times responses to satisfaction, 80% of the times the contacts recorded being satisfied with the response received from the bot.

What comes ahead

Challenges identified in the present offering (Glific + Jugalbandi) 

1. At present, there is only a very limited control over the narrative that the bot is pushing through its answers. Some level of fine-tuning the model will be needed in order for the bot to answer questions based on the narrative that the SNEHA team wants to propagate like. 
2. The next iteration will have to have conversational memory, and more of a conversational nature than question and answer. Present tech integrations only enables question answer format in a loop, rather than an actual conversation. 
3. Simplifying the hindi words used for everyday terms like delivery, pregnancy etc. to sound more conversational is a definite need of the hour as identified by the SNEHA tea,
4. Humanizing the voice of the responses, the voice notes generated sound like a machine speaking in an accent which seems foreign. A more Indian accent to the Hindi will make the responses more relatable. 
5. Optimizing the costs for scale, the SNEHA team plans to launch this bot in the entire district of Bhiwandi (~700,000 citizens). The Jugalbandi platform will have served its purpose, if the present pilot is successful, and the team will need to graduate to something more robust and customizable. All this of course will have associated costs. 
6. Automating tests: for tweaks to prompt, knowledge base and fine-tuning, it becomes increasingly important to have a way to generate responses of already asked questions (ranging in 100s) to see the differences in the responses

The data and pictures shared in the post is possible due to the untiring efforts of SNEHA team. Blog on learnings from puting together the program coming soon. For more information related to on-ground efforts in making this pilot successful, reach out to Snehal Kulkarni or Clipsy Banji from the SNEHA team. To know more on how LLM powered chatbot can help your organization, get in touch with Tejas Mahajan