The blog is put together by Sunderrajan Krishnan , Kiran Kumar Sen and Anurag Shri Rathore from INREM foundation. Glific team is extremely grateful for the contributions made by NGO partners using the platform and contributing with research and knowledge for the Glific community to draw inspiration from.

What Does INREM Do?

INREM engages citizens and ensures that villages are Water-Safe anywhere in India. Enabling students, school teachers, health workers and water champions to detect water contamination problems and their health impacts is an important step towards enhancing the agency of citizens to sense and solve their own local problems. 

We are working on a score card for Water-Safe communities that citizens can report about and helping measure their success. Part of this score-card are indicators of water contamination such as unclean water bodies, vegetation growth over ponds, stagnant water; but also symptoms of diseases and specific health problems that people face. 

Normally, measuring these indicators needs a bit of expertise and training that takes time and effort. Our online courses for Water Quality Champions help gain such expertise. Water testing is one effort that helps as an indicator here. We however see that a bit of help and support makes “Water Championing” more accessible. What role can technology play in this is a question we are exploring. 

Some of the indicators for Water-Safe Communities are visual indicators requiring expertise for interpretation. Our hypothesis here is:

Does a bit of screening for visual indicators of Water contamination, lead to greater agency in Water Quality Champions (WQC) for sensing and solving their own problems? 

Use cases for WhatsApp chatbot with image recognition

INREM uses the Glific’s WhatsApp chatbot creation platform. This provides an empowerment interface for water quality champions (WQCs) who comprise of students, health workers, water operators, teachers, NGO community mobilisers, govt dept engineers, chemists and any citizen broadly.

GPT 4 with vision brings digital vision recognition capabilities that have the potential to make Bot based interactions even more potent. 

We are now working on two use cases that are useful for WQCs for enhancing their ability to sense indicators for Water-Safe Communities:

1. Water bodies with visible signs of water contamination and decay. To be able to share photographs, get a problem diagnostic and then actionable suggestions that make local sense.

Example:

To be able to say that this Water body is covered with Algae

2. Health problems from water that show visible symptoms such as Dental fluorosis and Arsenicosis. To be able to share photographs of suspected cases and get a first level screening done

Examples:

Identify this as Mild Dental Fluorosis

Identify this as Arsenicosis

Construct of experiment

In order to do a fidelity check as a first phase test, we have constructed an experiment with 3 different feature implementations within Glific that integrates gpt-4-vision-preview along with specific prompting. 

The manner in which GPT 4 Vision is being adopted here is through a Webhook from Glific that calls by providing an Image and prompt as an input.

Implementation of Webhook that calls GPT 4 Vision

We are testing three different calls for GPT Vision, each of which have a different prompt as input. Primarily the focus here is on prompt engineering. Facility to train the model is currently not available from OpenAI.

1. Waterdetect 

Here we are prompting GPT Vision to recognise water bodies and detect algae, water hyacinth and any water contamination that is visible. The reply is limited to 20 words.

2. Vision

With this prompt , we are focussing GPT Vision towards recognizing Dental fluorosis with teeth and Arsenicosis on skin. The prompt asks to also provide a Mild/Moderate/Severe Rating and a confidence score from 1-10 on this prediction. No word limit is given on the response.

3. Waterdoc

Here, we prompt the same as Vision, without the confidence score and replying only in 2 words ie No/medium/Severe and either of Fluorosis or Arsenicosis

For each of the 3 features above, we are requesting the users to test with images and then provide feedback. First very simply as 👍 or 👎. Then specific inputs if they would like to.

Insights from experiment

We obtained a total of 43 responses from a user base comprising mostly field practitioners within INREM. 

Most of the responses were 👍 indicating a positive response to the fidelity check. As an example, we share one such interaction here:

We received 4 thumbs down and 3 short feedback responses. From these, we infer the following: 

• Arsenicosis detection is weaker. Many types of skin problems seem to be getting shown as Arsenicosis. 

• Fluorosis detection is at the stage of narrative analysis rather than clear objective one ie not still at Yes/No, but it works quite good too

• The ability to distinguish between other dental issues is present, but in cases like tobacco stains, there is one case where it classified as fluorosis. 

• Water body detection is pretty strong. To be able to detect algae, water hyacinth and other water pollution is quite precise. It is able to clearly point out when any other images are provided eg a park

Next steps

Our next steps here are as follows:

• Robustness of image recognition with WaterDetect on water bodies for deployment on field conditions shows a high readiness level. We will do further tests by different prompts for further detection (type of water bodies; types of problems) and differing light conditions. We are now ready with this to go to the next step of providing action points to the user and further engagement.

• Dental Fluorosis detection: We understand here that unless the model is further trained, its inherent vision recognition is providing more chances for false positives (classifying tobacco stains as Dental Fluorosis). However, the ability to detect Dental Fluorosis and provide insights is still present. This is shown by the observation till now that we have not observed any False Negatives. Further model training would be needed for greater accuracy here on false positives. Next step here is to deploy this in field conditions and look at accuracy in bulk image settings ie whether it is able to provide a Percentage of Dental Fluorosis from a set of images from a School.

• Arsenicosis detection: This shows the greatest vision detection weakness amongst all examples that we tested here. For eg. one of the user responses was: “This is a picture of vitiligo and not arsenicosis ”

Here there is a bigger need for model reinforcement with both positive and negative cases. 

For model reinforcement in detecting Dental Fluorosis and Arsenicosis, we plan to strengthen these experiments now with a Custom GPT Model and wait for the next feature from Open AI releases to be able to train the GPT Vision model. 

The experiments as a whole show a high degree of promise in being able to empower Water Quality Champions by using AI Vision recognition capabilities towards solving urgent problems that we face.