Summary

A program (tech enabled or otherwise), any product adoption, or running a campaign could be objectively broken down into 3 stages: Onboarding, Delivery, Engagement. To simplify and objectively measure the success of a program, the framework suggests breaking down the program into these buckets and then reflecting on the numbers, or setting goals and strategizing for each of these stages within a program. 

Premise 

1. The success for any product/ program/ campaign is based on its ability to “bring in new participants, and retain existing ones”. 
2. Each program/ product/ campaign is nuanced in its own way, and the framework itself is only one of the ways to objectively reflect on its success/ failure. 

The framework

A program can be broken down into the following phases. 

Onboarding -> Delivery -> Engagement 

Consider the scenario of a restaurant: Walking into a restaurant, and being greeted by clean tables and smiling faces is the onboarding phase. The quality and taste of food itself and whether you think it was worth it for the money that was spent, is the delivery. And the restaurant’s ability to strike the balance of making you feel comfortable (whether it’s with multiple interactions or no interactions at all) and thus getting you to order the 2nd dish or turning up again with more friends is the engagement. 

The ultimate proof of the pudding lies in the restaurant’s ability to attract new customers, and make a place in the hearts or wallets of the previous ones. This example is just to break in the idea of 3 different phases. The framework with which to break down the program or a product at an operational level.

For the remainder of the blog, we will work with the following hypothetical scenario: 

1. We are running a program whose aim is to nurture the next generation of badminton and chess maestros in India. 
2. The main target audience of the program is 5th-9th graders across the country
3. Objective of the program is to reach 10 million of these young athletes. Get them to explore their talent in chess or badminton by getting them to play either of the sports at least once. 
4. With that as the premise, the success of this program would mean getting as many young athletes to try at least one sport, and then getting more of them to keep on regularly playing the sport. 

Applying the onboarding -> delivery -> engagement framework to this: 

Onboarding 

Measure of how many participants started the program. Onboarding is the base of the funnel. Participants that can be onboarded are reached through channels. And once on the channel there can be various methods to deliver the call to action. 

Example of channels (that give access to the 5-9th graders considering our scenario): 

1. Private schools
2. Government schools
3. Large apartments 

Methods of onboarding onto the program: 

1. In-person individual sessions in schools/ apartments 
2. Through arranging tournaments and competitions 
3. Digital introductions and do-it-yourself model for recording and sharing athletic events. 

Some metrics that help to analyze the onboarding: 

1. The total number of 5th-9th graders reached
2. Channels vs students reached 

Decisions that need to be made: Which onboarding channel or method to double down on? And which ones to drop? The list of decisions of course can be expanded and more contextualized.  

Delivery 

Delivery might be the most crucial aspect of the program. Indicator of good delivery of the program is the direct output.

In our example, good delivery would be indicated by 

1. How many students actually spent time playing the sport. (Either self reported/ otherwise)

Then there are indicators which signal a good delivery like feedback rating of the program, Net Promoter Score etc. Teams will push to measure this, but eventually a good Net Promoter Score is not as good as an actual new athlete being referred to by an existing athlete. Or a principal of a school recommending the program to another fellow principal. Feedback rating and NPS can serve as good metrics to help triangulate what is going well or going off as a secondary measure of delivery. 

Engagement/ Retention 

This is the returning participants. Most programs will require the participants to return in order to reach the desired outcomes. In the simplest sense, engagement/ retention is a measure of loyalty or how much have the participants fallen in love with the program, how useful do they actually find it.   

Engagement and retention, depending on the context, might be the most difficult to define and arrive at a consensus as a team. What does engagement look like, and what does retention mean for a particular context. Some orgs use it interchangeably, some hold these in different regard, and there is no right or wrong definition. The only way to stay sane is to arrive at a definition and stick with it at least for a year (hopefully longer) till the consensus evolves. 

In the example we have taken: 

1. Engagement could mean, number of students who played the sports multiple times in the short term 
2. It could even mean, number of students who continued in the program from 5th grade to 6th grade. 
3. It could mean lots of other things too. 

A decent definition of the metrics and subsequent monitoring system should help teams identify 

1. What strategies create the desired engagement and retention
2. Result of those strategies on engagement from a particular channel (ex school, apartment) or demographic group.

Framework chatbot assisted programs

1. Onboarding:
   1. How many people entered the keyword and onboarded to chatbot. Read here to get the scoop on least frictional onboarding method. 
2. Delivery
   1. If the chatbot is for participants to take a particular action, then the basic measurement of delivery is the total number of first time action takers. 
3. Engagement
   1. Measure of repeat action takers, 
   2. Measure of how many times the actions are repeated

The framework itself is simple enough and serves as a foundation only. In my experience, grounding in this framework can enable

1. Iterations of the program to
   1. Improve the onboarding, 
   2. Get creative in different methods of delivery and 
   3. Try strategies for keeping the audiences engaged. 
2. Very objective goal setting, reflection and investigation exercises
3. Being at peace with the results knowing that the process was followed, considering many external factors can potentially disrupt the outputs. 

Credits

The framework was introduced and drilled into my conscience by Kuldeep Dantewadia, co-founder and CEO of Reap Benefit.