International Development

How to Use Premise for Surveys

High quality survey data is critical for making development and humanitarian programs more impactful. Yet, as most readers are aware, surveying populations in low and middle countries poses significant challenges not found in high-income countries. Two sets of methods — face-to-face interviews and computer assistance telephone interviews (CATI) — have traditionally been used to address these concerns. 

Traditional survey methods leave much to be desired; face-to-face interviews are costly, slow to complete, and a burden to manage due to the logistics of enumerator training and travel. These challenges are part of the reason notable data collection programs like the Demographic Health Surveys can take almost two years to implement. CATI surveys often rely on partnerships with telecommunications companies to implement, lack the data quality control capabilities of purely digital survey methodologies, and frequently struggle to meet sample quotas because respondents’ demographic data isn’t known in advance. SMS and Interactive Voice Response (IVR) often struggle with these same issues and have worse data quality control.

At Premise, we’re bringing the best practices and innovations in polling and market research from developed countries and tailoring them to the realities of low and middle-income countries to give our customers the highest quality insights and information. 

Premise offers three methods for surveying people in low and middle-income countries. First, our network of five million app users, whom we call ‘Contributors’, serve as an opt-in survey panel. You can survey these panelists using probability sampling or quotas, targeted based on the population’s age, gender, and geographic makeup. Second, an average of 20% of our Contributors also complete enumeration tasks, and this portion of our network can be tasked with conducting face-to-face interviews. Third, a mix of these two methodologies can be implemented to ensure our partners can reach all groups in the sampling frame. 

App-Based Opt-In Panel Surveys at Premise

App-based opt-in panel surveys are the most common type of survey at Premise. They are used and trusted by commercial customers like Serta Simmons; international development partners like the World Health Organization; research institutions, like the Vaccine Confidence Project at the London School of Hygiene and Tropical Medicine; and media organizations, like The Economist

To better understand app-based opt-in panel surveys, let’s start by defining two key terms:

    1. App-based surveysA type of online survey that uses an app instead of a website. This is the most popular type of survey in the world because the process of collecting survey data is fully digital and easy to manage. 
    2. Opt-in panels Panels where respondents have signed up or were recruited to take surveys in exchange for money or other rewards. Opt-in panels have become extremely common in polling and market research because they make data collection fast, panelists provide their demographic data in advance, and research has shown that they are as reliable as telephone surveys.

App-based opt-in panel surveys have become very popular with pollsters in recent years. Pew Research has shown that this methodology provides results consistent with probability-based surveys over time, and YouGov uses this approach as its primary methodology

Meet Our Team

Leah Taub

Business Development Executive: 11+ years driving strategic partnerships and business development, including in global health and international development

Sarah McKay

Business Development Executive: 12 years working in the charitable and social enterprise sector

Bias

All survey methodologies have biases. The bias inherent in our app-based opt-in panels is that our panelists must own their own smartphone, have at least intermittent internet access, and download the Premise app. Fortunately, smartphone ownership and internet availability is growing rapidly in low and middle income countries. In 2019 the GSMA estimated that 65% of the world owned a smartphone and forecasts an increase to 80% by 2025. These numbers hold up well across low and middle income countries, and internet availability usually exceeds smartphone ownership.If smartphone ownership is particularly low in the country you want to survey, we suggest you consider mixed methodology or face-to-face surveys. 

Sampling

Premise offers three main sampling options for app-based opt-in panel surveys. In countries where our network is sufficiently large (e.g. Ukraine, Colombia, or the Philippines) stratified random samples are available and recommended when doing national surveys. In countries with smaller networks, quota samples with quotas for age, gender, and geography are most commonly used for national surveys. Lastly, convenience samples can be used when targeting niche demographic or geographic populations (e.g. mothers of children under five living in the Nairobi metro area.) Premise has an automated process for creating sample frames based on Worldpop modeled census data, but custom sample frames based on other census data are available as well. 

When Should I Use App-Based Opt-in Panel Surveys? 

App-based opt-in panel surveys in low and middle income countries are useful in several scenarios:

    1. When time or budget is an issue. There is no faster or cost-effective survey methodology than app-based panel surveys. 
    2. Countries with relatively high smartphone ownership. Nationally representative surveys can be conducted using our app-based opt-in panels in countries with relatively high smartphone ownership with the application of post-stratification weights. If the target population is narrower and smartphone ownership is lower, then one of the other methodologies will be more appropriate (see below).
    3. Recurring surveys. When making inferences from change over time, even a non-representative, homogenous sample is statistically valid. Premise’s panel is homogenous in two ways: all respondents opted in, and they’re all interested in generating income by offering their opinions and collecting data in their community.

Regional Smartphone Adoption

2020 actual – 2025 estimate

Latin America: 72% – 81%

M. East and N. Africa: 77% – 85%

Europe: 77% – 83%

Sub-Saharan Africa: 48% – 64%

Russia and CIS: 71% – 85%

North America: 81% – 85%

Asia Pacific: 68% – 83%

Source: GSMA

Face-to-Face Surveys at Premise

At Premise, we believe anyone should be able to provide insights and information about the communities in which they live. To make that a reality, we’ve established a method for Contributors to survey other community members. Our platform and policies enable us to replicate the best practices in training, sampling, and ethics from traditional data collection but without the significant overhead requirements encountered when using professional enumerators to conduct face-to-face surveys.

Qualification and training

Premise’s approach to qualifying and training Contributors leverages a unique advantage of our platform: each of our contributors has a history of data collection on the platform already. As such, Premise’s approach to training begins by offering our most prolific and trusted Contributors—those who have greater than 50 lifetime completed tasks and have a 70%+ trust score, derived from the percentage of tasks they complete that pass our data quality control modules—the opportunity to access interview tasks. Push notifications are sent to Contributors who pass this threshold informing them that they’re eligible to complete higher paying tasks by participating in a training series.

Premise staff then screen interested Contributors before enrolling them in a two-part training series. The first part of the training is standardized and applies to every face-to-face interview with Premise. The second is customized to our partner’s needs and usually involves topics such as how to perform a random walk within the context of the country in question, training on how to appropriately use local languages, and donor-specific compliance topics such as prevention of security exploitation and abuse. Contributors who pass the training then gain access to the interview tasks.

Sampling

Premise offers standardized sampling for nationally representative surveys and a variety of custom sampling options appropriate for both household and site surveys. To create standard national samples, we follow three steps:

    1. Generation of sampling frames for each of the administrative level one units of a country (the largest subnational administrative unit of a country, e.g. state or department) using their latest census data or modeled census data from WorldPop 
    2. Random selection of administrative level two units which are overlayed with population density maps on a grid using an internally-generated process
    3. Random selection of hexagonal polygons that are only from populated areas 

Our platform then automatically generates face-to-face survey tasks for each of the selected polygons and makes those tasks available to Premise Contributors living in or near those areas. Sampling for site surveys or surveys targeting specific towns and villages can be developed in partnership with Premise in-house statisticians. 

When Should I Use Face-to-Face Surveys? 

Face-to-face surveys in low- and middle-income countries are useful in several scenarios that are inadequately covered by opt-in panel surveys:

    • National baseline surveys. Most baseline or endline national surveys seek to make statistical inferences between highly granular geographic units. This includes activities like immunization coverage surveys, humanitarian multi-sectoral needs assessments, and baseline surveys for USAID Mission Performance Management Plans. Premise’s opt-in panel alone is usually not large enough to fulfill requirements for these, so face-to-face interview surveys can be used to bridge the gap. 
    • When you need to target specific geographies. If your program has a very narrow geographic focus—e.g. only has interventions in 15 villages—then face-to-face surveys are the best option because our panel won’t be large enough to adequately represent your target population. 
    • Site surveys. Surveys that must take place at a specific site or time— e.g. patient, caregiver, project beneficiary, or teacher interviews—can only be conducted using the face-to-face method. 

Advantages and Limitations Face-to-Face Surveys at Premise

Advantages Limitations
  • Better Statistical Inferences. Premise’s opt-in panel alone is often not large enough to support sample frames capable of generating statistical inferences between lower level administrative units (e.g. communes in Mali or municipalities in Colombia) but face-to-face surveys makes those inferences possible. This makes them an important methodology for large baseline surveys.  
  • Targeted. You can use Premise to collect survey data from specific geographic areas (e.g. villages or settlements) and activity sites (e.g. health facilities), making interviews with direct beneficiaries possible.
  • Global Scale. Traditional field data collection often requires contracting with multiple providers to cover an entire country or region. Using Premise for face-to-face interviews allows you to access anywhere and everywhere through the same provider. 
  • Relative Speed. Since Premise contributors conduct interviews in the areas where they live, there is little travel time or costs required. Our use of virtual training also makes using Premise faster than traditional field data collection.  
  • Costly. While still less expensive than traditional providers, face-to-face surveys incur the most credit usage compared with Premise’s other survey options. 
  • Slower than Opt-in Panel Surveys. Face-to-face interviews are not as fast as opt-in panel surveys because most surveys require some custom training program for the Contributor and custom sample frames cannot be generated automatically.

Mixed Methodology Surveys at Premise

Smartphone ownership and internet access in low- and middle-income countries is growing at an incredible rate. Yet ownership and access is not equal across all demographic and geographic groups. Mixed methodology surveys—in this case a mix of app-based opt-in panel surveys and face-to-face surveys—offer Premise partners a way to increase representation in the sample without the full time and cost requirements of face-to-face interviews. 

Sampling 

Mixed methodology surveys involve creating two samples, one for opt-in panel surveys and one for face-to-face interviews. The opt-in panel samples typically use quota sampling for age, gender, and geography. Premise then compares those quotas to the number of respondents in our panel and builds the face-to-face survey quotas to cover the projected delta. Sometimes the samples are launched simultaneously. Otherwise, the face-to-face survey is launched after the panel surveys are completed so the actual delta is covered. Customers with time constraints typically launch their surveys concurrently while customers with more time to spare launch them sequentially. 

Enumerator selection

The face-to-face survey qualification and training features described above aren’t always necessary for mixed methodology surveys. This is because the face-to-face survey sample is typically used to cover age groups or genders that are underrepresented in our panel rather than geographies. We can use our existing demographic data on our Contributors to determine whether they live in a household that includes the underrepresented population (e.g. respondents aged 55+). Our partners can then publish face-to-face survey tasks just to those Contributors, now enumerators, who live in households with members of the underrepresented population. Since such face-to-face interviews do not involve performing a random walk or other standard enumerator duties, training is usually not necessary. They will be tasked with conducting the interview and taking a photo of the interviewee’s identification card to prove they are in the target population. 

When Should I Use Mixed-Methodology Surveys? 

Mixed methodology surveys in low and middle income countries are useful in several scenarios:

    • When certain demographics are not well-represented in our panel. Older demographics, women, and certain geographies are sometimes underrepresented in our panel—even in countries with high smartphone ownership. If you need a nationally representative sample but are on a limited budget or timeline, a mixed methodology survey is a great way to improve representation without the time and expense of only face-to-face interviews.
    • Programs targeting youth or children. Premise does not allow people under 18 on our app, however, we do allow our Contributors to interview people under 18 in their household. If your program exclusively targets these groups or If your audience includes ages under 18 then mixed methodology surveys are a good option.

Advantages and Limitations Mixed Methodology Surveys at Premise

Advantages Limitations
  • Best value. Mixed methodology surveys offer most of the advantages of opt-in panel surveys and face-to-face surveys but with fewer limitations. In particular, you can get highly representative data in countries with low smartphone ownership without the expense of using only face-to-face interviews.
  • Data is less real-time. The opt-in panel survey data is collected faster than the face-to-face survey data, which makes using the data in real time more fraught.