(Last updated on 2022-12-16)

1. Why this analysis?

What is this analysis for?

There’s rapidly growing interest to measure and understand quality of care - especially experience among clients. However, it is challenging to conduct conventional client exit interviews in many low-resource settings due to resource constraints, and innovative approaches have been used to obtain information from clients such as SMS, phone interviews, and surveys on social media platforms.

However, innovative approaches face challenges from getting a good quality sampling frame to achieving high contact and response rates. Critical remaining questions are how much bias is in samples of such assessments, and if (and how much) the sample bias matters to understand quality of care - especially experience of care.

We explore these questions, using PMA family planning client data from four countries/geographies: Burkina Faso, Kenya, and two states in Nigeria, Kano and Lagos. Implications from this analysis would be most appropriate for phone surveys, since other modes have additional barriers such as literacy and use of social media.

Note:
1. For more information on the PMA, see here.
2. For PMA client phone interview data, analytical weights are used to adjust for a woman’s propensity for phone ownership, to address the bias. Similar weighting approaches have been applied in other longitudinal analyses such as this and this But, such techniques cannot be used in most cross-sectional phone interview data, although different adjustment approaches can be considered, as explained here and here.

2. Data and methods

Detailed information about methods and implementation results of PMA client sampling is available here and here.

PMA client data are collected at two time points using different modes:
1. Face-to-face interview - at the end of facility visit. Response rates are typically very high in all settings.
2. Follow-up phone interview - 4 to 6 months after the visit. On average across countries/geographies, roughly 70% of clients completed the phone follow-up.

For illustrative purposes, we can treat the exit interview data (green left panel below) as sampling framework for a potential client phone survey, and the phone follow-up interview (navy blue part on the right panel) as a sample of clients who completed the phone survey.

Questions and Methods

1. How much background characteristics change between the sampling frame and final sample of a client phone survey? Considering that access to a phone is required but not necessarily universal in certain settings, it may result in a sample that is more likely to be wealthy and educated than those who are not included in the sample.

Analysis: We compare demographic and socioeconomic characteristics of those included (navy blue part on the right panel above) vs. excluded in the sample (the rest on the right panel).

2. Does the sample produce biased measures? Ideally, there should be equal (and high) quality of care - both experience and process of care - regardless of clients’ characteristics. If that holds true, biased sample may not be detrimental to assess and monitor.

Analysis: We compare both satisfaction (measured using a four-point Likert scale) and experience (using 10-item quality of contraceptive counseling (QCC) scale score - reference forthcoming) between those included vs. excluded in the sample.

All data on demographic and socioeconomic characteristics and quality of care come from exit interview data in four countries/geographies: Burkina Faso, Kenya, and two states in Nigeria, Kano and Lagos. The exit surveys were conducted in early 2021.

All data are accessible at www.PMAdata.org. See below table for analysis sample sizes.

3.1. How different are those who are included vs. excluded in the sample?

As expected, clients included in the sample are relatively advantaged in terms of background characteristics. They tend to:
- have higher education (in all countries/geographies),
- perceive themselves to be wealthier (in Kenya and Lagos, Nigeria), and
- be older (in Kenya and Lagos, Nigeria).

Note: The difference in perceived wealth median or above is statistically significant in Kenya and Lagos, Nigeria (Chi-square P-value < 0.05).

Note: The difference in attending secondary school or higher is statistically significant all countries/geographies (Chi-square P-value < 0.05).

Note: The difference in age is statistically significant in Kenya and Lagos, Nigeria (T-test with unequal variance P-value < 0.05).

3.2. Is reported quality of care different between those who are included vs. excluded in the sample?

Here, results are inconsistent.
- Clients in the sample are more likely to report to be very satisfied only in Burkina Faso and Kenya. Note that in the Nigerian states, about 70% of clients reported to be very satisfied and the measure may not be able to differentiate this perceived quality.
- In terms of experience of care score, clients in the sample have reported slightly higher score in Kenya, but the difference is too small to be useful for programmatic purposes. In other settings, it is comparable between the two groups.

Note: The difference in reporting to be very satisfied is statistically significant in Burkina Faso and Kenya (T-test P-value < 0.05).

Note: The difference in the QCC score is statistically significant in Kenya (T-test with unequal variance P-value < 0.05).

3.2.1. Controlling for background characteristics, do women in the sample still report to be ‘very satisfied’?

Satisfaction reflects how their experiences aligned with the care they expected, which may be confounded with clients background characteristics. So it is important to assess further whether or not the bivariate association remain, after holding background characteristics same.

Multivariate regression analysis showed that, in Burkina Faso and Kenya, clients who are included in the sample more likely have reported to be very satisfied, controlled for age, household wealth, and education.

Odds ratio of being included in the phone intervew sample by characteristics measured at exit interview
Characteristics.at.exit.interview Burkina.Faso Kenya Kano.Nigeria Lagos.Nigeria
very satisfied 1.88*** 1.43*** 1.31 0.94
(1.340 - 2.641) (1.160 - 1.769) (0.849 - 2.026) (0.573 - 1.537)
client’s age at interview 1.02 1.03*** 1.00 1.03*
(0.996 - 1.040) (1.019 - 1.047) (0.973 - 1.024) (0.999 - 1.067)
education secondary or higher 2.47*** 2.21*** 3.07*** 2.12**
(1.759 - 3.472) (1.832 - 2.675) (2.031 - 4.633) (1.104 - 4.074)
perceived HH wealth median or higher 1.18 1.18* 1.15 1.65**
(0.845 - 1.639) (0.972 - 1.439) (0.767 - 1.729) (1.041 - 2.624)
number of clients 903 4,114 695 495

Note: Results from a multivariable logistic regression model with robust standard errors to account for clustering of women within facilities. Analysis was done for each country/geography separately. 95% confidence interval in parentheses. *** p<0.01, ** p<0.05, * p<0.1

4. Implications to design a client phone surveys

4.1. Access to phone is still a significant barrier among female clients in certain low resource settings such as Kano, Nigeria (remote rural state) and Burkina Faso (with relatively less resources) - see below distribution.

Note: PMA phone follow up was done only women who received contraceptive method(s) or prescription at the facility visit, since an objective of the follow-up phone survey was to study contraceptive discontinuation. This group of women has comparable background characteristics with the rest clients (results not shown).

Note: concerning response rates among those who were eligible for the follow-up interviews (i.e., clients who consented for follow up, have a phone number, and received contraceptive methods at baseline: light and dark blue bars above), the response rates were fairly high in the case of PMA ranging from 85% to 90%. This may be due to: quality training of interviewers (both for the client exit and phone follow up interviews) and/or having been interviewed at exit, which might have created rapport and interest/willingness for continued participation.

Follow-up response rate among those eligible for phone follow-up
survey number.of.clients.lost number.of.clients.completed percent.completed
Burkina Faso 79 617 89
Kenya 354 3317 90
Nigeria, Kano 50 462 90
Nigeria, Lagos 60 351 85

4.2. Even in settings where access to phone is higher (Kenya and Lagos, Nigeria), the sample is still biased upward significantly. Our results suggest in all settings, client phone survey sample will have more educated women.

4.3. Meanwhile, specific experience during the visit is comparable in all settings. This may be because women received comparable care at facilities, regardless of their background characteristics that are studied here. This is a good news for potential client phone surveys!

4.4. However, in Burkina Faso and Kenya, reported satisfaction - which reflects expectation and may be confounded by clients background characteristics - is higher among clients included in the phone survey sample. This suggests that phone surveys should be careful in measuring subjective outcome of care quality and, rather, should focus more on specific experience.

5. Limitations and methodological note

5.1. How strong is the assumption?
This exercise is reasonably acceptable, if:
- sample recruitment and enrollment is done actively in-person - as opposed to more passive approaches such as relying on clients scanning QR code; and
- the data collection is done via computer assisted telephone interviewing (CATI) - as opposed to using short message services (SMS), self-administered web-based interview, or interactive voice response interviews.

5.2. What if the interval between sampling and interview is short?
In PMA client sample follow-up, length of follow-up interval ranged from 4 to 6 months. With incentives and shorter interval between sampling and interview, contact rate - and subsequently response rate - may be higher.
Mean length of follow-up interval in months by survey
survey number_of_clients_at_baseline interval_in_months
Burkina Faso 903 3.7
Kenya 4115 4.8
Nigeria, Kano 695 5.5
Nigeria, Lagos 509 5.8

5.3. Is PMA client sample representative of all clients in the country? PMA client sample is convenient random sample, conducted in a subset of facilities sampled for facility assessment (see below bar graph). The client sampling frame is FP clients who attended medium-to-high volume facilities. Thus, it systematically excludes those who visit small and/or lower-level facilities. In addition, no client data were collected from pharmacies. However, it is suggested that back ground characteristics of this sample are by and large similar to those who obtained contraception in the relatively recent time period at the population level. See here and here.

See GitHub for data, code, and more information. For typos, errors, and questions, contact YJ Choi at www.isquared.global.