Overview

This summer Eckington will be undertaking a neighborhood survey aiming to assess the communities interest in getting Historic Designation for the neighborhood. Historic Designation has a broad set of implications both positive and negative:

Pros Cons
Improved aesthetics Restricted property rights
Increased home values Increased repair costs
Limits on developers Increased pressure on low income neighbors
Slowing rate of change Tight restriction on growth/repair
Uniform code of rules Time consuming approval process

Eckington is an area of great historic interest and is also racially and economically diverse. According to 2014 statistics, Eckington has 4120 residents of which 74% are Black or African American, 21% are Caucasian, and 5% are from one of our many other racial groups. In all there are 1835 housing units.

Survey Challenges and Oportunities

A survey tool will be used to assess the favorability of moving to Historic Designation.

There are a number of challenges for such a survey:

  1. Obtaining an adequate sample size

  2. Ensuring the sample is representative of the neighborhood (ie.income,race)

  3. Developing a survey tool that does not bias responses in one direction or the other

  4. How the Eckington Community Association should interpret survey results.

Challenge #1: Minimum Sample Size

Determining Sample Size

The mathematics behind solving for the minimum sample size required for a survey is well established. The sample is determined as a function of two key variables: a) the population size \({N}\) (number of residents in Eckington), b) the maximum acceptable margin of error \({m}\) (a measure of survey accuracy), and c) the statistical level of significance (the likelihood our findings would be wrong, by random chance). For those interested the formula for the minimum sample size see the Suplimentary Materials section.

We can solve for the minimum sample size required for a survey with a True False question using a few variables including a,b,c above.

According to the American Community Survey (ACS) in 2014 Eckington has a total of 1835 housing units (which is our best estimate of the number of households).

If we want to have a 5% margin of error and be 95% confident that our result isn’t the result of random chance:

sample1 = round((1.96^2*1835)/(1.96^2+4*1835*0.05^2))
print(paste('Minimum Sample Size:',sample1))
## [1] "Minimum Sample Size: 318"

If we want to have a 5% margin of error and be 90% confident that our result isn’t the result of random chance:

sample2 = round((1.64^2*1835)/(1.64^2+4*1835*0.05^2))
print(paste('Minimum Sample Size:',sample2))
## [1] "Minimum Sample Size: 235"

I recommend a minimum sample size of 235 individuals.

Choosing the best survey method

Another issue is the method by which the survey will be implemented. The primary choices are mail, web, or a combination. A variety of studies seem to point to the advantages of combining a paper survey that also provides on online option. In a recent study of college student participation in surveys, results point to improved response rates from surveys that combined paper forms with online options (the increase participation rate was particularly clear among minorities)1. Minimizing the costs of participation (monetary or effort) should be minimized. To encourage the participation of low income individuals we should provide a prepaid option. Even with this option response rates will likely approach 20%, except in those groups that highly motivated/informed on this particular topic.

I recommend a prepaid paper survey that points to an easy to understand online option.

Participation rates are likely to be around 20% unless the community is highly motivated or informed

Challenge #2: A Representative Sample

Another issue is getting a representative sample. This this will likely be the most challenging part of implimenting this survey in a fair manner. Low income and minorities are significantly less likely to participate in groups like the Eckington Civic Association2, compounding this problem they are also likely to be under-represented in our survey3.

The first issue is identifying participation bias. This is problem is easily identified by asking two simple demographic questions: 1) race/ethnicity 2) income bracket. The question can be asked in a way that indicates this information will be used to ensure that the vote fairly represents the diversity of the neighborhood.

To ensure all groups are represented in the survey, I recommend asking participant for information on their race and income bracket

Considering the high proportion of minority populations in Eckington, and their likely under-representation in the survey, extra effort will likely be required to encourage participation. This may include the use of additional door-to-door participation drives.

Considering the low expected participation rates, minority and disaffected community participation could be improved by door-to-door sampling drives

Challenge #3: An Unbiased Survey Tool

This is likely a minor issue. However the survey should be reviewed by both proponents and opponents to the measure with an aim to avoid leading questions, or biased introductory materials.

To ensure an unbiased survey tool, I recommend the development of introductory materials and questions developed with both proponents and opponents to the measure

Challenge #4: Interpreting Results

The Eckington Community Association is in charge of making the final determination of whether or not the community should move forward on Historic Designation. Therefore there should be clear guidance about how to meaningfuly interpret survey results.

When surveys are collected and tallied we should look at 3 main indicators. The proportion for and against the measure, the racial/economic composition of respondents, and the actual survey margin of error. If a ‘winner’ can’t be determined outside the margin of error, more surveys should be collected. If the racial/economic composition of respondents is significantly different than the neighborhood composition, more surveys should be collected.

The survey can be considered valid and ‘in favor’ of Historic Designation if it meets two criteria: 1) the proportion of ‘fors’ is above 50% and outside of the margin of error, and 2) the racial and economic composition of the neighborhood roughly corresponds to that of respondents (+-10%).

Example: If ‘againsts’ are 60% and the margin of error is 10% then the survey cannot say meaningfully that ‘againsts’ win (because 50% is within the margin of error). Therefore addition surveys should be collected or 4. Surveys should be collected until a ‘winner’ can be determined.

Using the first formula in the Supplimental Materials we can estimate the margin of error for any survey results based on the # of people that participated, the % of ‘fors’ and ‘againsts’ etc.

Summary of Recommendations

The move towards Historic Designation has critical and long-lasting implications for our neighborhood (both pro and con).

To ensure that the survey represents the will of the neighborhood, I recommend the following:

1. A minimum sample size of 235 individuals.

2. A prepaid paper survey that points to an easy to understand online option.

3. Ask participant for information on their race and income bracket.

4. Participation could be improved by door-to-door survey drives.

5. Development of introductory materials and questions should include both proponents and opponents to the measure.

6. The survey can be considered valid and ‘in favor’ of Historic Designation if it meets two criteria: a) the proportion of ‘fors’ is above 50% and outside of the margin of error, and b) the racial and economic composition of the neighborhood roughly corresponds to that of respondents (+-10%).

Supplimental Materials

Including a finite correction factor, the margin of error \(m\) for binary data is given by \[m = z_{\alpha/2} \sqrt{{p(1-p)} \over {n}} \sqrt{{{N-n} \over {N-1}}},\] where \(N\) is the population size, \(n\) is the sample size, \(p\) is the success probability (say, the chance of a voter giving a true answer), and \(z_{\alpha/2}\) is the appropriate quantile from the standard normal distribution for the desired confidence.

Solving for \(n\) we find \[n ={ {z^2_{\alpha/2} \ p (1-p)N} \over {z^2_{\alpha/2} \ p(1-p)+(N-1)m^2} } \] Now \(p\) is unknown, but the worst case is \(p={{1} \over {2}},\) so using that we have \[n ={ {z^2_{\alpha/2} \ N} \over {z^2_{\alpha/2} \ +4(N-1)m^2} } \]

Using our numbers (\(z_{\alpha/2}=1.64\)), \(N=1835\), I get a minimum sample size of \(n=235\).

References Cited & Notes

  1. Sax, Linda J., Shannon K. Gilmartin, and Alyssa N. Bryant. “Assessing response rates and nonresponse bias in web and paper surveys.” Research in higher education 44.4 (2003): 409-432.↩

  2. Alesina, Alberto, and Eliana La Ferrara. Participation in heterogeneous communities. No. w7155. National bureau of economic research, 1999.↩

  3. Groves, Robert M., Robert B. Cialdini, and Mick P. Couper. “Understanding the decision to participate in a survey.” Public Opinion Quarterly 56.4 (1992): 475-495.↩

  4. Rather than collecting additional surveys, more advanced statistical methods could be used to simulate additional responses (controlling for race and income) based on results that were collected.↩