Survey Report
Version
Version 2023-09-A
Status
Ongoing - report will be updated should further surveying be
undertaken in the future.
Purpose
The purpose of the community surveys: Over the
period Aug 2021 to Oct 2022 Sustainable Tarras undertook a series of
surveys in the areas of Tarras, Upper Clutha and Lake Hawea. The purpose
was to understand community sentiment regarding the proposed development
of an international, jet-capable airport in Tarras.
The purpose of this report: To convey further detail
in plain English on the survey design, analytic decisions and analysis,
along with top line results as per best practice. More detail can be found later in the
report for more technical readers.
Summary
The surveys provide evidence of considerable local opposition to the
development of a jet capable airport at Tarras:
Overall 72% ±6% were opposed to development of a
jet capable airport at Tarras
Overall 21% ±5% were in favour of development of a jet capable
airport at Tarras.
Results are accordance with the Wanaka Stakeholders Group survey
(74% oppose in 2021, and 83% oppose in 2023).
Opposition was distributed uniformly across the district, with
the exception of increased opposition in Tarras.
Response data was excellent quality.
Response rates are commensurate with similar surveys conducted in
the South Island.
Sustainable Tarras would like to thank all respondents in the
community for their time and input.
Survey #1: Tarras & Bendigo
Target population
The Target population is residents of Tarras and Bendigo close to the
proposed airport site.
Target/Area Frame
All households in the address file from Tarras, Bendigo and
Queensberry within 10 km of the proposed airport site. Households in
Queensberry located over the ridge and out of sight of the proposed
airport site were not included and were sampled in the Upper Clutha
community survey. Households in Tarras which had a PO Box but not a
letterbox were also sent an invitation letter.
Sampling Frame
For the Tarras survey complete enumeration (all households) from the
address file.
Sampling Unit
Primary sampling unit = Households.
Secondary sampling unit = max 2x responses per household.
Sample Size
N = 76 households.
M = 76 responses (1x response randomly sampled per household).
Timeframe
02-Aug-2021 to 02-Oct-2021 (61 days).
Survey #1 Results
Response Rate
Unit response was 76 from 190 invited households = 40%.
Item response for Q2 was 100% (this question was mandatory).
Item response for Q’s 3-15 was 100% (these questions were
optional).
3x responses with inadmissible codes were removed from the analysis
dataset.
Response Time
Median response time was 04:36.
Primary Result
84% (± 7%) were opposed to development of a jet capable airport
at Tarras.
12% (± 6%) were in favour of development of a jet capable airport at
Tarras.
Refer to table below and Results section for barchart.
##
Design$TarrasZAggregated = survey::svydesign(id = ~HouseholdID + ResponderID,
probs = NULL,
weights = NULL,
fpc = ~FPC + FPC2,
data = Data$TarrasZResponse2 %>%
dplyr::mutate(FPC = 190,
FPC2 = 1))
Results$TarrasZAggregatedMean = survey::svymean(x = ~Q2Aggregated,
design = Design$TarrasZAggregated,
na.rm = TRUE,
method = "logit")
#
Results$TarrasZAggregatedMean %>%
base::data.frame() %>%
dplyr::mutate(MarginOfError = SE * 1.96) %>%
dplyr::select(Result = mean,
MarginOfError) %>%
dplyr::mutate(Result = scales::percent(x = Result,
accuracy = 0.1),
MarginOfError = scales::percent(x = MarginOfError,
accuracy = 0.1))
Margin of Error
Refer to table above
DEFF
Design effect = 1 (owing to the second stage sampling of 1x response
per household).
Secondary Results
Refer charts in All Results section below.
Survey #2: Upper Clutha
Target population
The Target population is residents from Mt Pisa, Northburn,
Queensberry, Luggate and Hawea Flat.
Target/Area Frame
All households in the address file from the above areas excluding any
Queensberry residents who were included in the Tarras survey.
An additional 1x household was removed from the sampling frame (the
household of the survey statistician).
Sampling Frame
A random sample of 450 households from 878 households in the address
file.
Primary sampling unit = Households.
Secondary sampling unit = max 2x responses per household.
Sample Size
N = 120 households.
M = 120 responses (1x response randomly sampled per household).
Timeframe
16-Oct-2021 to 08-Feb-2022 (115 days).
Survey #2 Results
Response Rate
Unit response was 120 from 450 invited households = 27%.
Item response for Q2 was 100% (this question was mandatory).
Item response for Q’s 3-15 was 99% (these questions were
optional).
No responses with inadmissible codes were received.
Response Time
Median response time was 04:35.
Primary Result
68% (± 8%) were opposed to development of a jet capable airport
at Tarras.
25% (± 8%) were in favour of development of a jet capable airport at
Tarras.
Refer to table below and Results section for barchart.
##
Design$TarrasR = survey::svydesign(id = ~HouseholdID + ResponderID,
probs = NULL,
weights = NULL,
fpc = ~FPC + FPC2,
data = Data$TarrasRResponse2 %>%
dplyr::mutate(FPC = 878,
FPC2 = 1))
Results$TarrasRAggregatedMean = survey::svymean(x = ~Q2Aggregated,
design = Design$TarrasR,
na.rm = TRUE,
method = "logit")
#
Results$TarrasRAggregatedMean %>%
base::data.frame() %>%
dplyr::mutate(MarginOfError = SE * 1.96) %>%
dplyr::select(Result = mean,
MarginOfError) %>%
dplyr::mutate(Result = scales::percent(x = Result,
accuracy = 0.1),
MarginOfError = scales::percent(x = MarginOfError,
accuracy = 0.1))
Margin of Error
Refer to table above.
DEFF
Design effect = 1 (owing to the second stage sampling of 1x response
per household).
Secondary Results
Refer charts in All Results section below.
Survey #3: Lake Hawea
Target population
The Target population is residents from Lake Hawea.
Target/Area Frame
All households in the address file from the above area.
Sampling Frame
A random sample of 300 households from 626 households in the address
file.
Primary sampling unit = Households.
Secondary sampling unit = max 2x responses per household.
Sample Size
N = 59 households.
M = 59 responses (1x response randomly sampled per household).
Timeframe
22-Jul-2022 to 10-Oct-2022 (80 days).
Survey #3 Results
Response Rate
Unit response was 59 from 300 invited households = 20%.
Item response for Q2 was 100% (this question was mandatory).
Item response for Q’s 3-15 was 99% (these questions were
optional).
No responses with inadmissible codes were received.
Response Time
Median response time was 06:23.
Primary Result
75% (± 10%) were opposed to development of a jet capable
airport at Tarras.
17% (± 9%) were in favour of development of a jet capable airport at
Tarras.
Refer to table below and Results section for barchart.
##
Design$TarrasH = survey::svydesign(id = ~HouseholdID + ResponderID,
probs = NULL,
weights = NULL,
fpc = ~FPC + FPC2,
data = Data$TarrasHResponse2 %>%
dplyr::mutate(FPC = 626,
FPC2 = 1))
Results$TarrasHAggregatedMean = survey::svymean(x = ~Q2Aggregated,
design = Design$TarrasH,
na.rm = TRUE,
method = "logit")
#
Results$TarrasHAggregatedMean %>%
base::data.frame() %>%
dplyr::mutate(MarginOfError = SE * 1.96) %>%
dplyr::select(Result = mean,
MarginOfError) %>%
dplyr::mutate(Result = scales::percent(x = Result,
accuracy = 0.1),
MarginOfError = scales::percent(x = MarginOfError,
accuracy = 0.1))
Margin of Error
Refer to table above.
DEFF
Design effect = 1 (owing to the second stage sampling of 1x response
per household).
Secondary Results
Refer charts in All Results section below.
All Results
Primary Result
# Primary result = combined analysis.
# 3 Surveys/Strata, N households, and M randomly chosen household residents
Design$TarrasZRH = survey::svydesign(id = ~HouseholdID + ResponderID,
strata = ~SurveyName,
probs = NULL,
weights = NULL,
fpc = ~FPC + FPC2,
data = Data$WideFinal %>%
dplyr::mutate(FPC = Population,
FPC2 = 1))
#
survey::svymean(x = ~Q2, #Q2Aggregated,
design = Design$TarrasZRH,
na.rm = TRUE,
method = "logit") %>%
base::data.frame() %>%
dplyr::mutate(MarginOfError = SE * 1.96) %>%
dplyr::select(Result = mean,
MarginOfError) %>%
dplyr::mutate(Result = scales::percent(x = Result,
accuracy = 0.1),
MarginOfError = scales::percent(x = MarginOfError,
accuracy = 0.1))
Q2 results are concordant with the Wanaka Stakeholders Group survey
in Jan 2021.
WSG: “82.7% of respondents are opposed to a new international
airport at Tarras (2023)” Ref
WSG: “74% of respondents are opposed to a new international
airport at Tarras (2021)” Ref
Secondary Results
>
Click on the relevant tab to explore the results
Q3
Q3 results are concordant with the Wanaka Stakeholders Group survey
in Jan 2021.
WSG: “76% said that they were concerned about the impacts on
Quality of Life” Ref
Q4
Q5
Q5 results are concordant with the Wanaka Stakeholders Group survey
in Jan 2021.
WSG: “83.5% were concerned about the negative impacts on the
unique character of the Upper Clutha” Ref
Q6
Q7
Q7 results are concordant with the Wanaka Stakeholders Group survey
in Jan 2021.
WSG: “68.7% were concerned about road safety…” Ref
Q8
Q9
Q10
Q11
Q11 results are concordant with the Wanaka Stakeholders Group survey
in Jan 2021.
WSG: “87% are very worried about environmental impacts…” Ref
Q12
Q13
Q14
Q15
Q16 Responses in word cloud
>
Click on the relevant tab to explore the results
Opposition
Insights: Respondents opposed to the airport
proposal (below) were still favourable to economic development and
prosperity for the region.
Support
Insights: Respondents in favour of the airport
proposal (below) were not primarily motivated by the convenience of a
nearby airport, but rather the possibility of economic development and
prosperity for the region and future generations.
Discussion
Strengths
All responses to date have been high quality and answered in good
faith. There have been no instances of random or systematic responses to
the survey questions.
Most respondents were able to access and complete the survey in 3 to
7 minutes
All surveys had good geographical coverage with similar response
rates in all areas.
Potential sources of bias
Surveys on contentious issues tend to elicit responses from those
with strongly held views on either side of the debate early in the
survey. The voice/viewpoints of the less engaged can often be
under-represented. To this end the surveys were kept open for a long
period of time to better capture these responses.
The survey flow works best for people in possession of a smartphone
and comfortable with web based forms. Some residents may have had
difficulty responding as there was no paper based alternative.
Non-response bias must also be considered carefully. The relationship
between non-response and non-response bias is nuanced. Studies show
non-response alone is not a good predictor of non-response bias. A
thoughtful examination should identify putative mechanisms of
non-response bias specific to a given situation/survey.
“…a low response rate in itself does not necessarily imply a high
level of bias… The results support the conclusions of prior research,
showing that even achieved samples with response rates as low as 10
percent may produce highly accurate estimates in certain cases.” An empirical examination of the relationship between
non-response rate and non-response bias
“But it is not necessarily true that representativeness increases
monotonically with increasing response rate. Remarkably, recent research
has shown that surveys with very low response rates can be more accurate
than surveys with much higher response rates… Although the mail surveys
had response rates of about 20% and the telephone surveys had response
rates of about 60%, the mail surveys predicted election outcomes much
more accurately than did the telephone surveys… Therefore, having a low
response rate does not necessarily mean that a survey suffers from a
large amount of nonresponse error.” Survey Research
No follow-up letters were sent to non-responding households. In
future surveys this is recommeneded to further increase the response
rate.
Interpreting survey results
A note on interpreting survey results:
When it comes to survey data the classic analogy is: You only
need to taste a teaspoon of soup to know if it’s salty enough.
A teaspoon is sufficient if you’re cooking for 2 or 20, you don’t
need to drink the entire pot.
Comparison with other surveys
City and District Council surveys
Sample size and response rates are reported for relevant council
surveys below.
Christchurch City Council Waterway Survey Page 4/61 “… response rate of just over 10% was
suitable for this type of survey and enables good confidence in the
results”
Christchurch City council Quality of Life survey page 27/58 “…28%”
Queenstown Lakes District Council Quality of Life report page 3/119 “…10%”
Dunedin City Council Quality of Life survey 2020 page 12/161 “…29%”
Dunedin Residents’ Opinion survey 2021 page 77/80 “…31%”
Black dots indicate response rates for various city and district
council surveys, the blue dots represent response rates from the
Sustainable Tarras community surveys to date.
CIAL surveys
The aviation industry relies on survey data for insights into
customer satisfaction: “CIAL’s average passenger survey ratings
historically are the highest ratings of the regulated New Zealand
airports”. Ref.
We were unable to compare with any CIAL surveys. There was no
commentary in the various reports and the required study documentation
had an oblique reference “Survey fieldwork documentation is
available on CIAL’s website www.christchurchairport.co.nz”. We
crawled the site but were unable to find any documentation.
2021 page 47/58
2020 page 46/58
2019 page 48/66
2018 page 53/64
2017 page 42/51
CIAL can improve their survey documentation using this report as a
template and disclose: The target population size, the sampling frame,
the sample size, the response rate, the results, and associated margin
of error.
Black dots indicate response rates for various CIAL passenger
satisfaction surveys, the blue dots represent response rates from the
Sustainable Tarras community surveys to date.
Methods and further details
Data Governance
Data owner: Sustainable Tarras Society
Data custodian: RJ Labs
Data Confidentiality Policy
The survey collection is designed to ensure responses are
non-identifiable and thus confidential, this ensures the privacy of all
respondents.
Responses are reported at the aggregate level only. Responses at
the household level are not reported or made known to the Sustainable
Tarras Society or any 3rd parties.
The survey response data is stored securely and will not be
shared, on-provided, data-mined, or linked to any other
datasets.
Data Retention Policy
Household identifiers will be deleted 24 months after the survey
closes.
Sample Design
The surveys reported are two-stage cluster, online surveys. This
design first selects a sample of households in the survey area and then
randomly selects one response per household. It is very difficult to
know the precise resident population of an area in the Upper Clutha
region at any point in time. People are transient, household sizes
change, etc. It is considerably easier to accurately ascertain the
number of households in the region. This makes a household based
two-stage design desirable.
“The technique is used frequently when a complete list of all
members of the population does not exist and is inappropriate.” Ref
Two-stage cluster sampling aims at minimizing survey costs and at
the same time controlling the uncertainty related to estimates of
interest. Ref
It is important to note in cluster surveys the sample size is the
number of clusters (i.e. households) and not the number of residents
surveyed within households. This is a common misconception. Ref
In the first stage households are sampled exhaustively (in the Tarras
survey) or selected randomly (in subsequent surveys). A maximum of 2x
responses per household are invited on the assumption most households
have a maximum of 2x adults in residence.
Second stage sampling involves randomly selecting 1 response per
household. Data to date indicate “within-household” responses are very
highly correlated, in other words responders in a given household hold
identical opinions on the project (occasionally they differ in degree
but not kind). Under these conditions a sample of 1 response per
household accurately captures the household’s opinion. This is a common
design which simplifies the analysis and also safeguards against a
potential bias whereby less motivated households may only respond once
while more motivated ones would ensure 2x responses were submitted.
Sampling Frame
The sampling frame was constructed from 2 readily
available datasets: The LINZ NZ Street Address database and an NZ Post address file. The LINZ file contains street
addresses and also geocodes (longitude and latitude), this information
is crucial when designing the survey areas. Not all LINZ addresses are
residences (addresses include vineyards, depots, empty lots, occasional
woolsheds, etc). Joining the LINZ dataset to the NZ Post address file
ensured the sampling frame was limited to known street addresses which
also had a confirmed postal address. The NZ Post address file also
included PO boxes which ensured households which didn’t have a postal
address but collected their mail from a PO box were included.
Invitation
Invitation letters were delivered via post. The invitation letter had
a QR code which can be scanned by a smart phone and automatically
directs the reader to the online survey form. Most New Zealanders have
become accustomed to this technology recently. The letter also gave the
web address of the survey form in the event household residents didn’t
have a smartphone with a camera and QR scanner.
Questionnaire
The 16 item questionnaire was designed by Sustainable Tarras Society
and reviewed by RJ Labs. Closed questions were posed in a neutral manner
and reponses were invited on 4 and 5 level Likert scales covering a range of responses.
Questions 1 (Enter your 4 letter code) and 2 (Are you in favour of
the development of a jet capable airport at Tarras?) are compulsory the
remainder are optional (Ref Appendix I).
Survey Flow
Each selected household address is assigned a unique 4 letter
code by the RJ Labs A single lookup table mapping addresses to codes is
held securely by RJ Labs (this is not shared with anyone else).
One invitation letter is sent per household with its associated
unique 4 letter code inviting reponses from max 2 individuals.
Survey respondents enter their unique household code into the
questionnaire form. This ensures only responses from invited households
can participate.
Respondents are not required to enter any personal data or
identifiable information.
Only RJ Labs has access to the Survey Monkey platform.
At the close of the survey period summary statistics are computed
and reported to the Sustainable Tarras Society (these are reported in
the Results section below).
Analysis
Methods:
The two-stage cluster design informs the analysis strategy. We used
the {survey} package in R. This packages specialises in
the analysis of complex survey data. When computing the margin of error
we utilised a finite population correction factor.
Further details on the {survey} package can be found here.
For the more technical a worked example can be found here.
Software:
R Packages:
{tidyverse, chron} for data wrangling
{survey} for survey analysis
{ggplot2, patchwork} for plots
{ggmap} for maps
Analyst
Survey design and analysis was undertaken by a professional
statistician at RJ Labs.
