Geospatial Case Study - Mapping Open Defectation Prevalence

---

In this case study, we learn how to explore crowdsourced data to estimate a prevalence of events reported by residents of a city municipality. The example we are using is open defecation (Human or Animal Waste) on the streets of San Francisco.

Before we begin, let us pause for a moment and put our “public health caps” on to better understand why we are interested and even care about open defecation in the first place. Open defectation poses threats to human health via the fecal-oral route. Prevention of disease acquired via the fecal-oral route is one of the main reason humans have developed sanitation systems over the centuries. Universal access to sanitation systems in the urban United States has become a resurgent issue, and the root of this problem can be attributed mostly to lack of appropriate housing and public sanitation infrastructure for those who are experiencing homelessness.

So while we explore these data and look to understand the events better, please consider that we are evaluating the symptoms of a deeply rooted social disease: the lack of housing for everyone. With that in mind, the fruit of this analysis hopes to triage the indignity of open defecation by humans in urban areas, providing insight into articluating evidence-based practices and deployment of sanitary interventions across an urban area.

# install pacman if needed, install and load libraries
if (!require("pacman")) install.packages("pacman")

pacman::p_load(
  RSocrata, tidyverse, lubridate, ggplot2, leaflet, sf, 
  tmap, rgdal, prettydoc, hrbrthemes, raster, sp, spdep, 
  spatstat, fields, car, pgirmess, geoR)

# switch off scientific notation
options(scipen = 99)

# library(viridis)?

# set tmap to view mode 
tmap_mode("view")

A Note about accessing 311 data from API:

It is possible to download a current dataset from website, as well as use an Rsocrata package to import updated data directly from an API source. Here is what the code to do that looks like:

data_from_api <- RSocrata::read.socrata(“https://data.sfgov.org/resource/vw6y-z8j6.json”, stringsAsFactors = FALSE)

And here is the website associated with the 311 data, which is updated daily with new reports. We need to download the CSV file from the website and load into our local environment to proceed. Keep in mind the file is large. Downloading and reading in to memory is intensive and can take some time.

Once you set your directory to where the .csv file is, you are set up and ready to go with the rest of the code.

# set your directory and import data. 
path <- "where_the_downloaded_csv_is"
setwd(path)

# read in data
report_data <- read_csv("311_Cases.csv")

# organize column names for legibility
report_data<- rename_with(report_data, ~ tolower(gsub(" ", "_", .x, fixed =TRUE)))

# evaluate imported object
dim(report_data)

## [1] 497973     47

Subsetting data:

Now that we’ve imported our data object, we can run a few lines of code to specify the kind of reports we are interested in.

hum_anim_waste <- report_data %>% 
  
  # looking for "Waste" related calls while ignoring "Medical Waste"
  filter(str_detect(request_type, "Waste") & 
                    request_type != "Medical Waste") %>% 
  
  # restructure date and time variable as date alone
  mutate(opened = as.character(opened)) %>% 
  mutate(opened = str_remove(opened, "(\\d+:\\d+:\\d+\\s\\w+)")) %>% 
  mutate(opened = as.character.Date(opened)) %>% 
  mutate(opened = lubridate::mdy(opened)) %>% 
  # watch out for extreme lat/longitude errors, or NA values in the coordinates column
  filter(latitude> 37) %>% 
  filter(longitude != 'NA'| latitude!= 'NA') %>% 
  
  # remove obvious duplicate values or animal care values upstream  
  filter(responsible_agency != "Duplicate" & 
         responsible_agency != "Animal Care") %>% 
  
  # finally make sure the latitude& longitude colums are treated as numeric values 
  mutate(latitude= as.numeric(latitude)) %>% 
  mutate(longitude = as.numeric(longitude)) 

# what is the date range of our reports?
range(hum_anim_waste$opened)

## [1] "2011-05-23" "2021-04-20"

Data Quality Part 1, Narrative Cues:

Once we get in closer to the kind of reports we are looking for, we run into a complicated data quality issue. The 311 calls we are wrangling up here are based upon the reports and complaints of city residents, not the events themselves. The good news is we have a column in the dataset called status_reports, where department of public works staff report on what they find when responding to a call.

# how many unique status notes values are there?
NROW(unique(hum_anim_waste$status_notes))

## [1] 2147

Here’s where we need to employ a really hands-on evaluation process. In order to improve the data quality of these reports, we must dig into that status_report variable and find the narrative cues that tell us if the observation is a non-event, and then remove those non-event calls from our analysis.

# create a list of character strings that are cues indicating a non-event
nar_cues <- c("Duplicate","nothing", "Nothing", "Case Transferred", 
              "Insufficient Information","gone", "no work needed ",
              "Unable to locate", "animal control", "not thing", 
              "does not match", "not see any", "Unable to Locate",
              "Case is Invalid", "noting", "see anything",
              "dont see any",
              "Not thing", "not at", "no poop", "see this", 
              "wasnt there","looked both", "Duplicate", 
              "Animal Care",
              "no feces", "Unable To Locate", "not locate feces",
              "No feces", "insufficient information", 
              "does not exist", 
              "didnt see any", "nothng", "WASTE NOT FOUND", 
              "not sure where", 
              "there is not", "did not find", "DUPLICATE",
              "already removed", "No encampments", "nohing here",
              "Cancelled", "dup", "duplicate", "incomplete", 
              "no human waste", "no bird found", "in progress",
              "no dead rat", "no human feces","invalid address", 
              "no debris in the area", "NOTHING FOUND", 
              "TRANS TO RECO",
              "Cancel per 311", "not remove homeless", 
              "INCORRECT CATEGORY", "Location not entered", 
              "No human waste found", "NO HUMAN WASTE", "not there",
              "no items visable", "GRAFFITI", "graffiti", 
              "didnt see piles", "recology", "Recology",
              "theres birds", "no encampment",
              "this stuff", "Animal Care", "nothin here", "debris", 
              "thats garbage", "does not have any feces", 
              "loose garbage","rat removed",  "no waste back", 
              "NOTHING BUT TRASH",  "unable to find", "not find any",
              "nor did i see", "any feces", "and nothim", 
              "couldn't find", "could not find", "wrong address",
              "Abandon Vehicles", "ntohing found", "no poo", 
              "vomit", "no pile of poo", "personal belonging",
              "claimed", "needles", "cant locate", "Trash", 
              "dog poop", "trash",
              "items", "glass", "Dup", "nothing", "uable to locate")

# add on new entries as needed.

# function to remove any observations that contain strings that match "nar_cues" 
hum_anim_waste_clean <- 
  hum_anim_waste %>% 
filter(!(str_detect(status_notes, 
                  paste(nar_cues, collapse = "|"))))

Okay! So we have imported our dataset, located observations of interest, cleaned up the data a bit, created a list of character strings and purged those observations from our dataset. (this last part is really an ongoing process, but we have done what we can for now.) Next we will begin to map these data, exploring different ways to adjust and look at crowdsourced reports like these, as well as which approach is most helpful to look at crowdsourced “Human and Animal Waste” in this case.

Mapping, Part 1:

Let us pull out a month of reports and see what that looks like.

# reports from January 2020
jan_calls <- hum_anim_waste_clean %>% 
  filter(opened >= "2020-01-01" & 
           opened < "2020-02-01") 

# format as simple feature object
map_jan_calls <-  st_as_sf(jan_calls, coords 
                       = c('longitude','latitude'), crs = 4326)

# small fix needed if using tidyverse with tmap package. 
map_jan_calls_fix = st_as_sf(as.data.frame(map_jan_calls))

# quickmap
qtm(map_jan_calls_fix, dots.col = "red", symbols.size = .002)

Wow that is a lot of “Human or Animal Waste” reports in January of 2020! Could it really be so intense on the streets of San Francisco? Well maybe, yes.. But let us take our data quality methods a step further to make more sense of the all the reports we have crowdsourced.

Data Quality Part 2, Distinct vs. Duplicated:

Now that we have seen how “loud” the mapped points can be on their own, let us explore two very different ways of reducing the noise to signal ratio of these reports. The first approach is to remove everything but distinct values: only one unique address, latitude, and longitude coordinate per event in a given timeframe.

# create distinct observations only 
jan_distinct_calls = jan_calls  %>% 
  distinct(address, .keep_all = TRUE) %>% 
      distinct(latitude, .keep_all = TRUE) %>% 
        distinct(longitude, .keep_all = TRUE) 

map_jan_distinct_calls <-  st_as_sf(jan_distinct_calls, coords 
                       = c('longitude','latitude'), crs = 4326)

# small fix needed if using tidyverse with tmap package. 
map_jan_distinct_calls_fix = st_as_sf(as.data.frame(map_jan_distinct_calls))

# quickmap
qtm(map_jan_distinct_calls_fix, dots.col = "red", symbols.size = .002)

# what percentage of calls have we removed via distinct?
round((nrow(map_jan_calls) - 
         nrow(map_jan_distinct_calls)) / nrow(map_jan_calls),2)

## [1] 0.15

Okay, this does not look like it has changed much from just looking at the map. We removed a small percentage of our calls by only looking at unique places where the event has occured. This approach may be helpful for certain types of analyses, especially useful when looking at short time intervals or other kinds of events that have a longer biological or social duration. Now let us try a different approach where we are only interested in points that have identical latitude and longitude coordinates instead.

# adjust for duplicated values only in the latitude column 
dupl_lat <- duplicated(jan_calls$latitude)

# create new object with only those duplicated latitude values 
lat_select <- jan_calls[dupl_lat, ]

# adjusting for duplicated values in the longitude column, from the lat_select object
dupl_longitude <- duplicated(lat_select$longitude)

# create a new object with only those duplicated latitude and longitude values. 
jan_dupl_calls <- lat_select[dupl_longitude, ]

# create a simple features map object
map_jan_dupl <-  st_as_sf(jan_dupl_calls, coords 
                       = c('longitude','latitude'), crs = 4326)

# small fix needed if using tidyverse with tmap package. 
map_jan_dupl_fix = st_as_sf(as.data.frame(map_jan_dupl))

# quickmap
m1 <- qtm(map_jan_dupl_fix, dots.col = "red", symbols.size = .002)
m1 

What a difference the duplicated approach makes! Almost all of our points are gone, and we are left with these hotspot locations.

In the the case of Human or Animal Waste, looking at where events have occured in the same place could help explain social patterns as well as adjust some potential randomness associated with human behavior. Most importantly, knowing a certain area, a hotspot where events occur over time would allow us to provide a more thoughtful mobile interventions to vulnerable populations. We must consider that duplicated() ignores wider areas in favor of these impacted hotspots. In other contexts, and with different sorts of reported events distinct(), or even another functional approach may work better. There is always a trade off of some sort. At the end of the day, using duplicated() seems to be more useful given the nature ofevents we are observing here.

# what percentage of calls have we removed via the duplicated approach?
round((nrow(map_jan_calls) - 
         nrow(jan_dupl_calls)) / nrow(map_jan_calls),3)

## [1] 0.996

When we expand our time interval of observations, the duplicated() approach scales well if we are looking at studying areas over longer periods of time. Let us do that now.

# set time to 6 months. 
jan_july_calls <- hum_anim_waste_clean %>% 
  filter(opened >= "2020-01-01" & 
           opened < "2020-07-01")

# duplicated calls  
jan_july_dupl_lat <- duplicated(jan_july_calls$latitude)
jan_july_lat_select <- jan_july_calls[jan_july_dupl_lat, ]
jan_july_dupl_longitude <- duplicated(jan_july_lat_select$longitude)
jan_july_dupl_calls <- jan_july_lat_select[jan_july_dupl_longitude, ]

# repeat code for 3 and 6 month intervals
jan_march_dupl_calls <- jan_july_dupl_calls %>% 
    filter(opened >= "2020-01-01" & 
           opened < "2020-04-01")

map_jan_march_waste_dupl <-  st_as_sf(jan_march_dupl_calls, coords 
                       = c('longitude','latitude'), crs = 4326)
map_jan_march_waste_dupl_fix = st_as_sf(as.data.frame(map_jan_march_waste_dupl))


# repeat code for 3 and 6 month intervals
jan_june_dupl_calls <- jan_july_dupl_calls %>% 
    filter(opened >= "2020-01-01" & 
           opened < "2020-07-01")

map_jan_june_waste_dupl <-  st_as_sf(jan_june_dupl_calls, coords 
                       = c('longitude','latitude'), crs = 4326)
map_jan_june_waste_dupl_fix = st_as_sf(as.data.frame(map_jan_june_waste_dupl))

# quickmap 2 and 3
m2 <- qtm(map_jan_march_waste_dupl_fix, dots.col = "red", symbols.size = .002)
m3 <- qtm(map_jan_june_waste_dupl_fix, dots.col = "red", symbols.size = .002)

# arrange at 1, 3, and 6 month intervals.
tmap_arrange(m1, m2, m3)

# toggle out the first map to get an even view of the events. 

Moving left to right at 1, 3, & 6 month interval maps, it seems like there is a clear shape visible, in least more focused than when using the distinct()approach.

Data Quality, Part 3: Participation Bias

As we start thinking about analyses that invlove a longer timeframe of observations, we need to remind ourselves that we are sourcing data from a shifting population, namely San Francisco residents who participate in 311 reporting on a given day. There are differences in contributions to the data source itself, as we can observe in the scatteplot below.

# organize data for chart 
plot_part <- report_data %>% 
  dplyr::select(opened) %>% 
  mutate(opened = as.character(opened)) %>% 
  mutate(opened= str_remove(opened, "(\\d+:\\d+:\\d+\\s\\w+)")) %>% 
  mutate(opened = lubridate::mdy(opened)) %>% 
  filter(opened >= "2019-01-01" & 
           opened < "2021-01-01") %>% 
  group_by(opened) %>% 
  summarise(opened, n = n()) %>% 
  distinct(opened, .keep_all = TRUE) 

# plot chart
ggplot(plot_part, aes(x= opened, y = n, group = 1))+
  geom_point(color = 'steelblue', size = .5, alpha = .5)+
  geom_smooth(method=lm , color="red", fill="#69b3a2", se=TRUE, aplha = .3) +
  ggtitle("Documented 311 reports in San Francisco") +
  ylab("Frequency of total reports") +
  xlab(" ") +
  theme_ipsum()

What can we make of this scatterplot of aggregate 311 reports, circa 2019-2020?

Reporting changes, system updates, and trends over time can bring up a host of interesting data quality challenges with crowdsourced 311 data. An even greater limitation to an analysis frameworks is spatial participation bias: where certain areas, communities, individuals, and populations may not participate in 311 reporting for one reason or another. Differences in participation to a reporting system like 311 can distort what we see and don’t see from the available data. However possible it may be to evaluate and chacterize users on a 311 system administrator level, external researchers have only the event reports themselves to work from.

Now that we have gotten a handle on our data, explored cleaning, as well as some limitations around our output, we can look at two differenct statistical analyses: one approach that usesdistinct(), and another that uses the duplicated() data structures.

Statistical Analyses, Part 1: Density Plotting.

Looking at hundreds and hundreds of coordinates together on a map will overwhelm the eyes of any human being. Density Plotting is a tool we can use to explore large amounts of coordinate data to get a stronger visual sense of the distribution of events in space.

To begin, we will use a Shapefile of San Francisco to help us establish a coordinate reference system for our density plot.

# import shapefile of San Francisco neighborhoods
sf_nhoods <- readOGR(dsn = "Analysis Neighborhoods", 
    layer = "geo_export_dfbc5470-1876-41c8-90fb-f3d80661c272")

## OGR data source with driver: ESRI Shapefile 
## Source: "/Users/avery/Desktop/geospatial_casestudy/Analysis Neighborhoods", layer: "geo_export_dfbc5470-1876-41c8-90fb-f3d80661c272"
## with 195 features
## It has 5 fields

# remove Treasure Island for spatial contiguity
sf_nhoods <- sf_nhoods[sf_nhoods$nhood != "Treasure Island", ]

# plot shapefile 
plot(sf_nhoods, main = "San Francisco Neighborhood Boundaries.") 

Next we will format our data as a spatial object, create a window to calculate our density from, and calculate the density of points within that window.

# coerce our distinct() points into a spatial data frame object
jan_waste_spdf <- sp::SpatialPointsDataFrame(coords = 
            jan_distinct_calls[,c("longitude", "latitude")],
            data = jan_distinct_calls[,c("neighborhood")],
             proj4string = CRS("+init=epsg:4326"))

# create a window object with the range of distinct() points.  
sf_owin <- owin(xrange=range(jan_distinct_calls$longitude),
                yrange=range(jan_distinct_calls$latitude))

# create point pattern density object
sf_ppp <- ppp(jan_distinct_calls$longitude, 
              jan_distinct_calls$latitude, 
              window = sf_owin)

# create a raster layer from the point pattern density object.
density_raster <- 
  raster(density(sf_ppp, bw.ppl), crs = crs(sf_nhoods))

# set a palette to define a color range for mapping.
raster_pal <- colorNumeric(palette = tim.colors(), 
   domain = values(density_raster), na.color = NA)

leaflet() %>% addProviderTiles("CartoDB.Positron") %>%
  addPolygons(data=sf_nhoods,
              fillOpacity=0, opacity = .1, 
              weight = 1, color = "black") %>% 
  addRasterImage(density_raster, opacity=0.5, col = raster_pal)

Here we are with a density plot of events from January 2020. To do this correctly, we needed to use the distinct() approach. Otherwise we would have more than one point per coordinate and our density function would be distorted.

Statistical Analyses, Part 2: LISA

Positive spatial autocorrelation is the tendency for areas or sites that are close together to have similar values. A common statistical method to detect spatial autocorrelation is the Moran’s I test. Based on the princpals of a Moran’s I test, Local Indicators of Spatial Autocorrelation (LISAs) can be used to understand how each coordinate itself gives an indication of the extent of spatial clustering of similar values in proximity.

Let us walkthrough how to conduct a LISA and map our results with this type of 311 data. So far we have been looking at events alone, single points on our maps, essentially x and y points. Now we need to generate z values associated with our points to be able to determine if spatial autocorrelation is detectable. To do that we well expand upon the duplicated() approach we used earlier, but this time we count the events at each point to measure the intensity of events a at particular coordinate over the year.

# subset a year's worth of reports
year_waste <- hum_anim_waste %>%   
filter(opened >= "2020-01-01" & 
           opened < "2021-01-01") 

# count the events if more than one occured at a certain point 
dup_waste <- year_waste %>% 
  dplyr::select(opened, latitude, longitude) %>% 
  group_by(latitude, longitude) %>% 
  mutate(events = n()) %>% 
  filter(events > 1) %>% 
  distinct(latitude, .keep_all = T) %>% 
  distinct(longitude, .keep_all = T)

head(dup_waste)

## # A tibble: 6 x 4
## # Groups:   latitude, longitude [6]
##   opened     latitude longitude events
##   <date>        <dbl>     <dbl>  <int>
## 1 2020-08-26     37.8     -122.      2
## 2 2020-08-26     37.8     -122.      2
## 3 2020-03-03     37.8     -122.     10
## 4 2020-08-27     37.8     -122.      2
## 5 2020-10-10     37.8     -122.      6
## 6 2020-07-14     37.8     -122.      3

# set palette to map events
pal = colorNumeric("Oranges", dup_waste$events)

# map duplicate coordinate event
leaflet(dup_waste) %>% addProviderTiles("CartoDB.Positron") %>% 
  addCircleMarkers(~longitude, ~latitude, fillOpacity=.5,
  fillColor= ~pal(events), radius=~events/2, stroke=TRUE, weight=1) %>% 
  addLegend(pal = pal, values = ~events)

This map is an abstract sort of visualization, but helps us evaluate how each of these duplicated() coordinates are impacted over a year’s time.

Next we will create an inverse distance matrix to from the coordinate of all these points together.

# take a log odds of the event intesity 
dup_waste$log_odds <- car::logit(dup_waste$events)

# create matrix
event_dists <- as.matrix(dist(cbind(dup_waste$longitude, 
                        dup_waste$latitude)))

# divide by 1 to invert all the values
event_dists_inv <- 1 / event_dists

# be sure to insert a 0 down the middle! 
diag(event_dists_inv) <- 0 

# calculate maximum distance in matrix 
max_dist <- max(dist(cbind(dup_waste$longitude, 
                        dup_waste$latitude)))

With that matrix put together, we can begin to look at how distance effects positive spatial autocorrelation.

# create an array with the latitude and longitude points
xy <- cbind(dup_waste$longitude, 
         dup_waste$latitude)

# create a correlogram
pgi_cor <- pgirmess::correlog(coords=xy, 
                    z=dup_waste$log_odds, 
                    method="Moran", nbclass=10)

# plot correlogram
plot(pgi_cor, main = "Bins of Spatial Cluster Distance, degrees")

The 2 red dots indicate statistcally significant values, which help us explain at what distance postive spatial autocorrelation has a good chance of happening. The values we get here are .01 and .02 degrees, or about 1 mile or so apart.

Once we have observable evidence that PSA exists in these data, we can assign weights to each point and run a local moran’s I on each point together and have a look at the coefficients. Be warned, the printCoefmat function is stubbornly verbose. Just keep scrolling! Feel free to check out those significant p-values indicating evidence of PSA at each point along the way if you like.

# set coordinates to create a spatial object
coords <- coordinates(xy)

# set row names for neighbors list.
matrix_ids <- row.names(as.data.frame(coords))

# create neighbours list
neigh_nb <- knn2nb(knearneigh(coords, k=1, 
                              longlat = TRUE), 
                              row.names= matrix_ids) 

# returns the distance between nearest neighbors for each point
dists <- unlist(nbdists(neigh_nb,coords)) 

# returns maximum distance between nearest neighbors for each point
max_one_nn <- max(dists)

# set spatial coordiates 
neigh_kd_one <- dnearneigh(coords, d1=0, d2=max_one_nn, row.names=matrix_ids)

# set weights 
weights <- nb2listw(neigh_kd_one, 
# style the weights to sum over all links                 
                    style="W")   

# run local moran function with the log_odds of events
lisa <-localmoran(dup_waste$log_odds, weights)                      

# print coefficients for each point 
Coef <- stats::printCoefmat(data.frame(lisa[matrix_ids,], row.names=row.names(coords), check.names=FALSE))

##                 Ii        E.Ii      Var.Ii    Z.Ii             Pr(z > 0)    
##   [1,] -0.11057522 -0.00112613  0.00253820 -2.1724             0.9850890    
##   [2,]  0.11774304 -0.00112613  0.02891449  0.6991             0.2422588    
##   [3,]  0.53521804 -0.00112613  0.00204105 11.8718 < 0.00000000000000022 ***
##   [4,] -0.03279781 -0.00112613  0.01127880 -0.2982             0.6172332    
##   [5,]  0.23813154 -0.00112613  0.00194429  5.4261 0.0000000288036781541 ***
##   [6,]  0.03397467 -0.00112613  0.00263440  0.6839             0.2470274    
##   [7,] -0.03938162 -0.00112613  0.01487372 -0.3137             0.6231172    
##   [8,]  0.16670224 -0.00112613  0.00188904  3.8614 0.0000563687828742679 ***
##   [9,] -0.17533058 -0.00112613  0.02248202 -1.1618             0.8773472    
##  [10,]  0.23167343 -0.00112613  0.00209177  5.0901 0.0000001789513496476 ***
##  [11,]  0.44979102 -0.00112613  0.00223164  9.5452 < 0.00000000000000022 ***
##  [12,]  0.03922142 -0.00112613  0.00232397  0.8370             0.2013090    
##  [13,] -0.17741062 -0.00112613  0.00921504 -1.8364             0.9668502    
##  [14,]  0.50666090 -0.00112613  0.00237208 10.4260 < 0.00000000000000022 ***
##  [15,]  0.47901366 -0.00112613  0.00231215  9.9853 < 0.00000000000000022 ***
##  [16,]  0.09288701 -0.00112613  0.03191632  0.5262             0.2993615    
##  [17,]  0.50411533 -0.00112613  0.00206114 11.1287 < 0.00000000000000022 ***
##  [18,] -0.09210095 -0.00112613  0.06965349 -0.3447             0.6348426    
##  [19,] -0.18254378 -0.00112613  0.02429631 -1.1639             0.8777643    
##  [20,]  0.00821215 -0.00112613  0.00344033  0.1592             0.4367522    
##  [21,] -0.10989978 -0.00112613  0.00238432 -2.2276             0.9870472    
##  [22,]  0.06871945 -0.00112613  0.01176123  0.6440             0.2597749    
##  [23,]  0.02389934 -0.00112613  0.00537080  0.3415             0.3663719    
##  [24,]  0.09424652 -0.00112613  0.00348240  1.6162             0.0530298 .  
##  [25,]  0.22562302 -0.00112613  0.00375217  3.7017             0.0001071 ***
##  [26,]  0.03329281 -0.00112613  0.00295612  0.6330             0.2633514    
##  [27,]  0.22380932 -0.00112613  0.00230041  4.6898 0.0000013673021695669 ***
##  [28,] -0.07803772 -0.00112613  0.00218732 -1.6445             0.9499643    
##  [29,]  0.22810786 -0.00112613  0.00253820  4.5500 0.0000026816660954535 ***
##  [30,]  0.57176732 -0.00112613  0.00276582 10.8934 < 0.00000000000000022 ***
##  [31,]  0.18506800 -0.00112613  0.00708307  2.2124             0.0134710 *  
##  [32,] -0.01402465 -0.00112613  0.00634441 -0.1619             0.5643220    
##  [33,]  0.31347111 -0.00112613  0.00213354  6.8109 0.0000000000048496274 ***
##  [34,] -0.52517885 -0.00112613  0.49419676 -0.7455             0.7720035    
##  [35,] -0.33569448 -0.00112613  0.02429631 -2.1464             0.9840803    
##  [36,] -0.00377453 -0.00112613  0.00798437 -0.0296             0.5118226    
##  [37,]  0.14302066 -0.00112613  0.02429631  0.9248             0.1775422    
##  [38,]  0.10609554 -0.00112613  0.03085115  0.6104             0.2707834    
##  [39,] -0.37700117 -0.00112613  0.02566928 -2.3460             0.9905131    
##  [40,]  0.25635548 -0.00112613  0.00242156  5.2324 0.0000000836718189558 ***
##  [41,] -0.02474961 -0.00112613  0.00833058 -0.2588             0.6021149    
##  [42,] -0.04760027 -0.00112613  0.00348240 -0.7875             0.7845168    
##  [43,] -0.11090637 -0.00112613  0.00194429 -2.4897             0.9936071    
##  [44,]  0.18113503 -0.00112613  0.00806852  2.0291             0.0212255 *  
##  [45,]  0.03740830 -0.00112613  0.00245960  0.7770             0.2185816    
##  [46,]  0.02110250 -0.00112613  0.00651629  0.2754             0.3915170    
##  [47,]  0.04922136 -0.00112613  0.01597561  0.3983             0.3451914    
##  [48,]  0.36630339 -0.00112613  0.49419676  0.5227             0.3006034    
##  [49,] -0.51049127 -0.00112613  0.02366131 -3.3114             0.9995358    
##  [50,]  0.03287708 -0.00112613  0.00314779  0.6061             0.2722369    
##  [51,]  0.24855640 -0.00112613  0.00237208  5.1265 0.0000001475633424409 ***
##  [52,]  0.01396592 -0.00112613  0.00493653  0.2148             0.4149611    
##  [53,]  0.30124748 -0.00112613  0.00232397  6.2723 0.0000000001778459437 ***
##  [54,]  0.02161919 -0.00112613  0.00413756  0.3536             0.3618170    
##  [55,] -0.01632211 -0.00112613  0.02429631 -0.0975             0.5388312    
##  [56,]  0.04076544 -0.00112613  0.00235992  0.8623             0.1942507    
##  [57,] -0.15534999 -0.00112613  0.02496473 -0.9761             0.8354891    
##  [58,] -0.00625509 -0.00112613  0.00335845 -0.0885             0.5352617    
##  [59,] -0.01050556 -0.00112613  0.00541340 -0.1275             0.5507197    
##  [60,]  0.38179627 -0.00112613  0.00226567  8.0447 0.0000000000000004321 ***
##  [61,]  0.02689032 -0.00112613  0.00333844  0.4849             0.3138780    
##  [62,]  0.17672988 -0.00112613  0.00377609  2.8943             0.0018999 ** 
##  [63,] -0.28354143 -0.00112613  0.00978201 -2.8554             0.9978512    
##  [64,]  0.02864239 -0.00112613  0.00207129  0.6541             0.2565271    
##  [65,] -0.30699481 -0.00112613  0.04392360 -1.4594             0.9277779    
##  [66,]  0.06638427 -0.00112613  0.02429631  0.4331             0.3324666    
##  [67,] -0.10010181 -0.00112613  0.00238432 -2.0270             0.9786671    
##  [68,]  0.06234132 -0.00112613  0.00563507  0.8455             0.1989224    
##  [69,] -0.04478021 -0.00112613  0.01513564 -0.3548             0.6386428    
##  [70,]  0.21302551 -0.00112613  0.02891449  1.2594             0.1039430    
##  [71,]  0.03332752 -0.00112613  0.00218732  0.7367             0.2306583    
##  [72,]  0.06638427 -0.00112613  0.02429631  0.4331             0.3324666    
##  [73,] -0.10426155 -0.00112613  0.00247246 -2.0742             0.9809680    
##  [74,] -0.31254631 -0.00112613  0.02366131 -2.0245             0.9785429    
##  [75,]  0.03564024 -0.00112613  0.00237208  0.7549             0.2251561    
##  [76,]  0.04074030 -0.00112613  0.00237208  0.8596             0.1950020    
##  [77,]  0.21610045 -0.00112613  0.00287460  4.0516 0.0000254367609843498 ***
##  [78,] -0.08634194 -0.00112613  0.00295612 -1.5673             0.9414805    
##  [79,] -0.10455125 -0.00112613  0.00339902 -1.7740             0.9619670    
##  [80,] -0.05732003 -0.00112613  0.00550029 -0.7577             0.7756842    
##  [81,]  0.11866329 -0.00112613  0.00634441  1.5039             0.0663019 .  
##  [82,] -0.09676081 -0.00112613  0.00255164 -1.8932             0.9708371    
##  [83,] -0.01892437 -0.00112613  0.03427489 -0.0961             0.5382940    
##  [84,]  0.03405521 -0.00112613  0.02305728  0.2317             0.4083892    
##  [85,]  0.02072729 -0.00112613  0.01083124  0.2100             0.4168412    
##  [86,] -0.02806912 -0.00112613  0.00782062 -0.3047             0.6196901    
##  [87,] -0.09782730 -0.00112613  0.00297283 -1.7736             0.9619323    
##  [88,] -0.03572006 -0.00112613  0.00486376 -0.4960             0.6900658    
##  [89,] -0.13481899 -0.00112613  0.00213354 -2.8944             0.9981005    
##  [90,]  0.03047019 -0.00112613  0.00541340  0.4294             0.3338019    
##  [91,]  0.41431290 -0.00112613  0.00220933  8.8385 < 0.00000000000000022 ***
##  [92,]  0.16055784 -0.00112613  0.01911263  1.1695             0.1210976    
##  [93,]  0.10866684 -0.00112613  0.00214416  2.3711             0.0088681 ** 
##  [94,]  0.17262033 -0.00112613  0.00227718  3.6410             0.0001358 ***
##  [95,]  0.13509473 -0.00112613  0.00682109  1.6494             0.0495365 *  
##  [96,]  0.02946545 -0.00112613  0.00524629  0.4224             0.3363836    
##  [97,] -0.09429307 -0.00112613  0.00281170 -1.7570             0.9605429    
##  [98,]  0.03565458 -0.00112613  0.00243414  0.7455             0.2279852    
##  [99,]  0.23811371 -0.00112613  0.00200162  5.3474 0.0000000446121864624 ***
## [100,]  0.03798207 -0.00112613  0.00244683  0.7906             0.2145837    
## [101,]  0.02434676 -0.00112613  0.02641298  0.1567             0.4377263    
## [102,] -0.03299465 -0.00112613  0.00486376 -0.4570             0.6761492    
## [103,]  0.04196588 -0.00112613  0.02248202  0.2874             0.3869050    
## [104,] -0.09602126 -0.00112613  0.00228875 -1.9836             0.9763473    
## [105,]  0.26519638 -0.00112613  0.09795638  0.8509             0.1974054    
## [106,]  0.12250573 -0.00112613  0.00359108  2.0631             0.0195520 *  
## [107,]  0.04783155 -0.00112613  0.00234786  1.0104             0.1561565    
## [108,] -0.13923164 -0.00112613  0.32909660 -0.2407             0.5951219    
## [109,] -0.08027877 -0.00112613  0.00292312 -1.4640             0.9284036    
## [110,] -0.09425666 -0.00112613  0.01487372 -0.7636             0.7774558    
## [111,]  0.02103054 -0.00112613  0.00657536  0.2732             0.3923343    
## [112,]  0.03927249 -0.00112613  0.02429631  0.2592             0.3977494    
## [113,]  0.14398721 -0.00112613  0.07509636  0.5295             0.2982158    
## [114,]  0.13920331 -0.00112613  0.00422210  2.1597             0.0153996 *  
## [115,]  0.25734092 -0.00112613  0.02891449  1.5200             0.0642538 .  
## [116,] -0.11606923 -0.00112613  0.00269169 -2.2155             0.9866369    
## [117,] -0.02942956 -0.00112613  0.00316612 -0.5030             0.6925210    
## [118,]  0.19662463 -0.00112613  0.00238432  4.0498 0.0000256279812562614 ***
## [119,] -0.10112146 -0.00112613  0.00361343 -1.6635             0.9518927    
## [120,]  0.22516692 -0.00112613  0.00231215  4.7061 0.0000012623644652533 ***
## [121,]  0.02238653 -0.00112613  0.00472335  0.3421             0.3661308    
## [122,] -0.07791348 -0.00112613  0.00520584 -1.0643             0.8563925    
## [123,]  0.23581337 -0.00112613  0.00305847  4.2844 0.0000091636412561734 ***
## [124,] -0.11911274 -0.00112613  0.00300666 -2.1517             0.9842912    
## [125,]  0.02296699 -0.00112613  0.00462230  0.3544             0.3615287    
## [126,] -0.06370037 -0.00112613  0.08144636 -0.2193             0.5867763    
## [127,] -0.06370037 -0.00112613  0.08144636 -0.2193             0.5867763    
## [128,] -0.15204269 -0.00112613  0.01265463 -1.3416             0.9101317    
## [129,]  0.13150766 -0.00112613  0.00247246  2.6674             0.0038219 ** 
## [130,] -0.13797861 -0.00112613  0.02566928 -0.8542             0.8034954    
## [131,]  0.03047120 -0.00112613  0.00701596  0.3772             0.3530012    
## [132,]  0.13845640 -0.00112613  0.02641298  0.8589             0.1952089    
## [133,] -0.08540465 -0.00112613  0.00220933 -1.7930             0.9635156    
## [134,] -0.01609418 -0.00112613  0.00628883 -0.1887             0.5748545    
## [135,] -0.09706895 -0.00112613  0.00281170 -1.8094             0.9648033    
## [136,] -0.00261780 -0.00112613  0.00344033 -0.0254             0.5101447    
## [137,]  0.22417730 -0.00112613  0.00348240  3.8179 0.0000672867983297604 ***
## [138,]  0.21911415 -0.00112613  0.03852032  1.1222             0.1308989    
## [139,]  0.32657110 -0.00112613  0.00253820  6.5044 0.0000000000389917998 ***
## [140,] -0.08200373 -0.00112613  0.00215484 -1.7423             0.9592712    
## [141,] -0.73443493 -0.00112613  0.02366131 -4.7672             0.9999991    
## [142,] -0.02901204 -0.00112613  0.01015311 -0.2767             0.6090134    
## [143,]  0.24544628 -0.00112613  0.00228875  5.1540 0.0000001274895130712 ***
## [144,]  0.36630339 -0.00112613  0.98949724  0.3694             0.3559244    
## [145,] -0.00552832 -0.00112613  0.00489993 -0.0629             0.5250725    
## [146,] -0.00736432 -0.00112613  0.00824158 -0.0687             0.5273919    
## [147,]  0.06984209 -0.00112613  0.01953380  0.5078             0.3058059    
## [148,] -0.09476695 -0.00112613  0.00335845 -1.6158             0.9469345    
## [149,] -0.11069407 -0.00112613  0.02305728 -0.7216             0.7647210    
## [150,] -0.10661643 -0.00112613  0.00262035 -2.0608             0.9803383    
## [151,]  0.07451484 -0.00112613  0.00486376  1.0846             0.1390487    
## [152,] -0.15024616 -0.00112613  0.00890236 -1.5805             0.9429992    
## [153,]  0.67252414 -0.00112613  0.00278099 12.7742 < 0.00000000000000022 ***
## [154,] -0.04042118 -0.00112613  0.00395037 -0.6252             0.7340801    
## [155,]  0.11978643 -0.00112613  0.00222045  2.5660             0.0051444 ** 
## [156,] -0.02953974 -0.00112613  0.00640084 -0.3551             0.6387602    
## [157,] -0.04669935 -0.00112613  0.00628883 -0.5747             0.7172456    
## [158,] -0.03358680 -0.00112613  0.00554461 -0.4359             0.6685581    
## [159,] -0.08460308 -0.00112613  0.00214416 -1.8028             0.9642872    
## [160,]  0.02420612 -0.00112613  0.00675820  0.3081             0.3789851    
## [161,]  0.13848302 -0.00112613  0.00207129  3.0676             0.0010791 ** 
## [162,] -0.00383055 -0.00112613  0.00806852 -0.0301             0.5120094    
## [163,]  0.12945016 -0.00112613  0.00242156  2.6535             0.0039832 ** 
## [164,] -0.08113522 -0.00112613  0.00279629 -1.5130             0.9348644    
## [165,] -0.11377619 -0.00112613  0.00260640 -2.2065             0.9863267    
## [166,]  0.01288880 -0.00112613  0.00493653  0.1995             0.4209472    
## [167,]  0.00725429 -0.00112613  0.00890236  0.0888             0.4646123    
## [168,] -0.06757172 -0.00112613  0.05392967 -0.2861             0.6126079    
## [169,] -0.12384082 -0.00112613  0.00191641 -2.8032             0.9974700    
## [170,]  0.11725057 -0.00112613  0.10896306  0.3586             0.3599421    
## [171,] -0.08540585 -0.00112613  0.00815424 -0.9333             0.8246731    
## [172,]  0.14618265 -0.00112613  0.00260640  2.8854             0.0019545 ** 
## [173,] -0.05891181 -0.00112613  0.00462230 -0.8499             0.8023223    
## [174,]  0.12126405 -0.00112613  0.00377609  1.9917             0.0232017 *  
## [175,] -0.04900406 -0.00112613  0.00572802 -0.6326             0.7365045    
## [176,]  0.18424795 -0.00112613  0.00222045  3.9339 0.0000417808920435151 ***
## [177,]  0.04929349 -0.00112613  0.00395037  0.8022             0.2112197    
## [178,]  0.16622196 -0.00112613  0.00213354  3.6230             0.0001456 ***
## [179,]  0.15345560 -0.00112613  0.00304106  2.8031             0.0025303 ** 
## [180,] -0.10819478 -0.00112613  0.00238432 -2.1927             0.9858357    
## [181,] -0.08123362 -0.00112613  0.00197269 -1.8036             0.9643543    
## [182,] -0.03892196 -0.00112613  0.00634441 -0.4745             0.6824330    
## [183,]  0.33907884 -0.00112613  0.02429631  2.1826             0.0145333 *  
## [184,] -0.00141951 -0.00112613  0.00352526 -0.0049             0.5019713    
## [185,]  0.02670871 -0.00112613  0.00333844  0.4817             0.3149937    
## [186,] -0.04721763 -0.00112613  0.00375217 -0.7525             0.7741107    
## [187,]  0.06629271 -0.00112613  0.01246616  0.6038             0.2729782    
## [188,]  0.09885634 -0.00112613  0.03852032  0.5094             0.3052277    
## [189,]  0.14656446 -0.00112613  0.00311161  2.6476             0.0040527 ** 
## [190,] -0.01228169 -0.00112613  0.01083124 -0.1072             0.5426807    
## [191,] -0.04425804 -0.00112613  0.00663535 -0.5295             0.7017710    
## [192,] -0.01034866 -0.00112613  0.01083124 -0.0886             0.5353064    
## [193,] -0.00141311 -0.00112613  0.02429631 -0.0018             0.5007345    
## [194,] -0.10904953 -0.00112613  0.00249847 -2.1591             0.9845799    
## [195,]  0.03480670 -0.00112613  0.00218732  0.7683             0.2211520    
## [196,]  0.15145628 -0.00112613  0.00253820  3.0286             0.0012285 ** 
## [197,]  0.02376798 -0.00112613  0.00978201  0.2517             0.4006366    
## [198,]  0.08291268 -0.00112613  0.01193051  0.7694             0.2208289    
## [199,]  0.03656559 -0.00112613  0.00275076  0.7187             0.2361773    
## [200,]  0.09288701 -0.00112613  0.03191632  0.5262             0.2993615    
## [201,]  0.20162662 -0.00112613  0.00214416  4.3786 0.0000059713099219625 ***
## [202,]  0.03510337 -0.00112613  0.00222045  0.7689             0.2209910    
## [203,]  0.03399387 -0.00112613  0.00337864  0.6042             0.2728541    
## [204,]  0.01431921 -0.00112613  0.02429631  0.0991             0.4605336    
## [205,]  0.05844249 -0.00112613  0.00582356  0.7806             0.2175218    
## [206,]  0.03183609 -0.00112613  0.00220933  0.7013             0.2415669    
## [207,]  0.02925781 -0.00112613  0.01015311  0.3015             0.3815015    
## [208,]  0.24964146 -0.00112613  0.03699632  1.3037             0.0961605 .  
## [209,]  0.13633883 -0.00112613  0.00234786  2.8370             0.0022771 ** 
## [210,]  0.02574900 -0.00112613  0.02366131  0.1747             0.4306516    
## [211,] -0.11821029 -0.00112613  0.00292312 -2.1656             0.9848285    
## [212,] -0.09737741 -0.00112613  0.00307603 -1.7354             0.9586690    
## [213,] -0.11629145 -0.00112613  0.00259256 -2.2618             0.9881457    
## [214,] -0.05247242 -0.00112613  0.01462011 -0.4247             0.6644550    
## [215,] -0.04328347 -0.00112613  0.00326017 -0.7383             0.7698445    
## [216,]  0.37360688 -0.00112613  0.00206114  8.2541 < 0.00000000000000022 ***
## [217,]  0.04954952 -0.00112613  0.00682109  0.6136             0.2697459    
## [218,]  0.03163965 -0.00112613  0.03558521  0.1737             0.4310528    
## [219,]  0.03896498 -0.00112613  0.00446146  0.6002             0.2741802    
## [220,]  0.12139819 -0.00112613  0.00194429  2.7787             0.0027288 ** 
## [221,]  0.01064598 -0.00112613  0.00365879  0.1946             0.4228456    
## [222,]  0.03026283 -0.00112613  0.00289064  0.5838             0.2796704    
## [223,] -0.11100113 -0.00112613  0.01228278 -0.9914             0.8392557    
## [224,] -0.10557580 -0.00112613  0.00251162 -2.0842             0.9814271    
## [225,]  0.05633336 -0.00112613  0.00890236  0.6090             0.2712659    
## [226,]  0.07525719 -0.00112613  0.00921504  0.7957             0.2131030    
## [227,] -0.03234352 -0.00112613  0.01346394 -0.2690             0.6060490    
## [228,] -0.06370037 -0.00112613  0.08144636 -0.2193             0.5867763    
## [229,] -0.02882184 -0.00112613  0.00377609 -0.4507             0.6738985    
## [230,] -0.07469351 -0.00112613  0.00520584 -1.0196             0.8460464    
## [231,]  0.66033461 -0.00112613  0.00202121 14.7129 < 0.00000000000000022 ***
## [232,]  0.07043136 -0.00112613  0.02566928  0.4466             0.3275710    
## [233,] -0.13105749 -0.00112613  0.00219829 -2.7712             0.9972077    
## [234,] -0.50199549 -0.00112613  0.01159630 -4.6512             0.9999983    
## [235,]  0.11505196 -0.00112613  0.00222045  2.4655             0.0068412 ** 
## [236,] -0.07905596 -0.00112613  0.00205106 -1.7207             0.9573506    
## [237,]  0.01569321 -0.00112613  0.03085115  0.0958             0.4618566    
## [238,]  0.10509990 -0.00112613  0.07509636  0.3876             0.3491436    
## [239,]  0.14531222 -0.00112613  0.00851377  1.5871             0.0562493 .  
## [240,]  0.13575094 -0.00112613  0.00220933  2.9121             0.0017953 ** 
## [241,] -0.10877926 -0.00112613  0.00230041 -2.2445             0.9876006    
## [242,]  0.03921753 -0.00112613  0.02891449  0.2373             0.4062290    
## [243,]  0.07000105 -0.00112613  0.00370503  1.1685             0.1212965    
## [244,] -0.09879376 -0.00112613  0.00266283 -1.8927             0.9708003    
## [245,]  0.03536630 -0.00112613  0.00943459  0.3757             0.3535700    
## [246,]  0.36630339 -0.00112613  0.49419676  0.5227             0.3006034    
## [247,] -0.04230103 -0.00112613  0.07509636 -0.1503             0.5597175    
## [248,]  0.04733533 -0.00112613  0.00195370  1.0964             0.1364524    
## [249,]  0.02574900 -0.00112613  0.02366131  0.1747             0.4306516    
## [250,] -0.06999906 -0.00112613  0.00339902 -1.1813             0.8812646    
## [251,] -0.00551555 -0.00112613  0.01127880 -0.0413             0.5164840    
## [252,] -0.05768381 -0.00112613  0.01724075 -0.4307             0.6666707    
## [253,]  0.03841107 -0.00112613  0.00218732  0.8454             0.1989504    
## [254,]  0.38358417 -0.00112613  0.00235992  7.9193 0.0000000000000011946 ***
## [255,] -0.23628836 -0.00112613  0.01568613 -1.8776             0.9697840    
## [256,] -0.11521278 -0.00112613  0.01127880 -1.0742             0.8586436    
## [257,] -0.10528249 -0.00112613  0.00213354 -2.2549             0.9879315    
## [258,] -0.11006940 -0.00112613  0.00978201 -1.1015             0.8646615    
## [259,] -0.11519139 -0.00112613  0.00270629 -2.1926             0.9858332    
## [260,] -0.07831016 -0.00112613  0.00295612 -1.4196             0.9221380    
## [261,]  0.00195924 -0.00112613  0.00833058  0.0338             0.4865167    
## [262,]  0.09273223 -0.00112613  0.00497359  1.3309             0.0916147 .  
## [263,]  0.02532122 -0.00112613  0.00458939  0.3904             0.3481222    
## [264,]  0.03319139 -0.00112613  0.00419362  0.5299             0.2980789    
## [265,]  0.00104064 -0.00112613  0.00597200  0.0280             0.4888158    
## [266,]  0.07361602 -0.00112613  0.04196589  0.3649             0.3576108    
## [267,]  0.00945031 -0.00112613  0.00475783  0.1533             0.4390680    
## [268,]  0.12546086 -0.00112613  0.08144636  0.4436             0.3286801    
## [269,]  0.03047019 -0.00112613  0.00541340  0.4294             0.3338019    
## [270,] -0.08905131 -0.00112613  0.00645812 -1.0941             0.8630461    
## [271,]  0.06565995 -0.00112613  0.00516591  0.9292             0.1763909    
## [272,] -0.23323520 -0.00112613  0.02366131 -1.5089             0.9343433    
## [273,] -0.06514600 -0.00112613  0.00563507 -0.8528             0.8031247    
## [274,] -0.07926686 -0.00112613  0.00563507 -1.0409             0.8510496    
## [275,]  0.14302066 -0.00112613  0.02429631  0.9248             0.1775422    
## [276,] -0.08880677 -0.00112613  0.00282724 -1.6490             0.9504268    
## [277,] -0.02666265 -0.00112613  0.01002663 -0.2550             0.6006484    
## [278,]  0.31657948 -0.00112613  0.00327947  5.5478 0.0000000144622611923 ***
## [279,] -0.04168199 -0.00112613  0.00541340 -0.5512             0.7092559    
## [280,] -0.05070071 -0.00112613  0.00512648 -0.6924             0.7556531    
## [281,]  0.09995779 -0.00112613  0.00541340  1.3739             0.0847403 .  
## [282,] -0.11721288 -0.00112613  0.00273582 -2.2194             0.9867708    
## [283,] -0.28035377 -0.00112613  0.04606776 -1.3009             0.9033618    
## [284,]  0.13385775 -0.00112613  0.00607455  1.7319             0.0416450 *  
## [285,]  0.09367436 -0.00112613  0.01143554  0.8865             0.1876721    
## [286,] -0.05249051 -0.00112613  0.00572802 -0.6787             0.7513272    
## [287,]  0.03368653 -0.00112613  0.00262035  0.6801             0.2482281    
## [288,] -0.39803007 -0.00112613  0.04842633 -1.8036             0.9643544    
## [289,]  0.03529868 -0.00112613  0.00234786  0.7517             0.2261069    
## [290,]  0.03606735 -0.00112613  0.01210430  0.3381             0.3676580    
## [291,] -0.01744228 -0.00112613  0.00634441 -0.2048             0.5811528    
## [292,] -0.06677818 -0.00112613  0.00413756 -1.0206             0.8462895    
## [293,]  0.07148539 -0.00112613  0.00389932  1.1628             0.1224520    
## [294,]  0.20553723 -0.00112613  0.00350373  3.4914             0.0002403 ***
## [295,] -0.06454698 -0.00112613  0.00462230 -0.9328             0.8245464    
## [296,] -0.00162186 -0.00112613  0.00910867 -0.0052             0.5020722    
## [297,]  0.10170689 -0.00112613  0.03085115  0.5855             0.2791194    
## [298,] -0.18611840 -0.00112613  0.03085115 -1.0532             0.8538792    
## [299,]  0.31171387 -0.00112613  0.00210211  6.8233 0.0000000000044487253 ***
## [300,] -0.02635621 -0.00112613  0.00669629 -0.3083             0.6210805    
## [301,]  0.36630339 -0.00112613  0.32909660  0.6405             0.2609271    
## [302,]  0.03200708 -0.00112613  0.00216559  0.7120             0.2382350    
## [303,] -0.03733075 -0.00112613  0.00452470 -0.5382             0.7047914    
## [304,]  0.01114228 -0.00112613  0.00413756  0.1907             0.4243691    
## [305,]  0.14507262 -0.00112613  0.02305728  0.9628             0.1678220    
## [306,]  0.17951377 -0.00112613  0.08144636  0.6330             0.2633791    
## [307,]  0.02574900 -0.00112613  0.02366131  0.1747             0.4306516    
## [308,]  0.00244835 -0.00112613  0.02429631  0.0229             0.4908522    
## [309,]  0.12546086 -0.00112613  0.08144636  0.4436             0.3286801    
## [310,] -0.19409499 -0.00112613  0.02566928 -1.2044             0.8857877    
## [311,] -0.07424531 -0.00112613  0.00408268 -1.1443             0.8737607    
## [312,]  0.10509990 -0.00112613  0.07509636  0.3876             0.3491436    
## [313,] -0.04171140 -0.00112613  0.03191632 -0.2272             0.5898565    
## [314,] -0.08204414 -0.00112613  0.00204105 -1.7911             0.9633609    
## [315,]  0.11895950 -0.00112613  0.04606776  0.5595             0.2879136    
## [316,]  0.10928468 -0.00112613  0.00212300  2.3963             0.0082813 ** 
## [317,]  0.11562685 -0.00112613  0.05716693  0.4883             0.3126652    
## [318,]  0.21768365 -0.00112613  0.00255164  4.3317 0.0000073985306338618 ***
## [319,]  0.13259289 -0.00112613  0.00244683  2.7033             0.0034329 ** 
## [320,]  0.01179633 -0.00112613  0.00978201  0.1307             0.4480235    
## [321,]  0.11725057 -0.00112613  0.10896306  0.3586             0.3599421    
## [322,]  0.04033180 -0.00112613  0.00252486  0.8251             0.2046669    
## [323,]  0.03916233 -0.00112613  0.00264856  0.7828             0.2168592    
## [324,] -0.05156259 -0.00112613  0.00458939 -0.7445             0.7717142    
## [325,]  0.03007476 -0.00112613  0.00309375  0.5610             0.2874156    
## [326,] -0.01632211 -0.00112613  0.02429631 -0.0975             0.5388312    
## [327,]  0.10609554 -0.00112613  0.03085115  0.6104             0.2707834    
## [328,] -0.01781902 -0.00112613  0.02429631 -0.1071             0.5426424    
## [329,] -0.01722810 -0.00112613  0.02140994 -0.1100             0.5438133    
## [330,] -0.09447859 -0.00112613  0.00276582 -1.7751             0.9620563    
## [331,]  0.12213601 -0.00112613  0.05103318  0.5456             0.2926579    
## [332,]  0.08965374 -0.00112613  0.01210430  0.8251             0.2046503    
## [333,] -0.00871457 -0.00112613  0.24654652 -0.0153             0.5060967    
## [334,]  0.10969994 -0.00112613  0.00281170  2.0901             0.0183065 *  
## [335,]  0.03629533 -0.00112613  0.02193352  0.2527             0.4002587    
## [336,] -0.03360562 -0.00112613  0.00479272 -0.4692             0.6805213    
## [337,]  0.07824567 -0.00112613  0.03699632  0.4127             0.3399298    
## [338,]  0.03629533 -0.00112613  0.02193352  0.2527             0.4002587    
## [339,]  0.15986694 -0.00112613  0.06965349  0.6100             0.2709282    
## [340,] -0.06409081 -0.00112613  0.00344033 -1.0735             0.8584739    
## [341,] -0.10753660 -0.00112613  0.00266283 -2.0621             0.9804016    
## [342,] -0.10603824 -0.00112613  0.14041071 -0.2800             0.6102532    
## [343,] -0.08005588 -0.00112613  0.00263440 -1.5378             0.9379512    
## [344,]  0.03055281 -0.00112613  0.00217642  0.6790             0.2485543    
## [345,]  0.03535648 -0.00112613  0.00232397  0.7568             0.2245903    
## [346,]  0.01572334 -0.00112613  0.00504909  0.2371             0.4062794    
## [347,] -0.00175599 -0.00112613  0.00380025 -0.0102             0.5040761    
## [348,] -0.09239599 -0.00112613  0.00320324 -1.6126             0.9465866    
## [349,]  0.31013283 -0.00112613  0.10896306  0.9429             0.1728568    
## [350,]  0.04087107 -0.00112613  0.00284290  0.7877             0.2154475    
## [351,] -0.01048897 -0.00112613  0.01487372 -0.0768             0.5305972    
## [352,]  0.02971341 -0.00112613  0.01368137  0.2637             0.3960212    
## [353,]  0.47557827 -0.00112613  0.00223164 10.0911 < 0.00000000000000022 ***
## [354,] -0.09196061 -0.00112613  0.00220933 -1.9325             0.9733513    
## [355,] -0.10229599 -0.00112613  0.00297283 -1.8555             0.9682391    
## [356,]  0.07617281 -0.00112613  0.01210430  0.7026             0.2411548    
## [357,] -0.09361651 -0.00112613  0.00284290 -1.7347             0.9585999    
## [358,]  0.04050703 -0.00112613  0.00256518  0.8220             0.2055337    
## [359,] -0.06252675 -0.00112613  0.00932366 -0.6359             0.7375747    
## [360,] -0.11721288 -0.00112613  0.00273582 -2.2194             0.9867708    
## [361,] -0.02151464 -0.00112613  0.00380025 -0.3307             0.6295775    
## [362,] -0.02415767 -0.00112613  0.00628883 -0.2904             0.6142555    
## [363,]  0.07635463 -0.00112613  0.06080884  0.3142             0.3766834    
## [364,]  0.36630339 -0.00112613  0.19701647  0.8278             0.2038933    
## [365,]  0.03912351 -0.00112613  0.00281170  0.7591             0.2239077    
## [366,]  0.09254538 -0.00112613  0.00354699  1.5728             0.0578810 .  
## [367,] -0.00975867 -0.00112613  0.00798437 -0.0966             0.5384817    
## [368,] -0.05125280 -0.00112613  0.00433915 -0.7610             0.7766621    
## [369,]  0.01912481 -0.00112613  0.00568122  0.2687             0.3940906    
## [370,]  0.01261276 -0.00112613  0.01112593  0.1303             0.4481836    
## [371,] -0.18761226 -0.00112613  0.02429631 -1.1964             0.8842297    
## [372,]  0.45020694 -0.00112613  0.00352526  7.6015 0.0000000000000146315 ***
## [373,]  0.01095999 -0.00112613  0.02193352  0.0816             0.4674792    
## [374,]  0.02888343 -0.00112613  0.00833058  0.3288             0.3711563    
## [375,]  0.03570246 -0.00112613  0.00285869  0.6888             0.2454703    
## [376,] -0.02544103 -0.00112613  0.00708307 -0.2889             0.6136747    
## [377,] -0.11916628 -0.00112613  0.00389932 -1.8903             0.9706424    
## [378,]  0.03640500 -0.00112613  0.00287460  0.7000             0.2419612    
## [379,]  0.01144625 -0.00112613  0.01690721  0.0967             0.4614863    
## [380,] -0.04120950 -0.00112613  0.00688500 -0.4831             0.6854779    
## [381,]  0.02113542 -0.00112613  0.01083124  0.2139             0.4153114    
## [382,]  0.02516104 -0.00112613  0.01127880  0.2475             0.4022524    
## [383,] -0.10324088 -0.00112613  0.00225425 -2.1507             0.9842515    
## [384,]  0.05013301 -0.00112613  0.00206114  1.1291             0.1294362    
## [385,] -0.37319523 -0.00112613  0.06965349 -1.4098             0.9206981    
## [386,] -0.07884473 -0.00112613  0.00311161 -1.3933             0.9182296    
## [387,] -0.09979949 -0.00112613  0.00259256 -1.9379             0.9736835    
## [388,] -0.07439169 -0.00112613  0.00267720 -1.4160             0.9216103    
## [389,]  0.00876755 -0.00112613  0.00628883  0.1248             0.4503571    
## [390,]  0.03191566 -0.00112613  0.00350373  0.5582             0.2883503    
## [391,] -0.03625355 -0.00112613  0.00486376 -0.5037             0.6927591    
## [392,]  0.44444655 -0.00112613  0.00309375  8.0108 0.0000000000000005698 ***
## [393,] -0.28035377 -0.00112613  0.04606776 -1.3009             0.9033618    
## [394,] -0.09105646 -0.00112613  0.00251162 -1.7944             0.9636287    
## [395,] -0.02762248 -0.00112613  0.00790175 -0.2981             0.6171768    
## [396,]  0.01854307 -0.00112613  0.00860806  0.2120             0.4160539    
## [397,]  0.12546086 -0.00112613  0.08144636  0.4436             0.3286801    
## [398,] -0.12001629 -0.00112613  0.09795638 -0.3799             0.6479773    
## [399,]  0.32375979 -0.00112613  0.00212300  7.0511 0.0000000000008876213 ***
## [400,]  0.28204755 -0.00112613  0.16399644  0.6993             0.2421963    
## [401,]  0.13142336 -0.00112613  0.12272140  0.3784             0.3525775    
## [402,]  0.17284078 -0.00112613  0.00220933  3.7011             0.0001073 ***
## [403,]  0.01184284 -0.00112613  0.08895092  0.0435             0.4826578    
## [404,] -0.08975683 -0.00112613  0.00194429 -2.0100             0.9777863    
## [405,]  0.23991963 -0.00112613  0.12272140  0.6881             0.2457010    
## [406,]  0.25841391 -0.00112613  0.00275076  4.9485 0.0000003738457190282 ***
## [407,] -0.13243864 -0.00112613  0.02566928 -0.8196             0.7937765    
## [408,] -0.10266998 -0.00112613  0.00222045 -2.1549             0.9844164    
## [409,]  0.04526480 -0.00112613  0.00205106  1.0243             0.1528378    
## [410,]  0.03420119 -0.00112613  0.00215484  0.7610             0.2233189    
## [411,] -0.03383309 -0.00112613  0.02429631 -0.2098             0.5831003    
## [412,]  0.02925781 -0.00112613  0.01015311  0.3015             0.3815015    
## [413,]  0.20334152 -0.00112613  0.01246616  1.8313             0.0335283 *  
## [414,] -0.05653715 -0.00112613  0.05103318 -0.2453             0.5968819    
## [415,]  0.01228802 -0.00112613  0.00708307  0.1594             0.4366821    
## [416,] -0.05496784 -0.00112613  0.02566928 -0.3361             0.6315858    
## [417,] -0.09087396 -0.00112613  0.00297283 -1.6460             0.9501216    
## [418,] -0.03831423 -0.00112613  0.00425088 -0.5704             0.7157901    
## [419,]  0.04414379 -0.00112613  0.01325282  0.3932             0.3470718    
## [420,] -0.30385250 -0.00112613  0.06965349 -1.1470             0.8743176    
## [421,] -0.04373891 -0.00112613  0.00489993 -0.6088             0.7286578    
## [422,]  0.03796228 -0.00112613  0.00384929  0.6300             0.2643391    
## [423,] -0.10345667 -0.00112613  0.24654652 -0.2061             0.5816395    
## [424,]  0.15986694 -0.00112613  0.06965349  0.6100             0.2709282    
## [425,] -0.06998266 -0.00112613  0.00782062 -0.7786             0.7818977    
## [426,] -0.00756909 -0.00112613  0.00532876 -0.0883             0.5351657    
## [427,] -0.08078727 -0.00112613  0.02248202 -0.5313             0.7023899    
## [428,]  0.06638427 -0.00112613  0.02429631  0.4331             0.3324666    
## [429,]  0.16668391 -0.00112613  0.05716693  0.7019             0.2413857    
## [430,] -0.10020367 -0.00112613  0.00257882 -1.9510             0.9744736    
## [431,] -0.00792787 -0.00112613  0.03191632 -0.0381             0.5151852    
## [432,] -0.12059052 -0.00112613  0.00199191 -2.6767             0.9962827    
## [433,] -0.05785485 -0.00112613  0.00640084 -0.7091             0.7608572    
## [434,] -0.01781902 -0.00112613  0.02429631 -0.1071             0.5426424    
## [435,] -0.09279404 -0.00112613  0.00273582 -1.7526             0.9601614    
## [436,] -0.08771233 -0.00112613  0.00224291 -1.8283             0.9662463    
## [437,]  0.01358684 -0.00112613  0.00860806  0.1586             0.4370000    
## [438,]  0.01040197 -0.00112613  0.00516591  0.1604             0.4362859    
## [439,]  0.01625400 -0.00112613  0.00392472  0.2774             0.3907261    
## [440,]  0.03334279 -0.00112613  0.01597561  0.2727             0.3925386    
## [441,]  0.32348045 -0.00112613  0.00363600  5.3833 0.0000000365746206249 ***
## [442,]  0.01932889 -0.00112613  0.00416544  0.3169             0.3756467    
## [443,] -0.05777784 -0.00112613  0.00602291 -0.7300             0.7672982    
## [444,] -0.07747608 -0.00112613  0.00436922 -1.1551             0.8759684    
## [445,] -0.04685269 -0.00112613  0.00316612 -0.8127             0.7917915    
## [446,]  0.02085584 -0.00112613  0.00657536  0.2711             0.3931625    
## [447,] -0.04396083 -0.00112613  0.00520584 -0.5937             0.7236359    
## [448,]  0.04929349 -0.00112613  0.00395037  0.8022             0.2112197    
## [449,] -0.08807937 -0.00112613  0.00348240 -1.4735             0.9296902    
## [450,]  0.28626595 -0.00112613  0.00235992  5.9160 0.0000000016496535842 ***
## [451,]  0.02355026 -0.00112613  0.00766266  0.2819             0.3890109    
## [452,] -0.01632211 -0.00112613  0.02429631 -0.0975             0.5388312    
## [453,] -0.07971724 -0.00112613  0.00218732 -1.6804             0.9535621    
## [454,]  0.06568776 -0.00112613  0.01068915  0.6462             0.2590614    
## [455,]  0.10170689 -0.00112613  0.03085115  0.5855             0.2791194    
## [456,] -0.01796953 -0.00112613  0.00688500 -0.2030             0.5804292    
## [457,]  0.02936103 -0.00112613  0.00537080  0.4160             0.3387036    
## [458,] -0.04016099 -0.00112613  0.00486376 -0.5597             0.7121629    
## [459,] -0.04589343 -0.00112613  0.00545656 -0.6060             0.7277561    
## [460,]  0.00594199 -0.00112613  0.00824158  0.0779             0.4689708    
## [461,] -0.05044895 -0.00112613  0.00501111 -0.6968             0.7570225    
## [462,]  0.03838738 -0.00112613  0.02366131  0.2569             0.3986366    
## [463,] -0.07981688 -0.00112613  0.00645812 -0.9792             0.8362589    
## [464,] -0.09957458 -0.00112613  0.00225425 -2.0735             0.9809379    
## [465,] -0.09034132 -0.00112613  0.00234786 -1.8412             0.9672045    
## [466,] -0.11622446 -0.00112613  0.00479272 -1.6626             0.9518000    
## [467,]  0.21485816 -0.00112613  0.00214416  4.6644 0.0000015477436183927 ***
## [468,]  0.04912559 -0.00112613  0.00751021  0.5799             0.2810037    
## [469,]  0.15673735 -0.00112613  0.02719917  0.9572             0.1692325    
## [470,]  0.33752018 -0.00112613  0.00213354  7.3315 0.0000000000001137544 ***
## [471,] -0.07433892 -0.00112613  0.04392360 -0.3493             0.6365798    
## [472,]  0.20845583 -0.00112613  0.06080884  0.8499             0.1976889    
## [473,]  0.03627193 -0.00112613  0.00235992  0.7698             0.2206977    
## [474,]  0.09341044 -0.00112613  0.05716693  0.3954             0.3462770    
## [475,]  0.44707106 -0.00112613  0.00231215  9.3210 < 0.00000000000000022 ***
## [476,]  0.03479112 -0.00112613  0.32909660  0.0626             0.4750387    
## [477,] -0.03503773 -0.00112613  0.00419362 -0.5237             0.6997444    
## [478,] -0.08244817 -0.00112613  0.00203110 -1.8044             0.9644190    
## [479,] -0.06017332 -0.00112613  0.00352526 -0.9945             0.8400096    
## [480,]  0.13518354 -0.00112613  0.01143554  1.2747             0.1012127    
## [481,]  0.09187147 -0.00112613  0.01083124  0.8936             0.1857737    
## [482,]  0.02604321 -0.00112613  0.01997290  0.1922             0.4237746    
## [483,]  0.03222952 -0.00112613  0.00218732  0.7132             0.2378599    
## [484,] -0.08327561 -0.00112613  0.03305494 -0.4518             0.6743086    
## [485,]  0.16668391 -0.00112613  0.05716693  0.7019             0.2413857    
## [486,] -0.11280142 -0.00112613  0.00257882 -2.1991             0.9860649    
## [487,] -0.04789219 -0.00112613  0.00497359 -0.6631             0.7463749    
## [488,]  0.01438369 -0.00112613  0.00395037  0.2468             0.4025442    
## [489,]  0.16673152 -0.00112613  0.00207129  3.6883             0.0001129 ***
## [490,]  0.04108371 -0.00112613  0.00405566  0.6628             0.2537293    
## [491,]  0.08121139 -0.00112613  0.01265463  0.7319             0.2321039    
## [492,]  0.25396227 -0.00112613  0.10896306  0.7728             0.2198288    
## [493,]  0.13738586 -0.00112613  0.16399644  0.3420             0.3661624    
## [494,]  0.12917123 -0.00112613  0.00234786  2.6891             0.0035827 ** 
## [495,]  0.21464288 -0.00112613  0.09795638  0.6894             0.2452851    
## [496,] -0.09404366 -0.00112613  0.00256518 -1.8346             0.9667168    
## [497,]  0.19779171 -0.00112613  0.10896306  0.6026             0.2733851    
## [498,] -0.03539582 -0.00112613  0.01176123 -0.3160             0.6239978    
## [499,]  0.04805389 -0.00112613  0.01265463  0.4372             0.3309891    
## [500,] -0.18630877 -0.00112613  0.01325282 -1.6086             0.9461473    
## [501,] -0.35696916 -0.00112613  0.07509636 -1.2985             0.9029458    
## [502,]  0.01532308 -0.00112613  0.01346394  0.1418             0.4436341    
## [503,] -0.06920284 -0.00112613  0.00397629 -1.0796             0.8598380    
## [504,] -0.18761226 -0.00112613  0.02429631 -1.1964             0.8842297    
## [505,] -0.11051585 -0.00112613  0.00216559 -2.3506             0.9906297    
## [506,] -0.02783005 -0.00112613  0.01246616 -0.2392             0.5945135    
## [507,] -0.08792814 -0.00112613  0.00247246 -1.7457             0.9595669    
## [508,]  0.31095806 -0.00112613  0.02366131  2.0289             0.0212362 *  
## [509,]  0.31277984 -0.00112613  0.00237208  6.4452 0.0000000000577310768 ***
## [510,] -0.03424947 -0.00112613  0.01083124 -0.3183             0.6248598    
## [511,] -0.09785423 -0.00112613  0.03558521 -0.5128             0.6959420    
## [512,] -0.12447195 -0.00112613  0.00230041 -2.5717             0.9949401    
## [513,] -0.11181021 -0.00112613  0.00239664 -2.2609             0.9881177    
## [514,]  0.01738708 -0.00112613  0.00978201  0.1872             0.4257584    
## [515,] -0.11107598 -0.00112613  0.00275076 -2.0964             0.9819753    
## [516,]  0.21464288 -0.00112613  0.09795638  0.6894             0.2452851    
## [517,] -0.04446083 -0.00112613  0.00350373 -0.7321             0.7679463    
## [518,]  0.02574900 -0.00112613  0.02366131  0.1747             0.4306516    
## [519,] -0.25770317 -0.00112613  0.04392360 -1.2242             0.8895704    
## [520,] -0.06693797 -0.00112613  0.00550029 -0.8874             0.8125637    
## [521,]  0.01343041 -0.00112613  0.00504909  0.2049             0.4188418    
## [522,]  0.03473632 -0.00112613  0.00302379  0.6522             0.2571441    
## [523,]  0.03105904 -0.00112613  0.00327947  0.5620             0.2870504    
## [524,]  0.04364871 -0.00112613  0.01346394  0.3859             0.3497942    
## [525,]  0.23116808 -0.00112613  0.00210211  5.0665 0.0000002025694150586 ***
## [526,] -0.09235361 -0.00112613  0.00264856 -1.7726             0.9618559    
## [527,] -0.01454969 -0.00112613  0.00694995 -0.1610             0.5639608    
## [528,] -0.02208255 -0.00112613  0.00335845 -0.3616             0.6411805    
## [529,] -0.05539280 -0.00112613  0.00486376 -0.7781             0.7817512    
## [530,] -0.10080094 -0.00112613  0.09795638 -0.3185             0.6249359    
## [531,]  0.12505364 -0.00112613  0.02803161  0.7536             0.2255318    
## [532,]  0.08366245 -0.00112613  0.01159630  0.7874             0.2155333    
## [533,]  0.12416164 -0.00112613  0.03558521  0.6642             0.2532934    
## [534,]  0.11774304 -0.00112613  0.02891449  0.6991             0.2422588    
## [535,]  0.01291174 -0.00112613  0.00932366  0.1454             0.4422050    
## [536,]  0.12492929 -0.00112613  0.00216559  2.7088             0.0033766 ** 
## [537,]  0.01456919 -0.00112613  0.00462230  0.2309             0.4087134    
## [538,]  0.04269662 -0.00112613  0.00252486  0.8721             0.1915688    
## [539,] -0.09112811 -0.00112613  0.00224291 -1.9004             0.9713100    
## [540,]  0.03989809 -0.00112613  0.00253820  0.8143             0.2077403    
## [541,] -0.13318667 -0.00112613  0.00212300 -2.8661             0.9979225    
## [542,]  0.03454988 -0.00112613  0.00344033  0.6082             0.2715134    
## [543,]  0.36630339 -0.00112613  0.49419676  0.5227             0.3006034    
## [544,] -0.02127339 -0.00112613  0.01304773 -0.1764             0.5700022    
## [545,]  0.30311151 -0.00112613  0.12272140  0.8685             0.1925697    
## [546,] -0.09603300 -0.00112613  0.00234786 -1.9587             0.9749245    
## [547,] -0.00774867 -0.00112613  0.00701596 -0.0791             0.5315093    
## [548,]  0.02465823 -0.00112613  0.00307603  0.4649             0.3210011    
## [549,] -0.04900406 -0.00112613  0.00572802 -0.6326             0.7365045    
## [550,] -0.09547972 -0.00112613  0.00239664 -1.9273             0.9730310    
## [551,] -0.18761226 -0.00112613  0.02429631 -1.1964             0.8842297    
## [552,]  0.26519638 -0.00112613  0.09795638  0.8509             0.1974054    
## [553,]  0.09027406 -0.00112613  0.01176123  0.8428             0.1996723    
## [554,]  0.03506189 -0.00112613  0.00314779  0.6450             0.2594626    
## [555,] -0.02891418 -0.00112613  0.01568613 -0.2219             0.5877928    
## [556,] -0.14205642 -0.00112613  0.00199191 -3.1577             0.9992049    
## [557,] -0.02786774 -0.00112613  0.01068915 -0.2587             0.6020482    
## [558,]  0.18770417 -0.00112613  0.00227718  3.9571 0.0000379377193481106 ***
## [559,] -0.07910538 -0.00112613  0.00215484 -1.6799             0.9535071    
## [560,] -0.06318870 -0.00112613  0.02305728 -0.4087             0.6586273    
## [561,]  0.00286752 -0.00112613  0.00462230  0.0587             0.4765793    
## [562,] -0.10160391 -0.00112613  0.00253820 -1.9944             0.9769445    
## [563,]  0.08139537 -0.00112613  0.01325282  0.7168             0.2367411    
## [564,]  0.22849241 -0.00112613  0.02305728  1.5122             0.0652443 .  
## [565,]  0.04514649 -0.00112613  0.02566928  0.2888             0.3863621    
## [566,] -0.09822470 -0.00112613  0.00329896 -1.6905             0.9545373    
## [567,]  0.00117653 -0.00112613  0.00504909  0.0324             0.4870742    
## [568,]  0.09059101 -0.00112613  0.02496473  0.5805             0.2807956    
## [569,]  0.02881293 -0.00112613  0.00205106  0.6611             0.2542832    
## [570,]  0.00581861 -0.00112613  0.00824158  0.0765             0.4695114    
## [571,] -0.08053387 -0.00112613  0.00214416 -1.7149             0.9568167    
## [572,] -0.11257756 -0.00112613  0.00230041 -2.3237             0.9899296    
## [573,]  0.04926420 -0.00112613  0.00966369  0.5126             0.3041168    
## [574,] -0.01665070 -0.00112613  0.00479272 -0.2242             0.5887179    
## [575,] -0.01524006 -0.00112613  0.00618011 -0.1795             0.5712413    
## [576,] -0.08031269 -0.00112613  0.00327947 -1.3828             0.9166320    
## [577,] -0.18761226 -0.00112613  0.02429631 -1.1964             0.8842297    
## [578,]  0.03087690 -0.00112613  0.00208150  0.7015             0.2415078    
## [579,] -0.06465046 -0.00112613  0.14041071 -0.1695             0.5673091    
## [580,]  0.03757392 -0.00112613  0.00242156  0.7864             0.2158055    
## [581,] -0.08254817 -0.00112613  0.00212300 -1.7671             0.9613963    
## [582,] -0.04171140 -0.00112613  0.03191632 -0.2272             0.5898565    
## [583,]  0.02997928 -0.00112613  0.02429631  0.1996             0.4209138    
## [584,]  0.00333252 -0.00112613  0.00640084  0.0557             0.4777787    
## [585,]  0.13954185 -0.00112613  0.02366131  0.9145             0.1802314    
## [586,] -0.02757648 -0.00112613  0.00482803 -0.3807             0.6482753    
## [587,] -0.08959294 -0.00112613  0.00257882 -1.7421             0.9592535    
## [588,] -0.01030775 -0.00112613  0.00577546 -0.1208             0.5480818    
## [589,]  0.00647922 -0.00112613  0.00943459  0.0783             0.4687950    
## [590,] -0.05055678 -0.00112613  0.00550029 -0.6665             0.7474559    
## [591,] -0.09697023 -0.00112613  0.00260640 -1.8773             0.9697648    
## [592,]  0.10771683 -0.00112613  0.00223164  2.3040             0.0106105 *  
## [593,] -0.03372108 -0.00112613  0.00572802 -0.4307             0.6666472    
## [594,] -0.10105538 -0.00112613  0.00218732 -2.1367             0.9836874    
## [595,]  0.14201385 -0.00112613  0.05103318  0.6336             0.2631617    
## [596,] -0.16991400 -0.00112613  0.02140994 -1.1535             0.8756561    
## [597,]  0.07044677 -0.00112613  0.01143554  0.6693             0.2516523    
## [598,]  0.36630339 -0.00112613  0.49419676  0.5227             0.3006034    
## [599,] -0.09208681 -0.00112613  0.00436922 -1.3761             0.9156056    
## [600,] -0.05966233 -0.00112613  0.02248202 -0.3904             0.6518786    
## [601,] -0.06209741 -0.00112613  0.00433915 -0.9256             0.8226730    
## [602,] -0.02001288 -0.00112613  0.00419362 -0.2917             0.6147231    
## [603,] -0.03236571 -0.00112613  0.00337864 -0.5374             0.7045199    
## [604,] -0.38627512 -0.00112613  0.16399644 -0.9511             0.8292151    
## [605,]  0.01099363 -0.00112613  0.00870421  0.1299             0.4483204    
## [606,]  0.01189586 -0.00112613  0.00320324  0.2301             0.4090142    
## [607,]  0.00226577 -0.00112613  0.00663535  0.0416             0.4833928    
## [608,]  0.08187768 -0.00112613  0.01176123  0.7654             0.2220255    
## [609,]  0.19779171 -0.00112613  0.10896306  0.6026             0.2733851    
## [610,]  0.08432757 -0.00112613  0.00554461  1.1476             0.1255643    
## [611,] -0.09882143 -0.00112613  0.00237208 -2.0059             0.9775665    
## [612,]  0.00806545 -0.00112613  0.00389932  0.1472             0.4414887    
## [613,] -0.00245628 -0.00112613  0.00545656 -0.0180             0.5071834    
## [614,]  0.03057490 -0.00112613  0.00204105  0.7017             0.2414356    
## [615,]  0.04136622 -0.00112613  0.00266283  0.8235             0.2051251    
## [616,] -0.11075454 -0.00112613  0.00504909 -1.5428             0.9385634    
## [617,]  0.00413876 -0.00112613  0.00657536  0.0649             0.4741158    
## [618,]  0.14507262 -0.00112613  0.02305728  0.9628             0.1678220    
## [619,]  0.03248140 -0.00112613  0.01176123  0.3099             0.3783216    
## [620,]  0.00817774 -0.00112613  0.00395037  0.1480             0.4411603    
## [621,] -0.01459101 -0.00112613  0.02366131 -0.0875             0.5348770    
## [622,]  0.03627193 -0.00112613  0.00235992  0.7698             0.2206977    
## [623,] -0.09483992 -0.00112613  0.00220933 -1.9938             0.9769109    
## [624,] -0.08480253 -0.00112613  0.00209177 -1.8296             0.9663419    
## [625,] -0.08610311 -0.00112613  0.00247246 -1.7090             0.9562725    
## [626,]  0.12546086 -0.00112613  0.08144636  0.4436             0.3286801    
## [627,] -0.03755752 -0.00112613  0.00400248 -0.5759             0.7176427    
## [628,]  0.03978806 -0.00112613  0.00202121  0.9101             0.1813963    
## [629,] -0.09667831 -0.00112613  0.00269169 -1.8417             0.9672433    
## [630,] -0.01562250 -0.00112613  0.00554461 -0.1947             0.5771787    
## [631,] -0.02037671 -0.00112613  0.00339902 -0.3302             0.6293728    
## [632,] -0.02081391 -0.00112613  0.00966369 -0.2003             0.5793670    
## [633,]  0.64403885 -0.00112613  0.98949724  0.6486             0.2583050    
## [634,] -0.11570645 -0.00112613  0.03699632 -0.5957             0.7243136    
## [635,] -0.05740682 -0.00112613  0.00363600 -0.9334             0.8246819    
## [636,]  0.04365893 -0.00112613  0.00255164  0.8866             0.1876495    
## [637,] -0.13511161 -0.00112613  0.00207129 -2.9440             0.9983800    
## [638,] -0.07146594 -0.00112613  0.00628883 -0.8870             0.8124565    
## [639,]  0.30311151 -0.00112613  0.12272140  0.8685             0.1925697    
## [640,]  0.03927249 -0.00112613  0.02429631  0.2592             0.3977494    
## [641,] -0.07918103 -0.00112613  0.00217642 -1.6731             0.9528488    
## [642,] -0.00165051 -0.00112613  0.02140994 -0.0036             0.5014297    
## [643,] -0.01807807 -0.00112613  0.00587235 -0.2212             0.5875373    
## [644,]  0.02675573 -0.00112613  0.01513564  0.2266             0.4103550    
## [645,] -0.01747468 -0.00112613  0.00751021 -0.1886             0.5748158    
## [646,] -0.09572909 -0.00112613  0.00295612 -1.7400             0.9590684    
## [647,] -0.12722274 -0.00112613  0.00238432 -2.5824             0.9950940    
## [648,] -0.08401500 -0.00112613  0.00213354 -1.7945             0.9636340    
## [649,] -0.00261624 -0.00112613  0.00966369 -0.0152             0.5060470    
## [650,]  0.01692349 -0.00112613  0.00790175  0.2031             0.4195473    
## [651,]  0.28204755 -0.00112613  0.16399644  0.6993             0.2421963    
## [652,]  0.09288701 -0.00112613  0.03191632  0.5262             0.2993615    
## [653,] -0.08001814 -0.00112613  0.00365879 -1.3043             0.9039276    
## [654,] -0.02004529 -0.00112613  0.00634441 -0.2375             0.5938746    
## [655,] -0.02448643 -0.00112613  0.04196589 -0.1140             0.5453942    
## [656,]  0.04926420 -0.00112613  0.00966369  0.5126             0.3041168    
## [657,] -0.09091364 -0.00112613  0.00452470 -1.3348             0.9090316    
## [658,]  0.03062103 -0.00112613  0.00318460  0.5626             0.2868635    
## [659,] -0.06370037 -0.00112613  0.08144636 -0.2193             0.5867763    
## [660,] -0.13515011 -0.00112613  0.03699632 -0.6968             0.7570336    
## [661,] -0.04803342 -0.00112613  0.00554461 -0.6299             0.7356357    
## [662,]  0.00555550 -0.00112613  0.00335845  0.1153             0.4541054    
## [663,]  0.04221888 -0.00112613  0.00990296  0.4356             0.3315749    
## [664,]  0.13095132 -0.00112613  0.02566928  0.8244             0.2048649    
## [665,] -0.00237677 -0.00112613  0.00356893 -0.0209             0.5083511    
## [666,]  0.03734306 -0.00112613  0.00279629  0.7275             0.2334655    
## [667,] -0.04047135 -0.00112613  0.00365879 -0.6505             0.7423038    
## [668,]  0.01627045 -0.00112613  0.00493653  0.2476             0.4022215    
## [669,]  0.00137744 -0.00112613  0.00900446  0.0264             0.4894758    
## [670,] -0.09415155 -0.00112613  0.00413756 -1.4462             0.9259400    
## [671,] -0.08893687 -0.00112613  0.00324102 -1.5424             0.9385158    
## [672,] -0.01397559 -0.00112613  0.00512648 -0.1795             0.5712130    
## [673,] -0.07581369 -0.00112613  0.00207129 -1.6411             0.9496089    
## [674,]  0.36630339 -0.00112613  0.32909660  0.6405             0.2609271    
## [675,]  0.00892578 -0.00112613  0.00640084  0.1256             0.4500082    
## [676,]  0.04357151 -0.00112613  0.01284841  0.3943             0.3466687    
## [677,]  0.00757576 -0.00112613  0.01997290  0.0616             0.4754513    
## [678,] -0.05085754 -0.00112613  0.00520584 -0.6893             0.7546712    
## [679,]  0.07812692 -0.00112613  0.02305728  0.5219             0.3008597    
## [680,]  0.21911415 -0.00112613  0.03852032  1.1222             0.1308989    
## [681,] -0.00982251 -0.00112613  0.00966369 -0.0885             0.5352461    
## [682,]  0.01730121 -0.00112613  0.00572802  0.2435             0.4038174    
## [683,] -0.03807893 -0.00112613  0.00397629 -0.5860             0.7210671    
## [684,]  0.22843020 -0.00112613  0.08895092  0.7697             0.2207428    
## [685,]  0.06626375 -0.00112613  0.01246616  0.6036             0.2730645    
## [686,] -0.06294985 -0.00112613  0.00572802 -0.8169             0.7929987    
## [687,] -0.01324073 -0.00112613  0.00806852 -0.1349             0.5536423    
## [688,] -0.02953974 -0.00112613  0.00640084 -0.3551             0.6387602    
## [689,]  0.15516699 -0.00112613  0.00278099  2.9637             0.0015196 ** 
## [690,] -0.03988416 -0.00112613  0.00344033 -0.6608             0.7456258    
## [691,] -0.11984288 -0.00112613  0.00331861 -2.0608             0.9803385    
## [692,] -0.09419003 -0.00112613  0.00312962 -1.6635             0.9518986    
## [693,]  0.17685989 -0.00112613  0.06080884  0.7218             0.2352161    
## [694,]  0.09141209 -0.00112613  0.03427489  0.4998             0.3085930    
## [695,] -0.02429702 -0.00112613  0.00634441 -0.2909             0.6144370    
## [696,]  0.00951655 -0.00112613  0.00623407  0.1348             0.4463881    
## [697,]  0.11230580 -0.00112613  0.00209177  2.4802             0.0065663 ** 
## [698,] -0.09751801 -0.00112613  0.00305847 -1.7430             0.9593300    
## [699,] -0.13082739 -0.00112613  0.01390539 -1.0999             0.8643118    
## [700,] -0.10260709 -0.00112613  0.00257882 -1.9984             0.9771614    
## [701,]  0.03590272 -0.00112613  0.00272100  0.7099             0.2388938    
## [702,]  0.02523775 -0.00112613  0.00462230  0.3878             0.3490912    
## [703,] -0.09191508 -0.00112613  0.00223164 -1.9219             0.9726881    
## [704,] -0.03111644 -0.00112613  0.00729121 -0.3512             0.6372890    
## [705,]  0.04055909 -0.00112613  0.00255164  0.8252             0.2046219    
## [706,] -0.05328188 -0.00112613  0.00558953 -0.6976             0.7572903    
## [707,] -0.08639373 -0.00112613  0.00260640 -1.6702             0.9525582    
## [708,] -0.10948024 -0.00112613  0.00259256 -2.1280             0.9833334    
## [709,]  0.03561278 -0.00112613  0.00239664  0.7505             0.2264903    
## [710,] -0.06016117 -0.00112613  0.02566928 -0.3685             0.6437389    
## [711,] -0.09785423 -0.00112613  0.03558521 -0.5128             0.6959420    
## [712,]  0.08388820 -0.00112613  0.03558521  0.4507             0.3261142    
## [713,] -0.11503069 -0.00112613  0.00329896 -1.9831             0.9763240    
## [714,] -0.10391211 -0.00112613  0.00482803 -1.4793             0.9304666    
## [715,]  0.23523139 -0.00112613  0.00211252  5.1424 0.0000001355995848261 ***
## [716,] -0.06999469 -0.00112613  0.00377609 -1.1207             0.8687979    
## [717,] -0.11063059 -0.00112613  0.00238432 -2.2426             0.9875384    
## [718,] -0.02942400 -0.00112613  0.00348240 -0.4795             0.6842187    
## [719,]  0.05728543 -0.00112613  0.06965349  0.2213             0.4124203    
## [720,]  0.04051542 -0.00112613  0.00245960  0.8396             0.2005542    
## [721,] -0.01847867 -0.00112613  0.00339902 -0.2976             0.6170098    
## [722,] -0.12012362 -0.00112613  0.00196316 -2.6857             0.9963813    
## [723,]  0.03352017 -0.00112613  0.00304106  0.6283             0.2649144    
## [724,] -0.12564699 -0.00112613  0.00225425 -2.6227             0.9956376    
## [725,]  0.03564463 -0.00112613  0.00248542  0.7376             0.2303881    
## [726,] -0.06752467 -0.00112613  0.00413756 -1.0323             0.8490234    
## [727,] -0.08409104 -0.00112613  0.00213354 -1.7962             0.9637651    
## [728,] -0.10682658 -0.00112613  0.00372848 -1.7311             0.9582790    
## [729,] -0.10080094 -0.00112613  0.09795638 -0.3185             0.6249359    
## [730,]  0.09952511 -0.00112613  0.02803161  0.6012             0.2738644    
## [731,]  0.02740327 -0.00112613  0.01513564  0.2319             0.4083096    
## [732,] -0.06241216 -0.00112613  0.00382465 -0.9910             0.8391529    
## [733,]  0.26519638 -0.00112613  0.09795638  0.8509             0.1974054    
## [734,] -0.10367280 -0.00112613  0.02043109 -0.7174             0.7634437    
## [735,] -0.08977246 -0.00112613  0.00219829 -1.8907             0.9706666    
## [736,]  0.21464288 -0.00112613  0.09795638  0.6894             0.2452851    
## [737,]  0.09288701 -0.00112613  0.03191632  0.5262             0.2993615    
## [738,]  0.21464288 -0.00112613  0.09795638  0.6894             0.2452851    
## [739,] -0.04565222 -0.00112613  0.00558953 -0.5956             0.7242660    
## [740,]  0.02937268 -0.00112613  0.01112593  0.2891             0.3862355    
## [741,]  0.05103722 -0.00112613  0.01597561  0.4127             0.3399124    
## [742,] -0.09055702 -0.00112613  0.00231215 -1.8599             0.9685471    
## [743,]  0.04379552 -0.00112613  0.02140994  0.3070             0.3794191    
## [744,] -0.07622674 -0.00112613  0.09795638 -0.2400             0.5948169    
## [745,] -0.09855057 -0.00112613  0.00209177 -2.1302             0.9834205    
## [746,]  0.07130444 -0.00112613  0.02566928  0.4521             0.3256058    
## [747,]  0.10433997 -0.00112613  0.02140994  0.7208             0.2355212    
## [748,] -0.06666825 -0.00112613  0.00524629 -0.9049             0.8172374    
## [749,]  0.01854307 -0.00112613  0.00860806  0.2120             0.4160539    
## [750,]  0.17008836 -0.00112613  0.14041071  0.4569             0.3238642    
## [751,]  0.03033168 -0.00112613  0.00207129  0.6912             0.2447177    
## [752,]  0.18287452 -0.00112613  0.07509636  0.6714             0.2509688    
## [753,] -0.03636629 -0.00112613  0.00623407 -0.4463             0.6723191    
## [754,]  0.14539005 -0.00112613  0.04606776  0.6826             0.2474195    
## [755,]  0.36630339 -0.00112613  0.49419676  0.5227             0.3006034    
## [756,] -0.07430239 -0.00112613  0.08895092 -0.2454             0.5969092    
## [757,]  0.25797445 -0.00112613  0.06965349  0.9817             0.1631137    
## [758,]  0.04910934 -0.00112613  0.00943459  0.5172             0.3025123    
## [759,] -0.11570645 -0.00112613  0.03699632 -0.5957             0.7243136    
## [760,] -0.07770576 -0.00112613  0.00206114 -1.6868             0.9541774    
## [761,]  0.02750181 -0.00112613  0.00640084  0.3578             0.3602368    
## [762,]  0.02574900 -0.00112613  0.02366131  0.1747             0.4306516    
## [763,]  0.03571102 -0.00112613  0.00225425  0.7759             0.2189152    
## [764,] -0.00810341 -0.00112613  0.00657536 -0.0860             0.5342848    
## [765,]  0.02867105 -0.00112613  0.00293955  0.5496             0.2913022    
## [766,] -0.10669014 -0.00112613  0.00255164 -2.0898             0.9816825    
## [767,] -0.02429702 -0.00112613  0.00634441 -0.2909             0.6144370    
## [768,]  0.00014402 -0.00112613  0.02366131  0.0083             0.4967059    
## [769,] -0.00936485 -0.00112613  0.02985256 -0.0477             0.5190158    
## [770,] -0.12330427 -0.00112613  0.00206114 -2.6912             0.9964398    
## [771,]  0.00005195 -0.00112613  0.05716693  0.0049             0.4980343    
## [772,] -0.08424584 -0.00112613  0.16399644 -0.2053             0.5813123    
## [773,] -0.02971090 -0.00112613  0.00582356 -0.3746             0.6460122    
## [774,]  0.10659230 -0.00112613  0.04842633  0.4895             0.3122453    
## [775,] -0.08744121 -0.00112613  0.00276582 -1.6413             0.9496273    
## [776,] -0.02617678 -0.00112613  0.00520584 -0.3472             0.6357776    
## [777,]  0.02997928 -0.00112613  0.02429631  0.1996             0.4209138    
## [778,]  0.02864239 -0.00112613  0.00207129  0.6541             0.2565271    
## [779,]  0.03177292 -0.00112613  0.00715129  0.3890             0.3486243    
## [780,] -0.11347628 -0.00112613  0.00234786 -2.3187             0.9897934    
## [781,] -0.06769520 -0.00112613  0.00402893 -1.0488             0.8528565    
## [782,]  0.12848631 -0.00112613  0.04017132  0.6467             0.2589199    
## [783,] -0.00712678 -0.00112613  0.02496473 -0.0380             0.5151475    
## [784,] -0.08851351 -0.00112613  0.00218732 -1.8685             0.9691537    
## [785,]  0.13738586 -0.00112613  0.16399644  0.3420             0.3661624    
## [786,] -0.03678925 -0.00112613  0.00550029 -0.4809             0.6846951    
## [787,] -0.08492060 -0.00112613  0.00284290 -1.5716             0.9419751    
## [788,] -0.09964728 -0.00112613  0.00715129 -1.1650             0.8779966    
## [789,] -0.08023205 -0.00112613  0.00302379 -1.4386             0.9248649    
## [790,] -0.08366312 -0.00112613  0.00592182 -1.0726             0.8582653    
## [791,]  0.11895950 -0.00112613  0.04606776  0.5595             0.2879136    
## [792,] -0.03488295 -0.00112613  0.00377609 -0.5493             0.7086135    
## [793,] -0.00072994 -0.00112613  0.00341958  0.0068             0.4972971    
## [794,]  0.01218185 -0.00112613  0.00363600  0.2207             0.4126635    
## [795,]  0.16587936 -0.00112613  0.03085115  0.9508             0.1708497    
## [796,] -0.10131418 -0.00112613  0.00233587 -2.0730             0.9809121    
## [797,]  0.05038467 -0.00112613  0.02140994  0.3520             0.3624046    
## [798,] -0.02833783 -0.00112613  0.00774094 -0.3093             0.6214477    
## [799,] -0.05078441 -0.00112613  0.00582356 -0.6507             0.7423879    
## [800,]  0.05159796 -0.00112613  0.01627525  0.4133             0.3397004    
## [801,] -0.06693797 -0.00112613  0.00550029 -0.8874             0.8125637    
## [802,] -0.01524006 -0.00112613  0.00618011 -0.1795             0.5712413    
## [803,] -0.05044694 -0.00112613  0.00475783 -0.7150             0.7627056    
## [804,] -0.11515383 -0.00112613  0.00235992 -2.3473             0.9905440    
## [805,]  0.15322833 -0.00112613  0.04842633  0.7014             0.2415204    
## [806,]  0.11895950 -0.00112613  0.04606776  0.5595             0.2879136    
## [807,]  0.01154473 -0.00112613  0.02366131  0.0824             0.4671749    
## [808,] -0.05936362 -0.00112613  0.00400248 -0.9205             0.8213523    
## [809,] -0.09930092 -0.00112613  0.00245960 -1.9796             0.9761233    
## [810,]  0.07240212 -0.00112613  0.07509636  0.2683             0.3942285    
## [811,] -0.03070760 -0.00112613  0.00322205 -0.5211             0.6988650    
## [812,] -0.09802038 -0.00112613  0.00262035 -1.8929             0.9708118    
## [813,]  0.02272094 -0.00112613  0.00634441  0.2994             0.3823207    
## [814,] -0.02118357 -0.00112613  0.00651629 -0.2485             0.5981149    
## [815,]  0.28204755 -0.00112613  0.16399644  0.6993             0.2421963    
## [816,] -0.05319090 -0.00112613  0.00798437 -0.5827             0.7199428    
## [817,] -0.05802221 -0.00112613  0.00326017 -0.9965             0.8404882    
## [818,]  0.02376798 -0.00112613  0.00978201  0.2517             0.4006366    
## [819,] -0.09830858 -0.00112613  0.00210211 -2.1196             0.9829813    
## [820,] -0.11764759 -0.00112613  0.00326017 -2.0407             0.9793613    
## [821,] -0.09719512 -0.00112613  0.00235992 -1.9776             0.9760120    
## [822,]  0.07034691 -0.00112613  0.01228278  0.6449             0.2594953    
## [823,]  0.03043750 -0.00112613  0.00592182  0.4102             0.3408423    
## [824,] -0.10080094 -0.00112613  0.09795638 -0.3185             0.6249359    
## [825,] -0.00894166 -0.00112613  0.00537080 -0.1066             0.5424645    
## [826,]  0.19779171 -0.00112613  0.10896306  0.6026             0.2733851    
## [827,] -0.08485889 -0.00112613  0.00215484 -1.8038             0.9643685    
## [828,]  0.21464288 -0.00112613  0.09795638  0.6894             0.2452851    
## [829,] -0.07649316 -0.00112613  0.00213354 -1.6317             0.9486249    
## [830,] -0.01251014 -0.00112613  0.00416544 -0.1764             0.5700048    
## [831,] -0.02604552 -0.00112613  0.00427999 -0.3809             0.6483630    
## [832,] -0.04541239 -0.00112613  0.00368180 -0.7299             0.7672619    
## [833,] -0.13923164 -0.00112613  0.16399644 -0.3410             0.6334599    
## [834,]  0.00028769 -0.00112613  0.00870421  0.0152             0.4939546    
## [835,] -0.09602126 -0.00112613  0.00228875 -1.9836             0.9763473    
## [836,] -0.10795554 -0.00112613  0.00302379 -1.9427             0.9739763    
## [837,]  0.09288701 -0.00112613  0.03191632  0.5262             0.2993615    
## [838,] -0.01073747 -0.00112613  0.01284841 -0.0848             0.5337870    
## [839,]  0.12795158 -0.00112613  0.04842633  0.5866             0.2787505    
## [840,] -0.06499924 -0.00112613  0.02248202 -0.4260             0.6649428    
## [841,] -0.09240745 -0.00112613  0.00213354 -1.9762             0.9759340    
## [842,]  0.03927249 -0.00112613  0.02429631  0.2592             0.3977494    
## [843,] -0.09999933 -0.00112613  0.00365879 -1.6346             0.9489329    
## [844,] -0.08177963 -0.00112613  0.00331861 -1.4001             0.9192516    
## [845,]  0.04803847 -0.00112613  0.00339902  0.8433             0.1995337    
## [846,] -0.04562409 -0.00112613  0.00339902 -0.7632             0.7773411    
## [847,] -0.04172931 -0.00112613  0.00607455 -0.5210             0.6988021    
## [848,] -0.11679051 -0.00112613  0.00276582 -2.1993             0.9860723    
## [849,]  0.04244137 -0.00112613  0.04196589  0.2127             0.4157906    
## [850,]  0.03019558 -0.00112613  0.01193051  0.2868             0.3871487    
## [851,]  0.09288701 -0.00112613  0.03191632  0.5262             0.2993615    
## [852,] -0.05312059 -0.00112613  0.00743595 -0.6030             0.7267325    
## [853,] -0.01209037 -0.00112613  0.02641298 -0.0675             0.5268937    
## [854,]  0.14961992 -0.00112613  0.04392360  0.7193             0.2359847    
## [855,]  0.04038858 -0.00112613  0.00416544  0.6432             0.2600349    
## [856,] -0.08402141 -0.00112613  0.00204105 -1.8349             0.9667368    
## [857,] -0.09530448 -0.00112613  0.00249847 -1.8841             0.9702272    
## [858,] -0.91681409 -0.00112613  0.49419676 -1.3026             0.9036374    
## [859,]  0.15318127 -0.00112613  0.04392360  0.7363             0.2307829    
## [860,]  0.36630339 -0.00112613  0.24654652  0.7400             0.2296537    
## [861,] -0.07918103 -0.00112613  0.00217642 -1.6731             0.9528488    
## [862,] -0.08315499 -0.00112613  0.00215484 -1.7671             0.9613936    
## [863,]  0.00487496 -0.00112613  0.01143554  0.0561             0.4776239    
## [864,]  0.16758878 -0.00112613  0.08144636  0.5912             0.2772009    
## [865,] -0.11824289 -0.00112613  0.00344033 -1.9967             0.9770727    
## [866,] -0.00924469 -0.00112613  0.02429631 -0.0521             0.5207693    
## [867,] -0.10948627 -0.00112613  0.00298968 -1.9818             0.9762486    
## [868,]  0.02997928 -0.00112613  0.02429631  0.1996             0.4209138    
## [869,] -0.06688415 -0.00112613  0.00397629 -1.0428             0.8514844    
## [870,]  0.14091058 -0.00112613  0.02496473  0.8990             0.1843388    
## [871,] -0.12295228 -0.00112613  0.00194429 -2.7629             0.9971352    
## [872,]  0.03262840 -0.00112613  0.00210211  0.7362             0.2308002    
## [873,] -0.11447589 -0.00112613  0.00256518 -2.2380             0.9873898    
## [874,]  0.36630339 -0.00112613  0.19701647  0.8278             0.2038933    
## [875,]  0.12309529 -0.00112613  0.01997290  0.8790             0.1897078    
## [876,] -0.09588223 -0.00112613  0.00232397 -1.9656             0.9753268    
## [877,] -0.05288189 -0.00112613  0.00577546 -0.6810             0.7520734    
## [878,] -0.07220040 -0.00112613  0.00766266 -0.8119             0.7915862    
## [879,] -0.10645509 -0.00112613  0.00237208 -2.1626             0.9847154    
## [880,] -0.13481642 -0.00112613  0.00226567 -2.8087             0.9975127    
## [881,] -0.02121280 -0.00112613  0.02366131 -0.1306             0.5519476    
## [882,] -0.04103193 -0.00112613  0.06080884 -0.1618             0.5642792    
## [883,]  0.15391250 -0.00112613  0.02305728  1.0210             0.1536216    
## [884,] -0.02218768 -0.00112613  0.00582356 -0.2760             0.6087227    
## [885,]  0.02997928 -0.00112613  0.02429631  0.1996             0.4209138    
## [886,] -0.05936362 -0.00112613  0.00400248 -0.9205             0.8213523    
## [887,] -0.06693797 -0.00112613  0.00550029 -0.8874             0.8125637    
## [888,]  0.04744826 -0.00112613  0.02366131  0.3158             0.3760838    
## [889,]  0.11895950 -0.00112613  0.04606776  0.5595             0.2879136    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Okay, you made it! Thank you for scrolling through all that.

Next we are going to plot the spatial data against its spatially lagged values.

# create a moran.plot
nci <- moran.plot(dup_waste$log_odds, listw=weights, 
     xlab="Log prevalence", ylab="Spatially lagged log prev",
     labels=T, pch=16, col="grey")

The output is a little crowded in this case, but the moran.plotfunction allows us to conveniently extract those statistically significant points we saw in the verbose printCoefmat output.

# find which points are statistically significant outliers
infl <- nci$is_inf == T 
outlier_points <- sum(infl==T)  

paste(outlier_points, "significant outlier points.")

## [1] "83 significant outlier points."

Now that we have extracted the significant outliers in our distribution of points, we coerce them into a factors with four total classes defined as:

• High value points correlated around other High value points.
• High value points correlated around Low value points.
• Low value points correlated around High value points.
• Low value points correlated around other Low value points.

Once we have created these classes, we can project the points onto a map and view our results.

# create vector of logged odds of events
x <- dup_waste$log_odds

# cut into factor with two levels.
lhx <- cut(x, breaks=c(min(x), 
                       mean(x), 
                       max(x)), labels=c("L", "H"),
                       include.lowest=T)

# lag, based upon weights of the data
wx <- stats::lag(weights, dup_waste$log_odds)

# create factor with 2 levels 
lhwx <- cut(wx, breaks=c(min(wx), mean(wx), 
                     max(wx)), labels=c("L", "H"), 
                     include.lowest=T)

# compute a factor interaction
lhlh <- interaction(lhx,lhwx,infl,drop=T)

# add column names to output
names <- rep("none", length(lhlh))
names[lhlh=="L.L.TRUE"]<-"Low with Low"
names[lhlh=="H.L.TRUE"]<-"High with Low"
names[lhlh=="L.H.TRUE"]<-"Low with High"
names[lhlh=="H.H.TRUE"]<-"High with High"

# bind names with coordinate observations
dup_waste_lisa <- as.data.frame(cbind(xy, names))

# tidy the dataframe
dup_waste_lisa <- dup_waste_lisa %>% 
# rename columns
  rename(longitude = V1) %>% 
  rename(latitude = V2) %>% 
# structure coordinates
  mutate(longitude = as.numeric(longitude)) %>% 
  mutate(latitude  = as.numeric(latitude)) %>% 
# filter non-significant values in the map
  filter(names != "none") 

# create palette for map
factpal <- colorFactor(c( "red","blue","orange","purple"), names)

# map LISA values
leaflet(dup_waste_lisa) %>% 
  addProviderTiles("CartoDB.Positron") %>% 
  addPolygons(data=sf_nhoods,
              fillOpacity=0, weight = 1, 
              color = "grey", opacity = .8) %>% 
  addCircleMarkers(~longitude, ~latitude, fillOpacity=.4,
  color= ~factpal(names), radius=3, stroke=TRUE, weight=.5) %>% 
  addLegend(pal = factpal, values = ~names, 
            title="Event Correlation Class", position = "bottomleft") 

Here we are with our mapped results. The LISA method allows us to articulate local PSA relationships in the duplicated point areas based on the intensity of events at the location. With this method, we are able to characterize the autocorrelative relationship in reference to other points around each significant PSA location. Analyses like these could help determine how much, and even what kind of sanitary intervention to apply to a certain area in San Francisco.

Conclusion:

Crowdsourcing data from 311 certainly has its limitations, some of which we have been able to account for in this case study, others may require increased access, research, understanding, and experimentation. That being said, there is a wealth of availble data with valuable analyses to put together from the observations of observations we are able to gather with 311, available to anyone who cares!

Questions, comments, concerns? Feel free to contact me: avery.richards@berkeley.edu