Atlas of Deprivation in Wales

1. Deprivation

People are described as being deprived when they are lacking in resources considered to be basic necessities in society. While income poverty plays a major role in determining the level of socio-economic deprivation experienced by an individual or community, health, access to services, and the quality of the built and natural environment are also important factors. People who are more deprived are likely to be more vulnerable to (and less able to recover from) a range of shocks and challenges. The current Coronavirus pandemic, for example, has thrown into sharp focus the link between deprivation and vulnerability to health and economic shocks. In Wales (and the wider UK), preliminary analysis of official data on Coronavirus infection and mortality rates has found the impact of the disease has been greater in more deprived areas of the country - a phenomenon that has been widely discussed and reported. The short and long-term economic impacts of the Coronavirus lockdown and social distancing measures are also likely to be more severe in areas where deprivation is higher.

2. Welsh Index of Multiple Deprivation (WIMD)

Produced by the Welsh Government, the Welsh Index of Multiple Deprivation (WIMD) is a dataset providing statistics on relative deprivation for small geographical areas across Wales. Used widely by researchers and policy/decision-makers, WIMD ranks these small areas (called Lower Super Output Areas - LSOAs) from 1 (most deprived) to 1909 (least deprived). The average population of a LSOA in Wales is around 1,600 people. As the measure is relative, it is not possible, for example, to say that LSOA A is x times more deprived than LSOA B, only that it is more deprived. When interpreting WIMD data and maps, it is important to keep in mind that there will be many people living in the most deprived areas who may not be deprived. Conversely, there will be people living in less deprived areas who are deprived.

As WIMD LSOA scores are relative rather than absolute, analysing deprivation across larger geographical areas (e.g. local authorities) is done by comparing the proportion of LSOAs within the larger area that are deprived. Typically, this is done by dividing the LSOAs for the whole of Wales into deciles based on deprivation rank, and calculating the proportion of LSOAs within the larger area that are in Decile 1 (10% most deprived in Wales), Decile 2 (20% most deprived in Wales), Decile 3 (30% most deprived in Wales), and so on. WIMD computes an overall index of deprivation based on the weighted sum of 8 sub-domains (listed below), each of which are underpinned by numerous indicator datasets. The figures in brackets refer to the weights (i.e. importance) assigned to each sub-domain when calculating overall (or “multiple”) deprivation.

1. Income (22%)
2. Employment (22%)
3. Health (15%)
4. Education (14%)
5. Access to Services (10%)
6. Housing (7%)
7. Community Safety (5%)
8. Physical Environment (5%)

The current version of WIMD (WIMD 2019 - used in this project) was released in November 2019. For more information on WIMD, visit the Welsh Government’s WIMD site. Those seeking in-depth background and technical information on WIMD should consult the guidance documents and technical reports. Across the rest of the UK, indices of deprivation datasets have also been produced for England, Scotland, and Northern Ireland

3. Project Aim

While there exists a useful set of resources on mapping WIMD data in Wales, these are mainly web-based interactive map tools (e.g. Welsh Government) and static maps of WIMD at national level (e.g. StatsWales). The main aim of this project is to produce a clear, consistent, and informative set of WIMD maps at local authority level, designed for informing decision makers and the general public. An objective of project is to create the maps using open data and free and open source software tools, and to make all of the maps, data, and technical resources (e.g. computer code and software files) accessible and freely available for anyone to use. This reflects the University of Gloucestershire’s commitment to promoting and advocating the use open source geospatial tools and open education as a member of The Open Source Geospatial Foundation (OSGeo)

5. Methodology

This section provides a brief overview of the work undertaken to produce the maps. Anyone with a good knowledge of QGIS and R should be able to reproduce this project using the online resources and the information provided below. A more detailed step-by-step tutorial is planned.

All of the resources necessary to reproduce this work, including input data, R code and QGIS project files are available via the project’s GitHub repository: https://github.com/robertberryuk/WalesDeprivationAtlas

5.3 Data Processing in R

5.3.1 WIMD data

The code for this section can be found in the file Scripts/1_WIMD_Pre-processing.R

Step 1 - WIMD LSOA shapefiles downloaded from the Lle Geoportal do not come with a local authority name or code column in their attribute tables (see below), so the first objective was to add this attribute using the LSOA lookup table downloaded from StatsWales.  

Original WIMD attribute table

 

The R code first loads the required libraries then clears any existing shapefile outputs generated from previous running of the script. The LSOA to local authority lookup is then imported, and each of the 9 x WIMD LSOA shapefiles are loaded using a loop.

#> LOAD LIBRARIES
library(sf) #> spatial data handling
library(tidyverse) #> General data processing 
library(here) #> Create relative paths to files (for better portability and reproducibility)


#> 1. CLEAR (DELETE) ANY PREVIOUS SHAPEFILE OUTPUTS GENERATED FROM THIS SCRIPT (otherwise error will be thrown when trying to overwrite files on export)
#> Create list of existing WIMD shapefile outputs
shapes.WIMD <- here("Data", "Out", "Shapefiles", "WIMD")
#> Delete shapefiles
do.call(file.remove, list(list.files(shapes.WIMD, full.names = TRUE)))
#> Create list of any existing local authority stats outputs
shapes.LA <- here("Data", "Out", "Shapefiles", "Local_Authority_Stats")
#> Delete shapefiles
do.call(file.remove, list(list.files(shapes.LA, full.names = TRUE)))


#> 2. IMPORT WIMD LSOA shapefiles (Overall deprivation + 8 deprivation domains) 
#> First import LSOA to local authority lookup table
LA.lookup <- read_csv(here("Data", "In", "3_lookups", "LSOA_LA_Lookup.csv"))
#> Create list of WIMD domain names for use in import loop, processing, and export
domain <- c("access_to_services", "community_safety", "education", "employment", 
            "health", "housing", "income", "overall", "physical_environment")
#> For loop to import WIMD shapefiles 
#> (controlled by iterating through the WIMD domains in the "domain" character list)
for (i in 1:length(domain)) {
  #> Import file and sore as sf object "shp"
  shp <- st_read(here("Data", "In", "1_Shapefiles", "WIMD", paste0("WIMD2019_", domain[i], ".shp")))

 

Step 2 - Continuing in the loop, merge each of the imported shapefiles with the LSOA to local authority lookup table to add a local authority name attribute, and export updated LSOA shapefiles for use in QGIS.

#> 3. MERGE
#> Merge WIMD LSOA domain shapefiles with LSOA-locaL authority lookup file to add local authority attribute to each imported WIMD shapefile
wimd.merge <- merge(shp, LA.lookup, by.x = "lsoa_code", by.y = "LSOA")

#> 4. EXPORT MERGED WIMD SHAPEFILES 
st_write(wimd.merge, here("Data", "Out", "Shapefiles", "WIMD", (paste0("WIMD_", domain[i], ".shp"))))

 

Step 3 - Now that a local authority name attribute has been appended to each of the 9 WIMD shapefiles, continue processing the shapefile attribute tables in the loop to generate a set of summary statistics for each domain per local authority (no of LSOAs in each WIMD decile, total number of LSOAs in each local authority, and percentage of LSOAs in each decile) - these statistical tables (one for each domain) are then joined to a local authority boundary shapefile and the resulting exported shapefiles are then used to drive the Atlas layout in QGIS. Also generated are statistics showing how the target local authority ranks alongside other local authorities for a particular deprivation domain, and a “Map_No” column is added to populate the “Map No” label in the QGIS Atlas layout. Note the local authority boundary shapefile was created by dissolving a WIMD LSOA polygon layer in QGIS rather than downloading an existing shapefile of local authorities - this was to ensure the LSOA boundaries and local authority boundaries were an exact fit when mapped.

#> 5. GENERATE LOCAL AUTHORITY STATS FOR VISUALISTION IN QGIS Atlas:
#> "Bargend" stats - group by local authrity and count number of LSOAs in each decile, 
#> total number of LSOAs, and percentage of LSOAs in each decile 
#> Create non-geo version of LSOA data
LSOA.stats <- wimd.merge
st_geometry(LSOA.stats) <- NULL
#> Create a pivoted table showing number of LSOAs in each decile, with deciles 1-10 as columns
LSOA.sum <- LSOA.stats %>% 
  group_by(LA_Name, decile) %>% 
  tally() %>% 
  pivot_wider(names_from = decile, values_from = n) %>% 
  ungroup()
#> Add the "decile" prefix to the newly-generated LSOA-decile count columns
colnames(LSOA.sum)[2:11] <- paste("Dec_", colnames(LSOA.sum[,c(2:11)]), sep = "")
#> Change the NAs that are generated to zeros
LSOA.sum[is.na(LSOA.sum)] <- 0
#> Add a column for total mumber of LSOAs in each LA
LSOA.sum$Total_LSOAs <- rowSums(LSOA.sum[2:11])
#> Create a new dataframe of percentage columns (i.e percentage of LSOAs in each decile, for each LA)
df.percent <- (LSOA.sum[, 2:11] / LSOA.sum[[12]] * 100)
#> Round (to 1 decimal point)
df.percent <- round(df.percent, digits = 1)
#> Change column prefix from "Decile_1" to "PC_Decile_1 ("PC" = percent)
colnames(df.percent)[1:10] <- paste("PC_", colnames(df.percent[,c(1:10)]), sep = "")
#> Append percentage columns to LSOA sum dataframe using cbind
LSOA.sum <- data.frame(cbind(LSOA.sum, df.percent))
#> Add percentage total column (for checking)
LSOA.sum$PC_Total <- rowSums(LSOA.sum[13:22])
#> Add columns showing the percentage of LSOAs in: decile 1, (most deprived 10% LSOAs in Wales); 
#> deciles 1-2 (most deprived 20% LSOAs in Wales); 
#> deciles 1-3 (most deprived 30% LSOAs in Wales); deciles 1-5 (most deprived 50% LSOAs in Wales)
LSOA.sum$Dep_10 <- LSOA.sum$PC_Dec_1
LSOA.sum$Dep_20 <- LSOA.sum$PC_Dec_1 + LSOA.sum$PC_Dec_2
LSOA.sum$Dep_30 <- LSOA.sum$PC_Dec_1 + LSOA.sum$PC_Dec_2 + LSOA.sum$PC_Dec_3
LSOA.sum$Dep_50 <- LSOA.sum$PC_Dec_1 + LSOA.sum$PC_Dec_2 + LSOA.sum$PC_Dec_3 + LSOA.sum$PC_Dec_4 + LSOA.sum$PC_Dec_5
#> Add rank columns based on deprivation summary columns (i.e columns Dep_10 to Dep_50) - for "Rank" label in QGIS map layout
LSOA.sum <- arrange(LSOA.sum, desc(Dep_10)) %>%
  mutate(Rank_10 = 1:nrow(LSOA.sum))
LSOA.sum <- arrange(LSOA.sum, desc(Dep_20)) %>%
  mutate(Rank_20 = 1:nrow(LSOA.sum))
LSOA.sum <- arrange(LSOA.sum, desc(Dep_30)) %>%
  mutate(Rank_30 = 1:nrow(LSOA.sum))
LSOA.sum <- arrange(LSOA.sum, desc(Dep_50)) %>%
  mutate(Rank_50 = 1:nrow(LSOA.sum))
#> Add a column containing integers 1:22 for map page numbering in QGIS (based on alphabetical order of LAs)
LSOA.sum <- LSOA.sum %>% 
  arrange(LA_Name) %>% 
  mutate(Map_No = 1:nrow(LSOA.sum))

#> 6. MERGE THE LOCAL AUTHORITY DEPRIVATION STATS WITH LOCAL AUTHORITY SPATIAL BOUNDARY LAYER, THEN EXPORT AS SHAPEFILE
#> Import the LA boundary data (this file was created by dissolving an LSOA shapefile in QGIS
LAs <- st_read(here("Data", "In", "1_Shapefiles", "Local_Authorities.shp"))
#> Remove unwanted columns (i.e. all but geom and LA_Name)
LAs <- LAs %>% 
  select(LA_Name)
#> Merge
LA_stats.geo <- merge(LAs, LSOA.sum, by = "LA_Name")
#> Export
st_write(LA_stats.geo, here("Data", "Out", "Shapefiles", "Local_Authority_Stats", paste0("LA_Stats_", domain[i], ".shp")))

#> End the loop
}

The output from this code comprises 9 x local authority boundary shapefiles with summary WIMD statistics - one shapefile for each deprivation theme (see example attribute table for overall deprivation below):

Click HERE for a larger version

 

5.3.2 Place name data

The code for this section can be found in the file Scripts/Placename_Preparation.R. This script processes the OS Open Names place name dataset ready for visualisation and further (manual) editing in QGIS.

Step 1 - There are two place name columns in the OS Open Names dataset - NAME1 and NAME2. Confusingly, both columns contain a mix of place names in Welsh and English, indicated by the columns NAME1_LANG and NAME2_LANG:

 

The code below extracts the English place names (unfortunately I don’t have the resources to produce a Welsh version of the Atlas at the moment) from the NAME1 and NAME2 columns and consolidates them into a new column called “NAME_USE”

#> PREPARATION OF PLACENAME DATA FOR MAPPING IN QGIS

#> LIBRARIES ----
library(tidyverse)
library(here)
library(sf)

#> PLACENAMES processing using OS OpenNames
#> Merge NAME1 and NAME2 into a new column such that only english placenames are retained
#> Import data
placenames <- st_read(here("Data", "In", "1_Shapefiles", "OS_OpenNames_Populated_Place.shp"))
#> Get English placenames from column NAME1_LANG
places.eng1 <- placenames %>% 
  filter(NAME1_LANG == "eng") %>% 
  mutate(NAME_USE = NAME1)
#> Get English placenames from column NAME2_LANG
places.NA1 <- placenames %>% 
  filter(is.na(NAME1_LANG)) %>% 
  mutate(NAME_USE = NAME1)
#> Get English placenames from column NAME2_LANG
places.eng2 <- placenames %>% 
  filter(NAME2_LANG == "eng") %>% 
  mutate(NAME_USE = NAME2)
#> Bind the above dataframes together and use the new consolidated English language placename column "NAME_USE" for cartographic labelling in QGIS
placenames.new <- rbind(places.eng1, places.NA1, places.eng2)
#> Keep only required fields
placenames.new <- placenames.new %>% 
  select(NAME_USE, COUNTY_UNI, LOCAL_TYPE, MOST_DETAI, LEAST_DETA)
#> Remove text before hyphen in "COUNTY_UNI" place name field (i.e. all welsh text - makes string too long)
placenames.new$COUNTY_UNI <- gsub(".*- ", "", placenames.new$COUNTY_UNI)

#> Change LOCAL_TYPE for the city of St Davids to "Town" to reduce size of map label
placenames.new <- placenames.new %>% 
  mutate(LOCAL_TYPE=replace(LOCAL_TYPE, NAME_USE =="St Davids", "Town"))

Step 2 - The following code then sorts the place names into separate shapefiles based on hierarchy (i.e. cities, towns, and “other” minor settlements). Shapefiles for minor settlements are then created for each local authority, to allow for faster map rendering and easier place name editing in QGIS.

#> Create hierarchy subsets of the placename data for mapping - 
#> e.g. cities, towns, other places etc (per LA for "other" settlements
#> Get unique categories
places.unique <- as.data.frame(unique(placenames.new$LOCAL_TYPE))

#> Cities
places.cities <- placenames.new %>% 
  filter(LOCAL_TYPE == "City")
#> Towns
places.towns <- placenames.new %>% 
  filter(LOCAL_TYPE == "Town")
#> Change placename "Bargod" to "Bargoed"
places.towns<- places.towns %>% 
  mutate(NAME_USE = replace(NAME_USE, NAME_USE == "Bargod", "Bargoed"))

#> Export placename shapefiles
#> Remove any existing exported placename files prior to export
placenames.files <- here("Data", "Out", "Shapefiles", "Placenames")
#> Delete shapefiles
do.call(file.remove, list(list.files(placenames.files, full.names = TRUE)))
#> Export as shapefiles
#> Export data
st_write(placenames.new, here("Data", "Out", "Shapefiles", "Placenames", "Placenames_All.shp"))
st_write(places.cities, here("Data", "Out", "Shapefiles", "Placenames", "places.cities.shp"))
st_write(places.towns, here("Data", "Out", "Shapefiles", "Placenames", "places.towns.shp"))

#> All other settlements 
other.all <- c("Hamlet", "Other Settlement", "Suburban Area", "Village")
places.other <- placenames.new %>% 
  filter(LOCAL_TYPE %in% other.all)
#> Export as shapefile
st_write(places.other, here("Data", "Out", "Shapefiles", "Placenames", "places.other.all.shp"))
#> Create a layer of placenames for each LA 
#> Use a loop to create files based on local authority name variable
LA.names <- unique(places.other$COUNTY_UNI)
#> Check fields
glimpse(places.other)
#> Remove any existing previously exported placename files prior to export
LA.placename.files <- here("Data", "Out", "Shapefiles", "Placenames", "Placenames_by_LA")
#> Delete shapefiles
do.call(file.remove, list(list.files(LA.placename.files, full.names = TRUE)))
#> loop to create and export new shapefiles
for (i in 1:length(LA.names)) {
  df <- places.other %>% 
    filter(COUNTY_UNI== LA.names[i])
  st_write(df, here("Data", "Out", "Shapefiles", "Placenames", "Placenames_by_LA", paste0("LA.names", LA.names[i], ".shp")))
  df <- NULL
}

6. Acknowledgements

Thanks to Prof. Alasdair Rae for producing an extremely useful set of learning materials on QGIS Atlas, and for making the data and QGIS project resources freely available for all to use. Thanks also to Dr Richard Fry, Swansea University, for reviewing drafts of the map design. This work was funded by the Countryside and Community Research Institute (CCRI), University of Gloucestershire www.ccri.ac.uk.

Research collaboration/business enquries

If your organisation is looking for help with a geospatial/mapping project, please do get touch: rberry@glos.ac.uk

© Robert Berry, 2020