Analysing the frontline of diplomacy: accompanying code

Github repository link

0.0 Read me

|| 0.0|READ ME || 1.0|EXTRACT || 2.0|TRANSFORM || 3.0|LOAD and ANALYSE || 4.0|ACLED data || 5.0|ACLED data ||

Purpose of section| brief outline of document, its content and prerequisites
Inputs | none
Outputs | none

---

0.1 Outline of document

This notebook outlines the process and code outputs associated with the project Analysing the frontlines of diplomacy: the case for unleashing GIS (A proof of concept algorithm supplemented with i| indicative clustering analysis of diplomatic post locations and ii| conflict events within cities)

Final submission can be found here 13.01.2020

---

0.2 Document index

0. Read me
   • || 0.1-Outline || 0.2-Index || 0.3-Setup ||
1. Extract
   • || 1.1-Load || 1.2-Tidy || 1.3-Summarise ||
2. Transform
   • || 2.1-Reload data || 2.2-ISO encoding || 2.3-Geocoding ||
3. Load and analyse
   • || 3.1-Load and segment data frame || 3.2-Define visualisation function || 3.3-Visualise 1.0 || 3.4-Visualise 2.0, clustering and outlier removal ||
4. ACLED data
   • || 4.1-ACLED api function || 4.2-ACLED function output
5. Case studies
   • || 5.1-Case 1 - Cairo || 5.2-Case 2 - Ankara

• Reference code section

Overview of the steps of this workflow in the chart below (Appendix G.2 in the submitted work)

Appendix G.2 | complete process flow

---

0.3 Pre-requisites and set up

In order for the code below to work, please ensure you have installed and loaded the relevant packages.

#Installing packages (uncomment as required)
#install.packages(c("countrycode","tidyverse","sf","mapview","ggmap","RCurl","jsonlite","RCurl","jsonlite"))

library(countrycode) #iso tagging of countries
library(tidyverse) #number of packages for data manipulation, specifically want readr, dplyr, tibble
library(sf) #creation of spatial objects
library(mapview) #mapping package
library(ggmap) #for google api access 
library(RCurl) #to download ACLED content
library(jsonlite) #to process ACLED downloaded info

Setting to use local files or refresh geocoding via API calls.

#Set this to TRUE or FALSE to determine whether to use local, pre-loaded files or refresh via relevant API, i.e. (Google) connections.
use_local_files <- TRUE

---

1.0 Extract GDI dataset

|| 0.0|READ ME || 1.0|EXTRACT || 2.0|TRANSFORM || 3.0|LOAD and ANALYSE || 4.0|ACLED data || 5.0|ACLED cases ||

Purpose of section | load GDI dataset from Lowy Institute site and summarise
Inputs | GDI dataset available here
Outputs| Renamed and separated .csv files

Note | downloading from the Lowy Institute site, there may be incompatibility with downloads in Safari, chrome is recommended, have not tested other browsers.

---

1.1 Load .csv file and check data types

Back to index

Load 2019 GDI dataset

###open latest csv document based on relative path
lowy_raw_data <- read.csv("./Data/Raw_data/2019_LowyInstituteGlobalDiplomacyIndex_FullDataset.csv")

1.2 Tidy up dataset

Back to index
Check for| factors; rename columns; seperate out relevant columns
Output|

1. renamed data frame lowy_renamed
2. other meta data lowy_meta
3. ranking data lowy_rank
4. posting data lowy_posting <=== this is the relevant one for the rest of the work

Tidy dataset, see preview for table.

###check for factor columns and change them to character columns
#more of a pre-emptive measure
i <- sapply(lowy_raw_data, is.factor)
lowy_raw_data[i] <- lapply(lowy_raw_data[i], as.character)

###column rename
#new data frame (a)
lowy_renamed <- lowy_raw_data

#rename columns
lowy_renamed <- lowy_renamed %>% dplyr::rename(
  send_country=COUNTRY,
  #metadata
  meta_gdp_usd_billions = GDP..B..USD.,
  meta_population_millions = POPULATION..M.,
  #ranking columns
  rank_g20 = G20.RANK,
  rank_OECD = OECD.RANK,
  rank_asia = ASIA.RANK,
  rank_overall = OVERALL.RANK,
  #post country info
  post_city = POST.CITY,
  post_country = POST.COUNTRY,
  post_type = TYPE.OF.POST,
  post_title = POST.TYPE.TITLE
  )

#data frame with all metadata (GDP and population) (b)
lowy_meta <- lowy_renamed[, c("send_country","meta_population_millions","meta_gdp_usd_billions")]

#data frame with all rankings (G20, OECD, Asia and overall) (c)
lowy_rank <- lowy_renamed[,c("send_country","rank_g20","rank_OECD","rank_asia","rank_overall")]

#data frame with information on sending country, post city, country as well as post type and title (d)
lowy_posting <- lowy_renamed[,c("send_country","post_city","post_country","post_type","post_title")]

###save data frames as .csv files ready for part 2
write.csv(lowy_posting,"./Data/Processing/Part_1_Out_Lowy_Posting.csv", row.names = FALSE)
write.csv(lowy_rank,"./Data/Processing/Part_1_Out_Lowy_Rank.csv", row.names = FALSE)
write.csv(lowy_meta,"./Data/Processing/Part_1_Out_Lowy_Meta.csv", row.names = FALSE)

#quick preview of data frame to work on
head(lowy_posting,3)

1.3 Raw key information

Back to index

#Length of dataframe:
cat("Lowy Institute's GDI dataset has",length(lowy_posting[[1]]),"entries.")

Lowy Institute's GDI dataset has 7317 entries.

###report all unique values/entries
#Summarise all unique values in crude print out
for(column in colnames(lowy_posting)){
  cat(column,"has",length(apply(lowy_posting[column],2,unique)),"unique values\n")
}

send_country has 61 unique values
post_city has 686 unique values
post_country has 207 unique values
post_type has 4 unique values
post_title has 141 unique values

###summary stats
lowy_type_table <- ftable(table(lowy_posting$send_country,lowy_posting$post_type))

###summarise count by send country
lowy_count_per_send_country <- lowy_posting %>% 
  group_by(send_country) %>% 
  summarise(count = n()) %>% 
  arrange(desc(count))

per_country_count <- lowy_count_per_send_country$count
par(family = "Times New Roman")
per_country_box <- boxplot(data=per_country_count, x=per_country_count, main="Diplomatic post count by sending country", 
   xlab="All post types", ylab="Count of posts",notch = TRUE)

cat("Per country count summary",
    "\nMin value:",per_country_box$stats[1,1],
    "\n25th percentile:",per_country_box$stats[2,1],
    "\nMedian value:",per_country_box$stats[3,1],
    "\n75th percentile:",per_country_box$stats[4,1],
    "\nMax value:",per_country_box$stats[5,1]
    )

Per country count summary 
Min value: 9 
25th percentile: 67 
Median value: 107 
75th percentile: 152 
Max value: 276

###summarise count by post city
lowy_count_per_post_city <- lowy_posting %>% 
  group_by(post_city) %>% 
  summarise(count = n()) %>% 
  arrange(desc(count))

per_post_city_count <- lowy_count_per_post_city$count

per_post_city_box <- boxplot(data=per_post_city_count, x=per_post_city_count, main="Diplomatic post count by post city ", 
   xlab="All post types", ylab="Count of posts",notch = TRUE)

cat("Per post city count summary",
    "\nMin value:",per_post_city_box$stats[1,1],
    "\n25th percentile:",per_post_city_box$stats[2,1],
    "\nMedian value:",per_post_city_box$stats[3,1],
    "\n75th percentile:",per_post_city_box$stats[4,1],
    "\nMax value:",per_post_city_box$stats[5,1]
    )

Per post city count summary 
Min value: 1 
25th percentile: 1 
Median value: 2 
75th percentile: 14 
Max value: 33

###summarise count by post country
lowy_count_per_post_country <- lowy_posting %>% 
  group_by(post_country) %>% 
  summarise(count = n()) %>% 
  arrange(desc(count))

per_post_country_count <- lowy_count_per_post_country$count

per_post_country_box <- boxplot(data=per_post_country_count, x=per_post_country_count, main="Diplomatic post count by post country", 
   xlab="All post types", ylab="Count of posts",notch = TRUE)

cat("Per post country count summary",
    "\nMin value:",per_post_country_box$stats[1,1],
    "\n25th percentile:",per_post_country_box$stats[2,1],
    "\nMedian value:",per_post_country_box$stats[3,1],
    "\n75th percentile:",per_post_country_box$stats[4,1],
    "\nMax value:",per_post_country_box$stats[5,1]
    )

Per post country count summary 
Min value: 1 
25th percentile: 9 
Median value: 23 
75th percentile: 44.5 
Max value: 91

---

2.0 Transform GDI data

|| 0.0|READ ME || 1.0|EXTRACT || 2.0|TRANSFORM || 3.0|LOAD and ANALYSE || 4.0|ACLED data || 5.0|ACLED data ||

Purpose of section | Load,.csv files; add country iso codes to main data frame; and geocode send country+post city+post+country+exact embassy location with Google API
Inputs | Part_1_Out_Lowy_Posting.csv from end of part 1
Outputs | lowy_data_iso master geocoded table

---

2.1 Re-load

Back to index

lowy_data <- read_csv("./Data/Processing/Part_1_Out_Lowy_Posting.csv")

2.2 ISO codes

Back to index

###Set up

#iso dataframe
lowy_data_iso <- lowy_data

#country_iso list [column 1]
iso_list_send_country <- c()
iso_list_send_country_num <- c()
#post_country_iso list [column 3]
iso_list_post_country <- c()
iso_list_post_country_num <- c()

###ending country column (1) three digit numbers
for(country in lowy_data_iso[1]){
  country_iso <- countrycode(
    #call the entry of 'Country' for the row_number currently being taken by for loop
    sourcevar = country, 
    #three letter iso code
    destination = "iso3n",
    #based on english language names
    origin = "country.name")
  
  #add value to a list
  iso_list_send_country_num <- append(iso_list_send_country_num,country_iso)
}

###sending country column (1) three digit letters
for(country in lowy_data_iso[1]){
  country_iso <- countrycode(
    #call the entry of 'Country' for the row_number currently being taken by for loop
    sourcevar = country, 
    #three letter iso code
    destination = "genc3c", 
    #based on english language names
    origin = "country.name")
  
  #add value to a list
  iso_list_send_country <- append(iso_list_send_country,country_iso)
}
#===================================
###Post country column (3) three digit numbers
for(country in lowy_data_iso[3]){
  post_country_iso <- countrycode(
    #call the entry of 'Country' for the row_number currently being taken by for loop
    sourcevar = country, 
    #three letter iso code
    destination = "iso3n", 
    #based on english language names
    origin = "country.name")
  
  #add value to a list
  iso_list_post_country_num <- append(iso_list_post_country_num,post_country_iso)
}

Some values were not matched unambiguously: Abkhazia*, Cura�ao, Jerusalem*, Kosovo*, South Ossetia*

###Post country column (3) three letters
for(country in lowy_data_iso[3]){
  post_country_iso <- countrycode(
    #call the entry of 'Country' for the row_number currently being taken by for loop
    sourcevar = country, 
    #three letter iso code
    destination = "genc3c", 
    #based on english language names
    origin = "country.name")
  
  #add value to a list
  iso_list_post_country <- append(iso_list_post_country,post_country_iso)
}

Some values were not matched unambiguously: Abkhazia*, Cura�ao, Jerusalem*, South Ossetia*

###create new columns
lowy_data_iso$send_country_iso = iso_list_send_country
lowy_data_iso$send_country_iso_num = iso_list_send_country_num

lowy_data_iso$post_country_iso = iso_list_post_country
lowy_data_iso$post_country_iso_num = iso_list_post_country_num

Click arrows to see all iso columns in data frame

###preview data frame
head(lowy_data_iso,3)

2.3 Geo-coding

Back to index
This section requires a google api key; you can set one up here through the google cloud platform

register_google(key = "")

Code to run the geocoding for each country column (send_country post_country), city post_city + post country (to avoid finding wrong city)

Part 1 cities and countries
Back to index
Generate geocoding queries:

###========Preparation

###concatenate post city and post country columns
lowy_data_iso$post_city_country = paste(
  lowy_data_iso$post_city,
  lowy_data_iso$post_country,
  sep=", ")

###geo-c - concatenated city and country names
geo_city_country <- distinct(lowy_data_iso,post_city_country)
geo_city_country_df <- as.data.frame(geo_city_country)
#geo_city_country_df <- head(geo_city_country_df,4) #for sample work only

###geo-code - sending countries
geo_send_country <- distinct(lowy_data_iso,send_country)
geo_send_country_df <- as.data.frame(geo_send_country)

Cities hosting diplomatic outposts geocoded

###========POST CITIES
#if local file tag is set to TRUE, google api will not be used
if(use_local_files) {
  #print("Local file flag set to TRUE, using local files")
  geo_city_country_df_dl <- read.csv("./Data/GoogleAPI/Geo_City_Country_DL.csv")
} else {
  #print("Local file flag set to FALSE, calling google now!")
  geo_city_country_df_dl <- mutate_geocode(geo_city_country_df, post_city_country)
  #Save .csv file
  write.csv(geo_city_country_df_dl,"./Data/GoogleAPI/Geo_City_Country_DL.csv", row.names = FALSE)
}
head(geo_city_country_df_dl,3)

Sending countries geocoded

###========SEND COUNTRIES
#if local file tag is set to TRUE, google api will not be used
if(use_local_files) {
  #print("Local file flag set to TRUE, using local files")
  geo_send_country_df_dl <- read.csv("./Data/GoogleAPI/Geo_Send_Country_DL.csv")
} else {
  #print("Local file flag set to FALSE, calling google now!")
  geo_city_country_df_dl <- mutate_geocode(geo_city_country_df, post_city_country)
  #Save .csv file
  write.csv(geo_send_country_df_dl,"./Data/GoogleAPI/Geo_Send_Country_DL.csv", row.names = FALSE)
}

head(geo_send_country_df_dl,3)

Data frame with all countries and cities geocoded (first 50 records)

###========JOIN DATA TO MASTER DATA FRAME
lowy_data_geo <- lowy_data_iso

#Join send lon/lat
lowy_data_geo <- merge(lowy_data_geo,geo_send_country_df_dl, by = "send_country")

lowy_data_geo <- lowy_data_geo %>% rename(
  send_lon =lon,
  send_lat = lat)

#Join post city lon/lat
lowy_data_geo <- merge(lowy_data_geo,geo_city_country_df_dl, by = "post_city_country")

lowy_data_geo <- lowy_data_geo %>% rename(
  post_city_lon =lon,
  post_city_lat = lat)

head(lowy_data_geo,50)

Part 2 exact embassy locations
Back to index
Code to run the geocoding for exact embassy locations. Using combination of send_country+post_country+“Embassy”+ post_city

###master dataframe with all data
lowy_data_geo_master <- lowy_data_geo

###Exact geocoding query
lowy_data_geo_master$full_geo_query = paste(
  lowy_data_geo_master$send_country,
  lowy_data_geo_master$post_title,
  lowy_data_geo_master$post_city_country,
  sep=", ")

###RUN GEOCODING
if(use_local_files) {
  print("Local file flag set to TRUE, using local files")
  lowy_data_geo_master <- read.csv("./Data/GoogleAPI/lowy_data_geo_master.csv")
} else {
  print("Local file flag set to FALSE, calling google now!")
  lowy_data_geo_master <- mutate_geocode(lowy_data_geo_master, full_geo_query)
  
  #rename columns
  lowy_data_geo_master <- lowy_data_geo_master %>% rename(
  exact_post_lon = lon,
  exact_post_lat = lat)
  
  #Save .csv file
  write.csv(geo_send_country_df_dl,"./Data/GoogleAPI/lowy_data_geo_master.csv", row.names = FALSE)
}

[1] "Local file flag set to TRUE, using local files"

First 50 records of all geocoded diplomatic outposts

head(lowy_data_geo_master,50)

---

3.0 Load and analyse Lowy Institute data

|| 0.0|READ ME || 1.0|EXTRACT || 2.0|TRANSFORM || 3.0|LOAD and ANALYSE || 4.0|ACLED data || 5.0|ACLED data ||

Purpose of section | visualise coordinates; and run clustering analysis
Inputs | lowy_data_geo_master (final output of part 2)
Outputs | maps and visuals

---

3.1 Load data and segmet usable part for analysis

Back to index

Load data

###Load data from section 2
lowy_data_complete <- read_csv("./Data/GoogleAPI/lowy_data_geo_master.csv")

Segmentation code

###Filter out unusable datapoints
#names(lowy_data_complete)

#Re-arrange columns and sort table
lowy_data_complete <-lowy_data_complete
lowy_data_complete = select(lowy_data_complete, 
                            #sending data
                            send_country_iso,
                            send_country,
                            send_lon,
                            send_lat,
                            #posting data
                            post_country_iso, 
                            post_country,
                            post_city_country,
                            post_city,
                            post_city_lon,
                            post_city_lat,
                            post_type,
                            post_title,
                            #other columns
                            everything())

lowy_data_complete <- lowy_data_complete[order(lowy_data_complete$send_country),]

#Filter out NA values (around 44 values)
geo_lowy_NA_values <- lowy_data_complete %>%
  dplyr::select(everything()) %>%
  dplyr::filter(is.na(exact_post_lon))

geo_lowy_no_NA <- lowy_data_complete %>%
  dplyr::select(everything()) %>%
  dplyr::filter(!is.na(exact_post_lon))

Summarise change to master GDI dataframe

#Preview table
report_full_length <-length(lowy_data_complete$send_country)
report_na_length <- length(geo_lowy_NA_values$send_country)
report_non_na_length <- length(geo_lowy_no_NA$send_country)

cat("There are,", report_full_length,"entries in the complete data frame.","\nFrom these,",report_na_length,"are NA/blank. (",report_na_length/report_full_length*100,"% )","\nIn total,",report_non_na_length,"entries have valid coordinates.","\nTotal coverage is:",report_non_na_length/report_full_length*100,"%")

There are, 7317 entries in the complete data frame. 
From these, 43 are NA/blank. ( 0.5876725 % ) 
In total, 7274 entries have valid coordinates. 
Total coverage is: 99.41233 %

all points visualised

geo_lowy_no_NA_vis <-
    st_as_sf(geo_lowy_no_NA,coords=c("exact_post_lon","exact_post_lat"),crs = 4326,remove = FALSE)

3.2 Geo-coding analysis function

Back to index
Function to take in data frame, and a filter to visualise and return filtered dataframe for further analysis f_exact_city_postings <- function(p_city_filter,p_df) expand to see function

###FUNCTION
f_exact_city_postings <- function(p_city_filter,p_df,p_colour = "red"){
  
  ###Subset of dataframe for chosen country
  sub_city_df <- p_df[p_df$post_city == p_city_filter,]

  #create spatial objects
  sub_city_sf <- 
    st_as_sf(sub_city_df,coords=c("post_city_lon","post_city_lat"),crs = 4326, remove = FALSE)

  sub_city_geo_sf <-
    st_as_sf(sub_city_df,coords=c("exact_post_lon","exact_post_lat"),crs = 4326,remove = FALSE)

  #map view options
  mapviewOptions(leafletWidth = 750,leafletHeight = 400)
  
  #City lat longs
  exact_map <- mapview(
      #spatial object
      sub_city_sf,
      legend = FALSE,
      layer.name = "City coordinates",
      lwd = 0.5,
      alpha.regions = 0.2,
      label = sub_city_df$post_city_country,
      cex = 10,
      colour = "black",
      col.regions = "orange",
      popup = leafpop::popupTable(subset(sub_city_sf, select = c(post_city_country)))
      )+ mapview(
      #spatial object
      sub_city_geo_sf,
      legend = FALSE,
      layer.name = "Diplomatic post locations",
      label = sub_city_df$full_geo_query,
      color = "black",
      alpha= 1,
      lwd = "0.5",
      alpha.regions = 0.5,
      col.regions = p_colour,
      cex = 6,
      popup = leafpop::popupTable(subset(sub_city_sf, select = c(send_country_iso,send_country,post_type,post_title,exact_post_lon,exact_post_lat)))
      )
  
  mapshot(exact_map, url = paste("/cloud/project/Data/MapFiles/",p_city_filter,"map.html",sep="_"))

  #return dl table,map
  return_list <- list(df = sub_city_df,map = exact_map)
  return(return_list)
}

3.3 Visualise

Back to index

Run function on array of a city of interest - Tehran

###run function
f_out <- f_exact_city_postings(p_city_filter = "Tehran",p_df = geo_lowy_no_NA)
f_out$map

3.4 Clustering

Use function output from 3.2 to run DBSCAN spatial clustering analysis

#============== function start
f_exact_city_cluster <- function(p_city_filter,p_df,p_eps,p_k = 4,p_outlier_remove = FALSE){
  
    #run original function
  f_out <- f_exact_city_postings(p_city_filter = p_city_filter,p_df = p_df)
  
  f_out$df
  
  #assign coordinates of filtered dataset to variable
  db_coordinates <- f_out$df[c("exact_post_lon","exact_post_lat")]*1000
  
  #boxplot
  db_coordinates_box <- boxplot(data=db_coordinates$exact_post_lon, x=db_coordinates$exact_post_lon, main="Boxplot for coordinate values", 
   xlab="Posts", ylab="Longitude values",notch = TRUE) #[RETURN 1]
  
  db_coordinates <- db_coordinates/1000
  db_coordinates_out <- db_coordinates_box$out/1000
  
  if(length(db_coordinates_out) == 0){
    zero_outliers <- FALSE
  }else{
    zero_outliers <- TRUE
  }
  
  #negation for filtering
  `%nin%` <- Negate("%in%")

  
  #if outlier parameter set to true, they will be removed, else can keep going as is
  if(p_outlier_remove & zero_outliers){
    
    #clustering objects
    db_coordinates_keep <- db_coordinates%>% filter(exact_post_lon %nin% db_coordinates_out)
    db_coordinates_out_cluster <- db_coordinates%>% filter(exact_post_lon %in% db_coordinates_out)
    
    #full dataframe without outliers
    f_out_df <- f_out$df %>% filter(exact_post_lon %nin% db_coordinates_out)
      
    #outlier dataframe
    f_out_outlier_df <- f_out$df %>% filter(exact_post_lon %in% db_coordinates_out)
    
    f_out_outlier_df_sf <- st_as_sf(f_out_outlier_df,
                                      coords=c("exact_post_lon","exact_post_lat"),
                                      crs = 4326,remove = FALSE)
  }else{
    db_coordinates_keep <- db_coordinates
    f_out_df <- f_out$df
  }
  
  #Indetify optimal eps 
  kNNdist_df <- dbscan::kNNdist(db_coordinates_keep, k = p_k)
  db_kNN_plot <- dbscan::kNNdistplot(db_coordinates_keep,k = p_k) #[RETURN 2]

  #now run the dbscan analysis
  db <- fpc::dbscan(db_coordinates_keep, eps = p_eps, MinPts = p_k)
  
  #now plot the results [RETURN 3]
  db_plot <- plot(db, db_coordinates_keep, main = "DBSCAN Output", frame = F)
  
  #you can add '2' to cluster to avoid 0 values as leaflet doesn't like to colour by 0 (zero) values and colour '1' is black
  db_coordinates_keep$cluster <- db$cluster+1 #+2 #not used
  dbscan_test_df <- db_coordinates_keep
  
  dbscan_test_df
  
  f_out_cluster <- f_exact_city_postings(p_city_filter = p_city_filter,p_df = f_out_df,p_colour = dbscan_test_df$cluster)
  
  if(p_outlier_remove & zero_outliers){
    f_out_cluster_map <- f_out_cluster$map + mapview(f_out_outlier_df_sf,
      legend = FALSE,
      layer.name = "Outliers",
      label = f_out_outlier_df_sf$full_geo_query,
      color = "yellow",
      alpha.regions = 1,
      col.regions = "yellow",
      cex = 4)
  }else{
    #no outliers to plot
    f_out_cluster_map <- f_out_cluster$map #[RETURN]
  }
  
  f_out_cluster$df #[RETURN]
  
   #mapshot(f_out_cluster_map, url = paste("/cloud/project/R_Notebook/Final/Data/MapFiles/",p_city_filter,"cluster_map.html",sep="_"))

  #return list of relevant elements
  return_list <- list(df_sub = f_out$df,map = f_out$map,box_plot = db_coordinates_box, k_plot = db_kNN_plot,f_cluster_map = f_out_cluster_map, f_cluster_df = f_out_cluster$df)
  
    return(return_list)
}
#============== function end

Run function

sample_anlaysis <- f_exact_city_cluster(p_city_filter = "Tehran",p_df = geo_lowy_no_NA,p_eps = 0.015,p_k =4,p_outlier_remove = TRUE)

sample_anlaysis$f_cluster_map

---

4.0 ACLED data

|| 0.0|READ ME || 1.0|EXTRACT || 2.0|TRANSFORM || 3.0|LOAD and ANALYSE || 4.0|ACLED data || 5.0|ACLED cases ||

Purpose of section | add data layer to output from function in 3.4
Inputs | f_exact_city_cluster() (outputs of function call in 3.4)
Outputs | maps and visuals

---

4.1 Set up ACLED API

Back to index

For details on how to use ACLED API please see documentation here

API structure for JSON output Full url call: https://api.acleddata.com/acled/read?terms=accept&country=Iran&year=2019&admin1=Tehran URL base: https://api.acleddata.com/acled/read?terms=accept terms=accept is required for output to be provided

after this, filters and criteria can be concatenated using &
In this case, country iso &iso=364, year &year=2019, admin1 admin1=Tehran for city

Define general purpose function

#define function, parameters same as those for clustering function in 3.4
f_acled_merge <- function(p_city_filter,p_df,p_eps,p_k,p_outlier_remove = TRUE,p_acled_year = FALSE,p_acled_city = TRUE){
  
  #function from 3.4
  f_exact_output <- f_exact_city_cluster(p_city_filter = p_city_filter,
                       p_df = p_df,p_eps = p_eps,
                       p_k = p_k,
                       p_outlier_remove = p_outlier_remove)
  
  #relevant outputs from f_exact_city_cluster() are: dataset, cluster analysis stats and map to add to.

  ###ISO
  distinct_country_iso <- distinct(f_exact_output$f_cluster_df,post_country_iso_num)
  distinct_country_iso_call <- paste("&iso=",distinct_country_iso,sep="")
  ###YEAR
  if(p_acled_year){
    acled_year <- p_acled_year 
    acled_year_call <- paste("&year=",acled_year,sep="")
  }else{
    acled_year_call <- ""
  }
  ###CITY
  if(p_acled_city){
    city_call <- paste("&admin1=",p_city_filter,sep="")
  }else{
    city_call <- ""
  }
  
  ###API
  api_base <-"https://api.acleddata.com/acled/read?terms=accept"
  
  url_call <- paste(api_base,distinct_country_iso_call,acled_year_call,city_call,sep="")
  acled_data <- getURL(url_call)
  acled_df <- fromJSON(acled_data)$data #get data right away
  acled_df <- type_convert(acled_df)
  acled_df$point_size <- acled_df$fatalities+6
  acled_df_sf <- st_as_sf(acled_df,
                                      coords=c("longitude","latitude"),
                                      crs = 4326,remove = FALSE)
  
  
  f_acled_map <- f_exact_output$f_cluster_map + mapview(acled_df_sf,
      legend = TRUE,
      layer.name = "Incidents (by type)",
      label = acled_df_sf$event_type,
      lwd = 0.5,
      color = "red",
      alpha.regions = 0.5,
      #col.regions = acled_df_sf$event_type,
      col.regions = "white",
      #zcol = "event_type",
      cex = "point_size",
      popup = leafpop::popupTable(subset(acled_df_sf, select = c(event_date,event_type,sub_event_type,fatalities,source_scale,source)))
      )
    
  mapshot(f_acled_map, url = paste("/cloud/project/Data/MapFiles/",p_city_filter,"ACLED_map.html",sep="_"))
  
  #return list of relevant elements
  return_list <- list(f_data = f_exact_output$f_cluster_df,
                      f_map = f_acled_map,
                      f_k_plot = f_exact_output$k_plot,
                      f_acled_data = acled_df_sf,
                      f_boxplot = f_exact_output$box_plot)
  
    return(return_list)
}

Run function and see completed function results

---

5.0 ACLED cases

|| 0.0|READ ME || 1.0|EXTRACT || 2.0|TRANSFORM || 3.0|LOAD and ANALYSE || 4.0|ACLED data || 5.0|ACLED cases ||

Purpose of section: run two case studies using function from section 4
Inputs: f_acled_merge()
Outputs: maps and plots to assess clustering

---

In this section, different cities are examined for qualitative assessment in the final assessment. Deselect outlier layer and city layer and focus in on ‘exact embassy locations’ layer.

5.1 | Case 1 Cairo

Back to index

5.2 | Case 2 Ankara

Back to index

---

REFERENCE CODE FUNCTIONS

Back to index

Calculate global population and global economy proportions for this dataset.

#ECONOMY
economy_summary <- distinct(lowy_meta_df,send_country,meta_gdp_usd_billions)
economy_summary_sum <- sum(economy_summary$meta_gdp_usd_billions)
economy_2019 <- 85.91*1000
economy_coverage <- economy_summary_sum/economy_2019

###POPULATION
people_summary <- distinct(lowy_meta_df,send_country,meta_population_millions)
people_summary_sum <- sum(people_summary$meta_population_millions)
people_2019 <- 7.594*1000
people_coverage <- people_summary_sum/people_2019

cat("Coverage of the world economy is:",economy_coverage*100,"%","\nCoverage of global population is:",people_coverage*100,"%")

Coverage of the world economy is: 91.78675 % 
Coverage of global population is: 74.7577 %

#economy from: https://data.worldbank.org/indicator/ny.gdp.mktp.cd
#people from: https://data.worldbank.org/indicator/sp.pop.totl 
#85.91 GDP in current dollars in thousands of billions for 2018
#7.594thousand million population in the world in 2018