AOC / Dark Objects Integration v2
Purpose
To develop a baseline connection methodology between AIS signal data and sarscape ship detection output. This will chiefly involve validation, conservative feature selection, data articulation, and deriving novel probability metrics.
This is ultimately to derive a “target” by which we can train models. We need some degree in both size and accuracy of known connections between the AIS signals and dark object ship detections through which we can teach an algorithm to catch similar scenarios.
Query Sources
click to run query url - must have shiny server running locally
- Example query Values
- uuid = 0e683f0b-8ea8-4bc2-b447-53a1995cf5b6
- file = S1A_IW_GRDH_1SDV_20191223T142452_20191223T142517_030477_037D47_221F.SAFE
- type = GRD
- sat_time = 2019-12-23T14:24:52.805Z
- wkt = [view above hyperlink]
- Associated interpolated query (Cassandra)
aisSnapshot.bat -U pans_user -P pans -d 2019-12-23T14:24:52.805Z -a 1 -l 0e683f0b-8ea8-4bc2-b447-53a1995cf5b6 -p Sentinel-1Initialization
Libraries required
require(rgdal)
require(raster)
require(sf)
require(rgeos)
require(leaflet)
require(dplyr)
require(htmlwidgets)
require(htmltools)
require(ggpubr)
require(gridExtra)Data Loading
Loading ship detections and AIS interpolation via Sarscape+R/Cassandra+Bash processes, respectively
# Hardcode the filename which has run ship detection in the shiny subsystem + output the above bash script with interpolated AIS signals
filename <- 'S1A_IW_GRDH_1SDV_20191223T142452_20191223T142517_030477_037D47_221F.SAFE'
setwd(paste0('../shiny_ship_detection/import/', filename))
darkobj_pt <- shapefile("sentinel1_SENTINEL-1_130_20191223_142452805_IW_SIW_A_VV_gr_geo_ship_mask.shp")
darkobj_poly <- shapefile("ship_vectors_multi.shp")
# AIS data load
ais_signals_uncorrected <- read.csv('data-1579797017584.csv', header = TRUE, stringsAsFactors=F)
coordinates(ais_signals_uncorrected) <- ~lng+latPreparing test shorelines using GIS Software
- Shoreline data retrieved from National Geospatial-Intelligence Agency, Prototype Global Shoreline Data; Satellite Derived High Water Line Data
- Retrieval date unknown, circa 2004.
- Data is said to be at the high water line.
- Required geoprocessing Inside ArcGIS
- Used Feature to Polygon with the footprint/envelope and shoreline polyline as inputs.
- Manually removed ocean features inside an edit session.
- Perform a ‘Select By Location’ and delete the inverse (using switch selection inside attribute table).
- Target layer: shoreline polygon output
- Source layer: footprint/envelope
- Spatial selection method: ’are within (Clementini) the source layer feature.
- Visual spot check against 1/9/2019 Digital Globe Satellite Imagery
- Various infill and “made land” not included from between approx 2004 and 2019.
- In the United Arab Emirites, there are numerous such examples of recent development.
- In the SChina Sea the Chinese are performing even more recent “made land” developments.
Loading test shorelines
- Retrieved from http://www.soest.hawaii.edu/pwessel/gshhg/
setwd('../shoreline_data/')
shoreline_poly <- shapefile("nga_zone10_example.shp")
plot(main="Area shoreline extraction",shoreline_poly, col="#f2f2f2", bg="skyblue", lwd=0.25, border=0 )
Apply secondary shoreline filter
# NAs are points not on land
darkobj_pt_filtered<-darkobj_pt[!complete.cases(over(darkobj_pt, shoreline_poly)),]Analyses
Analysis #1 - Baseline
Data Visualization
html_legend <- "<img src='http://pngimg.com/uploads/target/target_PNG38.png' height='20' width='20'>Ship Detection Point<br/><br/>
<img src='http://www.pngplay.com/wp-content/uploads/2/Red-Target-Board-Transparent-Free-PNG.png' height='20' width='20'>AIS Signal via Cassandra<br/>"
leaflet(ais_signals_uncorrected) %>%
addProviderTiles("CartoDB.Positron") %>%
setView(55,25.5, zoom = 8) %>%
# Currently representing AIS signals
addMarkers(label=~mmsi,
icon = list(
iconUrl='http://www.pngplay.com/wp-content/uploads/2/Red-Target-Board-Transparent-Free-PNG.png',
iconSize=c(20, 20)
)) %>%
# Currently representing dark object points
addMarkers(data = darkobj_pt_filtered,
icon = list(
iconUrl='http://pngimg.com/uploads/target/target_PNG38.png',
iconSize=c(20, 20)
)) %>%
# Currently representing dark object polygon shapes
addPolygons(data = darkobj_poly,
col = 'purple',
stroke = TRUE,
smoothFactor = 0.5) %>%
addLegend(color = "purple",
labels = "Dark Object Polygon Vectors") %>%
addPolygons(data = shoreline_poly,
col = 'green',
stroke = TRUE) %>%
addLegend(color = "green",
labels = "Shoreline Mask") %>%
addControl(html = html_legend, position = "topright")Discussion
- The shoreline mask shown is not the one used in the current sarscape process, due to existing processing and database limitations.
- The shoreline has substantially shifted, leaving many false positives along made land. Further details here: https://en.wikipedia.org/wiki/Land_reclamation_in_the_United_Arab_Emirates.
- AIS signals themselves appear within the land mask.
- An improved product may be available for purchase at approx $200 from https://shoreline.noaa.gov/data/datasheets/wvs.html.
Analysis #2 - Simple Minimized Input Hypothesis
- Due to limitations in the underlying data, a conservative approach to detection ought to be initially considered. In this example:
- High levels of signal clustering are removed. (incomplete)
- Coastal signals are removed.
- Signals not within 15 minutes of satellite image time are removed.
Data tidying
` - Remove “distant” points greater than 30 seconds.
plotA <- ggdensity(ais_signals_uncorrected@data, main = "Time Density (All)", x = "delta_secs",
fill = "#0073C2FF", color = "#0073C2FF",
add = "mean", rug = TRUE)
ais_signals_uncorrected_30s <- subset(ais_signals_uncorrected, delta_secs < 30)
#http://www.sthda.com/english/articles/32-r-graphics-essentials/133-plot-one-variable-frequency-graph-density-distribution-and-more/
plotB <- ggdensity(ais_signals_uncorrected_30s@data, main = "Time Density (<30s)", x = "delta_secs",
fill = "#0073C2FF", color = "#0073C2FF",
add = "mean", rug = TRUE)
grid.arrange(plotA, plotB, ncol=2)
- Conforming projections in UTM for geoprocessing accuracy
# calculate UTM Zone
long2UTM <- function(long) {
(floor((long + 180)/6) %% 60) + 1
}
centroid <- shoreline_poly@polygons[[1]]@labpt
if (centroid[2] > 0){ hemisphere = 'N'}else{hemisphere = 'S'}
utm_zone <- paste0("+proj=utm +zone=",paste0(long2UTM(centroid[1]) , hemisphere)," +datum=WGS84 +units=km")
proj4string(ais_signals_uncorrected_30s) <- CRS("+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" )
ais_signals_uncorrected_30s_UTM <- spTransform(ais_signals_uncorrected_30s, CRS(utm_zone))
shoreline_poly_UTM <- spTransform(shoreline_poly, CRS(utm_zone))
g = rep(NA, dim(ais_signals_uncorrected_30s_UTM)[1])
for(i in 1:dim(ais_signals_uncorrected_30s_UTM)[1]){
g[i] = gDistance(ais_signals_uncorrected_30s_UTM[i,],shoreline_poly_UTM)
}
ais_signals_uncorrected_30s_UTM@data$dShore <- gplotA <- ggdensity(ais_signals_uncorrected_30s_UTM@data, main ="Distance from Shoreline (All)",x = "dShore",
fill = "#0073C2FF", color = "#0073C2FF",
add = "mean", rug = TRUE)
# Remove signals within 3 km of shoreline and reproject UTM dataset into decimal degrees for mapping.
ais_signals_uncorrected_30s_UTM_3km <- subset(ais_signals_uncorrected_30s_UTM, dShore > 3)
ais_signals_uncorrected_30s_wgs84_3km <- ais_signals_uncorrected_30s_UTM_3km
ais_signals_uncorrected_30s_wgs84_3km <- spTransform(ais_signals_uncorrected_30s_wgs84_3km, CRS("+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"))
plotB <- ggdensity(ais_signals_uncorrected_30s_wgs84_3km@data, main ="Distance from Shoreline (>3km)",x = "dShore",
fill = "#0073C2FF", color = "#0073C2FF",
add = "mean", rug = TRUE)
grid.arrange(plotA, plotB, ncol=2)
This leaves 11 candidate signals out of an original 326 vessels.
Visualization
html_legend <- "<img src='http://pngimg.com/uploads/target/target_PNG38.png' height='20' width='20'>Ship Detection Point<br/><br/>
<img src='http://www.pngplay.com/wp-content/uploads/2/Red-Target-Board-Transparent-Free-PNG.png' height='20' width='20'>AIS Signal via Cassandra<br/>"
leaflet(ais_signals_uncorrected_30s_wgs84_3km) %>%
addProviderTiles("CartoDB.Positron") %>%
setView(55,25.5, zoom = 8) %>%
# Currently representing AIS signals
addMarkers(label=~mmsi,
icon = list(
iconUrl='http://www.pngplay.com/wp-content/uploads/2/Red-Target-Board-Transparent-Free-PNG.png',
iconSize=c(20, 20)
)) %>%
# Currently representing dark object points
addMarkers(data = darkobj_pt_filtered,
icon = list(
iconUrl='http://pngimg.com/uploads/target/target_PNG38.png',
iconSize=c(20, 20)
)) %>%
# Currently representing dark object polygon shapes
addPolygons(data = darkobj_poly,
col = 'purple',
stroke = TRUE,
smoothFactor = 0.5) %>%
addLegend(color = "purple",
labels = "Dark Object Polygon Vectors") %>%
addPolygons(data = shoreline_poly,
col = 'green',
stroke = TRUE) %>%
addLegend(color = "green",
labels = "Shoreline Mask") %>%
addControl(html = html_legend, position = "topright")Current Discussion and Suggestions
- Distances and clustering metrics need to be calculated for every vessel.
- Rather than binary filters, employ probabilistic gradients based upon distances.
- Continue to refine data inputs for:
- Estimated travel location
- Shorelines
- Streamline shoreline generation / investigate higher resolution products.
- Drop detections based upon dimensional data (examples)
- “rectangularize” polygon detections and approximate according to vessel type.
- remove irregular, repetitive shapes. This may be possible in sarscape.
- Spot check “hits” case by case and develop an explanation for each result.
- Determine a significant target, based upon known hits. This will likely derive (at a minimum) from those points most accurately estimated in spacetime according to their corresponding dark object. This may require an expansion of our geographic zone.