1.1.1 GeoJSON

GeoJSON (Geospatial Data Interchange format for JavaScript Object Notation). See: http://geojson.org/ is becoming an increasingly popular spatial data format, particularly for web-based mapping as it is based on JavaScript Object Notation.

1.1.2 Shapefiles

Perhaps the most commonly used GIS data format is the Shapefile. Shapefiles were developed by ESRI (http://www.esri.com/) – one of the first and now certainly the largest commercial GIS company in the world. Despite being developed by a commercial company, they are an open format and Shapefiles can be used (read and written) by a host of GIS Software applications.

A shapefile is actually a collection of files – at least three of which are needed for the shapefile to be displayed by GIS software. They are:

1. .shp - the main part of the file which contains the feature geometry (e.g. image below).
2. .shx - an index file which stores the position of the feature IDs in the .shp file
3. .dbf - the part of the file that stores all of the attribute information associated with the coordinates – this might be the name of the shape or some other information associated with the feature – note how in a shapefile attribute data are held in a separate file whereas in a GeoJSON the attributes and boundaries are all contained in the same file.
4. .prj - the part of the file which contains all of the coordinate system information (the location of the shape on Earth’s surface). Data can be displayed without a projection, but the .prj file allows software to display the data correctly where data with different projections might be being used. Again, in file formats like GeoJSON, this information is contained all in the same file.

1.1.3 GeoJSON and Shapefiles

We’re now going to explore GeoJSON and shapefiles in action. Go to: http://geojson.io/#map=16/51.5247/-0.1339

1. Using the drawing tools to the right of the map window, create 3 objects: a point, line and a polygon as I have done above. Click on your polygon and colour it red and colour your point green.
2. Using the ‘Save’ option at the top of the map, save two copies of your new data – one in GeoJSON format and one in shp format.
3. Open your two newly saved files in a text editor such as notepad or notepad++. For the shapefile you might have to unzip the folder then open each file individually. What do you notice about the similarities or differences between the two ways that the data are encoded?

1.1.4 Raster data

Most raster data is now provided in GeoTIFF (.tiff) format, which stands for Geostarionary Earth Orbit Tagged Image File. The GeoTIFF data type was created NASA and is a standard public domain format. All necesary inforamtion to establish the location of the data on Earth’s surface is embedded into the image. This includes: map projection, coordinate system, ellipsoid and datum type.

1.1.5 Other types of data

Aforementioned data types and formats are likely to be the ones you predominately encounter. However there are several more used within spatial analysis. Theres include:

Vector

• GML (Geography Markup Language – gave birth to KML) - http://www.opengeospatial.org/standards/gml

Raster

• Band SeQuential (BSQ) - technically a method for encoding data but commonly refered to as BSQ.
• Hierarchical Data Format (HDF)
• Arc Grid

There are normally valid reasons for storing data in one of these other formats. For example, BSQ are just data with a seperate text header file (.hdr) providing geographic spatial reference information. Earth observation data often monitors the electromagenitc spectrum in bands. Humans see in the visible range of the spectrum and our vision is composed of red, green and blue wavelengths. If we wanted to analyse just the red wavelength the BSQ format would let us ‘read in’ only that data. In comaprsion a GeoTIFF might come with all the data ‘packaged’ in one file and when doing analysis over thousands of images would significantly slow things down. That said you can now often find GeoTIFFs seperated in a similar format to BSQ and it’s fairly straightforward to convert between raster formats. Often the most successful projects and reports will combine numerous datasets from various sources (and formats) to glean new insights.

1.1.6 Geodatabases

A geodatabase is a collection of geographic data held within a database. Geodatabases were developed by ESRI to overcome some of the limitations of shapefiles. They come in two main types: Personal (upto 1 TB) and File (limited to 250 - 500 MB), with Personal Geodatabases storing everything in a Microsoft Access database (.mdb) file and File Geodatabases offering more flexibility, storing everything as a series of folders in a file system. In the example below we can see that the FCC_Geodatabase (left hand pane) holds multiple points, lines, polygons, tables and raster layers in the contents tab.

1.1.7 GeoPackages

A GeoPackage is an open, standards-based, platform-independent, portable, self-describing, compact format for transferring geospatial data. A GeoPackage stores multiple spatial data layers (vector and raster) as a single file. One of the major advantages of a GeoPackage is the ability to analyse geographic data in SQLite producing code based, reproducible and transparent workflows. A GeoPackage is an SQLite database, a widely used relational database management system and as it stores data in a single file it is very easy to share, copy or move.

1.1.8 SpatiaLite

SpatialLite is an open source library that extends SQLite core. Support is fairly limited and most software that supports SpatiaLite also supports GeoPackage as they both build upon SQLite. It doesn’t have any clear advantage over GeoPackage, however it is unable to support raster data.

1.1.9 PostGIS

PostGIS is an opensource database extender for PostrgeSQL. Essentially PostgreSQL is a database and PostGIS is an add on which permits spatial functions. The advantage of using PostGIS over GeoPackages is that it allows users to access the data at the same time, can handle large data more efficiently and reduces processing time. In this example (https://medium.com/@GispoLearning/learn-spatial-sql-and-master-geopackage-with-qgis-3-16b1e17f0291) calcualting the number of bars per neighbourhood in Leon, Mexico the processing time reduced from 1.443 seconds (SQLite) to 0.08 seconds in PostGIS. However, data stored in PostGIS is much harder to share, move or copy.

1.1.10 What will I use

The variety of data types and formats can see a bit overwhelming. But don’t worry, most of the time you’ll be using shapefiles, GeoPackages or raster data. You just need to be aware of the all the different types so that later on in the course if someone is talking about PostGIS data you are aware that it is a type of spatial data.

2.4.1 Basics

ArcGIS should be installed as a standard programme in the UCL desktop and you should be able to navigate to it from the Windows start button.

2.4.2 ArcCatalog

• Find and run the ArcCatalog piece of software

• Once ArcCatalog Opens, go to File > Connect To Folder… and navigate to the N:folder, right click in the contents area and create a new Flie Geodatabase.

You can import data layers into a Geodatabase within ArCatalog, however we will do this in ArcMap.

2.4.3 ArcMap

ArcMap is the core of the ArcGIS suite and where you would normally produce maps, carry out spatial analysis functions and automate processes. Find ArcMap in your start menu and run the programme. When the programme starts, you should see something similar to the image below. ArcMap has various elements to the graphical user interface, but here some of the key buttons are highlighted:

1. Search for and open ArcMap
2. Upon opening select the database you just created in the dialogue box

3. Click ok
4. It’s important to now set the map document up properly. Go File > Map Document Properties, enter the details you wish. You can see the connection to the Geodatabase we just made. Click store relative pathnames, this means as long as the data stays in the same position relative to the path then Arc Map can load all the layers. For example if you moved your work from the C: drive to an external drive, H:.
5. Using the Plus icon (add data layer) navigate to the extracted folder you saved earlier.
6. Open London_Borough_Excluding_MHW.shp, but feel free to explore the other data layers. In the left hand Table Of Contents you can unselect layers to turn them off or drag layers above or below to change the display order. In the example below the wards layer is showing above the borough layer.

Note, while we have loaded the shapefile, it is not stored in our Geodatabase yet.

Useful tips

• To the right of the document you will see the Catalog and Search tabs.The Catalog tab is a small version of ArcCatalog and will let you see what data is stored in the current Geodatabase. The search tab will let you find any analysis tool within ArcGIS. Try searching for Clip.

• If you right click on the boroughs layer you will see various options. Zoom to layer is very useful if you ever get lost in your ArcMap document.

7. Now right click on the borough layer and open the attribute table. You’ll see the GSS_CODE field, which is the same code we output in our .csv. If you now right click on a field you’ll also be able to see the data type (e.g. string, integer). For example:

We’re now going to join our fly tipping data to the lonon borough shapefile. So:

8. Right click on the london borough layer > Joins and Relates > Join
   
9. Select the GSS_CODE as the field in the layer to base the join on
10. Navigate to the .csv we created earlier
11. Select the code field that matches (in my case this is called Row Labels)
12. Select only to join matching records
13. Validate join and click OK.

You will get errors, ArcGIS does not like fields starting with numbers (e.g. 2012), dashes (e.g. -) or spaces.

14. The join should work, so reopen the attribute table for the layer london boroughs.

Note, the join we have made is not permanent. To do so we need to export the layer.

15. Right click on the london boroughs layer > Data > Export Data.
16. The location should default to our GeoDatabase. Be sure to change the filename - again avoid all of the characters (e.g. spaces and -) previously mentioned. Add the data layer to the map.

Now lets use the data we’ve joined to create a basic thematic map.

17. Right click on the new london borough layer > properties.
18. Under the symbology tab select graduated colors.
19. Select the Value as one of the years of data we joined and change the classificaiton to something of your choice.

You should have something that looks like this:

We haven’t talked about the spatial reference of our map document

A spatial reference is a series of paramters that define the coordinate system. Within GIS we use geographic or projected cooridnate systems. The former uses a three-dimesional spherical surface to define locations of earth, whereas the latter is defined on a flat, two-dimesional surface giving it constat lengths, angles and areas.

In ArcMap we can specify what cooridate system we want to use by:

20. Right clicking on the map document > Data Frame Properties

You’ll see that it is already set to Projected Coordinate Systems, National Grids, Europe, British National Grid. This is because ArcMap will default to the cooridnate system of the first data layer loaded.

Save your ArcMap document. We’re now going to replicate this task in QGIS.