• Introduction
• The need for performance testing
• Performance testing in the literature
• Testing the IPF algorithm
• Broadening the tests
• Wider context/implications

"An approach to the analysis of individual-level phenomena over geographical space that involves the creation, analysis and modelling of spatial microdata" (Lovelace and Dumont 2015).

The MAUP plagues 'ecological' studies (Openshaw 1983)

Population synthesis can be used as a method of small area estimation. E.g.:

• Local variations in average income
• The rate of smoking across a city (Tomintz 2008)
• Geographical variability use in transportation

Spatial microdata can also shed light on intra-zone variability at the individual level.

Many real-world processes operate at the individual or household level.

Geographical factors almost always play a role.

"Everything is related to everything else, but near things are more related than distant things." (Tobler's '1st rule of geography', 1979)

Spatial microdata provide a foundation for:

• Spatially explicit and realistic agent-based models (ABM)
• 'What if' scenarios of change to external influences
• Full dynamic spatial microsimulation models

It's important to define terms early on.

Spatial microsimulation in geographical analysis is not:

• Agent-based modelling (ABM) (but can be an input for ABM)
• A replacement for real data

Sometimes, 'spatial microsimulation' research is not even:

• Spatial: space is often just a mutually exclusive categorical variable!
• Simulation: it's often just reweighting in the 1st instance

Reweighting algorithms, also labelled as synthetic reconstruction (Barthelemy and Toint, 2012):

• Iterative proportional fitting (IPF) (Deming and Stephan 1940; Beckman and Baggerly 1996)
• Generalised Regression Reweighting (GREGWT)

Combinatorial optimisation algorithms

• 'Hill climbing' algorithms
• Simulated annealing via the 'FMF' (Harland 3013)
• Genetic algorithms

Synthetic reconstruction, a refinement of IPF without reliance on a microdata sample (Barthelemy and Toint, 2012).

• Dozens of methods available for (spatial) microsimulation
• Difficult to choose from options
• Testing can be time consuming and tricky (Harland et al. 2013)

Need for fast and consistent testing framework

Broader motivations

• Assuming that climate science is correct, we must radically alter how we live
• This is the greatest challenge in planning, engineering and logistics of the 21^st century

• Harland et al. (2012): CO approaches work better
• A long (86 pages!) working paper, Williamson (2001): CO performs slightly better than IPF-based approaches
• Ryan, et al (2009): CO methods slightly outperform IPFP
• Barthelemy and Toint (2012): new method outperforms IPF

• Constant testing, skepticism = cornerstone of science
• Methods, algorithms and software are constantly shifting
• Many useful findings - often researcher's own model 'best'
• Few conclusive results from past research
• Many past tests are not reproducible or compare different things
• Testing YOUR model will help understand and improve it

Lovelace et al. (2015) revisited an established technique

• 'Scrambled' versions of official datasets used
• 3 example datasets used, progressively larger/more comples
• Saved weight matrices after every constraint:iteration

|-- data-big (just README links)
|-- figure
|-- input-data
|   |-- sheffield
|   |-- simple
|   `-- small-area-eg
|-- literature
|-- models
|   |-- ipfinr
|   |-- FMF
|   |-- simSALUD
|   `-- GREGWT
`-- output

Here: https://github.com/Robinlovelace/IPF-performance-testing

Reproducible example:

source("models/etsim.R") # run the code 'live'

• Additional iterations had big impact up to 3, very little benefit after 10
• 'Empty cells' found to have largest impact on fit
• Initial weights had very little impact

• C code (ipfp package): 50 fold speed increase
• R code can also be made faster with dplyr package
• Not yet tested mipfp!

• New project to test techniques on very large microdatasets
• Challenge: allocate 569,741 individuals to 7,787 zones
• Almost 60 million people in output spatial microdata!
• New methodology for IPF developed

• More important that 'internal validation' is how well results fit reality
• Opportunity provided by Census variable on census well-being
• Simulated at small area level with FMF (Harland, 2013) and R

• Compare different approaches in terms of timing, model fit and ease of use
• External validation
• Use alternative methods to generate same output: GREGWT? SimObesity? simSALUD?

Esteban Muñoz and I are working on package to facilitate testing: testmsim

The aim of the R package testmsim is to provide functions, data and documentation for the testing of spatial microsimulation.

library(devtools) # package for R developers
install_github("robinlovelace/testmsim") # install latest version
library(testmsim) # load the package
help(package = testmsim) # show package documentation

• Very early days
• Main benefit: facilitate others to do testing

• "Little attention is paid to the choice of programming language used" for microsimulation (Clarke and Holm 1987)
• Lack of reproducibility (Lovelace and Ballas, 2013)
• Hard to get started
• Few simple examples - uses tend to be big and complicated
• Few introductory teaching resources

• Two courses in May (Leeds) and August (Cambridge)
• Taught basic principles of spatial microsimulation
• And implementation in R
• Feedback: students grateful for first rung on ladder
• More success with latter course focussing on applications

• Contract with CRC Press as part of their R Series
• Draft of book available online in its entirety
• Open 'wiki' style allows anyone to contribute
• Any feedback/input gratefully received
• Check it out here: robinlovelace.net/spatial-microsim-book/

• What would YOU most like to see in a practical textbook on spatial microsimulation?
• Allocating individuals to households (Pritchard and Miller, 2012)
• Alternative methods of population synthesis
• Thinking about smsim-tools package
• Applications
• Code sharing and collaboration can make the world a better place