Sequence Analysis - R Tutorial

All code is available on GitHUb

We will look at data of transitions school-to-work in Northern Ireland.

• Cohort survey of 712 students from Northern Ireland.
• Followed from September 1993 to June 1999 (70 months), after the end of high school. The time sequence includes the monthly occupation state for each student (6 possible states: “Employment”, “Further Education”, “Higher Education”, “Gap”, “School”, “Training”)

This is an extract of the first 10 individuals in the dataset for the first 5 months available:

Background characteristic (at survey time):

• Binary GCSE score, measuring how well each student has done in their final GCSE exam (Lower/Higher score),

We first define the STS object’s attributes:

Then we use TraMineR::seqdef to create the STS object:

We can now set up the colout palette using TraMineR::cpal:

This is how the first 4 sequences of the STS object appear:

We can have a more compact view using this command: complete SPS (State Permanent Sequence) format

They display actually observed sequences.

• All observed sequences are displayed
  
• First glance at the data, although the result can be a bit messy…

To facilitate visual identification of patterns, the set of sequences can be sorted according to some specified criterion

• In this example the most common sequence was first identified as reference sequence (refseq = 0) and all other sequences were sorted according to their similarity to the reference (dist.mostfreq).

• In the plot, the reference sequence is represented at the bottom.

• An interesting pattern emerges: many individuals start in “School” (light blue) or “Further Education” (blue) and then progress into “Employment” (green) or “Higher Education” (red)

We can also visualise the same plot by group, adding it as a parameter. For example here we look at the same data by “GCSE results” and notice the difference in the most common sequences.

We can also choose to represent only the most frequent sequences in the data and extract their frequency as well. For example, to visualise the top 10 most frequent sequences:

A table with counts and percentages of the most frequent sequences can also be extracted:

Again, we can look at the same plot by GCSE group. The most frequent sequence of events for those with “Higher score” is to start in “School” (light blue) at the beginning of their trajectories and then go into “Higher Education”, while those with a “Lower Score” most commonly start directly into “Employment” (green) and remain there

These plots provide an aggregated overview of the data (rather than representing directly individual trajectories).

A chronogram is a series of stacked bar charts that visualise the cross-sectional state distributions of individuals (i.e. at each time point of the sequence).

The entropy curve shows the degree of diversity across state distributions at each time point of the sequence. Entropy values range from 0 to 1:

• entropy = 0 –> all individuals are in the same state
• entropy = 1 –> each state is associated to the same number of individuals
  

• Note how the cross-sectional entropy of the student cohort decreases at the end of the time sequence (as more and more individuals enter in the same state “Employment”)

The following code allows to show the transition rates between states as a matrix.

• The transition rates are first calculated using the function TraMiner::seqtrate()
  
• ggseqplot has its own function ggseqtrplot() to create the matrix, however I prefer aesthetically to build it directly with ggplot2 (more customisation on colours, font sizes and labels)

• The Start State is represented vertically while the End State is represented horizontally
• The higher the transition rates, the more intense the blue
• The highest rates (in the diagonal, all > 0.9) indicate that individuals in a particular state tend to remain in the same state
• The next highest transition rate (0.042) is [GA] –> [EM] (from Gap to Employment)

We can also look at sequences of transitions from one state into another, without considering the time spent in each state. These are called DSS (Discrete Successive State) sequences and can be visualised using all the tools shown previously and also using Sankey diagrams.

These are the explicit State sequences (STS):

They can also be written in “compact” SPS (State Permanent Sequence) format:

These are the same sequences in DSS (Discrete Successive State) format, removing the information of time spent in each state. The DSS states can be extracted using TraMineR::seqdss function:

We can use a Sankey diagram to visualise the DSS sequences. Since plotting the whole dataset would look a bit messy, I will just focus on those students starting their trajectories from a “Further Education” state (FE)

A Sankey diagram represents flows of individuals moving from one state to another. It uses weighted connections going from one node to another. Each node represents a state and the weighted connections correspond to the number of people moving between different states (the flows). I will use a custom made function created with the networkD3 library.

• On the plot, ST1 = State 1, ST2 = State 2 etc, going forward in the sequence of states.
• At the beginning (State 1 = ST1), we can see there are, for example, 275 individuals who start their trajectories in the “Further Education” (FE/ST1 = blue) state. If you follow the connections, you can see that 154 of them flows into the “Employment” state (EM/ST2 = green), 38 into the “Higher Education” state (HE/ST2 = red), 51 into the “Gap” state (GA/ST2 = yellow), 25 into the “Training” state (TR/ST2 = grey), 6 into the “School” state (SC/ST2 = light blue), while 1 remains into the “Further Education” state (FE/ST2 = blue). And so on…

• Clustering creates group objects (e.g. state sequences) in such a way that the groups obtained are as internally homogeneous as possible and as different from one another as possible.

• The method discussed here is based on comparing state sequences and quantifying how dissimilar they are, by using a metric of sequence dissimilarity.

• Sequence dissimilarity can be computed with the Optimal Matching (OM) algorithm, which expresses distances between sequences in terms of amount of effort required to transform one sequence into the other one. Three operations are possible to transform one sequence into another:

  • insertion (one state is inserted into the first sequence to obtain the second)
  • deletion (one state is deleted from the first sequence to obtain the second)
    
  • substitution (one state is replaced by another state)

• Each operation is associated to a cost. The distance between two sequences can be defined as the minimum cost to transform one sequence into the other. Definition of costs is often subjective, although it is possible to use data driven methods (e.g. between-states transition rates to define the substitution costs)

The following code uses TraMineR::seqdist() to compute the pairwise sequence dissimilarity matrix (e.g. distances between each pair of sequences) using OM. The pairwise distances for the first 4 sequences are shown:

• We can now carry out clustering. Different methods are available and we will focus on hierarchical clustering since it does not require to specify in advance the number of clusters (e.g. compare to k-means)

• We will use the Ward algorithm, a hierarchical clustering method that attempts to minimise the within-cluster variance.

The function cluster::agnes is used to compute hierarchical clustering (“agnes” = Agglomerative Nesting). The function’s output can be visualised as a dendrogram, a diagram with a tree structures showing the grouping hierarchy of the sequences:

• …but, how many clusters should we use?
• There might be knowledge domain-driven reasons to expect a specific number of clusters
• Otherwise, we can use data-driven clustering quality metrics, which give an indication on the optimal number of clusters for the specific dataset.

The following code will use the WeightedCluster::as.clustrange function to compare different clustering solutions (from 2 to 25 clusters). Multiple clustering quality metrics for each number of cluster considered. We will focus only on metrics:

• Point Bi-serial Correlation (PBC): to maximise
• Average Silhouette Width (ASW): to maximise
• Hubert’s C coefficient (HC): to minimise

We will plot these metrics so it is easier to visually inspect them:

Looking at both metrics simultaneously, 7 clusters seems a good compromise. We can use the cutree function to cut the clustering tree into 7 groups:

The following code assigns each individual to one of the 7 clusters and add this new information to the original dataset. The new cluster variable can be used as a group variable to look at the sequences now. We can have a quick look at how many individuals belong to each cluster:

We can use all the plotting and analysis tools available in TraMineR to analyse the sequence data divided by cluster group. For example we can look at their chronograms to have an idea of how individuals in each cluster might differ in their trajectories:

Some interesting trajectory patterns emerge from each cluster group:

• Cluster 1: individuals who enter the “Employment” state relatively early
• Cluster 2: individuals who start in “Further Education” and mainly progress into “Higher Education”
• Cluster 3: individuals who start in “Further Education” and mainly progress into “Employment”
• Cluster 4: individuals who start in “Training” and mainly progress into “Employment”
• Cluster 5: individuals who start in “School” and mainly progress into “Further Education” and “Employment”
• Cluster 6: individuals who start in “School” and mainly progress into “Higher Education”
• Cluster 7: individuals who end into “Gap” (Unemployment) state.

Finally, we can carry out some statistical analysis to identify whether there is any relevant association between clusters an other variables of interest. In this example we have a binary variable “GCSE results”. We carry out a \(\chi^2\) test to assess its statistical association with the clusters

The test Post-hoc tests reveal a significant statistical difference between the two GCSE groups in Clusters 1 and 7 (more individuals with GCSE Lower scores), and Clusters 2 and 6 (more individuals with GCSE Higher scores)