19/03/2018

Burn-in for a Markov Chain

  • Recall the invariant (unconditional) probability \(\mu\) for a Markov matrix \(\mathbf{P}\): \(\mu=\mathbf{P}\mu\). \(\mu\) is an eignvector associated with the unit eigenvalue. The other eigenvectors are in fact in a “decaying mode”.

  • Recall that the starting matrix \(\mathbf{P}\) is unrecognizable after a few steps \[ \mathbf{P} = \left[\begin{array}{cc} 0.8 & 0.3\\ 0.2 & 0.7 \end{array}\right], \mathbf{P}^2 = \left[\begin{array}{cc} 0.7 & 0.45\\ 0.3 & 0.55 \end{array}\right],\dots \mathbf{P}^{30} = \left[\begin{array}{cc} 0.6 & 0.6\\ 0.4 & 0.4 \end{array}\right].\]

  • The initial \(\mathbf{P}\) is different from the stationary \(\mathbf{P}^{30}\). \(\mathbf{P}\) is pseudostationary (disguised) at the beginning.

  • The eigenvector \(\mathbf{x}_1=[0.6, 0.4]^T\) is for \(\lambda_1=1\). The other eigenvector is \(\mathbf{x}_2=[1,-1]^T\) for \(\lambda_2=0.5\).

Burn-in for a Markov Chain

  • Each column of \(\mathbf{P}\) can be represented by a combination of eigenvectors \[ \left[\begin{array}{c} 0.8 \\ 0.2 \end{array}\right]=\mathbf{x}_1 + 0.2 \mathbf{x}_2,\,\, \left[\begin{array}{c} 0.3 \\ 0.7 \end{array}\right]=\mathbf{x}_1 - 0.3 \mathbf{x}_2.\] The powers of \(\mathbf{P}\) do not affect the eigenvector \(\mathbf{x}_1\), but for \(\mathbf{x}_2\), \((0.5)^{30}\mathbf{x}_2\) becomes a very small vector that will disappear eventually.

  • The period that a Markov chain approaches its steady state is called a burn-in time, namely, the moment of eliminating all the eignevectors except the one with the unit eigenvalue.

Gambler’s ruin problem

  • Consider a set \(\{0,1,2\dots,N \}\). The position \(x\) represents the gambler’s capital.

  • Each time step represents one game where the gambler may either increase his/her capital to \(x+1\) or decrease it to \(x−1\). (In each game, there is either a gain or loss, never a tie.)

  • If the gambler’s capital reaches zero, he/she is ruined, the opponent has won, and the games stop, whereas if the capital reaches \(N\), he/she has won all of the capital (the opponent is ruined), and the games stop.

  • Let \(w\) be the probability of moving to the right (winning a game), \(q\) be the probability of moving to the left (losing a game), and \(w + q = 1\).

Gambler’s ruin problem

  • The transition matrix is given by \[ p_{ij} = \Pr\{X_{t+1} = i | X_t=j \} =\begin{cases} w, & \mbox{ if }i=j+1\\ q, & \mbox{ if }i=j-1\\ 0, & \mbox{ if }i\neq j+1,\, j-1 \end{cases}\] for \(j=1,2,\dots,N-1\). The absorbing states are \(p_{00}=1\) and \(p_{NN}=1\). Or \[\mathbf{P}=\left(\begin{array}{ccccccc} 1 & q & 0 & \dots & 0 & 0 & 0\\ 0 & 0 & q & \dots & 0 & 0 & 0\\ 0 & w & 0 & \dots & 0 & 0 & 0\\ 0 & 0 & w & \dots & 0 & 0 & 0\\ \vdots & \vdots & \vdots & \dots & \vdots & \vdots & \vdots\\ 0 & 0 & 0 & \dots & 0 & q & 0\\ 0 & 0 & 0 & \dots & w & 0 & 0\\ 0 & 0 & 0 & \dots & 0 & w & 1 \end{array}\right)\]

Birth and Death

  • Modify the gambler’s ruin problem. The probability of birth/winning (or death/losing) is not constant but depends on the size of the population. \[ p_{ij} = \Pr\{X_{t+1} = i | X_t=j \} =\begin{cases} b_j, & \mbox{ if }i=j+1\\ d_j, & \mbox{ if }i=j-1\\ 1- (b_j+d_j), & \mbox{ if }i=j\\ 0, & \mbox{ if }i\neq j+1,\,j, \, j-1 \end{cases} \\ \mathbf{P}=\left(\begin{array}{cccccc} 1 & d_{1} & 0 & \dots & 0 & 0\\ 0 & 1-(b_{1}+d_{1}) & d_{2} & \dots & 0 & 0\\ 0 & b_1 & 1-(b_{2}+d_{2}) & \dots & 0 & 0\\ 0 & 0 & b_{2} & \dots & 0 & 0\\ \vdots & \vdots & \vdots & \dots & \vdots & \vdots\\ 0 & 0 & 0 & \dots & 1-(b_{N-1}+d_{N-1}) & d_{N}\\ 0 & 0 & 0 & \dots & b_{N-1} & 1-d_{N} \end{array}\right)\]

Stochastic Logistic Growth

  • Recall the logistic growth model \[y_{t+1} - y_t = r y_t \left(1 - \frac{y_t}{K} \right)=ry_t - r\frac{y^2_t}{K}.\]

  • A quadratic function of \(y\) equals the birth rate minus the death rate.

  • A stochastic logistic growth \[b_j - d_j = r \times j \left( 1 -\frac{j}{K} \right)= r \times j - r \times \frac{j^2}{K}\] for \(j=0,1,\dots, N\). Assume that \(\max \{b_j + d_j \}\leq 1\) for any time interval \(t\) to \(t+1\).

Stochastic Logistic Growth

K=50; N=2*K; r= 0.004; v=0:N+1; P=matrix(0,nrow=N+1,ncol=N+1);
b=r*v; d=r*v*v/K
for(j in 2:N){
  P[j,j]=1-b[j]-d[j]; 
  P[j,j+1]=d[j+1];
  P[j+1,j]=b[j]
}
P[1,1]=1; P[1,2]=d[2];P[N+1,N+1]=1-d[N+1]
P[1:5,1:5]
##      [,1]    [,2]    [,3]    [,4]  [,5]
## [1,]    1 0.00032 0.00000 0.00000 0.000
## [2,]    0 0.99168 0.00072 0.00000 0.000
## [3,]    0 0.00800 0.98728 0.00128 0.000
## [4,]    0 0.00000 0.01200 0.98272 0.002
## [5,]    0 0.00000 0.00000 0.01600 0.978

Stochastic Logistic Growth

library(markovchain); SimTime = 3000; set.seed(2019)
MC= new("markovchain", transitionMatrix=t(P))
is.irreducible(MC) # It has one absobing state
## [1] FALSE
absorbingStates(MC) # steadyStates(MC) # No steady state
## [1] "1"

Stochastic Logistic Growth

y=matrix(0,nrow=3,ncol=SimTime);
for(i in 1:3){
y[i,]=markovchainSequence(n=SimTime-1, markovchain=MC, t0=5, include=TRUE)
}
assessStationarity(y[1,], 1)
## The assessStationarity test statistic is:  2259.56 
## The Chi-Square d.f. are:  0 
## The p-value is:  0

Stochastic Logistic Growth

Probability Evolution (Eigenvector)

v = eigen(P, symmetric=FALSE); head(v$values)
## [1] -1.1813400 -1.0305484  1.0000000  1.0000000  0.9962551  0.9931675
v3= v$vectors[,3]; v4=v$vectors[,4]; c(v3[1:5],v4[1:5])
##  [1]  1.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00
##  [6] -1.470733e-03  1.364954e-14 -3.325741e-14 -3.956755e-14 -5.810998e-14
vProb=v4/sum(v4); which.max(vProb)
## [1] 48
vProb[46:50]
## [1] 0.05302102 0.05520522 0.05630741 0.05628396 0.05515828

Probability Evolution (Eigenvector)

plot(vProb, type="l")

Probability Evolution (Beginning)

library(Matrix)
P=Matrix(P, sparse=TRUE); ptemp=matrix(0,nrow=N+1,ncol=1); ptemp[6,1]=1 # X_0=5
for(t in 1:200){ p=P%*%ptemp; ptemp=p}; p = t(p);
pCum=cumsum(p); plot(p, type="l",col="grey", ylim=c(0,1)); lines(pCum)

Probability Evolution (Failure of Matrix Operation)

ptemp=matrix(0,nrow=N+1,ncol=1); ptemp[6,1]=1
for(t in 1:1000){ p=P%*%ptemp; ptemp=p}; p = t(p);
pCum=cumsum(p); plot(p, type="l",col="grey", ylim=c(-2,2)); lines(pCum)

Question One for the Exam

  • \(S_t\) and \(I_t\) are the number of susceptible and infected individuals at time \(t\).

  • Suppose the total number \(S_t + I_t = K\) is fixed at level \(K\).

  • The dynamics follows \[ I_{t+1} = I_t + \beta I_t S_t = I_t + r I_t \left(1- \frac{I_t}{K}\right)\] where \(r = \beta /K\). (It reduces a bivariate Markov chain model to a single chain model of \(I_t\).)

Question One for the Exam

  • The state space of \(I_t\) is \(\{0,1,2,\dots,2K\}\). From time \(t\) to \(t+1\), there is at most one change in the random variable \(I_t\). If \(I_t = i\), then \(I_{t+1}\) may change to either \(i+1\) or \(i-1\) or remain at the state \(i\).

  • I can tell you \(p_{i+1, i} = \Pr\{I_{t+1} = i +1 | I_t = i \}= r(i - i^2/2K)\). \(p_{0,0}=1\) ( \((0,0)\) state is an absorbing state). Think about the full transition matrix and simulation of \(I_t\) (and \(S_t\)).

  • Also, think about the deterministic continuous time version of the problem.

  • (For large \(K\), the stochastic model follows the deterministic model more closely, the probability distribution is approximately stationary for a long period of time - called quasistationary distribution. But MC sampler may have difficulty for the implementation.)

Estimation

  • A stationary (time homogeneous) Markov chain can be fit from given data.

  • The simplest way is MLE (empirical likelihood) \[\hat{p}_{ij} = \frac{\mbox{The number from state j to state i}}{\mbox{The number from any state to state i}} \\ = \sum_{t=0}^{T} \frac{1}{T} \frac{\# \{ X_{t+1} = s_i, X_{t}= s_j \}}{\sum_{k=0}^{N} \# \{ X_{t+1} = s_i, X_{t} = s_k \}}\]

mle=markovchainFit(data=y[1,], method="mle"); # other choices: laplace, map, bootstrap

Estimation

mle$estimate #  mle$confidenceInterval
## MLE Fit 
##  A  61 - dimensional discrete Markov Chain defined by the following states: 
##  10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 5, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 6, 60, 61, 62, 63, 64, 65, 7, 8, 9 
##  The transition matrix  (by rows)  is defined as follows: 
##           10         11         12         13         14     15        16
## 10 0.8333333 0.16666667 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 11 0.0000000 0.97959184 0.02040816 0.00000000 0.00000000 0.0000 0.0000000
## 12 0.0000000 0.02222222 0.95555556 0.02222222 0.00000000 0.0000 0.0000000
## 13 0.0000000 0.00000000 0.00000000 0.91666667 0.08333333 0.0000 0.0000000
## 14 0.0000000 0.00000000 0.00000000 0.00000000 0.93750000 0.0625 0.0000000
## 15 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 1.0000000
## 16 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.9333333
## 17 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 18 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 19 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 20 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 21 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 22 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 23 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 24 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 25 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 26 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 27 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 28 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 29 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 30 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 31 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 32 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 33 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 34 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 35 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 36 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 37 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 38 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 39 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 40 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 41 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 42 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 43 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 44 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 45 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 46 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 47 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 48 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 49 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 5  0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 50 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 51 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 52 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 53 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 54 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 55 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 56 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 57 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 58 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 59 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 6  0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 60 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 61 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 62 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 63 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 64 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 65 0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 7  0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 8  0.0000000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
## 9  0.2500000 0.00000000 0.00000000 0.00000000 0.00000000 0.0000 0.0000000
##            17         18  19     20        21         22         23
## 10 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 11 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 12 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 13 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 14 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 15 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 16 0.06666667 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 17 0.97142857 0.02857143 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 18 0.00000000 0.50000000 0.5 0.0000 0.0000000 0.00000000 0.00000000
## 19 0.00000000 0.00000000 0.9 0.1000 0.0000000 0.00000000 0.00000000
## 20 0.00000000 0.00000000 0.0 0.9375 0.0625000 0.00000000 0.00000000
## 21 0.00000000 0.00000000 0.0 0.0000 0.9166667 0.08333333 0.00000000
## 22 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.92307692 0.07692308
## 23 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.80000000
## 24 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 25 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 26 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 27 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 28 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 29 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 30 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 31 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 32 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 33 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 34 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 35 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 36 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 37 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 38 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 39 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 40 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 41 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 42 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 43 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 44 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 45 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 46 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 47 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 48 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 49 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 5  0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 50 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 51 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 52 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 53 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 54 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 55 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 56 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 57 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 58 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 59 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 6  0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 60 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 61 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 62 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 63 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 64 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 65 0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 7  0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 8  0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
## 9  0.00000000 0.00000000 0.0 0.0000 0.0000000 0.00000000 0.00000000
##           24         25        26        27        28         29
## 10 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 11 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 12 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 13 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 14 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 15 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 16 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 17 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 18 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 19 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 20 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 21 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 22 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 23 0.2000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 24 0.9166667 0.08333333 0.0000000 0.0000000 0.0000000 0.00000000
## 25 0.0000000 0.85714286 0.1428571 0.0000000 0.0000000 0.00000000
## 26 0.0000000 0.00000000 0.8888889 0.1111111 0.0000000 0.00000000
## 27 0.0000000 0.00000000 0.0000000 0.7142857 0.2857143 0.00000000
## 28 0.0000000 0.00000000 0.0000000 0.1000000 0.8000000 0.10000000
## 29 0.0000000 0.00000000 0.0000000 0.0000000 0.1000000 0.80000000
## 30 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.08333333
## 31 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 32 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 33 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 34 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 35 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 36 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 37 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 38 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 39 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 40 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 41 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 42 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 43 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 44 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 45 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 46 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 47 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 48 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 49 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 5  0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 50 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 51 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 52 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 53 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 54 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 55 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 56 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 57 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 58 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 59 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 6  0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 60 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 61 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 62 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 63 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 64 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 65 0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 7  0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 8  0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
## 9  0.0000000 0.00000000 0.0000000 0.0000000 0.0000000 0.00000000
##            30        31        32         33         34        35
## 10 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 11 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 12 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 13 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 14 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 15 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 16 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 17 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 18 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 19 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 20 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 21 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 22 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 23 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 24 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 25 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 26 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 27 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 28 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 29 0.10000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 30 0.62500000 0.2916667 0.0000000 0.00000000 0.00000000 0.0000000
## 31 0.06363636 0.7909091 0.1454545 0.00000000 0.00000000 0.0000000
## 32 0.00000000 0.1600000 0.7600000 0.08000000 0.00000000 0.0000000
## 33 0.00000000 0.0000000 0.1176471 0.80882353 0.07352941 0.0000000
## 34 0.00000000 0.0000000 0.0000000 0.08928571 0.73214286 0.1785714
## 35 0.00000000 0.0000000 0.0000000 0.00000000 0.11363636 0.7613636
## 36 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.1375000
## 37 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 38 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 39 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 40 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 41 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 42 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 43 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 44 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 45 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 46 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 47 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 48 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 49 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 5  0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 50 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 51 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 52 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 53 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 54 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 55 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 56 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 57 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 58 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 59 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 6  0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 60 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 61 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 62 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 63 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 64 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 65 0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 7  0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 8  0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
## 9  0.00000000 0.0000000 0.0000000 0.00000000 0.00000000 0.0000000
##           36        37        38        39        40         41        42
## 10 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 11 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 12 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 13 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 14 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 15 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 16 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 17 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 18 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 19 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 20 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 21 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 22 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 23 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 24 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 25 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 26 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 27 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 28 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 29 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 30 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 31 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 32 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 33 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 34 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 35 0.1250000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 36 0.7750000 0.0875000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 37 0.1372549 0.6666667 0.1960784 0.0000000 0.0000000 0.00000000 0.0000000
## 38 0.0000000 0.1030928 0.7835052 0.1134021 0.0000000 0.00000000 0.0000000
## 39 0.0000000 0.0000000 0.2391304 0.5652174 0.1956522 0.00000000 0.0000000
## 40 0.0000000 0.0000000 0.0000000 0.1666667 0.7037037 0.12962963 0.0000000
## 41 0.0000000 0.0000000 0.0000000 0.0000000 0.1228070 0.71929825 0.1578947
## 42 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.09473684 0.7473684
## 43 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.1200000
## 44 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 45 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 46 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 47 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 48 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 49 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 5  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 50 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 51 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 52 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 53 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 54 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 55 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 56 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 57 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 58 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 59 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 6  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 60 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 61 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 62 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 63 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 64 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 65 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 7  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 8  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
## 9  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000
##           43        44        45        46        47        48        49
## 10 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 11 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 12 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 13 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 14 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 15 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 16 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 17 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 18 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 19 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 20 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 21 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 22 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 23 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 24 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 25 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 26 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 27 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 28 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 29 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 30 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 31 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 32 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 33 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 34 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 35 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 36 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 37 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 38 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 39 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 40 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 41 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 42 0.1578947 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 43 0.7200000 0.1600000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 44 0.1315789 0.7105263 0.1578947 0.0000000 0.0000000 0.0000000 0.0000000
## 45 0.0000000 0.1188119 0.7079208 0.1732673 0.0000000 0.0000000 0.0000000
## 46 0.0000000 0.0000000 0.1502146 0.6866953 0.1630901 0.0000000 0.0000000
## 47 0.0000000 0.0000000 0.0000000 0.1603376 0.6793249 0.1603376 0.0000000
## 48 0.0000000 0.0000000 0.0000000 0.0000000 0.2405063 0.6139241 0.1455696
## 49 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2555556 0.5666667
## 5  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 50 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2318841
## 51 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 52 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 53 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 54 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 55 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 56 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 57 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 58 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 59 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 6  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 60 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 61 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 62 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 63 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 64 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 65 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 7  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 8  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 9  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##            5        50        51        52        53        54        55
## 10 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 11 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 12 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 13 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 14 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 15 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 16 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 17 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 18 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 19 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 20 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 21 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 22 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 23 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 24 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 25 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 26 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 27 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 28 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 29 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 30 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 31 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 32 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 33 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 34 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 35 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 36 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 37 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 38 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 39 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 40 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 41 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 42 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 43 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 44 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 45 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 46 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 47 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 48 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 49 0.0000000 0.1777778 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 5  0.9824561 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 50 0.0000000 0.6231884 0.1449275 0.0000000 0.0000000 0.0000000 0.0000000
## 51 0.0000000 0.3125000 0.4687500 0.2187500 0.0000000 0.0000000 0.0000000
## 52 0.0000000 0.0000000 0.1521739 0.5652174 0.2826087 0.0000000 0.0000000
## 53 0.0000000 0.0000000 0.0000000 0.2765957 0.5744681 0.1489362 0.0000000
## 54 0.0000000 0.0000000 0.0000000 0.0000000 0.2121212 0.5454545 0.2424242
## 55 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2500000 0.4687500
## 56 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.2812500
## 57 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 58 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 59 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 6  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 60 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 61 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 62 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 63 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 64 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 65 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 7  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 8  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
## 9  0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
##           56        57        58        59          6        60        61
## 10 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 11 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 12 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 13 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 14 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 15 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 16 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 17 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 18 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 19 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 20 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 21 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 22 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 23 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 24 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 25 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 26 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 27 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 28 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 29 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 30 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 31 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 32 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 33 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 34 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 35 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 36 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 37 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 38 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 39 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 40 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 41 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 42 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 43 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 44 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 45 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 46 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 47 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 48 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 49 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 5  0.0000000 0.0000000 0.0000000 0.0000000 0.01754386 0.0000000 0.0000000
## 50 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 51 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 52 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 53 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 54 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 55 0.2812500 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 56 0.5000000 0.2187500 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 57 0.2592593 0.5555556 0.1851852 0.0000000 0.00000000 0.0000000 0.0000000
## 58 0.0000000 0.3846154 0.3076923 0.3076923 0.00000000 0.0000000 0.0000000
## 59 0.0000000 0.0000000 0.4000000 0.3000000 0.00000000 0.3000000 0.0000000
## 6  0.0000000 0.0000000 0.0000000 0.0000000 0.88888889 0.0000000 0.0000000
## 60 0.0000000 0.0000000 0.0000000 0.2727273 0.00000000 0.5454545 0.1818182
## 61 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.1818182 0.3636364
## 62 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.4545455
## 63 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 64 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 65 0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 7  0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 8  0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
## 9  0.0000000 0.0000000 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000
##            62        63        64  65         7         8          9
## 10 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 11 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 12 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 13 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 14 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 15 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 16 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 17 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 18 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 19 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 20 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 21 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 22 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 23 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 24 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 25 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 26 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 27 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 28 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 29 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 30 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 31 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 32 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 33 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 34 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 35 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 36 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 37 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 38 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 39 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 40 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 41 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 42 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 43 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 44 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 45 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 46 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 47 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 48 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 49 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 5  0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 50 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 51 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 52 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 53 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 54 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 55 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 56 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 57 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 58 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 59 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 6  0.00000000 0.0000000 0.0000000 0.0 0.1111111 0.0000000 0.00000000
## 60 0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 61 0.45454545 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 62 0.09090909 0.4545455 0.0000000 0.0 0.0000000 0.0000000 0.00000000
## 63 0.38461538 0.3076923 0.3076923 0.0 0.0000000 0.0000000 0.00000000
## 64 0.00000000 0.4000000 0.4000000 0.2 0.0000000 0.0000000 0.00000000
## 65 0.00000000 0.0000000 1.0000000 0.0 0.0000000 0.0000000 0.00000000
## 7  0.00000000 0.0000000 0.0000000 0.0 0.9000000 0.1000000 0.00000000
## 8  0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.9722222 0.02777778
## 9  0.00000000 0.0000000 0.0000000 0.0 0.0000000 0.0000000 0.75000000

Estimation

pm=as.matrix(mle$estimate@transitionMatrix); image(pm, axes=FALSE)

Simulation

MCSim= new("markovchain", transitionMatrix=pm)
ys= markovchainSequence(n=SimTime-1, markovchain=MCSim, t0=5, include=TRUE);
ts.plot(ys)

Continuous Time Markov Chains

  • For the moment, let’s allow time to be continuous and assume a finite state space (\(N\) states).

  • Transitions probability under the continuous time. \[\Pr\left( X_{t+\Delta t}=S_i | X_{t}= S_j \right) =p_{ij}(t,t+\Delta t)\]

  • The transition probability matrix \[\mathbf{P}(t, t+\Delta t)=\{p_{ij}(t, t+\Delta t)\}_{i,j=1}^{N}.\]

  • For time homogenous or stationary: \[\Pr\left( X_{t+\Delta t}=S_i | X_{t}= S_j \right) = \Pr(X_{\Delta t} = S_i | X_{0} = S_j)=p_{ij}(\Delta t).\]

CK Equation

  • The interval from \(k\) to \(t\) can be partitioned (discretized) into \(k =t_0 < t_1 <\dots < t_m = t\). The transition probability matrix can be written as a product: \[ \mathbf{P}(k,t) = \mathbf{P}(t_0,t_m)=\mathbf{P}(t_0,t_1) \mathbf{P}(t_1,t_2) \dots \mathbf{P}(t_{m-1},t_m)\] This follows the CK equation in discrete time setting.

  • When the chain is stationary and \(t_1 - t_0 =t_2 -t_1=\cdots =\Delta t\), \[\mathbf{P}(k,t) = \mathbf{P}(t_0,t_m)=[\mathbf{P} (0,\Delta t)]^m=[\mathbf{P} (\Delta t)]^m.\]

CK Equation

  • Now consider the stationary CK \(\mathbf{P}(t + \Delta t)= \mathbf{P}(t) \mathbf{P}(\Delta t)\): \[ \frac{\mathbf{P}(t + \Delta t) - \mathbf{P}(t)}{t+\Delta t - t} = \frac{\mathbf{P}(t)\left[\mathbf{P}(\Delta t) - \mathbf{I} \right]}{\Delta t}\] and take the limit \(\Delta t \rightarrow 0\) \[ \lim_{\Delta t \rightarrow 0} \frac{\mathbf{P}(t + \Delta t) - \mathbf{P}(t)}{t+\Delta t - t} = \frac{d\mathbf{P} (t)}{dt}= \mathbf{P}(t) \times \lim_{\Delta t \rightarrow 0} \frac{\left[\mathbf{P}(\Delta t) - \mathbf{I} \right]}{\Delta t}.\]

CK Equation

  • For the moment, let’s assume the limit of \((\mathbf{P}(\Delta t) - \mathbf{I})/\Delta t\) exists and we denote it as a matrix \(\mathbf{Q}\). Then we will obtain the continuous time CK forward equation \[\frac{d\mathbf{P}(t)}{dt} = \mathbf{P}(t) \mathbf{Q}.\] Similarly, if we use \(\mathbf{P}(t + \Delta t)= \mathbf{P}(\Delta t)\mathbf{P}(t)\), we will have the CK backward equation \[\frac{d\mathbf{P}(t)}{dt} = \mathbf{Q} \mathbf{P}(t).\]

Remarks

  • Markov chain’s Burn-in

  • Construct a stochastic difference equation (based on a deterministic skeleton)

  • Probability evolution

  • Estimation and simulation

  • Continuous time CK (Kolmogorov forward equation for probability evolution)