Markov Model Notes
Markov Processes can be used to model a random process with multiple steps
Markov property governs the markov process, which states that only the most recent information (state) is needed for conditional probabilities
\[ P(X_t\in A| \{X_u\}_{u\leq s}) = P(X_t\in A|X_s)\]
- Transition probability of a continuous-time Markov Chain is:
\[ P_{i,j}(t,t+s) = P(X_{t+s}=j|X_t=i)\]
- Homogeneous Markov Chain property:
\[ P_{i,j}(t,t+s) = P_{i,j}(0,s)\]
2-State Model
- Model the random process by which a life passed from one state (alive) to another (death) by Markov Process
- States are alive a or dead d
- Dead state is absorbing
- Alive state is transient
- Time spent alive is the ‘future lifetime’
Kolmogorov Equations
- Backward Equations:
- Forward Equations:
Model Estimation
Assuming \(N\) identical and independent lives between ages \(x\) and \(x+1\)
Observation begins at age \(x+a_i\) for the ith life
Observation ends at age \(x+b_i\) for ith life
\(D_i = 1\) if the life dies
Random variable \(T_i\) such that \(x + T_i\) is the age at which observed life \(i\) dies.
- \(D_i\) and \(T_i\) are not independent
- Waiting time (time spent under observation) is:
\[ V_i = \min(T_i-a_, b_i-a_i)\]
The joint distribution of \(D_i\) and \(V_i\) is (surviving the waiting time and then dying/not dying in the next instant):
\[ f_i(d_i, v_i) = \exp\left(-\int^{v_i}_0\mu_{\mu_{x+a_i+t}}dt\right)[\mu_{x+a_i+v_i}]^{d_i}\] ### Piecewise Constant Force of Mortality Assumption
- Considering the case where \(\mu_{x+t}\) is constant for \(0\leq t<1\), then the joint distribution is a mixed distribution:
\[ f_i(d_i,v_i) = e^{-\mu v_i}\mu^{d_i}\]
If \(d_i = 0\) then \(v_i = b_i-a_i\) since the life survives to \(x+b_i\)
If \(d_i=1\) then \(v_i=t_i-a_i\) since the life dies at age \(x+t_i\)
Note that the following is true as a life either survives to \(x+b_i\) or dies between \(x+a_i\) and \(x+b_i\)
\[ \underbrace{e^{-\mu(b_i-a_i)}}_{d_i = 0} + \underbrace{\int^{b_i-a_i}_0e^{-\mu v_i}dv_i}_{d_i = 1} = 1\] * Probability Function of \(D_i\):
- Probability Function of \(v_i\):
Maximum Likelihood Estimation
\[ L(\mu) = \prod^N_{i=1}e^{\mu v_i}\mu^{d_i} = e^{-\mu v}\mu^d \]
Where \(d=\sum^N_{i=1}d_i\) is the total number of deaths
Where \(v=\sum^N_{i=1}v_i\) is the total waiting time or central exposed-to-risk
The maximum likelihood estimate is therefore:
- The total deaths / total wait time
\[ \hat{\mu} = \frac{D}{V} \]
- Therefore:
\[ \mathbb{E}[D] = \mu\mathbb{E}[V]\]
- With Distribution:
\[ \hat{\mu}\sim N\left(\mu, \frac{\mu}{\mathbb{E}[V]}\right) = N\left(\mu,\frac{\mu^2}{\mathbb{E}[D]}\right)\]
General Markov Model
- The markov process can be generalised to multiple decrements, such as alive, sick, retired, etc
- Transition probabilities:
Kolmogorov Equations
- The Kolmogorov forward equations are as follows:
- Note this is the rate of change of the probability, so we add people coming in to h from some state j, and people leaving h to some state j (in the next instant).
- Given the transition intensities (estimated from data) we can determine transition probabilities based on these equations
Solving Ordinary Differential Equations with Integrating Factor
- We can use an integration factor when the differential equation is in the form:
\[ P(t)' + a(t)P(t) = b(t)\]
- Integration factor:
\[ M(t) = e^{\int^t_0a(s)ds}\] * We multiple both sides of the differential equation by the integration factor:
\[ P(t)'M(t) + a(t)P(t)M(t) = b(t)M(t)\]
\[ P(t)'M(t)+P(t)M(t)' = b(t)M(t)\]
\[ (P(t)M(t))' = b(t)M(t)\]
\[ P(t)M(t) = \int^t_0b(s)M(s)ds + C\]
- Where we calculate C based initial condition
Holding Times and Jump Probabilities
- In a time homogenous Markov jump process, consider \(\mu_{ij}\) as the transition rates from state i to state j for \(j\neq i\) and \(\mu_i = \sum_{j\neq i}\mu_{ij}\)
- Consider \(W_i\) as the duration until the process leaves state i given that the current state is i.
- Consider \(W_i\) as the duration until the process leaves state i given that the current state is i.
- \(W_i\) is an exponential random variable
- \(S_{W_i}(t) = e^{-\mu_i t}\)
- \(F_{W_i}(t) = 1-e^{-\mu_it}\)
- \(f_{W_i}(t) = \mu_ie^{-\mu_i t}\)
- \(\mathbb{E}[W_i] = \frac{1}{\mu_i}\)
- \(Var(W_i) = \left(\frac{1}{\mu_i}\right)^2\)
- The probability that the Markov Process goes into state j when it leaves state i:
\[ P(X_{W_i}=j|X_0=i) = \frac{\mu_{ij}}{\mu_i}\]
Where \(\mu_i\) is defined by \(\mu_i=\sum_{j\neq i}\mu_{ij}\)
Assume that i can only go to i+1
- The probability that a person in state i will be in state i+1 t years from now is:
- Exponential density for the change (can also be interpreted to staying in state i for s the instanteously changing to i+1)
- Exponential survival from time s to t
- The probability that a person in state i will be in state i+1 t years from now is:
\[ \int^t_0\underbrace{\mu_ie^{-\mu_is}}_{\text{Change from i to i+1}}\underbrace{\left(e^{-\mu_{i+1}(t-s)}\right)}_{\text{ stays in i+1}}ds\]
- This idea can be extended to multiple jumps
\[ \int^t_0\int^t_s\mu_ie^{-\mu_is}\left(\mu_{i+1}e^{-\mu_{i+1}(r-s)}\right)\left(e^{-\mu_{i+2}(t-r)}\right)drds \]
Maximum Likeihood Estimates
- In a time-homogenous Markov Jump Process we observe
- Total number of samples \(N\)
- \(T_i = \sum^N_{I=1}T^{(I)}_{i}\): Total waiting time in state i, where \(T_i^{I}\) is the waiting time in state i of the Ith life
- \(N_{ij} = \sum^N_{i=1}N_{ij}^{(I)}\): The total number of transitions from state i to j, where \(N_{ij}^{I}\) concerns the Ith life
- The total likelihood for all lives is:
\[ \prod^m_{i=1}\left(e^{-\mu_it_i}\prod_{j\neq i}\mu_{ij}^{n_{ij}}\right)\]
- The maximum likelihood estimates of the transition rates \(\mu_{ij} (i\neq j)\) from state i to j is:
\[ \hat{\mu}_{ij}=\frac{n_{ij}}{t_i},\quad i\neq j\]
- As \(\mu_{ii} = -\sum_{j\neq i}\mu_{ij}\), the MLE of \(\mu_{ii}\) is:
\[ \hat{\mu}_{ii} = -\sum_{j\neq i}\hat{\mu}_{ij} \] * The properties of the MLE are (noting that we consider the expected waiting time for the state being left):
\[ \hat{\mu_{ij}}\sim N\left(\mu_{ij},\frac{\mu_{ij}}{\mathbb{E}[T_i]}\right)\]
Estimation of Transition Intensities
- The calculation of the estimates need to compute the total waiting time, which may not be possible in practice.
- We can use a simpler approach referred to as the Census approach
- Census data is available when number in each state recorded only at fixed dates
- We can estimate waiting times by making simplifying assumptions, e.g assuming transitions occur on average half way though the time interval.
- Below is an example of the total waiting time with this assumption, relying on the piecewise linear assumption
- Assuming the length of interval is 1 in this case
\[ v_i = \int^T_0 P_i(t)dt\approx \sum^{T-1}_{k=0}\frac{P_i(k)+P_i(k+1)}{2}*(\text{length of interval}) = \frac{1}{2}P_i(0)+\sum^{T-1}_{k=1}P_i(k)+\frac{1}{2}P_i(T)\]