Data 605 HW10

library(knitr)
library(rmdformats)

## Global options
options(max.print="85")
opts_chunk$set(cache=TRUE,
               prompt=FALSE,
               tidy=TRUE,
               comment=NA,
               message=FALSE,
               warning=FALSE)
opts_knit$set(width=31)

Central Theme of the Week

This week the reading focused on Markov Chains and Random Walks. A Markov chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.

Random Walk

as a Markov chain

Gambler’s Ruin

Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6.

Find the probability that he wins 8 dollars before losing all of his money if

(a) he bets 1 dollar each time (timid strategy).

Ans: For 0 ≤ z ≤ M, let the quantity p_z be the probability that the gambler’s stake reaches M without ever having reached 0.

q_z = \(\frac{(\frac{q}{p})^{M}-(\frac{q}{p})^{z}}{(\frac{q}{p})^{M}-1}\)

p_z = 1 - \(\frac{(\frac{q}{p})^{M}-(\frac{q}{p})^{z}}{(\frac{q}{p})^{M}-1}\)

p <- 0.4
q <- 0.6
M <- 8
z <- 1

# Calculate the probability of reaching M before reaching 0
p_z1 <- 1 - ((q/p)^M - (q/p)^z)/((q/p)^M - 1)
p_z1

[1] 0.02030135

So the probability that the gambler’s, or Smith’s, stake reaches M without ever having reached 0 if the gambler uses a timid strategy is ≅ 0.02.

(b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).

Ans: To answer this question, we need to use Theorem 11.2

Let P be the transition matrix of a Markov chain, and let u be the probability vector which represents the starting distribution. Then the probability that the chain is in state s_i after n steps is the i th entry in the vector

\[ \begin{equation} \boldsymbol{u}^{(n)}=\mathbf{u}\mathbf{P}^{n} \end{equation} \]

Let P be the transition matrix = \(\begin{bmatrix}1 & 0 & 0 & 0 & 0\\ .6 & 0 & .4 & 0 & 0\\ .6 & 0 & 0 & .4 & 0\\ .6 & 0 & 0 & 0 & .4\\ 0 & 0 & 0 & 0 & 1\end{bmatrix}\)

u = \(\begin{bmatrix}0\\ 1\\ 0\\ 0\\ 0\end{bmatrix}\)

# Trasition Matrix P
P <- matrix(c(1, 0, 0, 0, 0, 0.6, 0, 0.4, 0, 0, 0.6, 0, 0, 0.4, 0, 0.6, 0, 0, 0, 
    0.4, 0, 0, 0, 0, 1), nrow = 5, ncol = 5, byrow = T)
P

     [,1] [,2] [,3] [,4] [,5]
[1,]  1.0    0  0.0  0.0  0.0
[2,]  0.6    0  0.4  0.0  0.0
[3,]  0.6    0  0.0  0.4  0.0
[4,]  0.6    0  0.0  0.0  0.4
[5,]  0.0    0  0.0  0.0  1.0

# probability vector, u, that represents that starting distribution
u <- matrix(c(0, 1, 0, 0, 0), nrow = 5, ncol = 1, byrow = F)
u <- t(u)
u

     [,1] [,2] [,3] [,4] [,5]
[1,]    0    1    0    0    0

# u_z4 final distribution of the probability vector after 4 steps
u_z1 <- u %*% P
u_z2 <- u_z1 %*% P
u_z3 <- u_z2 %*% P
u_z4 <- u_z3 %*% P
u_z4

      [,1] [,2] [,3] [,4]  [,5]
[1,] 0.936    0    0    0 0.064

So the probability that the gambler’s, or Smith’s, stake reaches M without ever having reached 0 if the gambler uses a bold strategy is ≅ 0.06.

(c) Which strategy gives Smith the better chance of getting out of jail?

Of course, the bold strategy.