Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6. Find the probability that he wins 8 dollars before losing all of his money if

  1. he bets 1 dollar each time (timid strategy).

We first find the transition matrix, which considers the probability of moving from one state to the next. For example, the probability of moving from 1 to 2 is 0.4.

T = matrix(c(1,0,0,0,0,0,0,0,0,
             0.6,0,0.4,0,0,0,0,0,0,
             0,0.6,0,0.4,0,0,0,0,0,
             0,0,0.6,0,0.4,0,0,0,0,
             0,0,0,0.6,0,0.4,0,0,0,
             0,0,0,0,0.6,0,0.4,0,0,
             0,0,0,0,0,0.6,0,0.4,0,
             0,0,0,0,0,0,0.6,0,0.4,
             0,0,0,0,0,0,0,0,1), nrow = 9, ncol = 9, byrow = TRUE)
rownames(T) = c(0:8)
colnames(T) = c(0:8)

The probability of moving from 0 to 0 is 100% since you need at least A dollars to win A dollars, which makes this an absorbing state. We also want 8 to 8 to be an absorbing state since that is the final steady state outcome. All the steps in between are transient states as each state will continue to change until it reaches 0 or 8.

We define The transient states as T_new by removing the absorbing states and the absorbing states are defined as R.

#identifies when diagonal of matrix is 1, which is the index for the absorbing states. Returns array of indices. 
absorbing_states <- which(diag(T) == 1)

#T_new removes the absorbing states from the transition to isolate the transition states. 
T_new <- T[-absorbing_states, -absorbing_states]

#We calculate the fundamental matrix next (N = (I-Q)^-1)
N = solve(diag(nrow(T_new))-T_new)

#B = NR; bij is the probability that an absorbing chain will be absorbed in the absorbing state sj. (ref: pg 420 of text)
R = T[,absorbing_states]
R = R[-absorbing_states,]
prob_absorb = N %*% R
prob_absorb
##           0          8
## 1 0.9796987 0.02030135
## 2 0.9492466 0.05075337
## 3 0.9035686 0.09643140
## 4 0.8350515 0.16494845
## 5 0.7322760 0.26772403
## 6 0.5781126 0.42188739
## 7 0.3468676 0.65313243
print(paste0('There is a ',round(prob_absorb[1,2]*100,3), '% probability of absorption into 8 when starting from state 1'))
## [1] "There is a 2.03% probability of absorption into 8 when starting from state 1"
  1. he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).
T = matrix(c(1,0,0,0,0,
             0.6,0,0.4,0,0,
             0.6,0,0,0.4,0,
             0.6,0,0,0,0.4,
             0,0,0,0,1), nrow = 5, ncol = 5, byrow = TRUE)
rownames(T) = c(0,1,2,4,8)
colnames(T) = c(0,1,2,4,8)

#identifies when diagonal of matrix is 1, which is the index for the absorbing states. Returns array of indices. 
absorbing_states <- which(diag(T) == 1)

#T_new removes the absorbing states from the transition to isolate the transition states. 
T_new <- T[-absorbing_states, -absorbing_states]

#We calculate the fundamental matrix next (N = (I-Q)^-1)
N = solve(diag(nrow(T_new))-T_new)

#B = NR; bij is the probability that an absorbing chain will be absorbed in the absorbing state sj. (ref: pg 420 of text)
R = T[,absorbing_states]
R = R[-absorbing_states,]
prob_absorb = N %*% R
prob_absorb
##       0     8
## 1 0.936 0.064
## 2 0.840 0.160
## 4 0.600 0.400
print(paste0('There is a ',round(prob_absorb[1,2]*100,3), '% probability of absorption into 8 when starting from state 1'))
## [1] "There is a 6.4% probability of absorption into 8 when starting from state 1"
  1. Which strategy gives Smith the better chance of getting out of jail?

To my surprise, the bolder strategy gives Smith the better chance of getting out of jail. His probability increases from 2.03% to 6.4%.