Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6.

Timid Strategy

Find the probability that he wins 8 dollars before losing all of his money if he bets 1 dollar each time

This problem can be represented as an absorbing markov chain with a transition matrix in canonical form like this*:

\[ P = \begin{array}{c | c} Q & R \\ \hline 0 & I\\ \end{array} = \begin{array}{c | c c c c c c c | c c} & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 0 & 8 \\ \hline 1 & 0 & 0.4 & 0 & 0 & 0 & 0 & 0 & 0.6 & 0 \\ 2 & 0.6 & 0 & 0.4 & 0 & 0 & 0 & 0 & 0 & 0 \\ 3 & 0 & 0.6 & 0 & 0.4 & 0 & 0 & 0 & 0 & 0 \\ 4 & 0 & 0 & 0.6 & 0 & 0.4 & 0 & 0 & 0 & 0 \\ 5 & 0 & 0 & 0 & 0.6 & 0 & 0.4 & 0 & 0 & 0 \\ 6 & 0 & 0 & 0 & 0 & 0.6 & 0 & 0.4 & 0 & 0 \\ 7 & 0 & 0 & 0 & 0 & 0 & 0.6 & 0 & 0 & 0.4 \\ \hline 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 \\ 8 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\ \end{array} \]

# Create Matrix Q
Q <- matrix(c(rep(c(0, 0.4, 0, 0, 0, 0, 0, 0.6), 6), 0), nrow = 7, byrow = TRUE)
Q
##      [,1] [,2] [,3] [,4] [,5] [,6] [,7]
## [1,]  0.0  0.4  0.0  0.0  0.0  0.0  0.0
## [2,]  0.6  0.0  0.4  0.0  0.0  0.0  0.0
## [3,]  0.0  0.6  0.0  0.4  0.0  0.0  0.0
## [4,]  0.0  0.0  0.6  0.0  0.4  0.0  0.0
## [5,]  0.0  0.0  0.0  0.6  0.0  0.4  0.0
## [6,]  0.0  0.0  0.0  0.0  0.6  0.0  0.4
## [7,]  0.0  0.0  0.0  0.0  0.0  0.6  0.0
# Create identity matrix with same dimensions as Q
I <- diag(x=1, 7, 7)

# Find the fundamental matrix N
N <- solve(I - Q)
round(N, 2)
##      [,1] [,2] [,3] [,4] [,5] [,6] [,7]
## [1,] 1.63 1.05 0.67 0.41 0.24 0.13 0.05
## [2,] 1.58 2.64 1.67 1.03 0.60 0.32 0.13
## [3,] 1.51 2.51 3.18 1.96 1.15 0.60 0.24
## [4,] 1.39 2.32 2.94 3.35 1.96 1.03 0.41
## [5,] 1.22 2.03 2.58 2.94 3.18 1.67 0.67
## [6,] 0.96 1.61 2.03 2.32 2.51 2.64 1.05
## [7,] 0.58 0.96 1.22 1.39 1.51 1.58 1.63
# Create matrix R
R <- matrix(c(0.6, rep(0, 12), 0.4), nrow = 7, byrow = TRUE)
R
##      [,1] [,2]
## [1,]  0.6  0.0
## [2,]  0.0  0.0
## [3,]  0.0  0.0
## [4,]  0.0  0.0
## [5,]  0.0  0.0
## [6,]  0.0  0.0
## [7,]  0.0  0.4
# Find the absorption probability matrix B
B <- N %*% R
B
##           [,1]       [,2]
## [1,] 0.9796987 0.02030135
## [2,] 0.9492466 0.05075337
## [3,] 0.9035686 0.09643140
## [4,] 0.8350515 0.16494845
## [5,] 0.7322760 0.26772403
## [6,] 0.5781126 0.42188739
## [7,] 0.3468676 0.65313243
# Find the probability of being absorbed in state 8 given a starting point in state 1
Win <- round(B[1,2], 3)
Win
## [1] 0.02

Starting from state 1 with $1 he has a 0.02 chance of ending up (being absorbed) in state 8 with $8.

Bold Strategy

Find the probability that he wins 8 dollars before losing all of his money if he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars

This problem can be represented as an absorbing markov chain with a transition matrix in canonical form like this*:

\[ P = \begin{array}{c | c} Q & R \\ \hline 0 & I\\ \end{array} = \begin{array}{c | c c c | c c} & 1 & 2 & 4 & 0 & 8 \\ \hline 1 & 0 & 0.4 & 0 & 0.6 & 0 \\ 2 & 0 & 0 & 0.4 & 0.6 & 0 \\ 4 & 0 & 0 & 0 & 0.6 & 0.4 \\ \hline 0 & 0 & 0 & 0 & 1 & 0 \\ 8 & 0 & 0 & 0 & 0 & 1 \\ \end{array} \]

# Create Matrix Q
Q <- matrix(c(rep(c(0, 0.4, 0, 0), 2), 0), nrow = 3, byrow = TRUE)
Q
##      [,1] [,2] [,3]
## [1,]    0  0.4  0.0
## [2,]    0  0.0  0.4
## [3,]    0  0.0  0.0
# Create identity matrix with same dimensions as Q
I <- diag(x=1, 3, 3)

# Find the fundamental matrix N
N <- solve(I - Q)
round(N, 2)
##      [,1] [,2] [,3]
## [1,]    1  0.4 0.16
## [2,]    0  1.0 0.40
## [3,]    0  0.0 1.00
# Create matrix R
R <- matrix(c(rep(c(0.6, 0), 2), 0.6, 0.4), nrow = 3, byrow = TRUE)
R
##      [,1] [,2]
## [1,]  0.6  0.0
## [2,]  0.6  0.0
## [3,]  0.6  0.4
# Find the absorption probability matrix B
B <- N %*% R
B
##       [,1]  [,2]
## [1,] 0.936 0.064
## [2,] 0.840 0.160
## [3,] 0.600 0.400
# Find the probability of being absorbed in state 8 given a starting point in state 1
Win <- round(B[1,2], 3)
Win
## [1] 0.064

Starting from state 1 with $1 he has a 0.064 chance of ending up (being absorbed) in state 8 with $8.

Comparison and Conclusion

Which strategy gives Smith the better chance of getting out of jail?

Unfortunately for Smith neither strategy gives him a very good chance of getting out of jail, but the bold strategy gives him a better chance at 6.4% vs. a 2% chance with the timid strategy, so the bold strategy is definitely a better choice.

*Note: I couldn’t get the exact proper formatting of a transition matrix in canonical form to work in RStudio/LaTeX but I think the above format is clear enough…