Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6. Find the probability that he wins 8 dollars before losing all of his money if

(a) he bets 1 dollar each time (timid strategy).

0.0203 chance. The transition matrix is \[ P= \begin{pmatrix} 0 & 0.4 & 0& 0& 0& 0& 0& 0.6& 0\\ 0.6 & 0 & 0.4& 0& 0& 0& 0& 0& 0\\ 0 & 0.6 & 0& 0.4& 0& 0& 0& 0& 0\\ 0 & 0 & 0.6& 0& 0.4& 0& 0& 0& 0\\ 0 & 0 & 0& 0.6& 0& 0.4& 0& 0& 0\\ 0 & 0 & 0& 0& 0.6& 0& 0.4& 0& 0\\ 0 & 0& 0& 0& 0& 0.6& 0& 0& 0.4\\ 0 & 0 & 0& 0& 0& 0& 0& 1& 0 \\ 0 & 0& 0& 0& 0& 0& 0& 0& 1 \end{pmatrix} \]

The row names are 1, 2, 3, 4, 5, 6, 7, 0, 8, column names are 1, 2, 3, 4, 5, 6, 7, 0, 8. For example, if we look at the first row, 2nd element will show us if we had 1 dollar and what’s the chance to win 2, it is 0.4 (given at the beginning). Third element is 0 as using timid strategy you can’t get 3 dollars if you originally had 1, 8th element show us the chance to lose and have nothing if we had 1 dollar at the beginning, it is 0.6 (given at the beginning).

To find chance that we will reach 8 dollars faster than staying with nothing we will use the Markov chain with absorption probabilities, 0 and 8 are the absorb states. We will split the transition matrix to get Q - 7x7 matrix representing the transition probabilities from the non-absorbing states into the non-absorbing states, R - 7x2 matrix representing the transition probabilities from the non-absorbing states into the absorbing states.

\[ Q = \begin{pmatrix} 0 & 0.4 & 0& 0& 0& 0& 0\\ 0.6 & 0 & 0.4& 0& 0& 0& 0\\ 0 & 0.6 & 0& 0.4& 0& 0& 0\\ 0 & 0 & 0.6& 0& 0.4& 0& 0\\ 0 & 0 & 0& 0.6& 0& 0.4& 0\\ 0 & 0 & 0& 0& 0.6& 0& 0.4\\ 0 & 0& 0& 0& 0& 0.6& 0 \end{pmatrix} \] \[ R = \begin{pmatrix} 0.6& 0\\ 0& 0\\ 0& 0\\ 0& 0\\ 0& 0\\ 0& 0\\ 0& 0.4 \end{pmatrix} \] Fundamental matrix N, where I is identity matrix of the same size as Q (7x7): \[N=(I-Q)^{-1}\] Matrix B will give us the probability of absorption in state 0 and in state 8. Columns are 0 and 8, rows are states from 1 to 7. \[ B=N \cdot R = \begin{pmatrix} 0.9797 & 0.0203\\ 0.9493 & 0.0508\\ 0.9036 & 0.0964\\ 0.8351 & 0.1650\\ 0.7323 & 0.2677\\ 0.5781 & 0.4219\\ 0.3469 & 0.6531 \end{pmatrix} \] If we have state 1$ than it is 0.0203 chance to get 8 dollars before getting 0. Using R

Q <- matrix(c(0, 0.4, 0, 0, 0, 0, 0,    0.6, 0, 0.4, 0,0,0,0,  0, 0.6, 0, 0.4, 0,0,0,  0,0, 0.6, 0, 0.4, 0,0,  0,0,0, 0.6, 0, 0.4, 0,  0,0,0,0, 0.6, 0, 0.4,   0,0,0,0,0,0.6, 0), nrow = 7, byrow = TRUE)
Q
##      [,1] [,2] [,3] [,4] [,5] [,6] [,7]
## [1,]  0.0  0.4  0.0  0.0  0.0  0.0  0.0
## [2,]  0.6  0.0  0.4  0.0  0.0  0.0  0.0
## [3,]  0.0  0.6  0.0  0.4  0.0  0.0  0.0
## [4,]  0.0  0.0  0.6  0.0  0.4  0.0  0.0
## [5,]  0.0  0.0  0.0  0.6  0.0  0.4  0.0
## [6,]  0.0  0.0  0.0  0.0  0.6  0.0  0.4
## [7,]  0.0  0.0  0.0  0.0  0.0  0.6  0.0
R <- matrix(c(0.6, 0, 0,0,0,0,0,0,0,0,0,0, 0, 0.4), nrow = 7, byrow = TRUE)
R
##      [,1] [,2]
## [1,]  0.6  0.0
## [2,]  0.0  0.0
## [3,]  0.0  0.0
## [4,]  0.0  0.0
## [5,]  0.0  0.0
## [6,]  0.0  0.0
## [7,]  0.0  0.4
I <- diag(7)
N <- ginv(I-Q)
B<- N%*%R
B
##           [,1]       [,2]
## [1,] 0.9796987 0.02030135
## [2,] 0.9492466 0.05075337
## [3,] 0.9035686 0.09643140
## [4,] 0.8350515 0.16494845
## [5,] 0.7322760 0.26772403
## [6,] 0.5781126 0.42188739
## [7,] 0.3468676 0.65313243

(b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).

0.064 chance. The transition matrix is \[ \begin{pmatrix} 0 & 0.4 & 0& 0.6& 0\\ 0 & 0 & 0.4& 0.6& 0\\ 0 & 0 & 0& 0.6& 0.4\\ 0 & 0 & 0& 1& 0\\ 0 & 0 & 0& 0& 1 \end{pmatrix} \] The row names are 1, 2, 4,0,8, column names are 1, 2, 4,0,8.

To find chance that we will reach 8 dollars faster than staying with nothing we will use the Markov chain with absorption probabilities, 0 and 8 are the absorb states. We will split the transition matrix to get Q - 3x3 matrix representing the transition probabilities from the non-absorbing states into the non-absorbing states, R - 3x2 matrix representing the transition probabilities from the non-absorbing states into the absorbing states.

\[ Q = \begin{pmatrix} 0 & 0.4 & 0\\ 0 & 0 & 0.4\\ 0 & 0 & 0 \end{pmatrix} \] \[ R = \begin{pmatrix} 0.6& 0\\ 0.6& 0\\ 0.6& 0.4 \end{pmatrix} \] Fundamental matrix N, where I is identity matrix of the same size as Q (3x2): \[N=(I-Q)^{-1}\] Matrix B will give us the probability of absorption in state 0 and in state 8. Columns are 0 and 8, rows are states 1, 2, 4. \[ B=N \cdot R = \begin{pmatrix} 0.936 & 0.064 \\ 0.840 & 0.160 \\ 0.600 & 0.400 \end{pmatrix} \] If we have state 1$ than it is 0.064 chance to get 8 dollars before getting 0. Using R

Q <- matrix(c(0, 0.4, 0,    0, 0, 0.4,  0, 0, 0), nrow = 3, byrow = TRUE)
Q
##      [,1] [,2] [,3]
## [1,]    0  0.4  0.0
## [2,]    0  0.0  0.4
## [3,]    0  0.0  0.0
R <- matrix(c(0.6, 0, 0.6, 0, 0.6, 0.4), nrow = 3, byrow = TRUE)
R
##      [,1] [,2]
## [1,]  0.6  0.0
## [2,]  0.6  0.0
## [3,]  0.6  0.4
I <- diag(3)
N <- ginv(I-Q)
B<- N%*%R
B
##       [,1]  [,2]
## [1,] 0.936 0.064
## [2,] 0.840 0.160
## [3,] 0.600 0.400

(c) Which strategy gives Smith the better chance of getting out of jail?

The bold strategy since the chance of getting 8 dollars before losing all money is 0.64 comparing to the timid strategy where the chance is only 0.0203.