Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6. Find the probability that he wins 8 dollars before losing all of his money if

  1. he bets 1 dollar each time (timid strategy)
p <- c(.6,0,.4)
bb <- c(1,rep(0,8),p,rep(0,7),p,rep(0,7),p,rep(0,7),p,rep(0,7),p,rep(0,7),p,rep(0,7),p,rep(0,8),1)
P_bb <- matrix(bb,9,9,byrow=TRUE)
Build the Markov Chain matrix
(Pbb <-new('markovchain', transitionMatrix = P_bb, states = c('0','1','2','3','4','5','6','7','8'),name = 'Bet for Bail'))
## Bet for Bail 
##  A  9 - dimensional discrete Markov Chain defined by the following states: 
##  0, 1, 2, 3, 4, 5, 6, 7, 8 
##  The transition matrix  (by rows)  is defined as follows: 
##     0   1   2   3   4   5   6   7   8
## 0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
## 1 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0
## 2 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 3 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 4 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 5 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 6 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## 7 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
Identify transient states
# Determine transient states
(transientStates(Pbb))
## [1] "1" "2" "3" "4" "5" "6" "7"
Identify absorbing states
# determine absorbing states
(absorbingStates(Pbb))
## [1] "0" "8"
Create the Markov Chain in Conanical form
p <- c(.6,0,.4)
bb <- c(0,.4,rep(0,5),.6,0,p,rep(0,7),p,rep(0,7),p,rep(0,7),p,rep(0,7),p,rep(0,7),.6,0,0,.4,rep(0,7),1,rep(0,9),1)
P_bb <- matrix(bb,9,9,byrow=TRUE)

(Pbb <-new('markovchain', transitionMatrix = P_bb, states = c('1','2','3','4','5','6','7','0','8'),name = 'Betting on Bail Canonical Form'))
## Betting on Bail Canonical Form 
##  A  9 - dimensional discrete Markov Chain defined by the following states: 
##  1, 2, 3, 4, 5, 6, 7, 0, 8 
##  The transition matrix  (by rows)  is defined as follows: 
##     1   2   3   4   5   6   7   0   8
## 1 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.6 0.0
## 2 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0
## 3 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 4 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 5 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 6 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 7 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.0 0.4
## 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0
## 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
Contruct your Q matrix
(Q <- Pbb[1:7,1:7])
##     1   2   3   4   5   6   7
## 1 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 2 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 3 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 4 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 5 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## 6 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## 7 0.0 0.0 0.0 0.0 0.0 0.6 0.0
Construct the Identity matrix
(I <- diag(dim(Q)[2]))
##      [,1] [,2] [,3] [,4] [,5] [,6] [,7]
## [1,]    1    0    0    0    0    0    0
## [2,]    0    1    0    0    0    0    0
## [3,]    0    0    1    0    0    0    0
## [4,]    0    0    0    1    0    0    0
## [5,]    0    0    0    0    1    0    0
## [6,]    0    0    0    0    0    1    0
## [7,]    0    0    0    0    0    0    1
Calculate the Inverse martix
(I-Q)
##      1    2    3    4    5    6    7
## 1  1.0 -0.4  0.0  0.0  0.0  0.0  0.0
## 2 -0.6  1.0 -0.4  0.0  0.0  0.0  0.0
## 3  0.0 -0.6  1.0 -0.4  0.0  0.0  0.0
## 4  0.0  0.0 -0.6  1.0 -0.4  0.0  0.0
## 5  0.0  0.0  0.0 -0.6  1.0 -0.4  0.0
## 6  0.0  0.0  0.0  0.0 -0.6  1.0 -0.4
## 7  0.0  0.0  0.0  0.0  0.0 -0.6  1.0
Claculate the fundamental matrix
(N <- solve(I-Q))
##           1        2         3         4         5         6          7
## 1 1.6328311 1.054718 0.6693101 0.4123711 0.2410785 0.1268834 0.05075337
## 2 1.5820777 2.636796 1.6732752 1.0309278 0.6026963 0.3172086 0.12688343
## 3 1.5059477 2.509913 3.1792228 1.9587629 1.1451229 0.6026963 0.24107851
## 4 1.3917526 2.319588 2.9381443 3.3505155 1.9587629 1.0309278 0.41237113
## 5 1.2204600 2.034100 2.5765266 2.9381443 3.1792228 1.6732752 0.66931007
## 6 0.9635210 1.605868 2.0340999 2.3195876 2.5099128 2.6367962 1.05471848
## 7 0.5781126 0.963521 1.2204600 1.3917526 1.5059477 1.5820777 1.63283109
Calcualte the number of steps before the chain is absorbed.
c_calc <- c(rep(1,7))
c <- matrix(c_calc,7,1,byrow=TRUE)
(t <- N %*% c)
##        [,1]
## 1  4.187946
## 2  7.969865
## 3 11.142744
## 4 13.402062
## 5 14.291039
## 6 13.124504
## 7  8.874703
Construct the R matrix (non-zero t by r matrix)
(R <- Pbb[1:7,8:9])
##     0   8
## 1 0.6 0.0
## 2 0.0 0.0
## 3 0.0 0.0
## 4 0.0 0.0
## 5 0.0 0.0
## 6 0.0 0.0
## 7 0.0 0.4
Calculate the probabilities that an absorbing chain will be absorbed
The probability that Smith wins 8 dollars before losing all of his money if he bets 1 dollar each time (timid strategy) is 0.0203.
(round(B <- N %*% R,4))
##        0      8
## 1 0.9797 0.0203
## 2 0.9492 0.0508
## 3 0.9036 0.0964
## 4 0.8351 0.1649
## 5 0.7323 0.2677
## 6 0.5781 0.4219
## 7 0.3469 0.6531
  1. he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy)

p(3) = 0.4p(6)

p(6) = 0.4p(8) + 0.6p(4)

p(4) = 0.4p(8)

p(0) = 0

p(8) = 1

solve and find p(3) = 0.256, p(4) = 0.4, p(6) = 0.64

  1. By comparing the third components of the vector φ we find that the bold strategy gives Smith a better chance to get out jail
dbinom(3,3,0.4)
## [1] 0.064