Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6.

Find the probability that he wins 8 dollars before losing all of his money if


Answer -

We arrange the transition matrix for this problem in cannonical form fro the scenarios given below, so that we will get

\(Q^n\) = probabilities for being in each of the transient states after n steps for each possible transient strarting state
R = proablities for t (transient) state by r (absorbing) matrix
I = identifcal matrix
0 - zero matrix

  1. he bets 1 dollar each time (timid strategy)
P = matrix(c(1.0, rep(0,8),
             0, 0, 0.4,rep(0,6), 
             0, 0.6, 0, 0.4, rep(0,5),
             0, 0, 0.6, 0, 0.4, rep(0,4),
             0, 0, 0, 0.6, 0, 0.4, rep(0,3), 
             0, 0, 0, 0, 0.6, 0, 0.4, 0, 0,
             rep(0,5), 0.6, 0, 0.4, 0,
             rep(0,6), 0.6, 0, 0.4, 
             rep(0,8), 1.0),  nrow=9, ncol=9, byrow=TRUE)

rownames(P) = c(0:8) 
colnames(P) = c(0:8)
P
##   0   1   2   3   4   5   6   7   8
## 0 1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
## 1 0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0
## 2 0 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 3 0 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 4 0 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 5 0 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 6 0 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## 7 0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## 8 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0
Q = matrix(c(0,0.4,rep(0,5), 
             0.6, 0, 0.4, rep(0,4),
             0, 0.6, 0, 0.4, 0, 0, 0,
             0, 0, 0.6, 0, 0.4, 0, 0, 
             0, 0, 0, 0.6, 0, 0.4, 0,
             rep(0,4), 0.6, 0, 0.4,
             rep(0,5), 0.6, 0), nrow=7, ncol=7, byrow=TRUE)

rownames(Q) = c(1:7) 
colnames(Q) = c(1:7)
Q
##     1   2   3   4   5   6   7
## 1 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 2 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 3 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 4 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 5 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## 6 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## 7 0.0 0.0 0.0 0.0 0.0 0.6 0.0
I = diag(7)
rownames(I) = c(1:7) 
colnames(I) = c(1:7)
I
##   1 2 3 4 5 6 7
## 1 1 0 0 0 0 0 0
## 2 0 1 0 0 0 0 0
## 3 0 0 1 0 0 0 0
## 4 0 0 0 1 0 0 0
## 5 0 0 0 0 1 0 0
## 6 0 0 0 0 0 1 0
## 7 0 0 0 0 0 0 1
S = I - Q
S
##      1    2    3    4    5    6    7
## 1  1.0 -0.4  0.0  0.0  0.0  0.0  0.0
## 2 -0.6  1.0 -0.4  0.0  0.0  0.0  0.0
## 3  0.0 -0.6  1.0 -0.4  0.0  0.0  0.0
## 4  0.0  0.0 -0.6  1.0 -0.4  0.0  0.0
## 5  0.0  0.0  0.0 -0.6  1.0 -0.4  0.0
## 6  0.0  0.0  0.0  0.0 -0.6  1.0 -0.4
## 7  0.0  0.0  0.0  0.0  0.0 -0.6  1.0
N = solve(S) # inverse of I - Q
N
##           1        2         3         4         5         6          7
## 1 1.6328311 1.054718 0.6693101 0.4123711 0.2410785 0.1268834 0.05075337
## 2 1.5820777 2.636796 1.6732752 1.0309278 0.6026963 0.3172086 0.12688343
## 3 1.5059477 2.509913 3.1792228 1.9587629 1.1451229 0.6026963 0.24107851
## 4 1.3917526 2.319588 2.9381443 3.3505155 1.9587629 1.0309278 0.41237113
## 5 1.2204600 2.034100 2.5765266 2.9381443 3.1792228 1.6732752 0.66931007
## 6 0.9635210 1.605868 2.0340999 2.3195876 2.5099128 2.6367962 1.05471848
## 7 0.5781126 0.963521 1.2204600 1.3917526 1.5059477 1.5820777 1.63283109
R = matrix(c(0.6,rep(0,6),
             rep(0,6), 0.4), ncol=2)

rownames(R) = c(1:7) 
colnames(R) = c(0,8)
R
##     0   8
## 1 0.6 0.0
## 2 0.0 0.0
## 3 0.0 0.0
## 4 0.0 0.0
## 5 0.0 0.0
## 6 0.0 0.0
## 7 0.0 0.4
B = N%*%R
B
##           0          8
## 1 0.9796987 0.02030135
## 2 0.9492466 0.05075337
## 3 0.9035686 0.09643140
## 4 0.8350515 0.16494845
## 5 0.7322760 0.26772403
## 6 0.5781126 0.42188739
## 7 0.3468676 0.65313243

The absorption matrix B shows that using the timid strategy the probality of winning P[1,8] (\(S_{18}\)) = 0.02030135.
The probalibty of losing p[1,0] (\(S_{10}\)) = 0.9796987.



(b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).

P = matrix(c(1.0, rep(0,8),
             0.6, 0, 0.4, rep(0,6), 
             0.6, 0, 0, 0, 0.4, rep(0,4),
             0.6, 0, 0, 0, 0, 0,  0.4, 0, 0,
             0.6, 0, 0, 0, 0, 0, 0, 0, 0.4, 
             0, 0, 0.6, 0, 0, 0, 0, 0, 0.4,
             rep(0,4), 0.6, 0, 0, 0, 0.4,
             rep(0,6), 0.6, 0, 0.4, 
             rep(0,8), 1.0),  nrow=9, ncol=9, byrow=TRUE)

rownames(P) = c(0:8) 
colnames(P) = c(0:8)
P
##     0 1   2 3   4 5   6 7   8
## 0 1.0 0 0.0 0 0.0 0 0.0 0 0.0
## 1 0.6 0 0.4 0 0.0 0 0.0 0 0.0
## 2 0.6 0 0.0 0 0.4 0 0.0 0 0.0
## 3 0.6 0 0.0 0 0.0 0 0.4 0 0.0
## 4 0.6 0 0.0 0 0.0 0 0.0 0 0.4
## 5 0.0 0 0.6 0 0.0 0 0.0 0 0.4
## 6 0.0 0 0.0 0 0.6 0 0.0 0 0.4
## 7 0.0 0 0.0 0 0.0 0 0.6 0 0.4
## 8 0.0 0 0.0 0 0.0 0 0.0 0 1.0
Q = matrix(c(0, 0.4, rep(0,5), 
             0, 0, 0, 0.4, rep(0,3),
             0, 0, 0, 0, 0,  0.4, 0,
             0, 0, 0, 0, 0, 0, 0,  
             0, 0.6, 0, 0, 0, 0, 0, 
             rep(0,3), 0.6, 0, 0, 0,
             rep(0,5), 0.6, 0),  nrow=7, ncol=7, byrow=TRUE)


rownames(Q) = c(1:7) 
colnames(Q) = c(1:7)
Q
##   1   2 3   4 5   6 7
## 1 0 0.4 0 0.0 0 0.0 0
## 2 0 0.0 0 0.4 0 0.0 0
## 3 0 0.0 0 0.0 0 0.4 0
## 4 0 0.0 0 0.0 0 0.0 0
## 5 0 0.6 0 0.0 0 0.0 0
## 6 0 0.0 0 0.6 0 0.0 0
## 7 0 0.0 0 0.0 0 0.6 0
I = diag(7)
rownames(I) = c(1:7) 
colnames(I) = c(1:7)
I
##   1 2 3 4 5 6 7
## 1 1 0 0 0 0 0 0
## 2 0 1 0 0 0 0 0
## 3 0 0 1 0 0 0 0
## 4 0 0 0 1 0 0 0
## 5 0 0 0 0 1 0 0
## 6 0 0 0 0 0 1 0
## 7 0 0 0 0 0 0 1
S = I - Q
S
##   1    2 3    4 5    6 7
## 1 1 -0.4 0  0.0 0  0.0 0
## 2 0  1.0 0 -0.4 0  0.0 0
## 3 0  0.0 1  0.0 0 -0.4 0
## 4 0  0.0 0  1.0 0  0.0 0
## 5 0 -0.6 0  0.0 1  0.0 0
## 6 0  0.0 0 -0.6 0  1.0 0
## 7 0  0.0 0  0.0 0 -0.6 1
N = solve(S) # inverse of I - Q
N
##   1   2 3    4 5   6 7
## 1 1 0.4 0 0.16 0 0.0 0
## 2 0 1.0 0 0.40 0 0.0 0
## 3 0 0.0 1 0.24 0 0.4 0
## 4 0 0.0 0 1.00 0 0.0 0
## 5 0 0.6 0 0.24 1 0.0 0
## 6 0 0.0 0 0.60 0 1.0 0
## 7 0 0.0 0 0.36 0 0.6 1
R = matrix(c(0.6, 0.6, 0.6, 0.6, rep(0,6), 0.4, 0.4, 0.4, 0.4), ncol=2)

rownames(R) = c(1:7)  
colnames(R) = c(0,8)
R
##     0   8
## 1 0.6 0.0
## 2 0.6 0.0
## 3 0.6 0.0
## 4 0.6 0.4
## 5 0.0 0.4
## 6 0.0 0.4
## 7 0.0 0.4
B = N%*%R
B
##       0     8
## 1 0.936 0.064
## 2 0.840 0.160
## 3 0.744 0.256
## 4 0.600 0.400
## 5 0.504 0.496
## 6 0.360 0.640
## 7 0.216 0.784

The absorption matrix B shows that using the bold strategy the probality of winning P[1,8] (\(S_{18}\)) = 0.064
The probalibty of losing p[1,0] (\(S_{10}\)) = 0.936.




  1. Which strategy gives Smith the better chance of getting out of jail?
    Answer - Although the probality of winning the game is low whether the prisoner uses the bold (0.064) or timid (0.02030135) strategy,
    comparatively speaking, the prisoner has a
(0.064)/(0.02030135)
## [1] 3.1525

times chances of winning using the bold strategy. Therefore, the bold strategy is better strategy to use.


S.1 Game Simulation using Timid Strategy - note that probability of winning is approximately the same as what was computed using Markov Chain approach.

l = 0
w  = 0
reps = 1000000

for (j in 1:reps)
{
  m = 1
  while (m > 0 & m < 8)
  {
    x = runif(1, 0, 1)
  
    if (x <= 0.4)
       m = m + 1
    else
       m = m - 1
  
    if (m == 0)
    {
      l = l + 1
    }
    else
    {
      if (m == 8)
      {
         w = w + 1
      }
    }
  }
}

print(l)
## [1] 979914
print(w/reps)
## [1] 0.020086

S.2 Game Simulation using Bold Strategy - note that probability of winning is approximately the same as what was computed using Markov Chain approach.

l = 0
w  = 0
reps = 1000000

for (j in 1:reps)
{
  m = 1
  while (m > 0 & m < 8)
  {
    x = runif(1, 0, 1)
    
    if (x <= 0.4)
      m = m + min(c((8-m),m))
    else
      m = m - min(c((8-m),m))
    
    if (m == 0)
    {
      l = l + 1
    }
    else
    {
      if (m == 8)
      {
        w = w + 1
      }
    }
  }
}

print(l)
## [1] 935375
print(w/reps)
## [1] 0.064625