Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6. Find the probability that he wins 8 dollars before losing all of his money if

  1. he bets $1 each time (timid strategy)

The Markov chain (X_n, n = 0,1,…) represents the evolution of Smith’s money. Let φ(i) represent the probaility that the chain reaches state 8 before reaching state 0 starting from state i.

\[φ(i)=Pi(S_8 <S_0)=P(S_8 <S_0|X_0 =i).\] Calculated through simulation we obtain the following results.

p = 0.4
q = 0.6
r = q/p

for (i in seq(1, 7 , 1)){
  print ((1-r^i)/(1-r^8))
}
## [1] 0.02030135
## [1] 0.05075337
## [1] 0.0964314
## [1] 0.1649485
## [1] 0.267724
## [1] 0.4218874
## [1] 0.6531324

This means that the probability that the chain reaches state 8 before reaching state 0 starting from state 1 is the first component of this vector and equal to .0203. The probability increases as Smith wins.

  1. he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).

The markov chain has changed now since we are only concerned with the outputs of having 0, 1, 2, 4, and 8 dollars. The stakes have increased and any bet results in having $0. And win doubles earnings

earnings <- matrix(c(1, .6, .6, .6, 0, 0, 0, 0, 0, 0, 0, .4, 0, 0, 0, 0, 0, .4, 0, 0, 0, 0, 0, .4, 1),nrow = 5, byrow=FALSE)
earnings
##      [,1] [,2] [,3] [,4] [,5]
## [1,]  1.0    0  0.0  0.0  0.0
## [2,]  0.6    0  0.4  0.0  0.0
## [3,]  0.6    0  0.0  0.4  0.0
## [4,]  0.6    0  0.0  0.0  0.4
## [5,]  0.0    0  0.0  0.0  1.0

We solve using the following initial state

i = matrix(c(0,1,0,0,0),nrow=1)
i
##      [,1] [,2] [,3] [,4] [,5]
## [1,]    0    1    0    0    0
p1 = i %*% earnings
p2 = p1 %*% earnings
p3 = p2 %*% earnings
p3
##       [,1] [,2] [,3] [,4]  [,5]
## [1,] 0.936    0    0    0 0.064

Our resulting probability is .064 and this is the probability that Smith gets the $8.

  1. The probability is higher by taking a bold strategy approach.