Problem 2

Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars.A guard agrees to make a series of bets with him. If Smith bets A dollars,he wins A dollars with probability 0.4 and loses A dollars with probability 0.6.

Find the probability that he wins 8 dollars before losing all of his money if

  1. he bets 1 dollar each time (timid strategy).

Using Markov Chain, Let φ(i) represent the probaility that the Markov chain reaches state 8 before reaching state 0 starting from state i. In other words, if \(S_j\) is the first n >= 0 such that \(X_n = j\),

\(φ(i) = P_i(S_8 < S_0) = P(S_8 < S_0|X_0 = i)\).

Using first-step analysis (viz. the Markov property at time n = 1), we have

\(φ(i) = 0.4φ(i + 1) + 0.6φ(i − 1)\),

i = 1, 2, 3, 4, 5, 6, 7

\(φ(0) = 0 and φ(1) = 1\)

The system of Linear Equation can be solved using:

\(φ = (φ(1), φ(2), φ(3), φ(4), φ(5), φ(6), φ(7))\)

Which gives:

(0.0203, 0.0508, 0.0964, 0.1649, 0.2677, 0.4219, 0.6531, 1)

Starting from state 1, the probability that the chain reaches state 8 before reaching state 0 is the first component of this vector and equal to 0.0203. The winings willincrease the probability

  1. he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).

At this strategy, the chain changes and the equation becomes:

\(φ(0) = 0.6φ(3)\) \(φ(3) = 0.4φ(6)\) \(φ(6) = 0.4φ(8) + 0.6φ(4)\) \(φ(4) = 0.4φ(8)\) \(φ(0) = 0\) \(φ(8) = 1\)

Solving gives:

\(φ = (0, 0, 0.256, 0.4, 0, 0.64, 0)\).

The probability that Smith gets the $8 is 0.064.

  1. Which strategy gives Smith the better chance of getting out of jail?

The bold strategy approach gives Smith a better chance of geting out of jail.