Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability 0.4 and loses A dollars with probability 0.6.Find the probability that he wins 8 dollars before losing all of his money if (a) he bets 1 dollar each time (timid strategy). (b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy). (c) Which strategy gives Smith the better chance of getting out of jail?
Answer: Markov studied a new type of chance process that the outcome of given experiment can affect the outcome of the next experiment in 1907. Each outcome of a step is independent but the process is dependent. Markov Chains model is used for probability predition and the probability of transition between two states will be fixed after N steps if the probability matrix is convergen when N approche to a large number but a finit number.
${ lim }{ n->}P({ X }{ n }=8|_{ 0 }=\(1)\quad =\quad { r }_{ j }\)
\(\sum _{ n }^{ }{ { r }_{ n } } =\quad 1\)
Finite states Markov Chains model assumption gieven curent state, the past does not matter. The first step to apply this model is to identify the possible states, the possible transitions and the transition probabilities.
Simth has 1 dollar, which is the start state, and he will be free if he has 8 dollar, which is the last state. The inital state vector is
u <- c(0,1,0,0,0,0,0,0,0)
u## [1] 0 1 0 0 0 0 0 0 0The transition matrix is
P = matrix(
c(1, 0,0,0,0,0,0,0,0,
0.6,0,0.4,0,0,0,0,0,0,
0,0.6,0,0.4,0,0,0,0,0,
0,0,0.6, 0,0.4,0,0,0,0,
0,0,0,0.6,0,0.4,0,0,0,
0,0,0,0,0.6,0,0.4,0,0,
0,0,0,0,0,0.6,0,0.4,0,
0,0,0,0,0,0,0.6,0,0.4,
0,0,0,0,0,0,0,0,1
), # the data elements
nrow=9, # number of rows
byrow = TRUE) # fill matrix by rows
dimnames(P) = list(
c("$0","$1", "$2","$3","$4","$5","$6","$7","$8"), # row names
c("$0","$1", "$2","$3","$4","$5","$6","$7","$8")) # column names
P## $0 $1 $2 $3 $4 $5 $6 $7 $8
## $0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
## $1 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0
## $2 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## $3 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## $4 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## $5 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## $6 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## $7 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## $8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0#repeat the bet to 150 times, get the probability
Pn = diag(9)
for (i in 1:150) {
Pn <- P %*% Pn }
round(u%*%Pn,5)## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9]
## [1,] 0.9797 0 0 0 0 0 0 0 0.0203#check the probability in next bet (151th time), the probability is fixed
Pn2 = diag(9)
for (i in 1:151) {
Pn2 <- P %*% Pn2 }
round(u%*%Pn2,5)## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9]
## [1,] 0.9797 0 0 0 0 0 0 0 0.0203In this strategy, Smith puts all his money in each bet. If he wins 1 dollor at the first bet then he will put $2 (include $1 initial money he has) in the next run. So he will has money in these states $0,$1, $2,$4,$8. Answer:
u <-c(0,1,0,0,0) #The inital state vector
P = matrix(
c(1,0,0,0,0,
0.6,0,0.4,0,0,
0,0.6,0,0.4,0,
0,0,0.6,0,0.4,
0,0,0,0,1
), # the data elements
nrow=5, # number of rows
byrow = TRUE) # fill matrix by rows
dimnames(P) = list(
c("$0","$1", "$2","$4","$8"), # row names
c("$0","$1", "$2","$4","$8")) # column names
P## $0 $1 $2 $4 $8
## $0 1.0 0.0 0.0 0.0 0.0
## $1 0.6 0.0 0.4 0.0 0.0
## $2 0.0 0.6 0.0 0.4 0.0
## $4 0.0 0.0 0.6 0.0 0.4
## $8 0.0 0.0 0.0 0.0 1.0#repeat the bet to 150 times, get the probability
Pn = diag(5)
for (i in 1:150) { Pn <- P %*% Pn }
round(u%*%Pn,5) #round 5 digites## [,1] [,2] [,3] [,4] [,5]
## [1,] 0.87692 0 0 0 0.12308#check the probability in next bet (151th time), the probability is fixed
Pn2 = diag(5)
for (i in 1:151) { Pn2 <- P %*% Pn2 }
round(u%*%Pn2,5) #round 5 digites## [,1] [,2] [,3] [,4] [,5]
## [1,] 0.87692 0 0 0 0.12308Answer: The bold strategy is better by the above resoults. However, base on the property of Markov Chain and the probability of the transition, it can be tell by the following.
Since the game has been designed that Simth has 0.6 probability of loss and 0.4 probability of win in each bet,it’s Bernouli trail in each ste. Let p present the probability of win 0.4 and q is the probability of faulure 0.6.
The game is a special case in Markov Chain.All probability of to gain is 0.4 and 0.6 for loss for each step. Let rho $= $ and n be the min number steps to in steady-state.
\({ r }_{ 1 }={ r }_{ 0 }*p\)
\({ r }_{ 21 }\quad ={ \quad r }_{ 1 }*p\quad ={ \quad r }_{ 0 }*{ p }^{ 2 }\quad\)
\(\sum _{ i=0 }^{ n }{ { \quad r }_{ 0 }*{ p }^{ i } } \quad =1\)
\({ r }_{ 0 }\quad =\frac { 1 }{ \sum _{ i=0 }^{ n }{ { \rho }^{ i } } } =\frac { 1 }{ 1-\rho } \quad where\quad n->\infty\) #Geometry distribution but start from 0 and the game is in a stable side
(0.4/0.6)/(1-0.4/0.6) #E[Xn]:The best bet to win the game## [1] 2