DATA605 HW10
Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets a dollars, he wins a dollars with probability .4 and loses a dollars with probability .6. Find the probability that he wins 8 dollars before losing all of his money if:
Represent Smith’s ‘get out of jail’ money situation with a Markov Chain using the markovchain library:
Define a transition matrix between states in the system. In other words, how Smith’s money situation changes as he plays.
jailbreak<-matrix(rep(0, 9^2),byrow=TRUE,nrow = 9)
rownames(jailbreak)<-colnames(jailbreak)<-c("$0","$1","$2","$3","$4","$5","$6","$7","$8")
#2 absorption states of the system
jailbreak[1,1] = 1
jailbreak[9,9] = 1
#capture most of the behavior with this loop
for (link in seq( 1,dim(jailbreak)[1]-1) ) {
jailbreak[ link,link + 1 ] = 0.6 #gain 1
jailbreak[ link,link -1 ] = 0.4 #lose 1
}
jailbreak[2,1] = 0
jailbreak[8,9] = 0
jailbreak[9,8] = 0.4
print( jailbreak)## $0 $1 $2 $3 $4 $5 $6 $7 $8
## $0 1 0.6 0.0 0.0 0.0 0.0 0.0 0.0 0
## $1 0 0.0 0.6 0.0 0.0 0.0 0.0 0.0 0
## $2 0 0.4 0.0 0.6 0.0 0.0 0.0 0.0 0
## $3 0 0.0 0.4 0.0 0.6 0.0 0.0 0.0 0
## $4 0 0.0 0.0 0.4 0.0 0.6 0.0 0.0 0
## $5 0 0.0 0.0 0.0 0.4 0.0 0.6 0.0 0
## $6 0 0.0 0.0 0.0 0.0 0.4 0.0 0.6 0
## $7 0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0
## $8 0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 1
(a) he bets 1 dollar each time (timid strategy).
We need to find the probability that the ‘chain’ will propagate to the state ‘$8’ before reaching state ‘$0’, or: \[P( S_{8} \lt S_0 ) = P( S_8 < S_0|X_0 = i)\]
This can be found using the absorptionProbabilities method from the markovchain library. This method calculates the probability of reaching an absorption point in the markov chain from every possible state of the system. Absorption states are states that, once entered, cannot be left (e.g. reaching state ‘$8’ gets Smith out of jail)
## $1 $2 $3 $4 $5 $6 $7
## $0 0.97969865 0.94924663 0.9035686 0.8350515 0.732276 0.5781126 0.3468676
## $8 0.02030135 0.05075337 0.0964314 0.1649485 0.267724 0.4218874 0.6531324The probability that the chain propagates to state ‘$8’ before ‘$0’ when starting from the state ‘$1’ is given by 0.0203013
(b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).
This system behaves differently,so we need a new Markov Chain transition metric to describe propagation:
jailbreak<-matrix(rep(0, 9^2),byrow=TRUE,nrow = 9)
rownames(jailbreak)<-colnames(jailbreak)<-c("$0","$1","$2","$3","$4","$5","$6","$7","$8")
#the absorption states
jailbreak[1,1] = 1
jailbreak[9,9] = 1
jailbreak<-matrix(rep(0, 9^2),byrow=TRUE,nrow = 9)
rownames(jailbreak)<-colnames(jailbreak)<-c("$0","$1","$2","$3","$4","$5","$6","$7","$8")
#2 absorption states of the system
jailbreak[1,1] <- 1
jailbreak[9,9] <- 1
for (idx in seq( 2,dim(jailbreak)[1]-1) ) {
link <- idx-1
#smith will follow bold strategy and bet what he needs (but not more) to get to '$8' state
jailbreak[ idx + min( link,( 8 - link )),idx ] = 0.6
jailbreak[ idx - min( link,( 8 - link )),idx ] = 0.4
}
print( jailbreak)## $0 $1 $2 $3 $4 $5 $6 $7 $8
## $0 1 0.4 0.4 0.4 0.4 0.0 0.0 0.0 0
## $1 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0
## $2 0 0.6 0.0 0.0 0.0 0.4 0.0 0.0 0
## $3 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0
## $4 0 0.0 0.6 0.0 0.0 0.0 0.4 0.0 0
## $5 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0
## $6 0 0.0 0.0 0.6 0.0 0.0 0.0 0.4 0
## $7 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0
## $8 0 0.0 0.0 0.0 0.6 0.6 0.6 0.6 1jailMc<-as(jailbreak, "markovchain")
plot(jailMc, curved=TRUE, vertex.size=15, vertex.label.cex=0.8, vertex.label.dist=0, edge.arrow.size=.5 )
## $1 $2 $3 $4 $5 $6 $7
## $0 0.784 0.64 0.496 0.4 0.256 0.16 0.064
## $8 0.216 0.36 0.504 0.6 0.744 0.84 0.936The probability that the chain propagates to state ‘$8’ before ‘$0’ when starting from the state ‘$1’ using the Bold strategy is given by 0.216
(c) Which strategy gives Smith the better chance of getting out of jail?
0.216 \(\gt\) 0.0203013 evaluates TRUE. Therefore, the Bold strategy works out better for Smith.