library(markovchain)
## Package:  markovchain
## Version:  0.9.5
## Date:     2023-09-24 09:20:02 UTC
## BugReport: https://github.com/spedygiorgio/markovchain/issues

PROBLEM: Smith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6.

We can set up the transitional matrix \(P\)

\[P = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0.6 & 0 & 0.4 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & 0.6 & 0 & 0.4 & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0.6 & 0 & 0.4 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0.6 & 0 & 0.4 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0.6 & 0 & 0.4 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 0.6 & 0 & 0.4 & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0.6 & 0 & 0.4\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\ \end{bmatrix} \]

statenames <- c("0","1","2","3","4","5","6","7","8")
P <- matrix(c(1, 0, 0, 0, 0, 0, 0, 0, 0,
              0.6, 0, 0.4, 0, 0, 0, 0, 0, 0,
              0, 0.6, 0, 0.4, 0, 0, 0, 0, 0,
              0, 0, 0.6, 0, 0.4, 0, 0, 0, 0,
              0, 0, 0, 0.6, 0, 0.4, 0, 0, 0,
              0, 0, 0, 0, 0.6, 0, 0.4, 0, 0,
              0, 0, 0, 0, 0, 0.6, 0, 0.4, 0,
              0, 0, 0, 0, 0, 0, 0.6, 0, 0.4,
              0, 0, 0, 0, 0, 0, 0, 0, 1), nrow = 9, ncol= 9, byrow = T,
              dimnames = list(statenames, statenames))

markovB <- new("markovchain", states = statenames, transitionMatrix = P, 
               name = "A markovchain object")

P
##     0   1   2   3   4   5   6   7   8
## 0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
## 1 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0
## 2 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 3 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 4 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 5 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 6 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## 7 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0

Find the probability that he wins 8 dollars before losing all of his money if

  1. he bets 1 dollar each time (timid strategy).
absorptionProbabilities(markovB)
##           0          8
## 1 0.9796987 0.02030135
## 2 0.9492466 0.05075337
## 3 0.9035686 0.09643140
## 4 0.8350515 0.16494845
## 5 0.7322760 0.26772403
## 6 0.5781126 0.42188739
## 7 0.3468676 0.65313243

We can see that by betting 1 dollar each time the probability that we reach 8 dollars is 0.0203 or 2.03%

(b) If he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).

If we start with 1 dollars - We can reach 2 dollars with a prob. 0.4 - We can go down to 0 with a prob of 0.6 - At 0 dollars we get absorped

If we win and now have 2 dollars we can bet as much as we can, so 2 dollars - We can reach 4 dollars with a prob. 0.4 - We can go down to 0 with a prob of 0.6

If we win and now have 4 dollars and bet 4 dollars - We can go to 8 dollars with a prob of 0.4 - We can go down to 0 dollars with a prob of 0.6

If we win and now have 8 dollars we can get out of jail!

We can create a new transition matrix with the above information

statenames <- c("0","1","2","4","8")
X <- matrix(c(1, 0, 0, 0, 0, 
              0.6, 0, 0.4, 0, 0,
              0.6, 0, 0, 0.4, 0,
              0.6, 0, 0, 0, 0.4,
              0, 0, 0, 0, 1), nrow = 5, ncol= 5, byrow = T,
              dimnames = list(statenames, statenames))

markovC <- new("markovchain", states = statenames, transitionMatrix = X, 
               name = "A markovchain object")
absorptionProbabilities(markovC)
##       0     8
## 1 0.936 0.064
## 2 0.840 0.160
## 4 0.600 0.400

If we start with 1 dollar and each time, bets as much as possible but not more than necessary to bring his fortune up to 8 dollars, the probability that he wins 8 dollars before losing all of his money is 0.064 or 6.4%

(c) Which strategy gives Smith the better chance of getting out of jail?

The bold startegy gives us a better chance of getting out jail if we start with 1 dollar.