library("markovchain")## Package: markovchain
## Version: 0.9.5
## Date: 2023-09-24 09:20:02 UTC
## BugReport: https://github.com/spedygiorgio/markovchain/issuesSmith is in jail and has 1 dollar; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability .4 and loses A dollars with probability .6. Find the probability that he wins 8 dollars before losing all of his money if:
(a) he bets 1 dollar each time (timid strategy).
The best way to approach this problem is to manually set up the matrix of probabilities for each step, and then rely on the computational power of the markovchain package to calculate absorbption probabilities.
In the timid strategy, the stochastic matrix we need should be 8x8, where the top left and bottom right corners are valued at 1 (meaning 0 only moves to 0 and 8 only moves to 8). At every state beyond 0, the only possibilities are to the immediate left and right of the current state, with probabilities of 0.6 and 0.4 respectively.
#build matrix of probabilities for state movement
P_timid = matrix(c(1,0,0,0,0,0,0,0,0,
0.6,0,0.4,0,0,0,0,0,0,
0,0.6,0,0.4,0,0,0,0,0,
0,0,0.6,0,0.4,0,0,0,0,
0,0,0,0.6,0,0.4,0,0,0,
0,0,0,0,0.6,0,0.4,0,0,
0,0,0,0,0,0.6,0,0.4,0,
0,0,0,0,0,0,0.6,0,0.4,
0,0,0,0,0,0,0,0,1),
nrow=9, byrow=TRUE)
#create state names
states_names <- c("0","1","2","3","4","5","6","7","8")
#create Markov matrix object
markov_timid <- new("markovchain", states = states_names, transitionMatrix = P_timid,
name="prison game timid")
markov_timid ## prison game timid
## A 9 - dimensional discrete Markov Chain defined by the following states:
## 0, 1, 2, 3, 4, 5, 6, 7, 8
## The transition matrix (by rows) is defined as follows:
## 0 1 2 3 4 5 6 7 8
## 0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
## 1 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0
## 2 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0 0.0
## 3 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0 0.0
## 4 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0 0.0
## 5 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0 0.0
## 6 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4 0.0
## 7 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.4
## 8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0absorptionProbabilities(markov_timid)## 0 8
## 1 0.9796987 0.02030135
## 2 0.9492466 0.05075337
## 3 0.9035686 0.09643140
## 4 0.8350515 0.16494845
## 5 0.7322760 0.26772403
## 6 0.5781126 0.42188739
## 7 0.3468676 0.65313243This table describes the probability of reaching either absorption state (in this case, only 0 for losing all his money, or 8 for winning the game and getting out of jail) from any given starting point. Since Smith starts with 1 dollar, his chances of winning the game rather than losing all his money with the timid strategy are a hair over \(2\%\).
(b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy).
I can construct P_bold using similar logic to the above, but setting the probabilities based on this constraint.
P_bold = matrix(c(1,0,0,0,0,0,0,0,0,
0.6,0,0.4,0,0,0,0,0,0,
0.6,0,0,0,0.4,0,0,0,0,
0.6,0,0,0,0,0,0.4,0,0,
0.6,0,0,0,0,0,0,0,0.4,
0,0,0.6,0,0,0,0,0,0.4,
0,0,0,0,0.6,0,0,0,0.4,
0,0,0,0,0,0,0.6,0,0.4,
0,0,0,0,0,0,0,0,1),
nrow=9, byrow=TRUE)
markov_bold <- new("markovchain", states = states_names, transitionMatrix = P_bold,
name="prison game bold")
markov_bold## prison game bold
## A 9 - dimensional discrete Markov Chain defined by the following states:
## 0, 1, 2, 3, 4, 5, 6, 7, 8
## The transition matrix (by rows) is defined as follows:
## 0 1 2 3 4 5 6 7 8
## 0 1.0 0 0.0 0 0.0 0 0.0 0 0.0
## 1 0.6 0 0.4 0 0.0 0 0.0 0 0.0
## 2 0.6 0 0.0 0 0.4 0 0.0 0 0.0
## 3 0.6 0 0.0 0 0.0 0 0.4 0 0.0
## 4 0.6 0 0.0 0 0.0 0 0.0 0 0.4
## 5 0.0 0 0.6 0 0.0 0 0.0 0 0.4
## 6 0.0 0 0.0 0 0.6 0 0.0 0 0.4
## 7 0.0 0 0.0 0 0.0 0 0.6 0 0.4
## 8 0.0 0 0.0 0 0.0 0 0.0 0 1.0absorptionProbabilities(markov_bold)## 0 8
## 1 0.936 0.064
## 2 0.840 0.160
## 3 0.744 0.256
## 4 0.600 0.400
## 5 0.504 0.496
## 6 0.360 0.640
## 7 0.216 0.784Smith has a \(6.4\%\) chance of winning the game and getting out of jail using the bold strategy.
(c) Which strategy gives Smith the better chance of getting out of jail?
While Smith’s odds of getting out of jail through this game are not great any way you slice it, using the bold strategy, his odds are over 3 times better. Therefore, he should use the bold strategy (or try to convince the guard to play a different game.)