: person(“Annice”, “Najafi”, email = “”)

Summer 2024

This is an R implementation of the gambling model from Chapter 1, section 2 (“A Gambling Model”) of the “Introduction to Stochastic Dynamic Programming” book by Sheldon Ross.

A gambler that has x fortune, makes n bets wins with probability p and loses with probability q = 1-p. Goal is to maximize the expectation of the log of his fortune. Question is what strategy achieves this?

First let’s start by loading relevant libraries

#Load libraries
library(ggplot2)

Define the maximum expected return function

#Define V_n
#' Recursive function returns the maximal expected return
#'
#' @param n number of bets
#' @param x present fortune
#' @param p is the probability of winning, q is the probability of losing
#' hence p = 1-q
#' @param alpha is the percentage of fortune we are betting
#'
#' @return maximal expected return value
#' @export
V <- function(n, x, p, alpha) {
  
  #boundary condition
  if (n == 0) {
    return(log(x))
  }
  
  q <- 1 - p
  
  
  V_previous <- function(x) {
    V(n - 1, x, p, alpha)
  }
  
  expected_reward <- p * V_previous(x * (1 + alpha)) + q * V_previous(x * (1 - alpha))
  
  return(expected_reward)
}

Case I) Let’s try for when p is actually more than q

initial_fortune <- 1000  # Initial fortune
p <- 0.6  # Probability of winning
q <- 1- p #probability of losing
n <- 10  # Number of bets
alpha <- 0.3 #What fraction of his fortune is he betting

# Calculate V_n(x)
result <- V(n, initial_fortune, p, alpha)

res_vector <- c() #store results

#Go over all values
for(alpha in seq(0, p, 0.01)){
  
  result <- V(n, initial_fortune, p, alpha)
  
  res_vector <- c(res_vector, result)
  
}

reward.df <- data.frame(alpha = seq(0, p, 0.01), reward = res_vector)

Plot the first case

ggplot(reward.df, aes(x= alpha, y=reward))+
  geom_line(size=2, color = "#538392")+
  geom_point(x= seq(0, p, 0.01)[which.max(res_vector)], y = max(res_vector), color="darkred", shape=8, stroke=3)+
  labs(x="Alpha", y="Reward")+
  ggtitle("Maximal Expected Return - p > 1/2")+
  geom_vline(xintercept = (p - q), linetype="dashed")+
  theme(
    panel.grid.major = element_blank(),
    panel.grid.minor = element_blank(),
    panel.background = element_blank(),
    axis.line = element_line(colour = "black"),
    plot.title = element_text(hjust = 0.5, size=20),
    axis.text = element_text(size = 15),
    text = element_text(size=18))+labs(tag="A") ->plt.A
## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
plt.A

Case II) Let’s try p<1/2

p <- 0.3  # Probability of winning
q <- 1- p #probability of losing

# Calculate V_n(x)
result <- V(n, initial_fortune, p, alpha)

res_vector <- c() #store results

#Go over all values
for(alpha in seq(0, p, 0.01)){
  result <- V(n, initial_fortune, p, alpha)
  res_vector <- c(res_vector, result)
}

reward.df <- data.frame(alpha = seq(0, p, 0.01), reward = res_vector)

Plot the second case

ggplot(reward.df, aes(x= alpha, y=reward))+
  geom_line(size=2, color = "#538392")+
  geom_point(x= seq(0, p, 0.01)[which.max(res_vector)], y = max(res_vector), color="darkred", shape=8, stroke=3)+
  labs(x="Alpha", y="Reward")+
  ggtitle("Maximal Expected Return - p < 1/2")+
  geom_vline(xintercept = 0, linetype="dashed")+
  theme(
    panel.grid.major = element_blank(),
    panel.grid.minor = element_blank(),
    panel.background = element_blank(),
    axis.line = element_line(colour = "black"),
    plot.title = element_text(hjust = 0.5, size=20),
    axis.text = element_text(size = 15),
    text = element_text(size=18))+labs(tag="B")->plt.B

plt.B

Plot them side by side

#uncomment to try
#plt.A+plt.B

We see that we can come to the same conclusions as the book if we write the program in R!

Annice
Summer 2024

References:

Ross, S. M. (2014). Introduction to stochastic dynamic programming. Academic press.