1 Problem Statement in brief

Despite policy efforts to restrict menthol cigarette sales, menthol and menthol-like tobacco products remain widely visible in retail environments. Manual surveillance of these products on store shelves is time-consuming, inconsistent, and not scalable.

This project aims to automate the detection and classification of cigarette packs — especially menthol-labeled or menthol-implied products — using multimodal AI models (e.g., GPT-4o via Portkey) applied to thousands of retail images across New York City.

Primary goal Quantify the number and proportion of menthol vs. non-menthol cigarette packs visible on retail shelves using store photos and large language models.

2 The data, Store 3802 BROADWAY, Queens, NY 11103

The data used in this analysis consists of store-shelf photographs taken at a tobacco retail outlet in New York City: Store Location: 3802 BROADWAY, Queens, NY 11103

Each image captures a full or partial view of cigarette packs at the point-of-sale. These images are processed using multimodal LLMs to extract brand and menthol classification data automatically.

# Path to local image folder
img_dir <- "3802_BROADWAY_Queens_NY_11103"

# List all .jpg, .jpeg, .png files
files <- list.files(img_dir, pattern = "\\.jpg$|\\.jpeg$|\\.png$", full.names = TRUE)

# Total number of images
n_images <- length(files)

# Show summary and first few filenames
tibble(`Total Images` = n_images,
       `Sample Image Files` = list(head(files, 12))) %>%
  knitr::kable(caption = "Sample of Images in the Folder") %>%
  kable_styling()
Sample of Images in the Folder
Total Images Sample Image Files
42 3802_BROADWAY_Queens_NY_11103/IMG_2343.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2344.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2345.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2346.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2347.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2348.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2349.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2350.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2351.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2352.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2353.jpg, 3802_BROADWAY_Queens_NY_11103/IMG_2354.jpg

2.1 Sample Image Preview

Here are two sample images from the location: 3802 BROADWAY, Queens, NY 11103

2.1.1 📷 Image 1: Retail Shelf View – IMG_2360.jpg

Figure 1. Retail Shelf View – Menthol and Non-Menthol Packs Visible – IMG_2360.jpg

2.1.2 📷 Image 2: Store front – IMG_2346.jpg

Figure 2. Store front – IMG_2346.jpg

3 API-Based Image Classification: Identifying Cigarette Packs

To automate cigarette surveillance, we use an AI model via the Portkey API, which wraps multimodal capabilities from GPT-4o. This model can analyze images and respond with structured interpretations, making it possible to extract brand names, text, colors, and menthol classification from retail shelf photos.

Each image is encoded in base64 format and sent to the model with a structured prompt guiding its analysis.

3.1 Example: Classifying Cigarette Packs in One Image

We begin with a test image from the location 3802 BROADWAY, Queens, NY (file: IMG_2360.jpg), showing a retail shelf with dozens of cigarette packs.

The AI model is prompted to:

Count and list all visible cigarette packs

Identify brands and packaging text

Determine if each pack is Menthol, Non-Menthol, or Unclear

Extract the primary color of each pack

Return a structured table and summary statistics

# --- API Setup ---
#api_key <- "Replace with your actual API key"  
#portkey_config <- '{"virtual_key": "replace_with_your_keys"}'  # Proper JSON format
#url <- "https://ai-gateway.apps.cloud.rt.nyu.edu/v1/chat/completions" 

# --- Image Preparation ---
img_path <- "3802_BROADWAY_Queens_NY_11103/IMG_2360.jpg"
stopifnot(file.exists(img_path))  # Ensure image file exists

# Encode to base64 and convert to data URL
img_base64 <- base64encode(img_path)
data_url <- paste0("data:image/jpeg;base64,", img_base64)

# --- Prompt to Send ---
prompt_text <- paste(
  "You are analyzing a high-resolution store shelf photo from a NYC tobacco retailer.",
  "",
  "Your goal is to identify and classify all visible cigarette packs, using your knowledge of real-world branding and packaging.",
  "",
  "For each distinct pack (or pack group), create one row in a markdown table with:",
  "",
  "1. **Index** – running number",
  "2. **Visible Branding Text** – readable words/numbers **on the pack itself**, such as brand names, flavor names, or unique words like 'pleasure', 'crush', 'ice', etc.",
  "   - ❗️Do **NOT** include store price tags or shelf labels (e.g., '11.80', '1280') — these are not part of the cigarette packaging.",
  "3. **Key Color** – the most visually identifying color used by that pack (e.g., red for Marlboro Red, green for Newport Menthol)",
  "4. **Classification** – choose one:",
  "   - **Menthol** – if labeled as menthol or implied via green/blue, 'Ice', 'Cool', 'Crush', etc.",
  "   - **Non-Menthol** – if standard flavor (e.g., Marlboro Red) with no menthol cues",
  "   - **Unclear** – if pack is visible but classification is uncertain",
  "",
  "✅ Include all visible packs, even if partially covered or stacked",
  "✅ Use real-world knowledge of cigarette branding to classify packs accurately",
  "❌ Do not include shelf labels, digital price tags, or stickers as part of the cigarette pack text",
  "",
  "Return a markdown table like this:",
  "| Index | Packaging Text | Key Color | Classification |",
  "",
  "Then summarize:",
  "- 🔢 **Total packs**",
  "- ✅ **Menthol count**",
  "- ❌ **Non-Menthol count**",
  "- ❓ **Unclear count**",
  "- 📊 **Menthol %** (rounded)",
  "",
  "Your goal is **accurate, complete pack classification** using both visual reasoning and real-world tobacco product knowledge."
)


# --- API Request Body ---
body <- list(
  model = "gpt-4o",
  max_tokens = 1000,
  temperature = 0,
  messages = list(
    list(role = "user", content = list(
      list(type = "text", text = prompt_text),
      list(type = "image_url", image_url = list(url = data_url))
    ))
  )
)

# --- Make API Request ---
response <- POST(
  url,
  add_headers(
    Authorization = paste("Bearer", api_key),
    `x-portkey-config` = portkey_config
  ),
  content_type_json(),
  body = toJSON(body, auto_unbox = TRUE)
)

# --- Parse and Handle Response ---
parsed <- content(response, as = "parsed")

if (!is.null(parsed$error)) {
  stop(paste("API Error:", parsed$error$message))
}

if (!is.null(parsed$choices) && length(parsed$choices) > 0) {
  response_text <- parsed$choices[[1]]$message$content
  cat(response_text)
  
  # Save to text file
  dir.create("data/results", recursive = TRUE, showWarnings = FALSE)
  write_file(response_text, "data/results/IMG_2360_response.txt")
} else {
  print(parsed)
  stop("No usable response returned from the API.")
}
| Index | Visible Branding Text | Key Color | Classification |
|-------|-----------------------|-----------|-----------------|
| 1     | Newport pleasure      | Green     | Menthol         |
| 2     | Newport               | Green     | Menthol         |
| 3     | Newport               | Green     | Menthol         |
| 4     | Newport               | Green     | Menthol         |
| 5     | Marlboro              | Red       | Non-Menthol     |
| 6     | Marlboro              | Red       | Non-Menthol     |
| 7     | Marlboro              | Red       | Non-Menthol     |
| 8     | Marlboro              | Red       | Non-Menthol     |
| 9     | Camel                 | Tan       | Non-Menthol     |
| 10    | Lucky Strike          | Red/White | Non-Menthol     |
| 11    | Pall Mall             | Blue      | Non-Menthol     |
| 12    | American Spirit       | Yellow    | Non-Menthol     |

- 🔢 **Total packs**: 12
- ✅ **Menthol count**: 4
- ❌ **Non-Menthol count**: 8
- ❓ **Unclear count**: 0
- 📊 **Menthol %**: 33%