AI-POWERED SOCIAL MEDIA & SALES ANALYSIS FOR ODI

Proposal Presentation

Ceren Unal, Jeffrey Hsu, Emily Kuo, Eunice Won, Min Gong

IBM 6400, Cal Poly Pomona

2026-02-14

ABSTRACT

Purpose

This study’s purpose is to design and test an AI-powered app that improves how social media content is evaluated. Using ODI, a premium biking accessories company, as the client, the project applies the Structured Tactic Assessment and Measurement Prompting (STAMP) framework to classify marketing tactics such as classic timeless luxury and personalized service. The ultimate goal is to create a clear way to assess content performance while keeping human oversight to achieve a positive brand image.

Keywords

Generative AI, Social Media Analysis, AI Evaluation, Human Judgement Vs. AI Strategy

INTRODUCTION (CHAPTER 1)

Business Problem

With how advanced digital technology has become over the last decade, brands can now easily generate all kinds of content for their social media platforms. Entertaining that with the ever-increasing abilities of artificial intelligence, content managers can push out captions, visuals, calendars, and even videos with a quick click of a button. However, many of these brands do not have the reliable capability of evaluating how well their digital content performs in shaping consumer perception or how much important engagement is generated. Sometimes, depending only on human evaluation can lead to inaccurate interpretations, as it can be slow, inconsistent, and biased. For a brand like ODI, who is the client of this project, the real objective is knowing whether their content truly matches with the company and how successfully it connects with its audiences rather than content creation itself.
Blanchard et al. (2025) make an emphasis that while AI can speed up creative work and handle repetitive tasks efficiently, it still requires human direction to stay responsible and authentic. For ODI, this balance between automation and human review is essential. The company wants to expand its social media presence but has very limited time and resources to produce consistent, high-quality content for multiple audiences on a daily fashion. By using the app’s AI tools, ODI can generate caption ideas and image suggestions quickly while keeping its signature tone and credibility through human double-checking. This partnership between AI and human creativity allows ODI to save valuable tim and devote more attention to building stronger engagement with both new and loyal customers.
As AI continues to dominate more and more of the social media marketing landscape, marketing managers need a reliable way to assess post quality that is not just fully dependable on subjective opinion. Without a structured evaluation method, marketers may generate content that may evidently confuse their target audience or risk losing brand trust. Therefore, the business problem is to develop and refine a data-driven framework using an app to evaluate social media content quality using various Generative AI models (ChatGPT, Gemini, and Claude) instead of strictly on human judgement alone. Specifically, it will compare how consistently these AI platforms can key in on effective social media content styles such as “personalized service” or “classic timeless luxury”.

Analytics Objectives

The primary analytical objective of this study is to develop an AI-driven content generation workflow that achieves accurate and consistent text classification through the coordinated use of multiple large language models. Each analytical objective addresses a specific component of this workflow, including prompt engineering, measurement criteria, and application development. The workflow is grounded in the Structured Taxonomy AI Measurement Protocol (STAMP), which enables large language models to function as evaluators that audit one another against a shared construct definition, thereby producing transparent, reliable, and replicable classifications of digital marketing data. After defining the measurement criteria, three models—ChatGPT, Gemini, and Claude—iteratively refine the structured prompt until they achieve Krippendorff’s alpha greater than 0.67 and an F1 score greater than 0.8, demonstrating human-level inter-rater reliability and classification accuracy. Once validated, the resulting classifications are deployed to automate content generation for social media and customer support via a Streamlit application. The overarching objective is therefore divided into several subordinate objectives, each contributing to the development of a rigorous and operationally effective AI content workflow.

AOs:

AO1. Emily: Chatbot App Design and Development of Core Functions through Prompt Engineering

AO2. Ceren: LLM-Based Evaluation Framework For Chatbot App

AO3. Eunice: Social Media content generator App - Design and Development of Core Functions through Prompt Engineering

AO4. Jeffrey: LLM-Based Evaluation Framework for Social Media content generator App

AO5. Min: App Automated Optimization – Using LLM Feedback to Improve Prompts

Significance of the Topic (Emily)

The significance of this project lies in combining human marketing insight with AI driven tools to improve both the analysis and creation of social media communication. Traditional text analysis often fails to capture nuanced brand tactics or emotions in captions. From using the STAMP framework, this project helps marketing research by providing a reliable way to measure digital communication tactics across thousands of posts.
Our project will use generative AI classifiers to look at social media captions and interpret what the meaning and intent really are. It will provide transparent reasoning for every classification. This lets marketing teams see exactly which tactics are being used, like pushing personalized service or focusing on a classic luxury style and then track how those specific tactics drive likes, comments, and customer interaction. By using smart prompt engineering and checking our work with dual model validation, we’re moving marketing analytics toward a more human aligned, evidence based approach. It helps both academic researchers and real-world marketers by taking confusing, unstructured text and giving them measurable, strategic insights.
To extend this framework beyond analysis, this project also focuses on building practical AI applications, such as content creation tools, brand voice generators, and customer interaction chatbots. These tools make the analytic insights actionable. In these tools, businesses can analyze which tactics perform best, then use our content creation bot to generate new captions that follow the same high performing style. Our customer service chatbot can also respond to users with consistent tone, politeness, and empathy, helping companies scale personalized interactions.

BACKGROUND AND ANALYTICS OBJECTIVES (CHAPTER 2)

Company mission/vision

For over 40 years, ODI has been dedicated to providing the highest

product quality in its core products, while maintaining the highest level of comfort and longevity possible. Committed to its name,“Observe. Design. Innovate”, ODI aims to provide the best products through dedicated research, innovation, and continuous improvement.

Marketing team and expertise

ODI’s core marketing team is led by Colby Young, Vice President of Marketing and Customer Experience (MCX). He has been working with ODI since the 1990s, pioneering the brand’s social media strategy and athlete collaborations. Young is experienced in technical implementation, including database query programming, and is an expert in ODI’s brand strategy and positioning.

Comparison Among Competitors Summary Table

Category	ODI	ESI Grips	Lizard Skins
Founded/HQ	1980 in Riverside, CA	1999 in Chino Valley, AZ	1992 in American Fork, UT
Size (Employees)	~ 50-100	~ 10	~ 30-50
Positioning	High-level performance, durable, pro-graded grips	Ergonomic, handmade, USA-made	Premium, customizable, multi-sport performance
Target Market	BMX, MTB, motocross riders	Mountain, road cyclists, endurance	Cyclists, BMX, baseball, hockey
Website/Platform	Custom e-ecommerce site	Shopify	Shopify
SEO Focus	“Lock-on grips,” “bike grips”	“Made in USA,”, “silicone grips”	“Bike grips,” “handlebar tape,” “DSP grips”
Social Media Strategy	Frequent pro-rider features and product posts	Organic UGC, family-run storytelling	Giveaways, influencer collaborations, product highlights
Advertising	Active paid search & sponsorships	Minimal	Google ads and influencer content

SWOT Analysis

SWOT Analysis

Strengths

• Product Quality: ODI has a reputation for producing high-quality grips emphasizing comfort and durability.

• Domestic Manufacturing: Made in the USA, allowing strict quality control.

• Athlete Partnerships: Collaborations with professional athletes strengthen credibility.

Opportunities

• Expanding Digital Marketing Efforts: ODI can enhance digital presence with inbound strategies like blogs, newsletters, and SEO.

• Growing Online Community: Opportunity to strengthen engagement through collaborations and events.

• Influencer and Partnership Growth: Increasing collaboration with influencers can expand brand awareness.

Method

Analytics Objective #5: App automated optimization using LLM feedback to improve prompts (Min)

Method to be used: AO5 uses a coding script that processes evaluation data from several optimization rounds. The script compares pre- and post-scores for each evaluation rubric metric, calculates average differences, and applies a paired t-test to check whether the results improve consistently. A brief human review follows to see if higher scores also reflect better quality in the new prompts. Each run produces a refined prompt and a short report for AO1 and AO3 to review and decide on the final version.

Justification for the method: This method is practical and replicable. It produces clear numerical results without requiring another full system and includes a quick human check to ensure that statistical improvements correspond to genuine enhancements in prompt performance.

Variables to be used: The analysis focuses on three main variables: pre- and post-scores for each optimization loop, the mean difference and p-value from the t-test, and the average human ratings from the calibration step. These variables demonstrate both the measurable progress and the perceived quality of AO5’s optimized prompts.