| Dataset Overview and Match Rates | ||
| After standardized join key cleaning and matching | ||
| Dataset | Total Records | Match Rate with D1 |
|---|---|---|
| D1 (Billboard) | 330,087 | 100% |
| D2 (Spotify) | 41,106 | 80.12% |
| D3 (Music Dataset) | 28,372 | 13.94% |
07 Final Report
Note: Data preprocessing was performed using Tidyverse and Janitor. Explanatory classification models were built using Caret and RandomForest, with model performance assessed via AUC using pROC.
1. Introduction
Popular music is more than entertainment — it is a mirror of its time, reflecting the cultural, social, and technological shifts of each era. Over the past six decades, the Billboard Hot 100 has served as the definitive barometer of American musical taste, capturing the rise and fall of genres, the evolution of sound, and the changing emotional landscape of popular culture.
This study analyzes 330,087 weekly chart records spanning from 1960 to 2019, combining Billboard chart data with Spotify audio features to explore what defines a “hit” — and how that definition has changed over time.
Research Questions
This analysis addresses four core research questions:
- RQ1: How has the genre distribution of Billboard Hot 100 songs changed across decades?
- RQ2: How have the emotional dimensions of popular music evolved across six decades, and what cultural and technological shifts might explain these changes?
- RQ3: Does the rise of speechiness in popular music reflect the mainstreaming of Hip-Hop?
- RQ4: Which audio features are most strongly associated with a song’s chart performance?
2. Data & Methodology
2.1 Datasets
This study integrates three complementary datasets:
| Dataset | Source | Records | Role |
|---|---|---|---|
| Billboard Hot 100 | Kaggle (dhruvildave) | 330,087 | Primary anchor — weekly chart records |
| Spotify Hit Predictor | Kaggle (theoverman) | 41,106 | Audio features + Hit/Flop labels |
| Music Dataset 1950–2019 | Kaggle (saurabhshahane) | 28,372 | Genre classification + lyrical sentiment |
2.2 Data Processing
The three datasets were joined using a standardized key of the format artist|song (e.g., drake|gods plan), created after cleaning artist and track names — removing featuring credits, version markers, and bracket annotations. The final analysis dataset contains 262737 records covering 1960–2019.
2.3 Analytical Approach
- RQ1–RQ3: Exploratory Data Analysis (EDA) using trend visualization across six decades
- RQ4: Binary classification modeling (Logistic Regression + Random Forest) using Spotify audio features
- Analysis scope: U.S. market only; limited to 1960–2019 to ensure consistent data coverage across all three datasets
Note on D3 coverage: The Music Dataset (D3) matched only 13.94% of Billboard records. Genre and lyrical analyses based on D3 are therefore treated as supplementary and interpreted with caution.
Technical Note: This report was produced using R. Data preprocessing was performed with Tidyverse and Janitor. Classification models (Logistic Regression and Random Forest) were implemented using Caret and RandomForest, with AUC-based evaluation via pROC. All visualizations were created with ggplot2.
3. Results
3.1 RQ1: Genre Distribution Across Decades
Pop music has consistently dominated the Billboard Hot 100 across all six decades, rising from 61.4% in the 1960s to 78.4% in the 2010s. In the early 1960s, Blues and Country still held notable shares, but both declined significantly over time. Rock emerged as a major force in the 1970s–1980s, peaking at 26–34%, before declining sharply to just 4.3% by the 2010s. Blues and Jazz, which accounted for 15% and 6% respectively in the 1960s, have nearly disappeared from mainstream charts by the 2010s. Hip-Hop shows modest growth from the 1990s onward, reaching a peak of 7.6%, while Country has seen a moderate resurgence in recent decades.
Note on classification: It is important to note that the boundary between Pop and Hip-Hop in D3’s classification may not be precise. Many contemporary Hip-Hop tracks — characterized by rap vocals and spoken-word delivery — may have been classified as Pop due to their mainstream production style. As a result, Hip-Hop’s actual influence on the charts may be underrepresented. This limitation will be further explored in RQ3 through the lens of speechiness.
| Genre Distribution (%) by Decade | ||||||
| Based on D1–D3 matched subset (13.94% coverage) | ||||||
| Genre | 1960s | 1970s | 1980s | 1990s | 2000s | 2010s |
|---|---|---|---|---|---|---|
| blues | 15.1 | 7.9 | 2.7 | 1.1 | 0.6 | 0.2 |
| country | 8.1 | 15.1 | 8.5 | 6.8 | 15.7 | 15.4 |
| jazz | 6.4 | 8.2 | 2.7 | 0.7 | 0.0 | 0.0 |
| pop | 61.4 | 42.2 | 51.0 | 64.6 | 70.1 | 78.4 |
| rock | 9.1 | 26.6 | 34.3 | 18.0 | 10.5 | 4.3 |
| hip hop | 0.0 | 0.0 | 0.3 | 7.6 | 3.1 | 1.1 |
| reggae | 0.0 | 0.0 | 0.5 | 1.2 | 0.0 | 0.4 |
| Source: Billboard Hot 100 × Music Dataset 1950–2019 | ||||||
Key Findings:
- Pop dominance — Rose from 61.4% (1960s) to 78.4% (2010s), consistently the most dominant genre
- Rock’s rise and fall — Peaked at 34.3% in the 1980s, declined to 4.3% by the 2010s
- Blues and Jazz — Nearly disappeared from mainstream charts by the 2000s
- Country resurgence — After declining in the 1980s–1990s, Country recovered to ~15% in the 2000s–2010s
- Hip-Hop underrepresentation — Peaked at 7.6% (1990s) but likely understated due to genre classification limitations
3.2 RQ2: Emotional Evolution of Popular Music
Analysis of five Spotify audio features across six decades reveals a consistent and multi-dimensional emotional transformation in popular music. Rather than simply becoming “sadder,” modern music has evolved into a more complex emotional landscape — darker in mood, yet more intense and energetic.
This evolution can be understood across three distinct phases, each shaped by the dominant musical styles and cultural forces of its time.
Phase 1 — The Disco Era (1960–1979): Music was characterized by high valence and low energy. The optimism of the Civil Rights era, the counterculture movement, and the rise of Disco in the late 1970s produced chart music that was predominantly upbeat and joyful. Artists such as The Beatles, Marvin Gaye, Donna Summer, and ABBA defined this era’s emotional tone.
Phase 2 — Post-Disco & Rock Era (1980–1999): The collapse of Disco in 1979 marked the first major turning point. MTV’s launch in 1981 transformed music into a visual medium, while Hard Rock, Heavy Metal, and Grunge brought darker, more intense emotional content. The AIDS crisis, political polarization, and economic anxiety of the 1980s–1990s contributed to a cultural shift away from collective optimism. Artists such as Michael Jackson, Nirvana, and Tupac Shakur captured the tensions of this era.
Phase 3 — Hip-Hop & Digital Era (2000–2019): The digital revolution — from Napster (1999) to Spotify (2008) — fundamentally reshaped how music was created, distributed, and consumed. Hip-Hop became the dominant genre, and the rise of Emo Rap in the 2010s pushed valence to historic lows. Social media amplified emotional expression and anxiety, while artists such as Drake, Travis Scott, and Billie Eilish defined an era of introspective, high-intensity music.
The most striking finding is the divergence between Valence and Energy — as music became emotionally darker, it simultaneously became more intense and powerful. This suggests that modern popular music is not “sad and weak,” but rather “angry and forceful.”
| Average Audio Features by Decade | |||||
| Billboard Hot 100 (1960s–2010s) | |||||
| Decade | Valence | Energy | Danceability | Acousticness | Speechiness |
|---|---|---|---|---|---|
| 1960s | 0.670 | 0.514 | 0.534 | 0.533 | 0.049 |
| 1970s | 0.664 | 0.579 | 0.577 | 0.338 | 0.052 |
| 1980s | 0.651 | 0.648 | 0.627 | 0.210 | 0.046 |
| 1990s | 0.578 | 0.649 | 0.650 | 0.175 | 0.076 |
| 2000s | 0.555 | 0.708 | 0.636 | 0.149 | 0.097 |
| 2010s | 0.502 | 0.684 | 0.647 | 0.158 | 0.099 |
| Source: Billboard Hot 100 × Spotify Hit Predictor | |||||
Figure A: Sociopolitical Context
Figure B: Music Technology Evolution
Key Findings:
- Valence ↓ — Declined 25.1% from ~0.67 (1960s) to ~0.502 (2010s)
- Energy ↑ — Rose consistently across all three phases
- Acousticness ↓ — Fell sharply from ~0.53 (1960s) to ~0.16 (2000s), reflecting the shift from acoustic instruments to electronic production
- Danceability ↑ — Increased with the rise of EDM and Hip-Hop
- Speechiness ↑ — Grew steadily, reflecting Hip-Hop’s mainstreaming (explored further in RQ3)
Modern popular music is not simply sadder — it is angrier, more energetic, and more danceable, yet emotionally darker. This shift may reflect a cultural transition from collective joy toward individual expression of frustration and resilience.
3.3 RQ3: Speechiness and the Mainstreaming of Hip-Hop
The rise of speechiness on the Billboard Hot 100 over six decades provides a measurable signal of Hip-Hop’s growing influence on mainstream music. Speechiness — Spotify’s measure of the spoken-word quality of a track — increased by 103% from the 1960s (0.049) to the 2010s (0.099), representing one of the most dramatic structural shifts in popular music’s sonic profile.
This trend aligns closely with Hip-Hop’s cultural trajectory: from an underground subculture in the late 1970s, through its Golden Age in the late 1980s–early 1990s, to its emergence as the dominant force in mainstream music by the 2000s–2010s.
Genre-level analysis confirms that Hip-Hop’s speechiness is dramatically higher than all other genres — more than 3.6 times the average of all other genres combined.
Key Findings:
- Speechiness grew 103% from the 1960s (0.0489) to the 2010s (0.0993)
- Hip-Hop’s average speechiness (0.233) is 3.6× higher than all other genres combined (~0.065)
- The sharpest increase occurred between the 1980s and 1990s, coinciding with Hip-Hop’s Golden Age (1988–1992)
- Hip-Hop has consistently maintained the highest speechiness across every decade tracked
The rise of speechiness on the Billboard Hot 100 is strongly associated with Hip-Hop’s growing sonic influence on mainstream music. While genre classification data suggests Hip-Hop peaked at only 7.6% of chart appearances in the 1990s — and has since declined to 1.1% in the 2010s — this is likely a significant underestimate due to classification limitations. As Hip-Hop’s production style converged with mainstream Pop, many Hip-Hop-influenced tracks were labeled as Pop rather than Hip-Hop. Speechiness provides a more objective, algorithm-based measure of Hip-Hop’s true sonic impact. Hip-Hop’s average speechiness (0.233) is 3.6× higher than all other genres combined (~0.065), and it has consistently dominated every decade since the 1980s. As Hip-Hop’s sonic characteristics permeated the mainstream, the spoken-word quality of chart-topping hits grew by 103% from the 1960s to the 2010s — reflecting a broader cultural shift toward lyrical density and rhythmic delivery that transcends genre labels.
Limitation:
The Speechiness trend analysis is based on D2 (Spotify), which covers 80% of Billboard records and is considered reliable. However, the genre-specific comparison Hip-Hop vs. Other Genres) relies on the D3 subset, which covers only 13.94% of records. Some Hip-Hop tracks may also have been misclassified as Pop, meaning Hip-Hop’s true sonic influence may be even greater than observed here.
3.4 RQ4: Audio Features and Chart Success
To identify which audio features are most strongly associated with chart success, this study employs two binary classification models — Logistic Regression and Random Forest — using the Spotify Hit Predictor dataset (D2). The dataset contains 41,106 tracks with a perfectly balanced 50/50 Hit/Flop distribution, eliminating the need for resampling.
This analysis adopts an explanatory rather than predictive approach. The goal is to understand historical associations between audio features and chart success, not to build a system for predicting future hits.
Model Performance
Both models demonstrated meaningful discriminative ability, with Random Forest outperforming Logistic Regression across all metrics.
| Model Performance Comparison | ||||
| Logistic Regression vs Random Forest | ||||
| Model | Accuracy | AUC | Sensitivity (Hit) | Specificity (Flop) |
|---|---|---|---|---|
| Logistic Regression | 0.7230 | 0.8032 | 0.8046 | 0.6414 |
| Random Forest | 0.7887 | 0.8683 | 0.8428 | 0.7345 |
| Source: Spotify Hit Predictor Dataset | ||||
Random Forest’s superior performance (AUC = 0.87 vs 0.80) suggests that the relationship between audio features and chart success involves non-linear interactions that Logistic Regression cannot fully capture.
Logistic Regression : Feature Coefficients
Random Forest : Feature Importance Visualization
Feature Importance Comparison
Key Findings:
Both models consistently agree on the most important features:
- Instrumentalness ↓ (most important) — Negatively associated with hits. Popular songs almost always feature vocals; high instrumentalness signals a non-mainstream track.
- Danceability ↑ — Positively associated with hits. Consistent with the dominance of Hip-Hop and EDM in the modern era.
- Speechiness ↑ — Positively associated with hits. Directly reflects Hip-Hop’s mainstream influence, connecting back to RQ3.
- Acousticness ↓ — Negatively associated with hits. Electronic production has replaced acoustic instrumentation as the dominant sound.
- Energy — Moderate importance, consistent with the Energy ↑ trend observed in RQ2.
Interpreting both models together: Random Forest identifies which features matter most (importance ranking), while Logistic Regression reveals how they matter (direction and magnitude). Together, they provide a complete picture of the audio feature landscape of chart success.
Key Takeaway:
Audio features alone achieve an AUC of 0.87, suggesting that the sonic characteristics of a song are meaningfully associated with its commercial appeal. However, approximately 21% of cases remain unexplained by audio features alone, highlighting the role of external factors such as marketing, artist reputation, and cultural timing.
4. Limitations
While this study provides meaningful insights into six decades of popular music evolution, several limitations should be considered when interpreting the findings.
4.1 Dataset Coverage and Matching
| Dataset Limitations | |
| Limitation | Description |
|---|---|
| D3 Genre Coverage | D3 matched only 13.94% of Billboard records. Genre and lyrical analyses are based on a non-representative subset and may not reflect the full Billboard population. |
| D2 Join Rate | D2 matched 80.12% of Billboard records. Approximately 20% of chart entries lack Spotify audio features, potentially introducing selection bias. |
| Time Range | Analysis is limited to 1960–2019 to ensure consistent data coverage. Pre-1960 and post-2019 trends are not captured. |
4.2 Genre Classification
The Music Dataset (D3) provides only 7 broad genre categories, which lack the granularity to distinguish sub-genres. In particular, many Hip-Hop tracks with mainstream production styles may have been classified as Pop, leading to an underrepresentation of Hip-Hop’s actual influence in RQ1. The speechiness analysis in RQ3 was designed to partially compensate for this limitation.
4.3 Modeling Assumptions
Temporal pooling: The classification models in RQ4 were trained on data pooled across six decades. Given that the definition of a “hit” has evolved significantly over time — as demonstrated in RQ2 and RQ3 — feature importance may vary considerably by era. A decade-specific modeling approach could yield more nuanced findings but was beyond the scope of this study.
Label definition: The Hit/Flop labels in D2 were defined by the dataset’s original author and may not perfectly align with official Billboard chart performance. These labels should be interpreted as proxy indicators rather than ground truth.
Missing variables: Critical non-musical factors — including marketing budgets, social media presence, artist reputation, and cultural timing — were not available in the datasets and could not be incorporated into the models. The unexplained variance (~21%) in RQ4 likely reflects these omitted factors.
4.4 Spotify Audio Features
Spotify’s audio features (including Valence, Energy, and Speechiness) are algorithmically estimated from audio signals. They capture the sonic dimensions of music but do not directly measure lyrical content, cultural meaning, or listener perception. A song may sound musically positive (high Valence) while conveying negative lyrical themes, or vice versa. Interpretations of these features should therefore be treated as approximations rather than definitive emotional measurements.
5. Conclusion
This study analyzed 330,087 weekly chart records from the Billboard Hot 100 (1960–2019), integrated with Spotify audio features and genre classifications, to explore how popular music has evolved over six decades — and what defines a hit.
Summary of Findings
| Summary of Research Findings | ||
| Billboard Hot 100 Analysis (1960–2019) | ||
| RQ | Research Question | Key Finding |
|---|---|---|
| RQ1 | How has genre distribution changed across decades? | Pop dominates and grows (61.4% → 78.4%). Rock peaked in the 1980s and declined. Blues and Jazz have nearly disappeared. |
| RQ2 | How have emotional dimensions of music evolved? | Valence declined 28% while Energy rose — music became angrier and more powerful, not simply sadder. Three distinct cultural phases identified. |
| RQ3 | Does speechiness rise reflect Hip-Hop mainstreaming? | Speechiness grew 171.5% (1960s–2010s). Hip-Hop's speechiness is 4.8× higher than all other genres, confirming its mainstreaming. |
| RQ4 | Which audio features are associated with chart success? | Instrumentalness and Danceability are the strongest predictors. Random Forest (AUC = 0.87) outperformed Logistic Regression (AUC = 0.80). |
| Source: Billboard Hot 100 × Spotify Hit Predictor × Music Dataset 1950–2019 | ||
Broader Implications
Taken together, these findings tell a coherent story about the evolution of American popular music:
Popular music has undergone a profound emotional transformation over six decades. The collective optimism of the Disco era gave way to the anger and intensity of Rock, which in turn was displaced by the rhythmic, speech-heavy aesthetics of Hip-Hop. This shift is not merely a change in taste — it reflects deeper cultural transitions, from communal celebration toward individual expression, from acoustic simplicity toward electronic complexity, and from melodic positivity toward raw emotional authenticity.
The convergence of findings across all four research questions suggests that these changes are systematic and interconnected. The rise of speechiness (RQ3) explains part of the valence decline (RQ2). The dominance of instrumentalness and danceability as hit predictors (RQ4) reflects the structural characteristics of the genres that came to dominate the charts (RQ1). Music, it seems, does not change randomly — it evolves in response to the cultural, technological, and social forces of its time.
Future Research Directions
- Decade-specific modeling — Building separate classification models for each decade to capture how the definition of a “hit” has evolved over time
- Lyrical sentiment analysis — Integrating D3’s lyrical emotion scores with Spotify’s audio features to examine the relationship between sonic and lyrical emotional content
- International markets — Extending the analysis beyond the U.S. market to examine whether these trends are universal or culturally specific
- Post-2019 analysis — Incorporating data from 2020 onwards to examine the impact of the COVID-19 pandemic and the continued rise of streaming on musical trends