Part I: Dataset Chosen

The dataset I have chosen to analyze is the FBI Hate Crime Data, which is collected and maintained by the Federal Bureau of Investigation (FBI) as part of the Uniform Crime Reporting (UCR) program. This dataset is publicly available and can be accessed through the FBI’s Crime Data Explorer website in various formats, including CSV and Excel files. It provides comprehensive information on hate crimes reported across the United States from 1991 to 2023, capturing details on the nature of offenses, victim demographics, offender characteristics, and the types of biases involved.

I will be focusing on identifying trends and patterns in hate crimes over time, particularly examining whether notable spikes align with specific social and political climates. A key research question driving this analysis is whether major news events involving any ethnicity, race, or religion correspond to an increase in hate crimes. By examining the data, I aim to uncover potential correlations between significant political or social events and surges in hate crimes. The key variables of interest include the year of occurrence, type of bias motivation (e.g., race, religion, sexual orientation), and number of reported offenses.

To achieve this, I plan to conduct a regression analysis to quantify trends over time and identify periods of significant fluctuations. This will allow me to see whether certain years or events are statistically associated with an increase in hate crimes. Additionally, visualizing the data throughout the years will help illustrate patterns, while comparing spikes in hate crimes to news events will provide insight into the societal factors influencing these trends.

Part II: Pivot Longer/Wider Example

rm(list=ls())
gc()

##           used (Mb) gc trigger (Mb) limit (Mb) max used (Mb)
## Ncells  592230 31.7    1357374 72.5         NA   669422 35.8
## Vcells 1116071  8.6    8388608 64.0      16384  1851681 14.2

directory <- "/Users/ruthiemaurer/Desktop/DATA 712"
setwd(directory)

library(readxl)
library(tidyr)
library(ggplot2)
library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(knitr)
library(kableExtra)

## 
## Attaching package: 'kableExtra'

## The following object is masked from 'package:dplyr':
## 
##     group_rows

set.seed(123123)

hate_crime <- read_xlsx("hate_crime copy 2.xlsx")

print(hate_crime)

## # A tibble: 253,776 × 9
##    data_year pug_agency_name   state_abbr state_name incident_date offender_race
##        <dbl> <chr>             <chr>      <chr>      <chr>         <chr>        
##  1      1991 Pine Bluff        AR         Arkansas   1991-07-04    Black or Afr…
##  2      1991 Pine Bluff        AR         Arkansas   1991-12-24    Black or Afr…
##  3      1991 North Little Rock AR         Arkansas   1991-07-10    Black or Afr…
##  4      1991 North Little Rock AR         Arkansas   1991-10-06    Black or Afr…
##  5      1991 Sevier            AR         Arkansas   1991-10-14    White        
##  6      1991 Rogers            AR         Arkansas   1991-08-31    White        
##  7      1991 Hope              AR         Arkansas   1991-09-19    Black or Afr…
##  8      1991 Pine Bluff        AR         Arkansas   1991-12-23    Black or Afr…
##  9      1991 Pine Bluff        AR         Arkansas   1991-07-27    Black or Afr…
## 10      1991 Little Rock       AR         Arkansas   1991-11-14    Black or Afr…
## # ℹ 253,766 more rows
## # ℹ 3 more variables: offender_ethnicity <chr>, offense_name <chr>,
## #   bias_desc <chr>

colnames(hate_crime) <- c("Year", "Pug Agency Name", "State", "State Name", "Incident Date", "Offender Race", "Offender Ethnicity", "Offense", "Reason/Bias")
colnames(hate_crime)

## [1] "Year"               "Pug Agency Name"    "State"             
## [4] "State Name"         "Incident Date"      "Offender Race"     
## [7] "Offender Ethnicity" "Offense"            "Reason/Bias"

print(hate_crime)

## # A tibble: 253,776 × 9
##     Year `Pug Agency Name` State `State Name` `Incident Date` `Offender Race`   
##    <dbl> <chr>             <chr> <chr>        <chr>           <chr>             
##  1  1991 Pine Bluff        AR    Arkansas     1991-07-04      Black or African …
##  2  1991 Pine Bluff        AR    Arkansas     1991-12-24      Black or African …
##  3  1991 North Little Rock AR    Arkansas     1991-07-10      Black or African …
##  4  1991 North Little Rock AR    Arkansas     1991-10-06      Black or African …
##  5  1991 Sevier            AR    Arkansas     1991-10-14      White             
##  6  1991 Rogers            AR    Arkansas     1991-08-31      White             
##  7  1991 Hope              AR    Arkansas     1991-09-19      Black or African …
##  8  1991 Pine Bluff        AR    Arkansas     1991-12-23      Black or African …
##  9  1991 Pine Bluff        AR    Arkansas     1991-07-27      Black or African …
## 10  1991 Little Rock       AR    Arkansas     1991-11-14      Black or African …
## # ℹ 253,766 more rows
## # ℹ 3 more variables: `Offender Ethnicity` <chr>, Offense <chr>,
## #   `Reason/Bias` <chr>

hate_crime_summary <- hate_crime %>%
  group_by(Year, `Reason/Bias`) %>%
  summarise(count = n(), .groups = 'drop')

hate_crime_wide <- hate_crime_summary %>%
  pivot_wider(names_from = `Reason/Bias`, values_from = count, values_fill = 0)

hate_crime_long <- hate_crime_wide %>%
  pivot_longer(cols = -Year, names_to = "`Reason/Bias`", values_to = "count")

print(hate_crime_wide)

## # A tibble: 33 × 416
##     Year Anti-American Indian …¹ `Anti-Arab` `Anti-Asian` Anti-Atheism/Agnosti…²
##    <dbl>                   <int>       <int>        <int>                  <int>
##  1  1991                      11          73          269                      4
##  2  1992                      26          60          222                      1
##  3  1993                      28          55          260                      3
##  4  1994                      24          52          211                      3
##  5  1995                      41          55          355                      1
##  6  1996                      51          40          355                      2
##  7  1997                      38          34          346                      3
##  8  1998                      52          25          295                      2
##  9  1999                      49          24          303                      4
## 10  2000                      58          38          280                      4
## # ℹ 23 more rows
## # ℹ abbreviated names: ¹`Anti-American Indian or Alaska Native`,
## #   ²`Anti-Atheism/Agnosticism`
## # ℹ 411 more variables: `Anti-Bisexual` <int>,
## #   `Anti-Black or African American` <int>,
## #   `Anti-Black or African American;Anti-Gay (Male)` <int>,
## #   `Anti-Black or African American;Anti-Jewish` <int>, …

print(hate_crime_long)

## # A tibble: 13,695 × 3
##     Year `\`Reason/Bias\``                                    count
##    <dbl> <chr>                                                <int>
##  1  1991 Anti-American Indian or Alaska Native                   11
##  2  1991 Anti-Arab                                               73
##  3  1991 Anti-Asian                                             269
##  4  1991 Anti-Atheism/Agnosticism                                 4
##  5  1991 Anti-Bisexual                                            1
##  6  1991 Anti-Black or African American                        1624
##  7  1991 Anti-Black or African American;Anti-Gay (Male)           1
##  8  1991 Anti-Black or African American;Anti-Jewish               1
##  9  1991 Anti-Black or African American;Anti-Lesbian (Female)     1
## 10  1991 Anti-Black or African American;Anti-White                2
## # ℹ 13,685 more rows

dim(hate_crime_wide)

## [1]  33 416

dim(hate_crime_long)

## [1] 13695     3

kable(head(hate_crime_wide))

Year	Anti-American Indian or Alaska Native	Anti-Arab	Anti-Asian	Anti-Atheism/Agnosticism	Anti-Bisexual	Anti-Black or African American	Anti-Black or African American;Anti-Gay (Male)	Anti-Black or African American;Anti-Jewish	Anti-Black or African American;Anti-Lesbian (Female)	Anti-Black or African American;Anti-White	Anti-Catholic	Anti-Gay (Male)	Anti-Gay (Male);Anti-Jewish	Anti-Gay (Male);Anti-White	Anti-Heterosexual	Anti-Hispanic or Latino	Anti-Islamic (Muslim)	Anti-Jewish	Anti-Lesbian (Female)	Anti-Lesbian, Gay, Bisexual, or Transgender (Mixed Group)	Anti-Multiple Races, Group	Anti-Multiple Religions, Group	Anti-Other Race/Ethnicity/Ancestry	Anti-Other Religion	Anti-Protestant	Anti-White	Anti-Hispanic or Latino;Anti-Multiple Races, Group	Anti-Hispanic or Latino;Anti-White	Anti-American Indian or Alaska Native;Anti-Hispanic or Latino	Anti-Black or African American;Anti-Gay (Male);Anti-White	Anti-Asian;Anti-Gay (Male)	Anti-Multiple Races, Group;Anti-Other Race/Ethnicity/Ancestry
1991	11	73	269	4	1	1624	1	1	1	2	19	290	1	2	3	231	10	793	43	68	86	11	132	50	26	837	0	0	0	0	0	0
1992	26	60	222	1	3	2311	0	0	0	1	18	561	0	0	14	371	15	1020	93	100	145	14	241	69	28	1349	0	0	0	0	0	0
1993	28	55	260	3	2	2821	0	0	0	4	32	615	1	0	28	416	13	1140	121	94	161	14	225	64	30	1475	1	1	0	0	0	0
1994	24	52	211	3	7	2178	0	0	0	1	18	501	0	0	14	340	17	919	100	64	127	14	251	67	30	1014	0	0	1	0	0	0
1995	41	55	355	1	18	2989	0	0	0	4	31	735	0	0	17	516	29	1059	146	103	221	19	243	101	36	1228	0	1	0	1	0	0
1996	51	40	355	2	10	3675	1	0	0	1	35	757	0	0	15	564	27	1109	150	85	210	24	337	156	75	1106	1	1	0	0	1	1

kable(head(hate_crime_long))

Year	`Reason/Bias`	count
1991	Anti-American Indian or Alaska Native	11
1991	Anti-Arab	73
1991	Anti-Asian	269
1991	Anti-Atheism/Agnosticism	4
1991	Anti-Bisexual	1
1991	Anti-Black or African American	1624

Exaplanation

In this analysis, I transformed the FBI Hate Crime dataset between long and wide formats using the tidyverse functions pivot_wider() and pivot_longer(). I first summarized the dataset to count the number of hate crime incidents per Year and Reason/Bias, ensuring it was structured for further transformations. Using pivot_wider(), I reshaped the dataset so that each unique Reason/Bias became a separate column, with values representing the count of hate crimes per year. This wide format makes it easier to compare bias categories over time and identify trends at a glance.

Next, I applied pivot_longer() to revert the data to its original long format by gathering all bias-related columns into a single Reason/Bias column. This structure is better suited for data visualization and statistical modeling. To confirm the transformation, I used dim() to check the number of rows and columns in both formats and kable() to better illustrate the difference in formats. The wide format had fewer rows but many columns (one for each bias category), while the long format had more rows but only three columns: Year, Reason/Bias, and count. These transformations allow flexibility in data analysis, enabling us to choose the best format for specific tasks such as trend analysis, visualization, and regression modeling.

Part III: Reproducing Log-Likelihood Function and the MLE Procedure

# Summarize the number of hate crimes per year
hate_crime_year <- hate_crime %>%
  group_by(Year) %>% 
  summarise(count = n(), .groups = 'drop') %>%
  rename(Year = Year)  # Ensure consistent naming

head(hate_crime_year)

## # A tibble: 6 × 2
##    Year count
##   <dbl> <int>
## 1  1991  4589
## 2  1992  6662
## 3  1993  7604
## 4  1994  5953
## 5  1995  7949
## 6  1996  8789

# Center the Year variable
hate_crime_year <- hate_crime_year %>%
  mutate(centered_year = Year - mean(Year))

library(maxLik)

## Warning: package 'maxLik' was built under R version 4.3.3

## Loading required package: miscTools

## Warning: package 'miscTools' was built under R version 4.3.3

## 
## Please cite the 'maxLik' package as:
## Henningsen, Arne and Toomet, Ott (2011). maxLik: A package for maximum likelihood estimation in R. Computational Statistics 26(3), 443-458. DOI 10.1007/s00180-010-0217-1.
## 
## If you have questions, suggestions, or comments regarding the 'maxLik' package, please use a forum or 'tracker' at maxLik's R-Forge site:
## https://r-forge.r-project.org/projects/maxlik/

ols_hatecrime_year <- function(param) {
  sigma <- param[1]  # Standard deviation
  beta <- param[-1]  # Regression coefficients
  y <- as.vector(hate_crime_year$count)  # Dependent variable (hate crime counts)
  x <- cbind(1, hate_crime_year$centered_year)  # Independent variable (year with intercept)
  mu <- x %*% beta  # Predicted values
  sum(dnorm(y, mean = mu, sd = sigma, log = TRUE))  # Log-likelihood function
}
start_vals <- c(sigma = 500, beta1 = mean(hate_crime_year$count), beta2 = 0)

mle_hatecrime_year <- maxLik(logLik = ols_hatecrime_year, 
                             start = start_vals, 
                             method = "BFGS",
                             control = list(iterlim = 500))
mle_hatecrime_year <- maxLik(logLik = ols_hatecrime_year, 
                             start = start_vals, 
                             method = "Nelder-Mead")
summary(mle_hatecrime_year)

## Warning in sqrt(diag(vc)): NaNs produced

## Warning in sqrt(diag(vc)): NaNs produced

## --------------------------------------------
## Maximum Likelihood estimation
## Nelder-Mead maximization, 72 iterations
## Return code 0: successful convergence 
## Log-Likelihood: -288.8936 
## 3  free parameters
## Estimates:
##       Estimate Std. error t value Pr(> t)    
## sigma  1533.27        NaN     NaN     NaN    
## beta1  7691.11        NaN     NaN     NaN    
## beta2    60.32       0.00     Inf  <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## --------------------------------------------

lm(count ~ poly(centered_year, 2), data = hate_crime_year)

## 
## Call:
## lm(formula = count ~ poly(centered_year, 2), data = hate_crime_year)
## 
## Coefficients:
##             (Intercept)  poly(centered_year, 2)1  poly(centered_year, 2)2  
##                    7690                     3300                     3195

summary(lm(count ~ poly(centered_year, 2), data = hate_crime_year))

## 
## Call:
## lm(formula = count ~ poly(centered_year, 2), data = hate_crime_year)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3271.3 -1045.7   547.5   915.6  2777.2 
## 
## Coefficients:
##                         Estimate Std. Error t value Pr(>|t|)    
## (Intercept)               7690.2      260.9  29.470   <2e-16 ***
## poly(centered_year, 2)1   3300.2     1499.0   2.202   0.0355 *  
## poly(centered_year, 2)2   3194.9     1499.0   2.131   0.0414 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1499 on 30 degrees of freedom
## Multiple R-squared:  0.2384, Adjusted R-squared:  0.1876 
## F-statistic: 4.695 on 2 and 30 DF,  p-value: 0.01683

ggplot(hate_crime_year, aes(x = centered_year, y = count)) +
  geom_point() +
  geom_smooth(method = "lm", formula = y ~ poly(x, 2), color = "red") +
  labs(title = "Quadratic Curved Trend in Hate Crimes", x = "Year", y = "Hate Crime Count") +
  theme_minimal()

Analysis

In this analysis, using the FBI Hate Crime Dataset, I examined how hate crimes have changed over time using two different statistical approaches: Maximum Likelihood Estimation (MLE) and Ordinary Least Squares (OLS) regression. The goal was to determine whether hate crimes have increased steadily over the years or if the rate of increase has changed over time.

First, I applied MLE, a method that finds the most likely values for model parameters based on the data. I used the Nelder-Mead method, which successfully found the best estimates for the trend in hate crimes, producing a log-likelihood value of -288.89. The results showed that the year variable was highly significant (beta2 = 60.32, p < 2e-16), meaning that hate crimes have consistently increased over time. However, some values, such as the standard errors for the model, could not be accurately estimated, likely due to high variability in the data. Despite this, the MLE results aligned with those from the OLS regression, confirming that hate crimes have been rising each year.

To better illustrate changes in the rate of increase, I used a quadratic regression model, which allows for curved trends rather than a straight-line increase. The results showed that both the linear trend (general increase) and the quadratic trend (curvature) were statistically significant. Meaning that, rather than increasing at a constant rate, hate crimes have accelerated in recent years. The R² value (0.2384) improved, suggesting that this model better explains the data compared to a simple linear trend. The visualization further supports this, showing a U-shaped pattern where hate crimes initially declined slightly but then spiked and increased substantially. This suggests that recent years have seen a faster rise in hate crimes, potentially influenced by social or political factors. The chart provides a more accurate representation of the trend, highlighting that hate crimes are not just increasing but doing so at an accelerated rate.

DATA 712 HW#3

RMaurer

2025-03-02

Part I: Dataset Chosen

Part II: Pivot Longer/Wider Example

Exaplanation

Part III: Reproducing Log-Likelihood Function and the MLE Procedure

Analysis