Basic Data Analysis in R: Review
Sasha Tomic
October 10, 2026
NOTE: If you want to output from RMarkdown to PDF, you need to run this command once:
install.packages(“tinytex”) tinytex::install_tinytex()
0. Setup
Set the working directory
The working directory is the folder where R looks for files. Put the Excel file in one folder and point R to it. Replace the path below with your own folder.
# R Markdown version of "cd": all chunks below will look in this folder
knitr::opts_knit$set(root.dir = "/Users/aleksandartomic/Library/CloudStorage/GoogleDrive-tomica@bc.edu/My Drive/Data Analysis/Fall 26")Tip: in a regular R script or the console, use
setwd("/Users/aleksandartomic/Library/CloudStorage/GoogleDrive-tomica@bc.edu/My Drive/Data Analysis")
instead.
Install packages (once)
Run this chunk once on your computer. After that, you never need it
again. (eval = FALSE tells R to skip it when knitting.)
install.packages("readxl") # reads Excel files
install.packages("tidyr") # fill() for missing values
install.packages("psych") # describe(), skew(), kurtosi()Load packages (every time)
library(readxl)
library(tidyr)
library(psych)1. Read the data
The Excel file has four sheets. We read each one into its own data frame.
file <- "MSFT_example_3.xlsx" # file name
excel_sheets(file) # list the sheet names## [1] "MSFT" "SPY" "BIL" "30day Tbill"msft <- read_excel(file, sheet = "MSFT") # Microsoft stock
spy <- read_excel(file, sheet = "SPY") # S&P 500 fund
bil <- read_excel(file, sheet = "BIL") # T-bill fund
tbill <- read_excel(file, sheet = "30day Tbill") # 1-month T-bill yield from FRED
head(msft) # first 6 rows## # A tibble: 6 × 7
## Date Open High Low `Close ` `Adj Close ` Volume
## <dttm> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 2026-08-06 00:00:00 489. 502. 489. 500. 500. 32788866
## 2 2026-08-05 00:00:00 496. 498. 486. 487. 487. 33388900
## 3 2026-08-04 00:00:00 481. 499. 479. 493. 493. 50455500
## 4 2026-08-03 00:00:00 476. 492. 475 488. 488. 66762100
## 5 2026-07-31 00:00:00 450 467. 449. 465. 465. 60846000
## 6 2026-07-30 00:00:00 438. 459. 432. 451. 451. 110160700For a CSV file, one line does the job:
mydata <- read.csv("filename.csv")
2. Clean the data
Rename the columns
The column names contain a hidden character copied from the website,
so msft$"Adj Close" fails. We rename the columns
ourselves.
new_names <- c("Date", "Open", "High", "Low", "Close", "AdjClose", "Volume")
names(msft) <- new_names
names(spy) <- new_names
names(bil) <- new_namesRemove the dividend rows
BIL has extra rows for dividend payments. They have no price, so we drop them.
bil <- bil[!is.na(bil$AdjClose), ] # keep only rows where AdjClose is not missing3. Keep one column from each sheet and fix the dates
We keep the date and Adjusted Close for each stock, and the date and
yield for the T-bill. as.Date() converts the date column to
Date format.
msft2 <- data.frame(Date = as.Date(msft$Date), MSFT = msft$AdjClose)
spy2 <- data.frame(Date = as.Date(spy$Date), SPY = spy$AdjClose)
bil2 <- data.frame(Date = as.Date(bil$Date), BIL = bil$AdjClose)
tbill2 <- data.frame(Date = as.Date(tbill$observation_date), TBILL = tbill$DGS1MO)
class(msft2$Date) # check: should say "Date"## [1] "Date"4. Merge by date
stocks <- merge(msft2, spy2, by = "Date") # MSFT + SPY, matched on date
stocks <- merge(stocks, bil2, by = "Date") # add BIL
data <- merge(stocks, tbill2, by = "Date", all.x = TRUE) # add T-bill; keep every stock trading day
head(data)## Date MSFT SPY BIL TBILL
## 1 2021-08-06 277.79 413.56 76.83 0.04
## 2 2021-08-09 276.71 413.23 76.83 0.04
## 3 2021-08-10 274.90 413.74 76.83 0.05
## 4 2021-08-11 275.38 414.77 76.83 0.05
## 5 2021-08-12 278.13 416.01 76.83 0.05
## 6 2021-08-13 281.05 416.77 76.83 0.04all.x = TRUE keeps all rows of the first data frame
(stocks), even when the T-bill has no value for that
date.
5. Find missing values
colSums(is.na(data)) # number of NAs in each column## Date MSFT SPY BIL TBILL
## 0 0 0 0 9missing <- which(is.na(data$TBILL)) # row numbers with a missing T-bill yield
data[missing, ] # show those rows## Date MSFT SPY BIL TBILL
## 46 2021-10-11 282.91 407.58 76.82 NA
## 69 2021-11-11 319.64 434.84 76.81 NA
## 297 2022-10-10 222.28 342.67 77.26 NA
## 321 2022-11-11 239.59 379.31 77.47 NA
## 547 2023-10-09 322.88 417.94 80.76 NA
## 802 2024-10-14 413.40 572.56 85.12 NA
## 822 2024-11-11 412.28 586.71 85.43 NA
## 1051 2025-10-13 510.81 657.59 88.81 NA
## 1072 2025-11-11 505.48 677.39 89.07 NAThe bond market closes on Columbus Day and Veterans Day, but the stock market stays open. So the T-bill has no value on those days.
6. Fill missing values with the last known value
fill() copies the value from the row above. That only
makes sense if the rows run from oldest to newest, so we sort first.
data <- data[order(data$Date), ] # sort: oldest date first
data <- fill(data, TBILL, .direction = "down") # fill only the T-bill column
colSums(is.na(data)) # check: no NAs left## Date MSFT SPY BIL TBILL
## 0 0 0 0 0data[missing, ] # the same rows, now filled## Date MSFT SPY BIL TBILL
## 46 2021-10-11 282.91 407.58 76.82 0.02
## 69 2021-11-11 319.64 434.84 76.81 0.06
## 297 2022-10-10 222.28 342.67 77.26 3.03
## 321 2022-11-11 239.59 379.31 77.47 3.71
## 547 2023-10-09 322.88 417.94 80.76 5.59
## 802 2024-10-14 413.40 572.56 85.12 4.97
## 822 2024-11-11 412.28 586.71 85.43 4.70
## 1051 2025-10-13 510.81 657.59 88.81 4.19
## 1072 2025-11-11 505.48 677.39 89.07 4.047. Daily returns
Return = (price today - price yesterday) / price yesterday
data$MSFT_ret <- c(NA, diff(data$MSFT) / head(data$MSFT, -1))
data$SPY_ret <- c(NA, diff(data$SPY) / head(data$SPY, -1))
data$BIL_ret <- c(NA, diff(data$BIL) / head(data$BIL, -1))
data <- data[!is.na(data$MSFT_ret), ] # drop the first day: it has no returndiff()gives price today minus price yesterday.head(x, -1)gives yesterday’s prices (all prices except the last).NAgoes first because the first day has no earlier price.
8. Create new variables of different types
# Yes/No variable: did the stock go up that day?
data$MSFT_up <- ifelse(data$MSFT_ret > 0, "Yes", "No")
data$SPY_up <- ifelse(data$SPY_ret > 0, "Yes", "No")
# Count variable: how many of the 3 funds went down that day (0 to 3)
data$down_count <- (data$MSFT_ret < 0) + (data$SPY_ret < 0) + (data$BIL_ret < 0)
# Ordered categories: size of the MSFT move
data$move_size <- cut(data$MSFT_ret,
breaks = c(-Inf, -0.02, 0, 0.02, Inf),
labels = c("Big loss", "Small loss", "Small gain", "Big gain"))
head(data)## Date MSFT SPY BIL TBILL MSFT_ret SPY_ret BIL_ret
## 2 2021-08-09 276.71 413.23 76.83 0.04 -0.003887829 -0.0007979495 0.0000000000
## 3 2021-08-10 274.90 413.74 76.83 0.05 -0.006541144 0.0012341795 0.0000000000
## 4 2021-08-11 275.38 414.77 76.83 0.05 0.001746089 0.0024894862 0.0000000000
## 5 2021-08-12 278.13 416.01 76.83 0.05 0.009986201 0.0029896087 0.0000000000
## 6 2021-08-13 281.05 416.77 76.83 0.04 0.010498688 0.0018268792 0.0000000000
## 7 2021-08-16 282.73 417.75 76.82 0.04 0.005977584 0.0023514168 -0.0001301575
## MSFT_up SPY_up down_count move_size
## 2 No No 2 Small loss
## 3 No Yes 1 Small loss
## 4 Yes Yes 0 Small gain
## 5 Yes Yes 0 Small gain
## 6 Yes Yes 0 Small gain
## 7 Yes Yes 1 Small gainThe count works because R treats TRUE as 1 and FALSE as 0.
9. Measures of center, spread and shape
One variable at a time
x <- data$MSFT_ret # short name to save typing
# Center
mean(x) # average## [1] 0.0006244054median(x) # middle value## [1] 0.0004945681# Spread
sd(x) # standard deviation## [1] 0.01771387var(x) # variance## [1] 0.0003137811min(x) # smallest value## [1] -0.09991866max(x) # largest value## [1] 0.1550673range(x) # smallest and largest together## [1] -0.09991866 0.15506734IQR(x) # interquartile range: 75th minus 25th percentile## [1] 0.01819434quantile(x) # 0%, 25%, 50%, 75%, 100% percentiles## 0% 25% 50% 75% 100%
## -0.0999186636 -0.0081058256 0.0004945681 0.0100885175 0.1550673427# Shape (psych package)
skew(x) # 0 = symmetric; > 0 = long right tail; < 0 = long left tail## [1] 0.5779911kurtosi(x) # > 0 = fatter tails than a normal distribution## [1] 7.233472Mode
R has no function for the statistical mode. (mode(x)
exists, but it returns the data type, not the most common value.) Count
the values with table() and take the largest:
table(data$down_count) # how often each value appears##
## 0 1 2 3
## 481 314 423 36names(which.max(table(data$down_count))) # the mode: value with the highest count## [1] "0"names(which.max(table(data$MSFT_up))) # works for categories too## [1] "Yes"If two values tie, which.max() returns only the first
one, so always look at the table() output too. For
continuous data like returns, the mode is useless: almost every value
appears only once.
All variables at once
describe(data) # n, mean, sd, median, min, max, range, skew, kurtosis, se## vars n mean sd median trimmed mad min max range
## Date 1 1254 NaN NA NA NaN NA Inf -Inf -Inf
## MSFT 2 1254 361.60 83.22 371.56 359.59 96.07 207.73 538.66 330.93
## SPY 3 1254 506.56 116.37 479.56 497.87 128.70 339.38 771.33 431.95
## BIL 4 1254 82.67 4.95 82.15 82.41 7.21 76.78 91.45 14.67
## TBILL 5 1254 3.72 1.84 4.32 3.94 1.38 0.01 6.02 6.01
## MSFT_ret 6 1254 0.00 0.02 0.00 0.00 0.01 -0.10 0.16 0.25
## SPY_ret 7 1254 0.00 0.01 0.00 0.00 0.01 -0.06 0.11 0.16
## BIL_ret 8 1254 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
## MSFT_up* 9 1254 1.52 0.50 2.00 1.52 0.00 1.00 2.00 1.00
## SPY_up* 10 1254 1.54 0.50 2.00 1.55 0.00 1.00 2.00 1.00
## down_count 11 1254 1.01 0.91 1.00 0.98 1.48 0.00 3.00 3.00
## move_size* 12 1254 2.52 0.81 3.00 2.52 1.48 1.00 4.00 3.00
## skew kurtosis se
## Date NA NA NA
## MSFT 0.09 -1.06 2.35
## SPY 0.52 -0.99 3.29
## BIL 0.24 -1.40 0.14
## TBILL -1.02 -0.26 0.05
## MSFT_ret 0.58 7.23 0.00
## SPY_ret 0.31 8.87 0.00
## BIL_ret 0.60 0.80 0.00
## MSFT_up* -0.07 -2.00 0.01
## SPY_up* -0.17 -1.97 0.01
## down_count 0.20 -1.32 0.03
## move_size* -0.05 -0.49 0.02Rows marked with * are not numbers (dates and
categories). psych converted them to numbers, so ignore their
statistics. Describe categories with counts and proportions:
table(data$MSFT_up) # counts##
## No Yes
## 605 649prop.table(table(data$MSFT_up)) # proportions##
## No Yes
## 0.4824561 0.5175439table(data$move_size)##
## Big loss Small loss Small gain Big gain
## 124 481 522 12710. Levels of measurement
| Variable | Level | Why |
|---|---|---|
MSFT_up |
Nominal | Categories with no order |
move_size |
Ordinal | Categories with an order, but unequal gaps between them |
Date |
Interval | Equal gaps between days, but no true zero |
MSFT, TBILL |
Ratio | Equal gaps and a true zero; $200 is twice $100 |
down_count |
Ratio (discrete count) | Whole numbers with a true zero |
The level decides which statistics make sense: mode for nominal, median for ordinal, mean and sd for interval and ratio.
11. Graph for a count variable
For a count, use a bar chart: one bar for each value.
barplot(table(data$down_count), # bar heights = counts from table()
col = "steelblue",
main = "Number of Funds Down per Day",
xlab = "Funds down (out of 3)",
ylab = "Number of days")
For a continuous variable, use a histogram:
hist(data$MSFT_ret,
breaks = 50, # number of bins
col = "lightblue",
main = "MSFT Daily Returns",
xlab = "Daily return")
12. Side-by-side boxplots
y ~ group draws one box of y for each
group.
boxplot(MSFT_ret ~ SPY_up, data = data, # MSFT returns, split by SPY direction
col = c("salmon", "lightgreen"),
main = "MSFT Return by SPY Direction",
xlab = "Did SPY go up?",
ylab = "MSFT daily return")
abline(h = -0.05, col = "red", lty = 2) # dashed line marking one value of interest
13. Sort the data and find the first match
order() sorts the rows. Here: worst SPY day first. Then
we find the first day MSFT went up.
sorted <- data[order(data$SPY_ret), ] # sort by SPY return, smallest first
head(sorted[, c("Date", "SPY_ret", "MSFT_up")]) # look at the first rows## Date SPY_ret MSFT_up
## 920 2025-04-04 -0.05853566 No
## 919 2025-04-03 -0.04929135 No
## 924 2025-04-10 -0.04382544 No
## 278 2022-09-13 -0.04348384 No
## 198 2022-05-18 -0.04032279 No
## 215 2022-06-13 -0.03797124 Nowhich(sorted$MSFT_up == "Yes")[1] # row number of the first "Yes"## [1] 88which() lists the row numbers where the condition is
TRUE, and [1] takes the first one.
14. Count observations that meet a condition
How many days did MSFT go up while the T-bill yield was above 4%?
sum(data$MSFT_up == "Yes" & data$TBILL > 4) # & means "and"; TRUE counts as 1## [1] 397nrow(subset(data, MSFT_up == "Yes" & TBILL > 4)) # same answer: count the rows of the subset## [1] 39715. Save the result
write.csv(data, "merged_data.csv", row.names = FALSE) # saves to the working directory