Working with Dates and Times in R

Dates and Times in R

R doesn’t know something is a date or time unless you tell it.

ISO 8601 dates = YYYY-MM-DD

Specifying dates

If you have a character string that represents a date in the ISO 8601 standard you can turn it into a Date using the as.Date() function. Just pass the character string (or a vector of character strings) as the first argument.

# The date R 3.0.0 was released
x <- "2013-04-03"

# Examine structure of x
str(x)

 chr "2013-04-03"

# Use as.Date() to interpret x as a date
x_date <- as.Date(x)

# Examine structure of x_date
str(x_date)

 Date[1:1], format: "2013-04-03"

# Store April 10 2014 as a Date
april_10_2014 <- as.Date("2014-04-10")

Automatic import

Sometimes you’ll need to input a couple of dates by hand using as.Date() but it’s much more common to have a column of dates in a data file.

Some functions that read in data will automatically recognize and parse dates in a variety of formats. In particular the import functions, like read_csv(), in the readr package will recognize dates in a few common formats.

There is also the anytime() function in the anytime package whose sole goal is to automatically parse strings as dates regardless of the format.

# Load the readr package
library(readr)

# Use read_csv() to import rversions.csv
releases <- read_csv("rversions.csv")

Parsed with column specification:
cols(
  major = col_double(),
  minor = col_double(),
  patch = col_double(),
  date = col_date(format = ""),
  datetime = col_datetime(format = ""),
  time = col_time(format = ""),
  type = col_character()
)

# Examine the structure of the date column
str(releases$date)

 Date[1:105], format: "1997-12-04" "1997-12-21" "1998-01-10" "1998-03-14" "1998-05-02" "1998-06-14" "1998-06-14" ...

# Load the anytime package
library(anytime)

package 㤼㸱anytime㤼㸲 was built under R version 3.6.3

# Various ways of writing Sep 10 2009
sep_10_2009 <- c("September 10 2009", "2009-09-10", "10 Sep 2009", "09-10-2009")

# Use anytime() to parse sep_10_2009
anytime(sep_10_2009)

[1] "2009-09-10 01:00:00 BST" "2009-09-10 01:00:00 BST" "2009-09-10 01:00:00 BST"
[4] "2009-09-10 01:00:00 BST"

Sometimes these functions won’t work, especially if dates are ambiguous (e.g. Is 2004-10-4, Oct 4th or April 10th?)

Plotting

If you plot a Date on the axis of a plot, you expect the dates to be in calendar order, and that’s exactly what happens with plot() or ggplot().

We’ll make some plots with the R version releases data using ggplot2. There are two big differences when a Date is on an axis:

1. If you specify limits they must be Date objects.
2. To control the behavior of the scale you use the scale_x_date() function.

library(ggplot2)

package 㤼㸱ggplot2㤼㸲 was built under R version 3.6.3Registered S3 method overwritten by 'dplyr':
  method           from
  print.rowwise_df     

# Set the x axis to the date column
ggplot(releases, aes(x = date, y = type)) +
  geom_line(aes(group = 1, color = factor(major)))

# Limit the axis to between 2010-01-01 and 2014-01-01
ggplot(releases, aes(x = date, y = type)) +
  geom_line(aes(group = 1, color = factor(major))) +
  xlim(as.Date("2010-01-01"), as.Date("2014-01-01"))

# Specify breaks every ten years and labels with "%Y"
ggplot(releases, aes(x = date, y = type)) +
  geom_line(aes(group = 1, color = factor(major))) +
  scale_x_date(date_breaks = "10 years", date_labels = "%Y")

Arithmetic and logical operators

Since Date objects are internally represented as the number of days since 1970-01-01 you can do basic math and comparisons with dates. You can compare dates with the usual logical operators (<, ==, > etc.), find extremes with min() and max(), and even subtract two dates to find out the time between them.

Sys.date() in the code, it simply returns today’s date.

library(dplyr)

Attaching package: 㤼㸱dplyr㤼㸲

The following objects are masked from 㤼㸱package:lubridate㤼㸲:

    intersect, setdiff, union

The following objects are masked from 㤼㸱package:stats㤼㸲:

    filter, lag

The following objects are masked from 㤼㸱package:base㤼㸲:

    intersect, setdiff, setequal, union

# Find the largest date
last_release_date <- max(releases$date)

# Filter row for last release
last_release <- filter(releases, date == last_release_date)

# Print last_release
print(last_release)

# How long since last release?
Sys.Date() - last_release$date

Time difference of 1061 days

Date times

ISO 8601 = HH:MM:SS

Getting datetimes into R

Just like dates without times, if you want R to recognize a string as a datetime you need to convert it, although now you use as.POSIXct(). as.POSIXct() expects strings to be in the format YYYY-MM-DD HH:MM:SS.

The only tricky thing is that times will be interpreted in local time based on your machine’s set up. You can check your timezone with Sys.timezone(). If you want the time to be interpreted in a different timezone, you just set the tz argument of as.POSIXct().

# Use as.POSIXct to enter the datetime 
as.POSIXct("2010-10-01 12:12:00")

[1] "2010-10-01 12:12:00 BST"

# Use as.POSIXct again but set the timezone to `"America/Los_Angeles"`
as.POSIXct("2010-10-01 12:12:00", tz = "America/Los_Angeles")

[1] "2010-10-01 12:12:00 PDT"

# Use read_csv to import rversions.csv
releases <- read_csv("rversions.csv")

Parsed with column specification:
cols(
  major = col_double(),
  minor = col_double(),
  patch = col_double(),
  date = col_date(format = ""),
  datetime = col_datetime(format = ""),
  time = col_time(format = ""),
  type = col_character()
)

# Examine structure of datetime column
str(releases$datetime)

 POSIXct[1:105], format: "1997-12-04 08:47:58" "1997-12-21 13:09:22" "1998-01-10 00:31:55" "1998-03-14 19:25:55" ...

Datetimes behave nicely too

ust like Date objects, you can plot and do math with POSIXct objects.

As an example, in this exercise you’ll see how quickly people download new versions of R, by examining the download logs from the RStudio CRAN mirror.

R 3.2.0 was released at “2015-04-16 07:13:33” so cran-logs_2015-04-17.csv contains a random sample of downloads on the 16th, 17th and 18th.

# Import "cran-logs_2015-04-17.csv" with read_csv()
logs <- read_csv("cran-logs_2015-04-17.csv")

Parsed with column specification:
cols(
  datetime = col_datetime(format = ""),
  r_version = col_character(),
  country = col_character()
)

# Print logs
print(logs)

# Store the release time as a POSIXct object
release_time <- as.POSIXct("2015-04-16 07:13:33", tz = "UTC")

# When is the first download of 3.2.0?
logs %>% 
  filter(logs$datetime > release_time,
    r_version == "3.2.0")

# Examine histograms of downloads by version
ggplot(logs, aes(x = datetime)) +
  geom_histogram() +
  geom_vline(aes(xintercept = as.numeric(release_time)))+
  facet_wrap(~ r_version, ncol = 1)

Did you see how it takes about two days for downloads of the new version (3.2.0) to overtake downloads of the old version (3.1.3)?

Parsing and Manipulating Dates and Times with lubridate

Parsing dates with lubridate

Selecting the right parsing function

lubridate provides a set of functions for parsing dates of a known order. For example, ymd() will parse dates with year first, followed by month and then day. The parsing is flexible, for example, it will parse the m whether it is numeric (e.g. 9 or 09), a full month name (e.g. September), or an abbreviated month name (e.g. Sep).

All the functions with y, m and d in any order exist. If your dates have times as well, you can use the functions that start with ymd, dmy, mdy or ydm and are followed by any of _h, _hm or _hms.

To see all the functions available look at ymd() for dates and ymd_hms() for datetimes.

library(lubridate)

# Parse x 
x <- "2010 September 20th" # 2010-09-20
ymd(x)

[1] "2010-09-20"

# Parse y 
y <- "02.01.2010"  # 2010-01-02
dmy(y)

[1] "2010-01-02"

# Parse z 
z <- "Sep, 12th 2010 14:00"  # 2010-09-12T14:00
mdy_hm(z)

[1] "2010-09-12 14:00:00 UTC"

Did you notice the message after you called library(lubridate)? Whenever you see an object “is masked by”, it means an object in the package, in this case the date() function in lubridate has the same name as an object in another loaded package, in this case date() in the base package. If you ask for date() you’ll get the lubridate one, you can always get the one it masked with base::date().

Specifying an order with parse_date_time()

What about if you have something in a really weird order like dym_msh? There’s no named function just for that order, but that is where parse_date_time() comes in. parse_date_time() takes an additional argument, orders, where you can specify the order of the components in the date.

or example, to parse “2010 September 20th” you could say parse_date_time(“2010 September 20th”, orders = “ymd”) and that would be equivalent to using the ymd() function from the previous exercise.

One advantage of parse_date_time() is that you can use more format characters. For example, you can specify weekday names with A, I for 12 hour time, am/pm indicators with p and many others.

Another big advantage is that you can specify a vector of orders, and that allows parsing of dates where multiple formats might be used.

# Specify an order string to parse x
x <- "Monday June 1st 2010 at 4pm"
parse_date_time(x, orders = "AmdyIp")

[1] "2010-06-01 16:00:00 UTC"

# Specify order to include both "mdy" and "dmy"
two_orders <- c("October 7, 2001", "October 13, 2002", "April 13, 2003", 
  "17 April 2005", "23 April 2017")
parse_date_time(two_orders, orders = c("mdy", "dmy"))

[1] "2001-10-07 UTC" "2002-10-13 UTC" "2003-04-13 UTC" "2005-04-17 UTC" "2017-04-23 UTC"

# Specify order to include "dOmY", "OmY" and "Y"
short_dates <- c("11 December 1282", "May 1372", "1253")
parse_date_time(short_dates, orders = c("dOmY", "OmY", "Y"))

[1] "1282-12-11 UTC" "1372-05-01 UTC" "1253-01-01 UTC"

Did you notice that when a date component is missing, it’s just set to 1? For example, the input 1253 resulted in the date 1253-01-01.

Weather in Auckland

Import daily weather data

In practice you won’t be parsing isolated dates and times, they’ll be part of a larger dataset. We’ll practice your lubridate skills in context by working with weather data from Auckland NZ.

There are two data sets: akl_weather_daily.csv a set of once daily summaries for 10 years, and akl_weather_hourly_2016.csv observations every half hour for 2016.

library(lubridate)
library(readr)
library(dplyr)
library(ggplot2)

# Import CSV with read_csv()
akl_daily_raw <- read_csv("akl_weather_daily.csv")

Parsed with column specification:
cols(
  date = col_character(),
  max_temp = col_double(),
  min_temp = col_double(),
  mean_temp = col_double(),
  mean_rh = col_double(),
  events = col_character(),
  cloud_cover = col_double()
)

# Print akl_daily_raw
akl_daily_raw

# Parse date 
akl_daily <- akl_daily_raw %>%
  mutate(date = ymd(date))

# Print akl_daily
akl_daily

# Plot to check work
ggplot(akl_daily, aes(x = date, y = max_temp)) +
  geom_line() 

Can you see when it is hot in Auckland? Those temperatures are in farenheit. Yup, summer falls in Dec-Jan-Feb.

Import hourly weather data

The hourly data is a little different. The date information is spread over three columns year, month and mday, so you’ll need to use make_date() to combine them.

Then the time information is in a separate column again, time. It’s quite common to find date and time split across different variables. One way to construct the datetimes is to paste the date and time together and then parse them.

# Import "akl_weather_hourly_2016.csv"
akl_hourly_raw <- read_csv("akl_weather_hourly_2016.csv")

Parsed with column specification:
cols(
  year = col_double(),
  month = col_double(),
  mday = col_double(),
  time = col_time(format = ""),
  temperature = col_double(),
  weather = col_character(),
  conditions = col_character(),
  events = col_character(),
  humidity = col_double(),
  date_utc = col_datetime(format = "")
)

# Print akl_hourly_raw
akl_hourly_raw

# Use make_date() to combine year, month and mday 
akl_hourly  <- akl_hourly_raw  %>% 
  mutate(date = make_date(year = year, month = month, day = mday))

# Parse datetime_string 
akl_hourly <- akl_hourly  %>% 
  mutate(
    datetime_string = paste(date, time, sep = "T"),
    datetime = parse_date_time(datetime_string, orders = "ymdHMS")
  )
# Print date, time and datetime columns of akl_hourly
akl_hourly %>% select(date, time, datetime)

# Plot to check work
ggplot(akl_hourly, aes(x = datetime, y = temperature)) +
  geom_line()

it’s interesting how the day to day variation is about half the size of the yearly variation.

Extracting parts of a datetime

What can you extract?

Components of a datetime can be extracted by lubridate functions with the same name like year(), month(), day(), hour(), minute() and second(). They all work the same way just pass in a datetime or vector of datetimes.

There are also a few useful functions that return other aspects of a datetime like if it occurs in the morning am(), during daylight savings dst(), in a leap_year(), or which quarter() or semester() it occurs in.

release_time <- releases[["datetime"]]
head(release_time)

[1] "1997-12-04 08:47:58 UTC" "1997-12-21 13:09:22 UTC" "1998-01-10 00:31:55 UTC"
[4] "1998-03-14 19:25:55 UTC" "1998-05-02 07:58:17 UTC" "1998-06-14 12:56:20 UTC"

# Examine the head() of the months of release_time
head(month(release_time))

[1] 12 12  1  3  5  6

# Extract the month of releases 
month(release_time) %>% table()

.
 1  2  3  4  5  6  7  8  9 10 11 12 
 5  6  8 18  5 16  4  7  2 15  6 13 

# Extract the year of releases
year(release_time) %>% table()

.
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 
   2   10    9    6    6    5    5    4    4    4    4    6    5    4    6    4    4    4    5    5    3 

# How often is the hour before 12 (noon)?
mean(hour(release_time) < 12)

[1] 0.752381

# How often is the release in am?
mean(am(release_time))

[1] 0.752381

R versions have historically been released most in April, June, October and December, 1998 saw 10 releases and about 75% of releases happen in the morning (at least according to UTC).

Adding useful labels

Sometimes it’s nicer (especially for plotting or tables) to have named months. Both the month() and wday() (day of the week) functions have additional arguments label and abbr to achieve just that. Set label = TRUE to have the output labelled with month (or weekday) names, and abbr = FALSE for those names to be written in full rather than abbreviated.

head(month(release_time, label = TRUE, abbr = FALSE))

# Use wday() to tabulate release by day of the week
wday(releases$datetime) %>% table()

.
 1  2  3  4  5  6  7 
 3 29  9 12 18 31  3 

# Add label = TRUE to make table more readable
wday(releases$datetime, label = TRUE) %>% table()

.
Sun Mon Tue Wed Thu Fri Sat 
  3  29   9  12  18  31   3 

# Create column wday to hold labelled week days
releases$wday <- wday(releases$datetime, label = TRUE)

# Plot barchart of weekday by type of release
ggplot(releases, aes(x = wday)) +
  geom_bar() +
  facet_wrap(~ type, ncol = 1, scale = "free_y")

Looks like not too many releases occur on the weekends, and there is quite a different weekday pattern between minor and patch releases.

Extracting for plotting

Extracting components from a datetime is particularly useful when exploring data. Earlier in the chapter you imported daily data for weather in Auckland, and created a time series plot of ten years of daily maximum temperature. While that plot gives you a good overview of the whole ten years, it’s hard to see the annual pattern.

We’ll use components of the dates to help explore the pattern of maximum temperature over the year. The first step is to create some new columns to hold the extracted pieces, then you’ll use them in a couple of plots.

library(ggplot2)
library(dplyr)
library(ggridges)

package 㤼㸱ggridges㤼㸲 was built under R version 3.6.3

# Add columns for year, yday and month
akl_daily <- akl_daily %>%
  mutate(
    year = year(date),
    yday = yday(date),
    month = month(date, label = TRUE))

# Plot max_temp by yday for all years
ggplot(akl_daily, aes(x = yday, y = max_temp)) +
  geom_line(aes(group = year), alpha = 0.5)

# Examine distribution of max_temp by month
ggplot(akl_daily, aes(x = max_temp, y = month, height = ..density..)) +
  geom_density_ridges(stat = "density")

Both plots give a great view into both the expected temperatures and how much they vary. Looks like Jan, Feb and Mar are great months to visit if you want warm temperatures. Did you notice the warning messages? These are a consequence of some missing values in the max_temp column. They are a reminder to think carefully about what you might miss by ignoring missing values.

Extracting for filtering and summarizing

Another reason to extract components is to help with filtering observations or creating summaries. For example, if you are only interested in observations made on weekdays (i.e. not on weekends) you could extract the weekdays then filter out weekends, e.g. wday(date) %in% 2:6.

In the last exercise you saw that January, February and March were great times to visit Auckland for warm temperatures, but will you need a raincoat?

# Create new columns hour, month and rainy
akl_hourly <- akl_hourly %>%
  mutate(
    hour = hour(datetime),
    month = month(datetime, label = TRUE),
    rainy = weather == "Precipitation"
  )

# Filter for hours between 8am and 10pm (inclusive)
akl_day <- akl_hourly %>% 
  filter(hour >= 8, hour<=22)

# Summarise for each date if there is any rain
rainy_days <- akl_day %>% 
  group_by(month, date) %>%
  summarise(
    any_rain = any(rainy)
  )

# Summarise for each month, the number of days with rain
rainy_days %>% 
  summarise(
    days_rainy = sum(any_rain)
  )

At least in 2016, it looks like you’ll still need to pack a raincoat if you visit in Jan, Feb or March. Months of course are different lengths so we should really correct for that, take a look at days_in_month() for helping with that.

Rounding datetimes

round_date() rounds a date to the nearest value, floor_date() rounds down, and ceiling_date() rounds up.

All three take a unit argument which specifies the resolution of rounding. You can specify “second”, “minute”, “hour”, “day”, “week”, “month”, “bimonth”, “quarter”, “halfyear”, or “year”. Or, you can specify any multiple of those units, e.g. “5 years”, “3 minutes” etc.

r_3_4_1 <- ymd_hms("2016-05-03 07:13:28 UTC")

# Round down to day
floor_date(r_3_4_1, unit = "day")

[1] "2016-05-03 UTC"

# Round to nearest 5 minutes
round_date(r_3_4_1, unit = "5 minutes")

[1] "2016-05-03 07:15:00 UTC"

# Round up to week 
ceiling_date(r_3_4_1, unit = "week")

[1] "2016-05-08 UTC"

# Subtract r_3_4_1 rounded down to day
r_3_4_1 - floor_date(r_3_4_1, unit = "day")

Time difference of 7.224444 hours

That last technique of subtracting a rounded datetime from an unrounded one is a really useful trick to remember.

Rounding with the weather data

When is rounding useful? In a lot of the same situations extracting date components is useful. The advantage of rounding over extracting is that it maintains the context of the unit. For example, extracting the hour gives you the hour the datetime occurred, but you lose the day that hour occurred on (unless you extract that too), on the other hand, rounding to the nearest hour maintains the day, month and year.

We’ll explore how many observations per hour there really are in the hourly Auckland weather data.

# Create day_hour, datetime rounded down to hour
akl_hourly <- akl_hourly %>%
  mutate(
    day_hour = floor_date(datetime, unit = "hour")
  )

# Count observations per hour  
akl_hourly %>% 
  count(day_hour) 

# Find day_hours with n != 2  
akl_hourly %>% 
  count(day_hour) %>%
  filter(n != 2) %>% 
  arrange(desc(n))

92 hours that don’t have two measurements. Interestingly there are four measurements on 2016-04-03 and 2016-09-25, they happen to be the days Daylight Saving starts and ends.

Arithmetic with Dates and Times

Taking differenes of datetimes

How long has it been?

To get finer control over a difference between datetimes use the base function difftime(). For example instead of time1 - time2, you use difftime(time1, time2).

difftime() takes an argument units which specifies the units for the difference. Your options are “secs”, “mins”, “hours”, “days”, or “weeks”

We’ll also see the lubridate functions today() and now() which when called with no arguments return the current date and time in your system’s timezone.

# The date of landing and moment of step
date_landing <- mdy("July 20, 1969")
moment_step <- mdy_hms("July 20, 1969, 02:56:15", tz = "UTC")

# How many days since the first man on the moon?
difftime(today(), date_landing, units = "days")

Time difference of 18573 days

# How many seconds since the first man on the moon?
difftime(now(), moment_step, units = "secs")

Time difference of 1604763817 secs

How many seconds are in a day?

How many seconds are in a day? There are 24 hours in a day, 60 minutes in an hour, and 60 seconds in a minute, so there should be 246060 = 86400 seconds, right?

Not always!

# Three dates
mar_11 <- ymd_hms("2017-03-11 12:00:00", 
  tz = "America/Los_Angeles")
mar_12 <- ymd_hms("2017-03-12 12:00:00", 
  tz = "America/Los_Angeles")
mar_13 <- ymd_hms("2017-03-13 12:00:00", 
  tz = "America/Los_Angeles")

# Difference between mar_13 and mar_12 in seconds
difftime(mar_13, mar_12, units = "secs")

Time difference of 86400 secs

# Difference between mar_12 and mar_11 in seconds
difftime(mar_12, mar_11, units = "secs")

Time difference of 82800 secs

Why would a day only have 82800 seconds? At 2am on Mar 12th 2017, Daylight Savings started in the Pacific timezone. That means a whole hour of seconds gets skipped between noon on the 11th and noon on the 12th

Time spans

Adding or subtracting a time span to a datetime

A common use of time spans is to add or subtract them from a moment in time. For, example to calculate the time one day in the future from mar_11 you could do either of:

mar_11 + days(1)
mar_11 + ddays(1)

If you want to account for the fact that time units, in this case days, have different lengths (i.e. due to daylight savings), you want a period days(). If you want the time 86400 seconds in the future you use a duration ddays().

# Add a period of one week to mon_2pm
mon_2pm <- dmy_hm("27 Aug 2018 14:00")
mon_2pm + weeks(1)

[1] "2018-09-03 14:00:00 UTC"

# Add a duration of 81 hours to tue_9am
tue_9am <- dmy_hm("28 Aug 2018 9:00")
tue_9am + hours(81)

[1] "2018-08-31 18:00:00 UTC"

# Subtract a period of five years from today()
today() - years(5)

[1] "2015-05-26"

# Subtract a duration of five years from today()
today() - dyears(5)

[1] "2015-05-28"

Why did subtracting a duration of five years from today, give a different answer to subtracting a period of five years? Periods know about leap years, and since five years ago includes at least one leap year (assuming you aren’t taking this course in 2100) the period of five years is longer than the duration of 365*5 days.

When dealing with human interpretaions of dates and time you want to use periods.

Arithmetic with timespans

You can add and subtract timespans to create different length timespans, and even multiply them by numbers. For example, to create a duration of three days and three hours you could do: ddays(3) + dhours(3), or 3*ddays(1) + 3*dhours(1) or even 3*(ddays(1) + dhours(1)).

There was an eclipse over North America on 2017-08-21 at 18:26:40. It’s possible to predict the next eclipse with similar geometry by calculating the time and date one Saros in the future. A Saros is a length of time that corresponds to 223 Synodic months, a Synodic month being the period of the Moon’s phases, a duration of 29 days, 12 hours, 44 minutes and 3 seconds.

# Time of North American Eclipse 2017
eclipse_2017 <- ymd_hms("2017-08-21 18:26:40")

# Duration of 29 days, 12 hours, 44 mins and 3 secs
synodic <- ddays(29) + dhours(12) + dminutes(44) + dseconds(3)

# 223 synodic months
saros <- 223*synodic

# Add saros to eclipse_2017
eclipse_2017 + saros

[1] "2035-09-02 02:09:49 UTC"

2035 is a long way away for an eclipse, but luckily there are eclipses on different Saros cycles, so you can see one much sooner.

Generating sequences of datetimes

By combining addition and multiplication with sequences you can generate sequences of datetimes. For example, you can generate a sequence of periods from 1 day up to 10 days with:

1:10 * days(1)

Then by adding this sequence to a specific datetime, you can construct a sequence of datetimes from 1 day up to 10 days into the future

today() + 1:10 * days(1)

# Add a period of 8 hours to today
today_8am <- today() + hours(8)

# Sequence of two weeks from 1 to 26
every_two_weeks <- 1:26 * weeks(2)

# Create datetime for every two weeks for a year
every_two_weeks + today_8am

 [1] "2020-06-09 08:00:00 UTC" "2020-06-23 08:00:00 UTC" "2020-07-07 08:00:00 UTC"
 [4] "2020-07-21 08:00:00 UTC" "2020-08-04 08:00:00 UTC" "2020-08-18 08:00:00 UTC"
 [7] "2020-09-01 08:00:00 UTC" "2020-09-15 08:00:00 UTC" "2020-09-29 08:00:00 UTC"
[10] "2020-10-13 08:00:00 UTC" "2020-10-27 08:00:00 UTC" "2020-11-10 08:00:00 UTC"
[13] "2020-11-24 08:00:00 UTC" "2020-12-08 08:00:00 UTC" "2020-12-22 08:00:00 UTC"
[16] "2021-01-05 08:00:00 UTC" "2021-01-19 08:00:00 UTC" "2021-02-02 08:00:00 UTC"
[19] "2021-02-16 08:00:00 UTC" "2021-03-02 08:00:00 UTC" "2021-03-16 08:00:00 UTC"
[22] "2021-03-30 08:00:00 UTC" "2021-04-13 08:00:00 UTC" "2021-04-27 08:00:00 UTC"
[25] "2021-05-11 08:00:00 UTC" "2021-05-25 08:00:00 UTC"

The tricky thing about months

What should ymd(“2018-01-31”) + months(1) return? Should it be 30, 31 or 28 days in the future? Try it. In general lubridate returns the same day of the month in the next month, but since the 31st of February doesn’t exist lubridate returns a missing value, NA.

There are alternative addition and subtraction operators: %m+% and %m-% that have different behavior. Rather than returning an NA for a non-existent date, they roll back to the last existing date.

jan_31 <- as.Date("2020-01-31")

# A sequence of 1 to 12 periods of 1 month
month_seq <- 1:12 * months(1)

# Add 1 to 12 months to jan_31
jan_31 + month_seq

 [1] NA           "2020-03-31" NA           "2020-05-31" NA           "2020-07-31" "2020-08-31"
 [8] NA           "2020-10-31" NA           "2020-12-31" "2021-01-31"

# Replace + with %m+%
jan_31 %m+% month_seq

 [1] "2020-02-29" "2020-03-31" "2020-04-30" "2020-05-31" "2020-06-30" "2020-07-31" "2020-08-31"
 [8] "2020-09-30" "2020-10-31" "2020-11-30" "2020-12-31" "2021-01-31"

# Replace + with %m-%
jan_31 %m-% month_seq

 [1] "2019-12-31" "2019-11-30" "2019-10-31" "2019-09-30" "2019-08-31" "2019-07-31" "2019-06-30"
 [8] "2019-05-31" "2019-04-30" "2019-03-31" "2019-02-28" "2019-01-31"

use these operators with caution, unlike + and -, you might not get x back from x %m+% months(1) %m-% months(1). If you’d prefer that the date was rolled forward check out add_with_rollback() which has roll_to_first argument.

Intervals

Examining intervals. Reigns of kings and queens

You can create an interval by using the operator %–% with two datetimes. For example ymd(“2001-01-01”) %–% ymd(“2001-12-31”) creates an interval for the year of 2001. Once you have an interval you can find out certain properties like its start, end and length with int_start(), int_end() and int_length() respectively.

monarchs <- read.csv("monarchs.csv", stringsAsFactors=FALSE)

# Create an interval for reign
monarchs <- monarchs %>%
  mutate(reign = from %--% to) 

# Find the length of reign, and arrange
monarchs %>%
  mutate(length = int_length (reign)) %>% 
  arrange(desc(length)) %>%
  select(name, length, dominion)

The current queen, Elizabeth II, has ruled for 2070144000 seconds…you’ll see a better way to display the length later. If you know your British monarchs, you might notice George III doesn’t appear in the the top 5. In this data, his reign is spread over two rows for U.K. And Great Britain and you would need to add their lengths to see his total reign.

Comparing intervals and datetimes

A common task with intervals is to ask if a certain time is inside the interval or whether it overlaps with another interval.

The operator %within% tests if the datetime (or interval) on the left hand side is within the interval of the right hand side.

For example, if y2001 is the interval covering the year 2001,

y2001 <- ymd("2001-01-01") %--% ymd("2001-12-31")

Then ymd(“2001-03-30”) %within% y2001 will return TRUE and ymd(“2002-03-30”) %within% y2001 will return FALSE.

int_overlaps() performs a similar test, but will return true if two intervals overlap at all.

halleys <- read_csv("halleys.csv")

Parsed with column specification:
cols(
  designation = col_character(),
  year = col_double(),
  perihelion_date = col_date(format = ""),
  start_date = col_date(format = ""),
  end_date = col_date(format = ""),
  distance = col_character(),
  visible = col_character()
)

# Print halleys
head(halleys)

# New column for interval from start to end date
halleys <- halleys %>% 
  mutate(visible = start_date %--% end_date)

# The visitation of 1066
halleys_1066 <- halleys[14, ] 

# Monarchs in power on perihelion date
monarchs %>% 
  filter(halleys_1066$perihelion_date %within% reign) %>%
  select(name, from, to, dominion)

# Monarchs whose reign overlaps visible time
monarchs %>% 
  filter(int_overlaps(halleys_1066$visible, reign)) %>%
  select(name, from, to, dominion)

Looks like the Kings of England Edward the Confessor and Harold II would have been able to see the comet. It may have been a bad omen, neither were in power by 1067.

Converting to durations and periods

Intervals are the most specific way to represent a span of time since they retain information about the exact start and end moments. They can be converted to periods and durations exactly: it’s possible to calculate both the exact number of seconds elapsed between the start and end date, as well as the perceived change in clock time.

To do so you use the as.period(), and as.duration() functions, parsing in an interval as the only argument.

# New columns for duration and period
monarchs <- monarchs %>%
  mutate(
    duration = as.duration(reign),
    period = as.period(reign)) 
    
# Examine results    
monarchs %>%
  select(name, duration, period)

See how much easier it is to interpret the length of their reigns as periods or durations.

Problems in practice

Timezones

Setting the timezone

If you import a datetime and it has the wrong timezone, you can set it with force_tz(). Pass in the datetime as the first argument and the appropriate timezone to the tzone argument. Remember the timezone needs to be one from OlsonNames().

# Game2: CAN vs NZL in Edmonton
game2 <- mdy_hm("June 11 2015 19:00")

# Game3: CHN vs NZL in Winnipeg
game3 <- mdy_hm("June 15 2015 18:30")

# Set the timezone to "America/Edmonton"
game2_local <- force_tz(game2, tzone = "America/Edmonton")
game2_local

[1] "2015-06-11 19:00:00 MDT"

# Set the timezone to "America/Winnipeg"
game3_local <- force_tz(game3, tzone = "America/Winnipeg")
game3_local

[1] "2015-06-15 18:30:00 CDT"

# How long does the team have to rest?
as.period(game2_local %--% game3_local)

[1] "3d 22H 30M 0S"

Edmonton and Winnipeg are in different timezones, so even though the start times of the games only look 30 minutes apart, they are in fact 1 hour and 30 minutes apart, and the team only has 3 days, 22 hours and 30 minutes to prepare.

Viewing in a timezone

To view a datetime in another timezone use with_tz(). The syntax of with_tz() is the same as force_tz(), passing a datetime and set the tzone argument to the desired timezone. Unlike force_tz(), with_tz() isn’t changing the underlying moment of time, just how it is displayed.

# What time is game2_local in NZ?
with_tz(game2_local, tzone = "Pacific/Auckland")

[1] "2015-06-12 13:00:00 NZST"

# What time is game2_local in Corvallis, Oregon?
with_tz(game2_local, tzone = "America/Los_Angeles")

[1] "2015-06-11 18:00:00 PDT"

# What time is game3_local in NZ?
with_tz(game3_local, tzone = "Pacific/Auckland")

[1] "2015-06-16 11:30:00 NZST"

Timezones in the weather data

Did you ever notice that in the hourly Auckland weather data there was another datetime column, date_utc? Take a look:

tibble::glimpse(akl_hourly)

The datetime column you created represented local time in Auckland, NZ. I suspect this additional column, date_utc represents the observation time in UTC (the name seems a big clue). But does it really?

Use your new timezone skills to find out.

# Examine datetime and date_utc columns
head(akl_hourly$datetime)

[1] "2016-01-01 00:00:00 UTC" "2016-01-01 00:30:00 UTC" "2016-01-01 01:00:00 UTC"
[4] "2016-01-01 01:30:00 UTC" "2016-01-01 02:00:00 UTC" "2016-01-01 02:30:00 UTC"

head(akl_hourly$date_utc)

[1] "2015-12-31 11:00:00 UTC" "2015-12-31 11:30:00 UTC" "2015-12-31 12:00:00 UTC"
[4] "2015-12-31 12:30:00 UTC" "2015-12-31 13:00:00 UTC" "2015-12-31 13:30:00 UTC"

  
# Force datetime to Pacific/Auckland
akl_hourly <- akl_hourly %>%
  mutate(
    datetime = force_tz(datetime, tzone = "Pacific/Auckland"))

# Reexamine datetime
head(akl_hourly$datetime)

[1] "2016-01-01 00:00:00 NZDT" "2016-01-01 00:30:00 NZDT" "2016-01-01 01:00:00 NZDT"
[4] "2016-01-01 01:30:00 NZDT" "2016-01-01 02:00:00 NZDT" "2016-01-01 02:30:00 NZDT"

  
# Are datetime and date_utc the same moments
table(akl_hourly$datetime - akl_hourly$date_utc)

-82800      0   3600 
     2  17450      2 

Looks like for 17,450 rows datetime and date_utc describe the same moment, but for 4 rows they are different. Can you guess which? Yup, the times where DST kicks in.

Times without dates

Sometimes we just have a time without a date.

In this situation, the hms package provides an hms class of object for holding times without dates, and the best place to start would be with as.hms().

# Import auckland hourly data 
akl_hourly <- read_csv("akl_weather_hourly_2016.csv")

Parsed with column specification:
cols(
  year = col_double(),
  month = col_double(),
  mday = col_double(),
  time = col_time(format = ""),
  temperature = col_double(),
  weather = col_character(),
  conditions = col_character(),
  events = col_character(),
  humidity = col_double(),
  date_utc = col_datetime(format = "")
)

# Examine structure of time column
str(akl_hourly$time)

 'hms' num [1:17454] 00:00:00 00:30:00 01:00:00 01:30:00 ...
 - attr(*, "units")= chr "secs"

# Examine head of time column
head(akl_hourly$time)

00:00:00
00:30:00
01:00:00
01:30:00
02:00:00
02:30:00

# A plot using just time
ggplot(akl_hourly, aes(x = time, y = temperature)) +
  geom_line(aes(group = make_date(year, month, mday)), alpha = 0.2)

Using time without date is a great way to examine daily patterns.

More on importing and exporting datetimes

Fast parsing with fasttime

The fasttime package provides a single function fastPOSIXct(), designed to read in datetimes formatted according to ISO 8601. Because it only reads in one format, and doesn’t have to guess a format, it is really fast!

You’ll see how fast in this exercise by comparing how fast it reads in the dates from the Auckland hourly weather data (over 17,000 dates) to lubridates ymd_hms().

To compare run times you’ll use the microbenchmark() function from the package of the same name. You pass in as many arguments as you want each being an expression to time.

library(microbenchmark)
library(fasttime)

dates <- readRDS("dates.rds")

# Examine structure of dates
str(dates)

 chr [1:17454] "2015-12-31T11:00:00Z" "2015-12-31T11:30:00Z" "2015-12-31T12:00:00Z" ...

# Use fastPOSIXct() to parse dates
fastPOSIXct(dates) %>% str()

 POSIXct[1:17454], format: "2015-12-31 11:00:00" "2015-12-31 11:30:00" "2015-12-31 12:00:00" "2015-12-31 12:30:00" ...

# Compare speed of fastPOSIXct() to ymd_hms()
microbenchmark(
  ymd_hms = ymd_hms(dates),
  fasttime = fastPOSIXct(dates),
  times = 20)

Unit: milliseconds

To compare speed, you can compare the average run time in the mean column. You should see fasttime is about 20 times faster than ymd_hms().

Fast parsing with lubridate::fast_strptime

lubridate provides its own fast datetime parser: fast_strptime(). Instead of taking an order argument like parse_date_time() it takes a format argument and the format must comply with the strptime() style.

# Head of dates
head(dates)

[1] "2015-12-31T11:00:00Z" "2015-12-31T11:30:00Z" "2015-12-31T12:00:00Z" "2015-12-31T12:30:00Z"
[5] "2015-12-31T13:00:00Z" "2015-12-31T13:30:00Z"

# Parse dates with fast_strptime
fast_strptime(dates, 
    format = "%Y-%m-%dT%H:%M:%SZ") %>% str()

 POSIXlt[1:17454], format: "2015-12-31 11:00:00" "2015-12-31 11:30:00" "2015-12-31 12:00:00" "2015-12-31 12:30:00" ...

# Comparse speed to ymd_hms() and fasttime
microbenchmark(
  ymd_hms = ymd_hms(dates),
  fasttime = fastPOSIXct(dates),
  fast_strptime = fast_strptime(dates, 
    format = "%Y-%m-%dT%H:%M:%SZ"),
  times = 20)

Unit: milliseconds

fast_strptime() is much faster than ymd_hms() but just a little slower than fasttime.

Outputting pretty dates and times

An easy way to output dates is to use the stamp() function in lubridate. stamp() takes a string which should be an example of how the date should be formatted, and returns a function that can be used to format dates.

finished <- "I finished 'Dates and Times in R' on Thursday, September 4, 2017!"

# Create a stamp based on "Saturday, Jan 1, 2000"
date_stamp <- stamp("Saturday, Jan 1, 2000")

Multiple formats matched: "%A, %b %d, %Y"(1), "Saturday, Jan %Om, %Y"(1), "Saturday, %Om %d, %Y"(1), "Saturday, %b %d, %Y"(1), "Saturday, Jan %m, %Y"(1), "%A, Jan %Om, %Y"(0), "%A, %Om %d, %Y"(0), "%A, Jan %m, %Y"(0)
Using: "%A, %b %d, %Y"

# Print date_stamp
date_stamp

function (x, locale = "English_United Kingdom.1252") 
{
    {
        old_lc_time <- Sys.getlocale("LC_TIME")
        if (old_lc_time != locale) {
            on.exit(Sys.setlocale("LC_TIME", old_lc_time))
            Sys.setlocale("LC_TIME", locale)
        }
    }
    format(x, format = "%A, %b %d, %Y")
}
<environment: 0x000002aa3f3ecc10>

# Call date_stamp on today
date_stamp(today())

[1] "Tuesday, May 26, 2020"

# Create and call a stamp based on "12/31/1999"
stamp("12/31/1999")(today())

Multiple formats matched: "%Om/%d/%Y"(1), "%m/%d/%Y"(1)
Using: "%Om/%d/%Y"

[1] "05/26/2020"

# Use string finished for stamp()
stamp(finished)(today())

Multiple formats matched: "I finished 'Dates and Times in R' on %A, %B %d, %Y!"(1), "I finished 'Dates and Times in R' on Thursday, September %Om, %Y!"(1), "I finished 'Dates and Times in R' on Thursday, %Om %d, %Y!"(1), "I finished 'Dates and Times in R' on Thursday, %B %d, %Y!"(1), "I finished 'Dates and Times in R' on Thursday, September %m, %Y!"(1), "I finished 'Dates and Times in R' on %A, September %Om, %Y!"(0), "I finished 'Dates and Times in R' on %A, %Om %d, %Y!"(0), "I finished 'Dates and Times in R' on %A, September %m, %Y!"(0)
Using: "I finished 'Dates and Times in R' on %A, %B %d, %Y!"

[1] "I finished 'Dates and Times in R' on Tuesday, May 26, 2020!"