Parsing a vector

suppressPackageStartupMessages(library("tidyverse"))

package 㤼㸱tidyverse㤼㸲 was built under R version 3.6.3

1. What are the most important arguments to locale()?

The locale object has arguments to set the following:

• date and time formats: date_names, date_format, and time_format
• time zone: tz
• numbers: `decimal_mark, grouping_mark
• encoding: encoding

2. What happens if you try and set decimal_mark and grouping_mark to the same character? What happens to the default value of grouping_mark when you set decimal_mark to ","? What happens to the default value of decimal_mark when you set the grouping_mark to "."?

If the decimal and grouping marks are set to the same character, locale throws an error:

#locale(decimal_mark = ".", grouping_mark = ".")
#> Error: `decimal_mark` and `grouping_mark` must be different

If the decimal_mark is set to the comma “,”, then the grouping mark is set to the period “.”:

locale(decimal_mark = ",")

<locale>
Numbers:  123.456,78
Formats:  %AD / %AT
Timezone: UTC
Encoding: UTF-8
<date_names>
Days:   Sunday (Sun), Monday (Mon), Tuesday (Tue), Wednesday (Wed), Thursday (Thu), Friday
        (Fri), Saturday (Sat)
Months: January (Jan), February (Feb), March (Mar), April (Apr), May (May), June (Jun),
        July (Jul), August (Aug), September (Sep), October (Oct), November (Nov),
        December (Dec)
AM/PM:  AM/PM

If the grouping mark is set to a period, then the decimal mark is set to a comma

locale(grouping_mark = ",")

<locale>
Numbers:  123,456.78
Formats:  %AD / %AT
Timezone: UTC
Encoding: UTF-8
<date_names>
Days:   Sunday (Sun), Monday (Mon), Tuesday (Tue), Wednesday (Wed), Thursday (Thu), Friday
        (Fri), Saturday (Sat)
Months: January (Jan), February (Feb), March (Mar), April (Apr), May (May), June (Jun),
        July (Jul), August (Aug), September (Sep), October (Oct), November (Nov),
        December (Dec)
AM/PM:  AM/PM

3. I didn’t discuss the date_format and time_format options to locale(). What do they do? Construct an example that shows when they might be useful.

They provide default date and time formats. The readr vignette discusses using these to parse dates: since dates can include languages specific weekday and month names, and different conventions for specifying AM/PM

locale()

<locale>
Numbers:  123,456.78
Formats:  %AD / %AT
Timezone: UTC
Encoding: UTF-8
<date_names>
Days:   Sunday (Sun), Monday (Mon), Tuesday (Tue), Wednesday (Wed), Thursday (Thu), Friday
        (Fri), Saturday (Sat)
Months: January (Jan), February (Feb), March (Mar), April (Apr), May (May), June (Jun),
        July (Jul), August (Aug), September (Sep), October (Oct), November (Nov),
        December (Dec)
AM/PM:  AM/PM

Examples from the readr vignette of parsing French dates

parse_date("1 janvier 2015", "%d %B %Y", locale = locale("fr"))

[1] "2015-01-01"

parse_date("14 oct. 1979", "%d %b %Y", locale = locale("fr"))

[1] "1979-10-14"

Apparently the time format is not used for anything, but the date format is used for guessing column types.

4. If you live outside the US, create a new locale object that encapsulates the settings for the types of file you read most commonly.

Read the help page for locale() using ?locale to learn about the different variables that can be set.

As an example, consider Australia. Most of the defaults values are valid, except that the date format is “(d)d/mm/yyyy”, meaning that January 2, 2006 is written as 02/01/2006.

However, default locale will parse that date as February 1, 2006.

parse_date("02/01/2006")

1 parsing failure.
row col   expected     actual
  1  -- date like  02/01/2006

[1] NA

To correctly parse Australian dates, define a new locale object.

au_locale <- locale(date_format = "%d/%m/%Y")

Using parse_date() with the au_locale as its locale will correctly parse our example date.

parse_date("02/01/2006", locale = au_locale)

[1] "2006-01-02"

5. What’s the difference between read_csv() and read_csv2()?

The delimiter. The function read_csv() uses a comma, while read_csv2() uses a semi-colon (;). Using a semi-colon is useful when commas are used as the decimal point (as in Europe).

6. What are the most common encodings used in Europe? What are the most common encodings used in Asia? Do some googling to find out.

UTF-8 is standard now, and ASCII has been around forever.

For the European languages, there are separate encodings for Romance languages and Eastern European languages using Latin script, Cyrillic, Greek, Hebrew, Turkish: usually with separate ISO and Windows encoding standards. There is also Mac OS Roman.

For Asian languages Arabic and Vietnamese have ISO and Windows standards. The other major Asian scripts have their own:

• Japanese: JIS X 0208, Shift JIS, ISO-2022-JP
• Chinese: GB 2312, GBK, GB 18030
• Korean: KS X 1001, EUC-KR, ISO-2022-KR

The list in the documentation for stringi::stri_enc_detect() is a good list of encodings since it supports the most common encodings.

• Western European Latin script languages: ISO-8859-1, Windows-1250 (also CP-1250 for code-point)
• Eastern European Latin script languages: ISO-8859-2, Windows-1252
• Greek: ISO-8859-7
• Turkish: ISO-8859-9, Windows-1254
• Hebrew: ISO-8859-8, IBM424, Windows 1255
• Russian: Windows 1251
• Japanese: Shift JIS, ISO-2022-JP, EUC-JP
• Korean: ISO-2022-KR, EUC-KR
• Chinese: GB18030, ISO-2022-CN (Simplified), Big5 (Traditional)
• Arabic: ISO-8859-6, IBM420, Windows 1256

For more information on character encodings see the following sources.

• The Wikipedia page Character encoding, has a good list of encodings.
• Unicode CLDR project
• What is the most common encoding of each language (Stack Overflow)
• “What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text”, http://kunststube.net/encoding/.

Programs that identify the encoding of text include:

• readr::guess_encoding()
• stringi::str_enc_detect()
• iconv
• chardet (Python)

7. Generate the correct format string to parse each of the following dates and times:

d1 <- "January 1, 2010"
d2 <- "2015-Mar-07"
d3 <- "06-Jun-2017"
d4 <- c("August 19 (2015)", "July 1 (2015)")
d5 <- "12/30/14" # Dec 30, 2014
t1 <- "1705"
t2 <- "11:15:10.12 PM"

The correct formats are:

parse_date(d1, "%B %d, %Y")

[1] "2010-01-01"

parse_date(d2, "%Y-%b-%d")

[1] "2015-03-07"

parse_date(d3, "%d-%b-%Y")

[1] "2017-06-06"

parse_date(d4, "%B %d (%Y)")

[1] "2015-08-19" "2015-07-01"

parse_date(d5, "%m/%d/%y")

[1] "2014-12-30"

parse_time(t1, "%H%M")

17:05:00

The time t2 uses real seconds,

parse_time(t2, "%H:%M:%OS %p")

23:15:10.12