Load Packages and Libraries

packages=c('dplyr', 'tidyverse', 'tidytext', 'ggplot2', 'ggraph', 'knitr', 'quRan')
for (p in packages){
  if (! require (p,character.only = T)){
    install.packages(p)
  }
library(p,character.only = T)
}

Focus on Selected Quran version and Variables

The quRan packaage has 4 versions of the Quran.

1. quran_ar
2. quran_ar_min
3. quran_en_sahih
4. quran_en_yusufali

We will analyze selected variable (columns) from quran_en_sahih.

quranES <- quran_en_sahih %>% select(surah_id, 
                                   ayah_id,
                                   surah_title_en, 
                                   surah_title_en_trans, 
                                   revelation_type, 
                                   text,
                                   ayah_title)
quranES

To work with this as a tidy dataset, we need to restructure it in the one-token-per-row format. This is done with the unnest_tokens() function.

tidyES <- quranES %>%
  unnest_tokens(word, text)
tidyES

This function uses the tokenizers package to separate each line of text in the original data frame into tokens. The default tokenizing is for words, but other options include characters, n-grams, sentences, lines, paragraphs, or separation around a regex pattern.

Now that the data is in one-word-per-row format, we can manipulate it with tidy tools like dplyr.

Term frequency in English Quran

We start by looking at the Surahs of the Quran and examine first term frequency, and then tf-idf. We can start just by using dplyr verbs such as group_by() and join(). What are the most commonly used words in the Surahs? We also calculate the total words in each Surah here, for later use.

surah_words <- quranES %>%
  unnest_tokens(word, text) %>%
  count(surah_title_en, word, sort = TRUE)
surah_words

total_words <- surah_words %>% 
  group_by(surah_title_en) %>% 
  summarize(total = sum(n))

surah_words <- left_join(surah_words, total_words)

surah_words

There is one row in this surah_words data frame for each word-surah combination; n is the number of times that word is used in that surah and total is the total words in that surah. The usual suspects are here with the highest n, “the”, “and”, “to”, and so forth. In the following figures, we look at the distribution of n/total for selected long surahs (with > 150 verses), the number of times a word appears in a surah divided by the total number of terms (words) in that surah. This is exactly what term frequency is.

# library(ggplot2)
surah_words %>% filter(surah_title_en %in% 
            c("Al-Baqara", "Aal-i-Imraan", "An-Nisaa", "Al-Maaida", "Al-An'aam", "Al-A'raaf")) %>%
            ggplot(aes(n/total, fill = surah_title_en)) +
            geom_histogram(show.legend = TRUE) +
            xlim(NA, 0.01) +
            facet_wrap(~surah_title_en, ncol = 2, scales = "free_y") +
            labs(title = "Term frequency distribution of the 6 Long Surahs",
                 x = "(Word Frequency)/(Total Words)",
                 y = "Count")

There are long tails to the right for these selected surahs (those extremely rare words!) that we have not shown in these plots.

We repeat the same with the Hamim Surahs that are betwwen 4 to 10 pages each, medium length Surahs.

surah_words %>% filter(surah_title_en %in% 
            c("Ghafir", "Fussilat", "Ash-Shura", "Az-Zukhruf", "Ad-Dukhaan", "Al-Jaathiya")) %>%
            ggplot(aes(n/total, fill = surah_title_en)) +
            geom_histogram(show.legend = TRUE) +
            xlim(NA, 0.01) +
            facet_wrap(~surah_title_en, ncol = 2, scales = "free_y") +
            labs(title = "Term frequency distribution of 6 Medium Surahs",
                 x = "(Word Frequency)/(Total Words)",
                 y = "Count")

These plots exhibit similar distributions for all the surahs, with many words that occur rarely and fewer words that occur frequently.

The bind_tf_idf function

The idea of tf-idf is to find the important words for the content of each Surah by decreasing the weight for commonly used words and increasing the weight for words that are not used very much in a collection or corpus of documents, in this case, the group of Quran Surahs as a whole. Calculating tf-idf attempts to find the words that are important (i.e., common) in a Surah, but not too common. Let’s do that now.

The bind_tf_idf function in the tidytext package takes a tidy text dataset as input with one row per token (term), per document. One column (word here) contains the terms/tokens, one column contains the documents (Surah in this case), and the last necessary column contains the counts, how many times each document contains each term (n in this example). We calculated a total for each Surah for our explorations in previous sections, but it is not necessary for the bind_tf_idf function; the table only needs to contain all the words in each document.

surah_words <- surah_words %>%
  bind_tf_idf(word, surah_title_en, n)

surah_words

Notice that idf and thus tf-idf are zero for these extremely common words. These are all words that appear in all 114 of Quran Surahs, so the idf term (which will then be the natural log of 1) is zero. The inverse document frequency (and thus tf-idf) is very low (near zero) for words that occur in many of the documents in a collection; this is how this approach decreases the weight for common words. The inverse document frequency will be a higher number for words that occur in fewer of the documents in the collection.

Let’s look at terms with high tf-idf in Quran Surahs.

surah_words %>%
  select(-total) %>%
  arrange(desc(tf_idf))

Some of the values for idf are the same for different terms because there are 114 Surahs and we are seeing the numerical value for ln(114/1), ln(114/2) etc.

Let’s look at a visualization for these high tf-idf words. We focus on selected long Surahs.

surah_words %>%
  filter(surah_title_en %in% 
            c("Al-Baqara", "Aal-i-Imraan", "An-Nisaa", "Al-Maaida", "Al-An'aam", "Al-A'raaf")) %>%
  arrange(desc(tf_idf)) %>%
  mutate(word = factor(word, levels = rev(unique(word)))) %>% 
  group_by(surah_title_en) %>% 
  top_n(10) %>% 
  ungroup() %>%
  ggplot(aes(word, tf_idf, fill = surah_title_en)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "tf-idf") +
  facet_wrap(~surah_title_en, ncol = 2, scales = "free") +
  coord_flip()

Redo with the last 6 Surahs.

surah_words %>%
  filter(surah_title_en %in% 
            c("An-Naas", "Al-Falaq", "Al-Ikhlaas", "Al-Masad", "An-Nasr", "Al-Kaafiroon")) %>%
  arrange(desc(tf_idf)) %>%
  mutate(word = factor(word, levels = rev(unique(word)))) %>% 
  group_by(surah_title_en) %>% 
  top_n(10) %>% 
  ungroup() %>%
  ggplot(aes(word, tf_idf, fill = surah_title_en)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = "tf-idf") +
  facet_wrap(~surah_title_en, ncol = 2, scales = "free") +
  coord_flip()

Still all proper nouns in the two figures! These words are, as measured by tf-idf, the most important to each Surah and most readers would likely agree. This is the point of tf-idf; it identifies words that are important to one document within a collection of documents.