EDU identification using latent semantic analysis

```r
# Function to split text into sentences
split_into_sentences <- function(text) {
  # ensure there is one text 
  text <- str_c(text, collapse = " ")
  
  # remove new line
  text <- gsub("[\r\n]", "", text)
  
  # Create an NLP annotation object
  annotator <- Maxent_Sent_Token_Annotator()
  
  # Tokenize the sentences
  sentence_annotations <- NLP::annotate(text, list(sent_token_annotator=annotator))

  # split in letters
  text_split <- str_split_1(text, pattern = "")  
  
  # Extract the sentences
  sentences <- map_vec(seq(nrow(as_tibble(sentence_annotations))), 
    ~str_c(text_split[sentence_annotations$start[.]:sentence_annotations$end[.]], collapse = "")
  ) 
  
  return(sentences)
}

# Performs latent semantic analysis
get_lsa <- function(sentences) {
  # Lemmatize each sentence
  lemmatized_sentences <- map_vec(sentences, 
                              ~udpipe_annotate(ud_model, x = .) %>% 
              as.data.frame() %>% 
              pull(lemma) %>% 
              str_c(collapse = " "))

  # Preprocess the text: convert to lowercase, remove punctuation and numbers
  preprocessed_sentences <- tolower(lemmatized_sentences)
  preprocessed_sentences <- removePunctuation(preprocessed_sentences)
  preprocessed_sentences <- removeNumbers(preprocessed_sentences)
  preprocessed_sentences <- removeWords(preprocessed_sentences, stopwords("english"))
  preprocessed_sentences <- str_trim(preprocessed_sentences)
  preprocessed_sentences <- gsub("\\s{2,}", " ", preprocessed_sentences, perl = TRUE)
  
  # Create a document-term matrix
  corpus <- Corpus(VectorSource(preprocessed_sentences))
  dtm <- DocumentTermMatrix(corpus)
  
  # Perform latent semantic analysis
  lsa_space <- lsa(dtm)
  return(lsa_space)
}
  
# Finds keywords for topics
get_topics <- function(lsa_space, threshold = .4){
  # Normalize signs in the dk_matrix
  dk_matrix <- apply(lsa_space$dk, 2, function(col) col * sign(col[which.max(abs(col))]))
  
  # Ensure all values are non-negative
  dk_matrix <- pmax(dk_matrix, 0)

  topics <- dk_matrix %>% 
    as.data.frame() %>% 
    rownames_to_column("key_words") %>% 
    as_tibble() %>% 
    pivot_longer(-key_words, names_to = "topic", values_to = "weight") %>% 
    filter(weight >= threshold) %>% 
    mutate(across(topic, ~str_remove(., "V"))) %>% 
    arrange(topic, weight)
  
  return(topics)
}

# orders the sentences by topics
get_sentences_by_topics <- function(lsa_space, threshold = .4){
  lsa_space$tk
  # Normalize signs in the tk_matrix
  tk_matrix <- apply(lsa_space$tk, 2, function(col) col * sign(col[which.max(abs(col))]))
  
  # Ensure all values are non-negative
  tk_matrix <- pmax(tk_matrix, 0)
  
  # Identify related sentences
  tk_fin <- tk_matrix %>% 
    as.data.frame() %>% 
    rownames_to_column("sentence_id") %>% 
    pivot_longer(-sentence_id, names_to = "topic", values_to = "weight") %>% 
    filter(weight >= threshold) %>% 
    mutate(across(topic, ~str_remove(., "V"))) %>% 
    arrange(topic, weight)
  
  return(tk_fin)  
}
```

```r
prompt <- "Summarize the different viewpoints regarding genetically modified (GM) food based on the two provided texts. Include references to the texts and share your own perspective on the topic."

text_1 <- "TEXT 1: Genetically modified food: saving lives, or lining  corporate pockets? (Tom Chivers - The Telegraph, 2014) Genetically modified (GM) crops have not overcome widespread resistance mostly because the industry is tightly controlled by 
biotech companies. That is, the real problem is that genetic engineering is hurting the poor. It makes cotton cheaper to grow 
for highly subsidized American producers, further undercutting the price of cotton and forcing West African producers out of business. Major biotech companies have no financial interest in developing them for African crops and tightly control the technology. The 
methods of transferring genes were developed by universities, but companies now hold the licenses. The companies permit others to do 
research with the technologies but want control over any product commercialized as a result. Several poor nations are trying to 
develop improved versions of local crops, but these efforts have been damaged by the companies' control over the technology. In fact, the companies which develop GM technology will have unprecedented power over the food chain. They have a clear 
battle-plan to achieve their goal of 'consolidation of the entire food chain': an aggressive patenting regime, patenting technologies 
and genetic material. Academic work has shown that how well people are fed is less to do with the actual quantity of food available in 
the world, and more to do with who controls the food chain, and how well the food is distributed. GM, and the ability to patent GM 
technology, place far more power in the hands of major companies. As a result, there would be fewer competitors in the market. These biotech companies might also have more political power and 
might be able to influence safety and health standards."

text_2 <- 'TEXT 2: The Deadly Opposition to Genetically Modified Food (Bjorn Lomborg – Project Syndicate, 2013) Three billion people depend on rice as their staple food, with 
10% at risk for vitamin A deficiency, which causes 250,000 to 500,000 children to go blind each year. Of these, half die within a 
year. A British medical study estimates that, in total, vitamin A deficiency kills 668,000 children under the age of 5 each year. Yet, despite the cost in human lives, anti-GM campaigners have denied efforts to use golden rice to avoid vitamin A deficiency. 
Indian environmental activist, Vandana Shiva, called golden rice "a hoax" that is "creating hunger and malnutrition, not solving it. The NY Times Magazine reported in 2001 that one would need to 
"eat 15 pounds of cooked golden rice a day" to get enough vitamin A. However, two recent studies in the American Journal of Clinical 
Nutrition show that just 50 grams (roughly two ounces) of golden rice can provide 60 percent of the recommended daily intake of 
vitamin A. They show that golden rice is even better than spinach in providing vitamin A to children. Opponents maintain that there are better ways to deal with 
vitamin A deficiency, saying that golden rice is "neither needed nor necessary." They call for supplementation (vitamin pills) and 
fortification (adding vitamin A to staple products), which are described as "cost-effective." However, this is not a sustainable 
solution to vitamin A deficiency. And, while it is cost-effective, recent published estimates indicate that golden rice is much more 
so. Supplementation programs costs $4,300 for every life they save in India, whereas fortification programs cost about $2,700 for each 
life saved. Meanwhile, golden rice would cost just $100 for every life saved from vitamin A deficiency."'

texts <- c(prompt, text_1, text_2)
```