Introduction:

As an Information Retrieval (IR) Specialist, my role within the project revolves around optimizing the retrieval and searchability of product information to enhance user experience and satisfaction. With a focus on the electronics database, my objectives encompass refining search functionalities, improving product relevance, and facilitating efficient retrieval of relevant information for users. This involves implementing IR techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) and document similarity measures to enhance the search process and elevate the quality of search results.

Task 1:

Extract electronic product data from electronicsdb.xml into a CSV file named ‘Products.csv.’ Ensure accurate conversion of XML elements into CSV columns for ProductID, ProductName, Specifications, CategoryID, and SupplierID.

# Set the working directory
setwd("C:\\Users\\tiffany.ariel.chin\\Downloads")

# Install the xml2 package if not already installed
if (!requireNamespace("xml2", quietly = TRUE)) {
  install.packages("xml2")
}

# Load the xml2 package
library(xml2)
## Warning: package 'xml2' was built under R version 4.3.3
# Read the XML file
xml_data <- read_xml("electronicsdb.xml")

library(magrittr)
## Warning: package 'magrittr' was built under R version 4.3.3
# Extract relevant information
product_ids <- xml_data %>% xml_find_all("//ProductID") %>% xml_text()
product_names <- xml_data %>% xml_find_all("//ProductName") %>% xml_text()
specifications <- xml_data %>% xml_find_all("//Specifications") %>% xml_text()
category_ids <- xml_data %>% xml_find_all("//CategoryID") %>% xml_text()
supplier_ids <- xml_data %>% xml_find_all("//SupplierID") %>% xml_text()

# Create a data frame
products_df <- data.frame(
  ProductID = product_ids,
  ProductName = product_names,
  Specifications = specifications,
  CategoryID = category_ids,
  SupplierID = supplier_ids)

# Write to CSV
write.csv(products_df, file = "Products.csv", row.names = FALSE)

print(products_df)
##    ProductID       ProductName
## 1         41  Alpha Smartphone
## 2         42   Beta Smartphone
## 3         43      Gamma Laptop
## 4         44      Delta Laptop
## 5         45    Epsilon Tablet
## 6         46       Zeta Tablet
## 7         47     Eta Headphone
## 8         48   Theta Headphone
## 9         49  Iota Smart Watch
## 10        50 Kappa Smart Watch
## 11        51    Lambda Console
## 12        52        Mu Console
## 13        53         Nu Camera
## 14        54         Xi Camera
## 15        55   Omicron Printer
## 16        56        Pi Printer
## 17        57       Rho Monitor
## 18        58     Sigma Monitor
## 19        59       Tau Speaker
## 20        60   Upsilon Speaker
## 21        41  Alpha Smartphone
## 22        42   Beta Smartphone
## 23        43      Gamma Laptop
## 24        44      Delta Laptop
## 25        45    Epsilon Tablet
## 26        46       Zeta Tablet
## 27        47     Eta Headphone
## 28        48   Theta Headphone
## 29        49  Iota Smart Watch
## 30        50 Kappa Smart Watch
## 31        51    Lambda Console
## 32        52        Mu Console
## 33        53         Nu Camera
## 34        54         Xi Camera
## 35        55   Omicron Printer
## 36        56        Pi Printer
## 37        57       Rho Monitor
## 38        58     Sigma Monitor
## 39        59       Tau Speaker
## 40        60   Upsilon Speaker
##                                                                                                  Specifications
## 1                         64GB Storage, Black Color, 6.1-inch OLED Display, A14 Bionic Chip, Dual-camera system
## 2                                128GB Storage, White Color, Face ID, A14 Bionic Chip, Water and Dust Resistant
## 3  15-inch Retina display, 256GB SSD, 16GB RAM, 10th-generation Intel Core i7 processor, Touch Bar and Touch ID
## 4                    13-inch Retina display, 512GB SSD, 16GB RAM, 8-core CPU, 8-core GPU, 20 hours battery life
## 5                     10-inch Full HD Display, 64GB Internal Storage, Octa-core Processor, Long-lasting Battery
## 6                             8-inch HD Display, 32GB Internal Storage, Quad-core Processor, Lightweight Design
## 7            Wireless Connectivity, Noise Cancellation, Over-ear Design, 20 Hours Playtime, Built-in Microphone
## 8                        Wired Connectivity, In-Ear Design, Tangle-free Cable, Integrated Microphone, Deep Bass
## 9                          GPS Enabled, Heart Rate Monitor, OLED Retina Display, Water Resistant, Notifications
## 10   Fitness Tracking, Waterproof up to 50m, Sleep Monitoring, High and Low Heart Rate Notifications, Bluetooth
## 11                               1TB Storage, 2 Wireless Controllers, 4K Gaming, HDR Technology, Blu-ray Player
## 12              500GB Storage, VR Ready, 8-Core x86-64 AMD Jaguar CPU, GDDR5 8GB Memory, Blu-ray and DVD Player
## 13     DSLR Camera, 24MP Full-Frame CMOS Sensor, 4K Video Recording, Built-in Wi-Fi, 3.2 inches Touchscreen LCD
## 14                      Mirrorless Camera, 20MP Sensor, Compact Build, Interchangeable Lens, 4K Video Recording
## 15               Inkjet Printer, Wireless Printing, Color Printer, Mobile Printing, Print, Scan, Copy Functions
## 16             Laser Printer, Multifunction with Fax, High-speed Printing, Ethernet Networking, Duplex Printing
## 17               27-inch 4K UHD Monitor, IPS Display, 5ms Response Time, HDMI and DisplayPort, Adjustable Stand
## 18               24-inch Full HD 1080p Monitor, LED Display, 1ms Response Time, HDMI and VGA Ports, Slim Design
## 19                 Bluetooth Connectivity, Waterproof Design, Portable Speaker, 360° Sound, Built-in Microphone
## 20                         Bookshelf Speaker, Wired Connectivity, Wooden Enclosure, 60W RMS Power, 5.25" Woofer
## 21                        64GB Storage, Black Color, 6.1-inch OLED Display, A14 Bionic Chip, Dual-camera system
## 22                               128GB Storage, White Color, Face ID, A14 Bionic Chip, Water and Dust Resistant
## 23 15-inch Retina display, 256GB SSD, 16GB RAM, 10th-generation Intel Core i7 processor, Touch Bar and Touch ID
## 24                   13-inch Retina display, 512GB SSD, 16GB RAM, 8-core CPU, 8-core GPU, 20 hours battery life
## 25                    10-inch Full HD Display, 64GB Internal Storage, Octa-core Processor, Long-lasting Battery
## 26                            8-inch HD Display, 32GB Internal Storage, Quad-core Processor, Lightweight Design
## 27           Wireless Connectivity, Noise Cancellation, Over-ear Design, 20 Hours Playtime, Built-in Microphone
## 28                       Wired Connectivity, In-Ear Design, Tangle-free Cable, Integrated Microphone, Deep Bass
## 29                         GPS Enabled, Heart Rate Monitor, OLED Retina Display, Water Resistant, Notifications
## 30   Fitness Tracking, Waterproof up to 50m, Sleep Monitoring, High and Low Heart Rate Notifications, Bluetooth
## 31                               1TB Storage, 2 Wireless Controllers, 4K Gaming, HDR Technology, Blu-ray Player
## 32              500GB Storage, VR Ready, 8-Core x86-64 AMD Jaguar CPU, GDDR5 8GB Memory, Blu-ray and DVD Player
## 33     DSLR Camera, 24MP Full-Frame CMOS Sensor, 4K Video Recording, Built-in Wi-Fi, 3.2 inches Touchscreen LCD
## 34                      Mirrorless Camera, 20MP Sensor, Compact Build, Interchangeable Lens, 4K Video Recording
## 35               Inkjet Printer, Wireless Printing, Color Printer, Mobile Printing, Print, Scan, Copy Functions
## 36             Laser Printer, Multifunction with Fax, High-speed Printing, Ethernet Networking, Duplex Printing
## 37               27-inch 4K UHD Monitor, IPS Display, 5ms Response Time, HDMI and DisplayPort, Adjustable Stand
## 38               24-inch Full HD 1080p Monitor, LED Display, 1ms Response Time, HDMI and VGA Ports, Slim Design
## 39                 Bluetooth Connectivity, Waterproof Design, Portable Speaker, 360° Sound, Built-in Microphone
## 40                         Bookshelf Speaker, Wired Connectivity, Wooden Enclosure, 60W RMS Power, 5.25" Woofer
##    CategoryID SupplierID
## 1          31         50
## 2          32         50
## 3          33         50
## 4          34         40
## 5          35         40
## 6          36         40
## 7          37         35
## 8          38         35
## 9          39         35
## 10         40         45
## 11         41         45
## 12         42         45
## 13         43         33
## 14         44         33
## 15         45         33
## 16         46         33
## 17         47         50
## 18         48         33
## 19         49         33
## 20         50         33
## 21         31         31
## 22         31         32
## 23         32         33
## 24         32         34
## 25         33         35
## 26         33         36
## 27         34         37
## 28         34         38
## 29         35         39
## 30         35         40
## 31         36         41
## 32         36         42
## 33         37         43
## 34         37         44
## 35         38         45
## 36         38         46
## 37         39         47
## 38         39         48
## 39         40         49
## 40         40         50

Task 2:

Apply text pre-processing techniques to cleanse the ‘Specification’ attribute. Techniques include normalization (e.g., lowercasing), removing stop words, stemming/lemmatization, or eliminating punctuation or numbers, aiming to standardize the text data for analysis.

# Sample data
specifications <- xml_data %>% xml_find_all("//Specifications") %>% xml_text()

# Function to preprocess text
preprocess_text <- function(text) {
  # Convert text to lowercase
  text <- tolower(text)
  # Remove punctuation
  text <- gsub("[[:punct:]]", "", text)
  # Remove numbers
  text <- gsub("\\d+", "", text)
  # Remove stopwords (custom list)
  stopwords <- c("this", "is", "are", "with", "more", "and", "or", "the")  # Example custom stopword list
  text <- paste0(text[!tolower(text) %in% stopwords], collapse = " ")
  # Perform stemming (basic implementation)
  text <- gsub("(ing|ly|ed)$", "", text)
  return(text)
}

# Apply text preprocessing to each specification
cleaned_specifications <- sapply(specifications, preprocess_text)

# View the cleaned specifications
print(cleaned_specifications)
##                        64GB Storage, Black Color, 6.1-inch OLED Display, A14 Bionic Chip, Dual-camera system 
##                                   "gb storage black color inch oled display a bionic chip dualcamera system" 
##                               128GB Storage, White Color, Face ID, A14 Bionic Chip, Water and Dust Resistant 
##                                      "gb storage white color face id a bionic chip water and dust resistant" 
## 15-inch Retina display, 256GB SSD, 16GB RAM, 10th-generation Intel Core i7 processor, Touch Bar and Touch ID 
##               "inch retina display gb ssd gb ram thgeneration intel core i processor touch bar and touch id" 
##                   13-inch Retina display, 512GB SSD, 16GB RAM, 8-core CPU, 8-core GPU, 20 hours battery life 
##                                    "inch retina display gb ssd gb ram core cpu core gpu  hours battery life" 
##                    10-inch Full HD Display, 64GB Internal Storage, Octa-core Processor, Long-lasting Battery 
##                            "inch full hd display gb internal storage octacore processor longlasting battery" 
##                            8-inch HD Display, 32GB Internal Storage, Quad-core Processor, Lightweight Design 
##                                  "inch hd display gb internal storage quadcore processor lightweight design" 
##           Wireless Connectivity, Noise Cancellation, Over-ear Design, 20 Hours Playtime, Built-in Microphone 
##                 "wireless connectivity noise cancellation overear design  hours playtime builtin microphone" 
##                       Wired Connectivity, In-Ear Design, Tangle-free Cable, Integrated Microphone, Deep Bass 
##                           "wired connectivity inear design tanglefree cable integrated microphone deep bass" 
##                         GPS Enabled, Heart Rate Monitor, OLED Retina Display, Water Resistant, Notifications 
##                           "gps enabled heart rate monitor oled retina display water resistant notifications" 
##   Fitness Tracking, Waterproof up to 50m, Sleep Monitoring, High and Low Heart Rate Notifications, Bluetooth 
##       "fitness tracking waterproof up to m sleep monitoring high and low heart rate notifications bluetooth" 
##                               1TB Storage, 2 Wireless Controllers, 4K Gaming, HDR Technology, Blu-ray Player 
##                                     "tb storage  wireless controllers k gaming hdr technology bluray player" 
##              500GB Storage, VR Ready, 8-Core x86-64 AMD Jaguar CPU, GDDR5 8GB Memory, Blu-ray and DVD Player 
##                             "gb storage vr ready core x amd jaguar cpu gddr gb memory bluray and dvd player" 
##     DSLR Camera, 24MP Full-Frame CMOS Sensor, 4K Video Recording, Built-in Wi-Fi, 3.2 inches Touchscreen LCD 
##                "dslr camera mp fullframe cmos sensor k video recording builtin wifi  inches touchscreen lcd" 
##                      Mirrorless Camera, 20MP Sensor, Compact Build, Interchangeable Lens, 4K Video Recording 
##                              "mirrorless camera mp sensor compact build interchangeable lens k video record" 
##               Inkjet Printer, Wireless Printing, Color Printer, Mobile Printing, Print, Scan, Copy Functions 
##                   "inkjet printer wireless printing color printer mobile printing print scan copy functions" 
##             Laser Printer, Multifunction with Fax, High-speed Printing, Ethernet Networking, Duplex Printing 
##                   "laser printer multifunction with fax highspeed printing ethernet networking duplex print" 
##               27-inch 4K UHD Monitor, IPS Display, 5ms Response Time, HDMI and DisplayPort, Adjustable Stand 
##                      "inch k uhd monitor ips display ms response time hdmi and displayport adjustable stand" 
##               24-inch Full HD 1080p Monitor, LED Display, 1ms Response Time, HDMI and VGA Ports, Slim Design 
##                         "inch full hd p monitor led display ms response time hdmi and vga ports slim design" 
##                 Bluetooth Connectivity, Waterproof Design, Portable Speaker, 360° Sound, Built-in Microphone 
##                        "bluetooth connectivity waterproof design portable speaker  sound builtin microphone" 
##                         Bookshelf Speaker, Wired Connectivity, Wooden Enclosure, 60W RMS Power, 5.25" Woofer 
##                                  "bookshelf speaker wired connectivity wooden enclosure w rms power  woofer"

Task 3:

Create a TermDocumentMatrix to analyze the ‘Specification’ text data. Define the granularity of terms (unigrams or bigrams) and the weighting measure (TF or TF-IDF), justifying the choice based on the analysis objectives.
To choose the best granularity of terms and the weighting measure, we need to consider the analysis objectives and the nature of the text data.

Objective:

Identify key features or characteristics mentioned in the product specifications.

Justification:

Granularity of Terms:

Unigrams (Single Words): Unigrams provide a straightforward representation of individual terms present in the text data. They allow us to capture the most frequently occurring words or terms, which may represent important features or characteristics of the products.
Bigrams (Pairs of Words): Bigrams capture the relationship between adjacent words in the text data. They can help us identify meaningful phrases or combinations of words that may represent specific features or attributes unique to the products.

Weighting Measure:

Term Frequency (TF): TF measures the frequency of each term within each document (product specification). It helps in identifying the importance of terms based on their occurrence frequency in each specification. However, it may not account for the overall importance of terms across all specifications.
Term Frequency-Inverse Document Frequency (TF-IDF): TF-IDF considers both the frequency of a term within a document and its rarity across all documents. It helps in identifying terms that are frequent in a specific document (indicating importance for that document) while being rare across all documents (indicating uniqueness). This measure can help in identifying key features or characteristics that distinguish one product from another.

Choice

Based on the analysis objectives and justifications, the best choice would be to use unigrams and TF-IDF weighting. Unigrams allow me to capture individual terms that represent important features or characteristics mentioned in the product specifications. TF-IDF weighting helps in identifying terms that are both frequent within a specific specification and rare across all specifications,highlighting key distinguishing features of each product.
# Function to create TermDocumentMatrix with unigrams and TF-IDF weighting
cleaned_specifications <- sapply(specifications, preprocess_text)
create_TDM <- function(text_data) {
  # Tokenize the text
  tokens <- strsplit(text_data, "\\W+")
  
  # Create a Document-Term Matrix with unigrams
  dtm <- table(unlist(tokens))
  
  # Convert the Document-Term Matrix to a Term-Document Matrix
  tdm <- t(dtm)
  
  # Compute TF-IDF weighting
  n <- nrow(tdm)
  tf <- tdm / rowSums(tdm)  # Term Frequency
  idf <- log10(n / rowSums(tdm > 0))  # Inverse Document Frequency
  tf_idf <- tf * idf  # TF-IDF weighting
  
  return(tf_idf)
}

# Create TermDocumentMatrix with unigrams and TF-IDF weighting
tdm_unigrams_tfidf <- create_TDM(cleaned_specifications)

# Print the TermDocumentMatrix
print(tdm_unigrams_tfidf)
##       
##                   a   adjustable          amd          and          bar
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.052933213 -0.008822202
##       
##                bass      battery       bionic        black    bluetooth
##   [1,] -0.008822202 -0.017644404 -0.017644404 -0.008822202 -0.017644404
##       
##              bluray    bookshelf        build      builtin        cable
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.026466606 -0.008822202
##       
##              camera cancellation         chip         cmos        color
##   [1,] -0.017644404 -0.008822202 -0.017644404 -0.008822202 -0.026466606
##       
##             compact connectivity  controllers         copy         core
##   [1,] -0.008822202 -0.035288808 -0.008822202 -0.008822202 -0.035288808
##       
##                 cpu         deep       design      display  displayport
##   [1,] -0.017644404 -0.008822202 -0.044111010 -0.070577617 -0.008822202
##       
##                dslr   dualcamera       duplex         dust          dvd
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##             enabled    enclosure     ethernet         face          fax
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##             fitness         full    fullframe    functions       gaming
##   [1,] -0.008822202 -0.017644404 -0.008822202 -0.008822202 -0.008822202
##       
##                  gb         gddr          gps          gpu           hd
##   [1,] -0.088222021 -0.008822202 -0.008822202 -0.008822202 -0.026466606
##       
##                hdmi          hdr        heart         high    highspeed
##   [1,] -0.017644404 -0.008822202 -0.017644404 -0.008822202 -0.008822202
##       
##               hours            i           id         inch       inches
##   [1,] -0.017644404 -0.008822202 -0.017644404 -0.061755415 -0.008822202
##       
##               inear       inkjet   integrated        intel interchangeable
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202    -0.008822202
##       
##            internal          ips       jaguar            k        laser
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.035288808 -0.008822202
##       
##                 lcd          led         lens         life  lightweight
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##         longlasting          low            m       memory   microphone
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.026466606
##       
##          mirrorless       mobile      monitor   monitoring           mp
##   [1,] -0.008822202 -0.008822202 -0.026466606 -0.008822202 -0.017644404
##       
##                  ms multifunction   networking        noise notifications
##   [1,] -0.017644404  -0.008822202 -0.008822202 -0.008822202  -0.017644404
##       
##            octacore         oled      overear            p       player
##   [1,] -0.008822202 -0.017644404 -0.008822202 -0.008822202 -0.017644404
##       
##            playtime     portable        ports        power        print
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.017644404
##       
##             printer     printing    processor     quadcore          ram
##   [1,] -0.026466606 -0.026466606 -0.026466606 -0.008822202 -0.017644404
##       
##                rate        ready       record    recording    resistant
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.008822202 -0.017644404
##       
##            response       retina          rms         scan       sensor
##   [1,] -0.017644404 -0.026466606 -0.008822202 -0.008822202 -0.017644404
##       
##               sleep         slim        sound      speaker          ssd
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.017644404 -0.017644404
##       
##               stand      storage       system   tanglefree           tb
##   [1,] -0.008822202 -0.052933213 -0.008822202 -0.008822202 -0.008822202
##       
##          technology thgeneration         time           to        touch
##   [1,] -0.008822202 -0.008822202 -0.017644404 -0.008822202 -0.017644404
##       
##         touchscreen     tracking          uhd           up          vga
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##               video           vr            w        water   waterproof
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.017644404 -0.017644404
##       
##               white         wifi        wired     wireless         with
##   [1,] -0.008822202 -0.008822202 -0.017644404 -0.026466606 -0.008822202
##       
##              wooden       woofer            x
##   [1,] -0.008822202 -0.008822202 -0.008822202

Task 4:

Display the TDM in a matrix format and report its dimensions. Provide insights into the dataset’s sparsity.

Sparsity in a TermDocumentMatrix refers to the proportion of zero values compared to the total number of cells in the matrix. In text analysis, sparsity indicates how many terms are not present in each document or how many documents do not contain certain terms.
The output indicates that the TermDocumentMatrix (TDM) has dimensions of 1 row and 148 columns, with a sparsity of 0%. Here are some insights into the dataset’s sparsity based on this information:

Sparsity:

The sparsity of 0% indicates that there are no zero values in the TDM. In other words, all cells in the matrix contain non-zero values, meaning that every term from the vocabulary appears at least once in the document collection.

Interpretation:

A sparsity of 0% suggests that the dataset is dense, meaning that every term occurs in at least one document.
In this context, it implies that all terms extracted from the ‘Specifications’ attribute are present in the documents, indicating a comprehensive coverage of the vocabulary across the dataset.

Data Quality:

The absence of sparsity suggests that the text preprocessing and tokenization steps effectively captured all terms from the ‘Specifications’ attribute, indicating high data quality and completeness.
# Create TermDocumentMatrix with unigrams and TF-IDF weighting
tdm_unigrams_tfidf <- create_TDM(cleaned_specifications)

# Display the TermDocumentMatrix in a matrix format
cat("TermDocumentMatrix:\n")
## TermDocumentMatrix:
print(as.matrix(tdm_unigrams_tfidf))
##       
##                   a   adjustable          amd          and          bar
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.052933213 -0.008822202
##       
##                bass      battery       bionic        black    bluetooth
##   [1,] -0.008822202 -0.017644404 -0.017644404 -0.008822202 -0.017644404
##       
##              bluray    bookshelf        build      builtin        cable
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.026466606 -0.008822202
##       
##              camera cancellation         chip         cmos        color
##   [1,] -0.017644404 -0.008822202 -0.017644404 -0.008822202 -0.026466606
##       
##             compact connectivity  controllers         copy         core
##   [1,] -0.008822202 -0.035288808 -0.008822202 -0.008822202 -0.035288808
##       
##                 cpu         deep       design      display  displayport
##   [1,] -0.017644404 -0.008822202 -0.044111010 -0.070577617 -0.008822202
##       
##                dslr   dualcamera       duplex         dust          dvd
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##             enabled    enclosure     ethernet         face          fax
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##             fitness         full    fullframe    functions       gaming
##   [1,] -0.008822202 -0.017644404 -0.008822202 -0.008822202 -0.008822202
##       
##                  gb         gddr          gps          gpu           hd
##   [1,] -0.088222021 -0.008822202 -0.008822202 -0.008822202 -0.026466606
##       
##                hdmi          hdr        heart         high    highspeed
##   [1,] -0.017644404 -0.008822202 -0.017644404 -0.008822202 -0.008822202
##       
##               hours            i           id         inch       inches
##   [1,] -0.017644404 -0.008822202 -0.017644404 -0.061755415 -0.008822202
##       
##               inear       inkjet   integrated        intel interchangeable
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202    -0.008822202
##       
##            internal          ips       jaguar            k        laser
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.035288808 -0.008822202
##       
##                 lcd          led         lens         life  lightweight
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##         longlasting          low            m       memory   microphone
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.026466606
##       
##          mirrorless       mobile      monitor   monitoring           mp
##   [1,] -0.008822202 -0.008822202 -0.026466606 -0.008822202 -0.017644404
##       
##                  ms multifunction   networking        noise notifications
##   [1,] -0.017644404  -0.008822202 -0.008822202 -0.008822202  -0.017644404
##       
##            octacore         oled      overear            p       player
##   [1,] -0.008822202 -0.017644404 -0.008822202 -0.008822202 -0.017644404
##       
##            playtime     portable        ports        power        print
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.017644404
##       
##             printer     printing    processor     quadcore          ram
##   [1,] -0.026466606 -0.026466606 -0.026466606 -0.008822202 -0.017644404
##       
##                rate        ready       record    recording    resistant
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.008822202 -0.017644404
##       
##            response       retina          rms         scan       sensor
##   [1,] -0.017644404 -0.026466606 -0.008822202 -0.008822202 -0.017644404
##       
##               sleep         slim        sound      speaker          ssd
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.017644404 -0.017644404
##       
##               stand      storage       system   tanglefree           tb
##   [1,] -0.008822202 -0.052933213 -0.008822202 -0.008822202 -0.008822202
##       
##          technology thgeneration         time           to        touch
##   [1,] -0.008822202 -0.008822202 -0.017644404 -0.008822202 -0.017644404
##       
##         touchscreen     tracking          uhd           up          vga
##   [1,] -0.008822202 -0.008822202 -0.008822202 -0.008822202 -0.008822202
##       
##               video           vr            w        water   waterproof
##   [1,] -0.017644404 -0.008822202 -0.008822202 -0.017644404 -0.017644404
##       
##               white         wifi        wired     wireless         with
##   [1,] -0.008822202 -0.008822202 -0.017644404 -0.026466606 -0.008822202
##       
##              wooden       woofer            x
##   [1,] -0.008822202 -0.008822202 -0.008822202
# Report the dimensions of the TermDocumentMatrix
cat("\nDimensions of TermDocumentMatrix:", dim(tdm_unigrams_tfidf), "\n")
## 
## Dimensions of TermDocumentMatrix: 1 148
# Calculate sparsity of the TermDocumentMatrix
sparsity <- sum(tdm_unigrams_tfidf == 0) / length(tdm_unigrams_tfidf)
cat("\nSparsity of the TermDocumentMatrix:", sparsity * 100, "%\n")
## 
## Sparsity of the TermDocumentMatrix: 0 %

Task 5:

Apply a similarity measure, such as cosine similarity, to identify products with specifications most similar to those of the ‘Kappa Smart Watch’.Interpret the similarity scores to rank products by relevance.

To apply a similarity measure like cosine similarity to identify products with specifications most similar to those of the ‘Kappa Smart Watch’, you first need to represent the product specifications as numerical vectors. Then, you can compute the cosine similarity between the vector representing the ‘Kappa Smart Watch’ and the vectors representing other products. Higher cosine similarity scores indicate greater similarity between product specifications.

My Approach:

1. Preprocess the specifications data to prepare it for similarity calculation.
2. Compute the cosine similarity between the specifications of the ‘Kappa Smart Watch’ and all other products.
3. Interpret the similarity scores to rank products by relevance.
# Preprocess the specifications data
preprocess_text <- function(text) {
  text <- tolower(text)  # Convert text to lowercase
  text <- gsub("[[:punct:]]", "", text)  # Remove punctuation
  text <- gsub("\\d+", "", text)  # Remove numbers
  text <- gsub("\\s+", " ", text)  # Remove extra white spaces
  return(text)
}

# Apply preprocessing to the specifications
products_df$CleanedSpecifications <- sapply(products_df$Specifications, preprocess_text)

# Find the index of 'Kappa Smart Watch'
kappa_index <- which(products_df$ProductName == "Kappa Smart Watch")

# Extract the cleaned specifications of 'Kappa Smart Watch'
kappa_specifications <- products_df$CleanedSpecifications[kappa_index]

# Compute cosine similarity with other products
cosine_similarities <- sapply(products_df$CleanedSpecifications, function(spec) {
  # Compute dot product
  dot_product <- sum(sapply(strsplit(kappa_specifications, " "), function(word) {
    sum(word %in% strsplit(spec, " ")[[1]])
  }))
  # Compute magnitudes
  kappa_magnitude <- sqrt(sum(sapply(strsplit(kappa_specifications, " "), function(word) {
    length(word)^2
  })))
  spec_magnitude <- sqrt(sum(sapply(strsplit(spec, " "), function(word) {
    length(word)^2
  })))
  
  # Compute cosine similarity
  similarity <- dot_product / (kappa_magnitude * spec_magnitude)
  
  return(similarity)
})

# Rank products by relevance based on similarity scores
ranked_products <- order(cosine_similarities, decreasing = TRUE)

# Create a table to display the ranked products along with their relevance scores
ranked_table <- data.frame(
  Rank = 1:length(ranked_products),
  ProductID = products_df$ProductID[ranked_products],
  ProductName = products_df$ProductName[ranked_products],
  SimilarityScore = cosine_similarities[ranked_products]
)

# Print the table
print(ranked_table)
##    Rank ProductID       ProductName SimilarityScore
## 1     1        50 Kappa Smart Watch     0.094280904
## 2     2        50 Kappa Smart Watch     0.094280904
## 3     3        49  Iota Smart Watch     0.025712974
## 4     4        49  Iota Smart Watch     0.025712974
## 5     5        59       Tau Speaker     0.020951312
## 6     6        59       Tau Speaker     0.020951312
## 7     7        42   Beta Smartphone     0.007252377
## 8     8        42   Beta Smartphone     0.007252377
## 9     9        57       Rho Monitor     0.006734350
## 10   10        57       Rho Monitor     0.006734350
## 11   11        52        Mu Console     0.005892557
## 12   12        58     Sigma Monitor     0.005892557
## 13   13        52        Mu Console     0.005892557
## 14   14        58     Sigma Monitor     0.005892557
## 15   15        43      Gamma Laptop     0.005545936
## 16   16        43      Gamma Laptop     0.005545936
## 17   17        41  Alpha Smartphone     0.000000000
## 18   18        44      Delta Laptop     0.000000000
## 19   19        45    Epsilon Tablet     0.000000000
## 20   20        46       Zeta Tablet     0.000000000
## 21   21        47     Eta Headphone     0.000000000
## 22   22        48   Theta Headphone     0.000000000
## 23   23        51    Lambda Console     0.000000000
## 24   24        53         Nu Camera     0.000000000
## 25   25        54         Xi Camera     0.000000000
## 26   26        55   Omicron Printer     0.000000000
## 27   27        56        Pi Printer     0.000000000
## 28   28        60   Upsilon Speaker     0.000000000
## 29   29        41  Alpha Smartphone     0.000000000
## 30   30        44      Delta Laptop     0.000000000
## 31   31        45    Epsilon Tablet     0.000000000
## 32   32        46       Zeta Tablet     0.000000000
## 33   33        47     Eta Headphone     0.000000000
## 34   34        48   Theta Headphone     0.000000000
## 35   35        51    Lambda Console     0.000000000
## 36   36        53         Nu Camera     0.000000000
## 37   37        54         Xi Camera     0.000000000
## 38   38        55   Omicron Printer     0.000000000
## 39   39        56        Pi Printer     0.000000000
## 40   40        60   Upsilon Speaker     0.000000000

This code calculates the cosine similarity between the ‘Kappa Smart Watch’ and all other products based on their specifications. It then ranks the products by their similarity scores in descending order, indicating the relevance of each product to the ‘Kappa Smart Watch’. Finally, it prints the ranked list of similar products along with their specifications.