2024-04-23

Problem Statement

Aim is to utilize the Libraries.io API with R and H2O to analyze library usage patterns and dependencies. This will empower developers to make informed decisions for better project management.

Data Acquisition and Preprocessing

  • We collected data from various endpoints on the Libraries.io API from https://libraries.io/api

  • The collected Json data then transformed into a data frame for further analysis.

To ensure data quality, we will leverage the readr package for efficient data import and the tidyverse package to assess and potentially transform the data to adhere to tidy data principles and then used jsonlite to convert json to csv.

Data Exploration: Collection and Summary

To gather the necessary data for this project, we utilized the Libraries.io API and leveraged the capabilities of the httr and jsonlite packages within the R programming environment. After loading these packages, we executed the provided R script to facilitate the data collection process. Subsequently, the retrieved data was transformed and stored in a CSV file format, enabling further analysis and exploration of the dataset.

Machine Learning Platform

  • The h2o package will provide a scalable and distributed machine learning platform for training and deploying large-scale models.

Exploratory Data Analysis (EDA)

  • ggplot2: This package will be used for visualizing the data.

Data Modeling: Linear Regression | Gradient Boosting |Random Forest

Addressed Peer Comments

**** 1.How would you define the metrics or indicators used to measure the adoption, popularity, and activity of a package, considering factors like downloads, GitHub stars, update frequency, and community engagement?

In our analysis, we’ve highlighted the top 25 most popular R libraries and examined their usage across different platforms. We’ve also investigated the distribution of sourceRank on Libraries.io to understand the impact of these libraries in the software development landscape. When fetching download statistics from the Libraries.io API, it’s crucial to manage request frequency to avoid exceeding the rate limit. We need to ensure that requests are spaced out appropriately and consider implementing logic to introduce delays between requests. The API has a rate limit of 60 requests per minute per API key, exceeding which will result in a 429 response.

****2.How will preprocessing methods be employed such as feature importance analysis ,hyper-tuning your model using various methods such as cross-validation, grid searches, gradient-based boosting, and Random searches?

We have utilized techniques such as bagging (e.g., Random Forest), boosting (e.g., Gradient Boosting Machines), these methods help mitigate overfitting and capture diverse patterns in the data. Build a sequence of weak learners (decision trees) that are combined to create a powerful ensemble model. Tune hyperparameters such as learning rate, tree depth, and regularization parameters to optimize performance.

Plots

## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ stringr   1.5.1
## ✔ forcats   1.0.0     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

## Selecting by n

## `geom_smooth()` using formula = 'y ~ x'

## Load necessary libraries
library(dplyr)
library(ggplot2)

# Split the data by Platform and find the top 5 libraries based on SourceRank within each group
top_libraries_baseR <- Libraries_data %>%
  group_by(Platform) %>%
  top_n(5, SourceRank)

# View the top libraries
print(top_libraries_baseR)
## # A tibble: 117 × 16
## # Groups:   Platform [14]
##       ID Platform Name       `Created Timestamp` `Updated Timestamp` Description
##    <dbl> <chr>    <chr>      <chr>               <chr>               <chr>      
##  1    59 Alcatraz FuzzyAuto… 2021-01-11 23:56:1… 2023-06-14 01:03:3… Enables fu…
##  2   114 Alcatraz RealmPlug… 2021-01-11 23:56:1… 2023-06-14 01:03:5… File and c…
##  3   126 Alcatraz SCXcodeMi… 2021-01-11 23:56:1… 2023-06-14 01:04:0… Sublime Te…
##  4   127 Alcatraz SCXcodeSw… 2021-01-11 23:56:1… 2023-06-14 01:03:5… Autocomple…
##  5   133 Alcatraz Slack Tem… 2021-01-11 23:56:1… 2023-06-14 01:04:0… A set of u…
##  6   287 Dub      derelict-… 2021-01-11 23:58:5… 2023-06-14 01:03:3… A dynamic …
##  7   288 Dub      derelict-… 2021-01-11 23:58:5… 2023-06-14 01:03:3… A dynamic …
##  8   289 Dub      derelict-… 2021-01-11 23:58:5… 2023-06-14 01:03:3… A dynamic …
##  9   290 Dub      derelict-… 2021-01-11 23:58:5… 2023-06-14 01:03:3… A dynamic …
## 10   296 Dub      derelict-… 2021-01-11 23:58:5… 2023-06-14 01:03:3… A dynamic …
## # ℹ 107 more rows
## # ℹ 10 more variables: `Homepage URL` <chr>, Licenses <chr>,
## #   `Repository URL` <chr>, SourceRank <dbl>,
## #   `Latest Release Publish Timestamp` <chr>, `Latest Release Number` <chr>,
## #   Language <chr>, `Last synced Timestamp` <chr>, `Repository ID` <dbl>,
## #   Release_Date <date>
# Fit a linear regression model to predict SourceRank
model <- lm(SourceRank ~ ID +`Repository ID`, data = Libraries_data)

# Summary of the model
summary(model)
## 
## Call:
## lm(formula = SourceRank ~ ID + `Repository ID`, data = Libraries_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -8.6912 -1.6892 -0.5999  1.3941 22.5166 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      8.737e+00  3.254e-02 268.528  < 2e-16 ***
## ID              -1.680e-06  3.286e-07  -5.114 3.18e-07 ***
## `Repository ID` -9.659e-08  8.217e-09 -11.754  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.19 on 40369 degrees of freedom
## Multiple R-squared:  0.004059,   Adjusted R-squared:  0.00401 
## F-statistic: 82.27 on 2 and 40369 DF,  p-value: < 2.2e-16
# Predict SourceRank using the model
Libraries_data$predicted_SourceRank <- predict(model, Libraries_data)

# View the predicted SourceRank
head(Libraries_data$predicted_SourceRank)
##        1        2        3        4        5        6 
## 8.736742 8.736735 8.736713 8.736671 8.736657 8.736638
# Calculate RMSE
rmse <- sqrt(mean((Libraries_data$SourceRank - Libraries_data$predicted_SourceRank)^2))
print(paste("RMSE:", rmse))
## [1] "RMSE: 3.18965413825859"
# Calculate R-squared
rsquared <- summary(model)$r.squared
print(paste("R-squared:", rsquared))
## [1] "R-squared: 0.00405926250476845"
# Calculate standard deviation of residuals
residuals <- Libraries_data$SourceRank - Libraries_data$predicted_SourceRank
std_dev_resid <- sd(residuals)
print(paste("Standard Deviation of Residuals:", std_dev_resid))
## [1] "Standard Deviation of Residuals: 3.18969364228854"
# Calculate cumulative proportions
Libraries_data <- Libraries_data %>% 
  arrange(SourceRank) %>%
  mutate(cum_actual = cumsum(SourceRank) / sum(SourceRank),
         cum_predicted = cumsum(predicted_SourceRank) / sum(predicted_SourceRank))

# Plot Lorenz Curve
ggplot(Libraries_data, aes(x = cum_actual, y = cum_predicted)) +
  geom_line(color = "blue") +
  geom_abline(slope = 1, intercept = 0, color = "red", linetype = "dashed") +
  labs(title = "Lorenz Curve of SourceRank",
       x = "Cumulative Actual Proportion",
       y = "Cumulative Predicted Proportion") +
  theme_minimal()

Findings/Futurework

Build predictive models to forecast future trends in package adoption, popularity, and activity. Utilize machine learning algorithms to predict download counts, identify emerging technologies, or forecast package maintenance efforts.