Cryptocurrency Scraping
Muhammad Hasan Itqoni
2023-06-11
Cryptocurrency
Pendahuluan
Cryptocurrency, atau mata uang kripto, adalah bentuk mata uang digital yang menggunakan kriptografi untuk mengamankan transaksi dan mengendalikan penciptaan unit baru. Cryptocurrency beroperasi secara terdesentralisasi, yang berarti tidak ada otoritas pusat seperti bank atau pemerintah yang mengatur atau mengendalikannya. Istilah yang sering digunakan dalam konteks cryptocurrency adalah “blockchain,” yaitu teknologi yang digunakan untuk mencatat dan memverifikasi transaksi.
Salah satu ciri utama cryptocurrency adalah bahwa mereka menggunakan teknologi blockchain untuk mencatat setiap transaksi yang terjadi. Blockchain adalah sebuah buku besar digital yang berisi catatan transaksi yang terhubung dalam blok-blok yang aman. Setiap blok memiliki tanda tangan kriptografis yang menghubungkannya dengan blok sebelumnya, membentuk rantai blok yang tidak dapat diubah. Keandalan dan keamanan blockchain menjadikannya sebagai dasar utama cryptocurrency.
Salah satu cryptocurrency yang paling terkenal adalah Bitcoin, yang diperkenalkan pada tahun 2009. Bitcoin adalah mata uang digital pertama yang menggunakan teknologi blockchain. Sejak saat itu, ratusan atau bahkan ribuan cryptocurrency lainnya telah dibuat, masing-masing dengan karakteristik dan tujuan yang berbeda.
Pada project ini akan dilakukan teknik scraping data cryptocurrency dari web https://goldprice.org/cryptocurrency-price/bitcoin-price lalu di masukkan ke mongo atlas dan tabel divisualisasikan dengan Rshiny
- Load Libraries
- Scraping
- Store to mongo atlas
- Get data from atlas
- Preprocessing Dataframe
- Visualization with Rhiny
1. Load Library
library(rvest)## Warning: package 'rvest' was built under R version 4.2.3library(mongolite)## Warning: package 'mongolite' was built under R version 4.2.3library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(shiny)
library(tidyverse)## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2
## ──## ✔ ggplot2 3.4.1 ✔ purrr 1.0.1
## ✔ tibble 3.1.8 ✔ stringr 1.5.0
## ✔ tidyr 1.3.0 ✔ forcats 1.0.0
## ✔ readr 2.1.4
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ readr::guess_encoding() masks rvest::guess_encoding()
## ✖ dplyr::lag() masks stats::lag()library(shinythemes)## Warning: package 'shinythemes' was built under R version 4.2.32. Scraping
Data yang akan diambil adalah data tabel dari web URL = https://goldprice.org/cryptocurrency-price/bitcoin-price
url <- "https://goldprice.org/cryptocurrency-price/bitcoin-price"
page <- read_html(url)
Bitcoin <- page %>% html_table() %>%
as.data.frame() %>%
select(-8)
Bitcoin## Rank Currency Market.Cap. Circulating.Supply Price
## 1 1 Bitcoin 499,077,887,917 19,399,343 $25,742.00
## 2 2 Ethereum 209,749,514,759 120,227,282 $1,745.14
## 3 3 Tether 83,443,057,204 83,361,513,300 $1.00
## 4 4 BNB 36,600,999,182 155,855,196 $234.84
## 5 5 USD Coin 28,399,231,188 28,377,992,304 $1.00
## 6 6 XRP 26,469,785,753 51,987,017,573 $0.51
## 7 8 Cardano 9,334,424,122 35,045,020,830 $0.27
## 8 9 Dogecoin 8,635,314,882 139,723,606,384 $0.06
## 9 10 TRON 6,244,973,503 90,115,374,978 $0.07
## 10 11 Solana 6,110,707,101 398,485,616 $15.35
## 11 12 Polygon 5,781,443,525 9,289,469,069 $0.62
## 12 13 Litecoin 5,653,426,900 73,122,671 $77.42
## 13 14 Polkadot 5,536,443,831 1,241,612,246 $4.46
## 14 15 Binance USD 4,779,541,131 4,780,579,213 $1.00
## 15 16 Dai 4,543,516,660 4,542,553,510 $1.00
## 16 17 Wrapped Bitcoin 4,028,800,671 156,748 $25,694.00
## 17 18 Avalanche 3,985,494,508 344,895,612 $11.55
## 18 19 Shiba Inu 3,912,248,711 589,340,193,200,510 $0.00
## 19 20 LEO Token 3,300,320,793 930,134,739 $3.55
## 20 21 Uniswap 3,046,120,515 753,766,667 $4.04
## 21 22 Chainlink 2,647,746,080 517,099,971 $5.11
## 22 23 Monero 2,478,659,431 18,147,820 $136.55
## 23 24 OKB 2,423,524,380 60,000,000 $40.37
## 24 25 Cosmos Hub 2,409,234,281 292,586,164 $8.23
## 25 26 Stellar 2,200,039,963 26,888,568,243 $0.08
## Volume..24h. Change..24h.
## 1 $12,641,605,548 0.33 %
## 2 $6,306,676,014 0.07 %
## 3 $19,227,462,861 0.03 %
## 4 $706,870,541 -0.67 %
## 5 $2,532,573,502 0.18 %
## 6 $1,007,349,226 3.02 %
## 7 $689,590,129 6.95 %
## 8 $439,580,569 3.39 %
## 9 $291,396,856 1.78 %
## 10 $668,419,650 3.53 %
## 11 $671,810,356 11.12 %
## 12 $646,846,222 0.70 %
## 13 $145,241,088 0.17 %
## 14 $1,133,804,451 -0.09 %
## 15 $91,905,440 0.00 %
## 16 $90,024,780 0.14 %
## 17 $204,701,048 1.64 %
## 18 $156,051,398 1.73 %
## 19 $609,483 0.95 %
## 20 $52,887,122 1.55 %
## 21 $226,524,375 2.12 %
## 22 $48,785,543 0.25 %
## 23 $9,273,503 -2.73 %
## 24 $94,643,459 4.92 %
## 25 $51,371,923 2.07 %Setelah data berhasil di scrap, langkah selanjutnya store data ke mongo atlas # 3. Store Data to Mongo Atlas
atlas <- mongo(
collection = "bitcoin",
db = "project_scrap_mds",
url = "mongodb+srv://itqonihsn:Wszdrfc13@atlascluster.bq4l1ym.mongodb.net/?retryWrites=true&w=majority"
)
newdata <- cbind(no = atlas$count() + 1, Bitcoin)
atlas$insert(newdata)## List of 5
## $ nInserted : num 25
## $ nMatched : num 0
## $ nRemoved : num 0
## $ nUpserted : num 0
## $ writeErrors: list()Setelah data dilakukan scrap selama 10 menit sekali menggunakan github action pada laman berikut : https://github.com/itqonihsn/Scraping-web-data-automation Data kembali ditarik ke environmet R untuk dilakukan praprocess dan visualisasi dengan Shiny # 4. Get data from Atlas
df <- atlas$find(limit = 100000, skip = 0)
df1 <- subset(df, select = -no)
str(df1)## 'data.frame': 2425 obs. of 7 variables:
## $ Rank : int 1 2 3 4 5 6 8 9 10 11 ...
## $ Currency : chr "Bitcoin" "Ethereum" "Tether" "BNB" ...
## $ Market_Cap_ : chr "510,702,494,841" "219,873,620,485" "83,351,125,139" "39,542,361,543" ...
## $ Circulating_Supply: chr "19,397,850" "120,227,838" "83,356,513,307" "155,855,196" ...
## $ Price : chr "$26,343.00" "$1,830.26" "$1.00" "$253.33" ...
## $ Volume__24h_ : chr "$9,712,774,079" "$5,049,707,336" "$16,805,016,458" "$775,691,057" ...
## $ Change__24h_ : chr "-0.45 %" "-0.23 %" "-0.01 %" "-2.22 %" ...Disconnect from Mongo Atlas
atlas$disconnect()Karena tipe data masih character maka perlu di convert menjadi numerik # 5. Preprocessing Dataframe
df1 <- df1 %>%
mutate(Market_Cap_ = parse_number(Market_Cap_),
Circulating_Supply = parse_number(Circulating_Supply),
Price = parse_number(Price),
Volume__24h_ = parse_number(Volume__24h_),
Change__24h_ = parse_number(Change__24h_)
)
str(df1)## 'data.frame': 2425 obs. of 7 variables:
## $ Rank : int 1 2 3 4 5 6 8 9 10 11 ...
## $ Currency : chr "Bitcoin" "Ethereum" "Tether" "BNB" ...
## $ Market_Cap_ : num 5.11e+11 2.20e+11 8.34e+10 3.95e+10 2.83e+10 ...
## $ Circulating_Supply: num 1.94e+07 1.20e+08 8.34e+10 1.56e+08 2.83e+10 ...
## $ Price : num 26343 1830 1 253 1 ...
## $ Volume__24h_ : num 9.71e+09 5.05e+09 1.68e+10 7.76e+08 2.52e+09 ...
## $ Change__24h_ : num -0.45 -0.23 -0.01 -2.22 0.04 ...Dataframe
head(df1)## Rank Currency Market_Cap_ Circulating_Supply Price Volume__24h_
## 1 1 Bitcoin 510702494841 19397850 26343.00 9712774079
## 2 2 Ethereum 219873620485 120227838 1830.26 5049707336
## 3 3 Tether 83351125139 83356513307 1.00 16805016458
## 4 4 BNB 39542361543 155855196 253.33 775691057
## 5 5 USD Coin 28332180278 28333677686 1.00 2517731277
## 6 6 XRP 27441256435 51987017573 0.53 1486887056
## Change__24h_
## 1 -0.45
## 2 -0.23
## 3 -0.01
## 4 -2.22
## 5 0.04
## 6 0.416 Visualisasi Shiny
# Subset 10 mata uang crypto tertinggi
bitcoin <- subset(df1, Rank == '1')
Ethereum <- subset(df1, Rank == '2')
Tether <- subset(df1, Rank == '3')
BNB <- subset(df1, Rank == '4')
USDCoin <- subset(df1, Rank == '5')
XRP <- subset(df1, Rank == '6')
Cardano <- subset(df1, Rank == '8')
Dogecoin <- subset(df1, Rank == '9')
Solana <- subset(df1, Rank == '10')
TRON <- subset(df1, Rank == '11')
# ui
ui <- fluidPage(
titlePanel("Table Dashboard"),
sidebarLayout(
sidebarPanel(
selectInput("tableSelect", "Select Table:",
choices = c("bitcoin", "Ethereum", "Tether","BNB", "USDCoin", "XRP","Cardano", "Dogecoin", "Solana","TRON"),
selected = "bitcoin")
),
mainPanel(
tableOutput("displayTable")
)
)
)
server <- function(input, output) {
# Render table based on sidebar selection
output$displayTable <- renderTable({
table_choice <- input$tableSelect
if (table_choice == "bitcoin") {
bitcoin
} else if (table_choice == "Ethereum") {
Ethereum
} else if (table_choice == "Tether") {
Tether
} else if (table_choice == "BNB") {
BNB
} else if (table_choice == "USDCoin") {
USDCoin
} else if (table_choice == "XRP") {
XRP
} else if (table_choice == "Cardano") {
Cardano
} else if (table_choice == "Dogecoin") {
Dogecoin
} else if (table_choice == "Solana") {
Solana
} else if (table_choice == "TRON") {
TRON
}
})
}
shinyApp(ui = ui, server = server)## PhantomJS not found. You can install it with webshot::install_phantomjs(). If it is installed, please make sure the phantomjs executable can be found via the PATH variable.Shiny applications not supported in static R Markdown documents
Selesai 11 Juni 2023