Focus On These

All tables

Connect to Database

con <- dbConnect(
         RMySQL::MySQL(),
         username = 'guest',
         password = 'guest1',
         host = '35.202.154.254',
         dbname = 'Octoparse')
#List of tables in selected database
res <- dbListTables(con)
print(res) 

##  [1] "Bitgur"                 "Blocktivity"           
##  [3] "CoinToBuy"              "CoinToBuyCountries"    
##  [5] "CryptoNews"             "CryptoNews2"           
##  [7] "CryptoNews3"            "CryptoNews4"           
##  [9] "CryptoSemanticAnalysis" "Forecasts"             
## [11] "Liftopia"               "Liftopia(OLD)"         
## [13] "LiftopiaFull"           "LiftopiaFullAll"       
## [15] "LiftopiaInfo"           "LiftopiaQuick"         
## [17] "LiftopiaScatter"        "Markets"               
## [19] "NewPrices"              "PredictCryptoPrices"   
## [21] "PredictCryptoSocial"    "PredictCryptoTA"       
## [23] "Prices"                 "PricesView1"           
## [25] "PricesView2"            "Ratings"               
## [27] "SemanticData"           "Social"                
## [29] "TechnicalAnalysis"      "joinTarget"            
## [31] "newsCheck"              "priceTarget"

24h Percent Price Change (Target)

SELECT Date, Symbol, InitialPrice, 
Price24hLater,Target24hChange, pkey
FROM priceTarget 
WHERE symbol = 'BTC' 
ORDER BY Date DESC

Technical Analysis

SELECT Date, count(Symbol), Summary_Day, 
avg(Target24hChange) as 'Average Target 24h Change'
FROM PredictCryptoTA
GROUP BY Summary_Day, Date 
ORDER BY Date DESC,Summary_Day

Large Datasets Challenges

• Lots of data = lots of problems
• Load times
• Good approach is to take user inputs first and then execute query
• Usually can get a bit creative and get major performance benefits

Current issues

• Right now, uploading things slowly through MySQL because a number was larger than the pre-determined size and caused trouble

Web Scraping Issues

SELECT Date, Symbol, InitialPrice, 
Price24hLater,Target24hChange, BTCdominance24hChange
FROM priceTarget 
WHERE symbol = 'BTC' 
ORDER BY Date DESC

Google Cloud Console

Profiling

#Disconnect from database
#dbClearResult(dbListResults(con)[[1]])
dbDisconnect(con)

## [1] TRUE