Forex Scraping Using R

Disclaimer: Forex Trading is a risky activity. Trade money you can afford to lose. Past performance does not guarantee future results.

Sample for Quantmod

### References
# https://rpubs.com/krish469/BitCoinPrediction - Bitcoin Prediction in R
# http://www.r-chart.com/2017/07/investigating-cryptocurrencies-part-ii.html - Investigating Cryptocurrencies (Part II)
# https://stackoverflow.com/questions/34319504/r-how-can-i-change-date-format-when-i-plot-an-xts-zoo-object - Rotate xts date labels

#Load required libraries
library('ggplot2')
library('forecast')
## Registered S3 method overwritten by 'xts':
##   method     from
##   as.zoo.xts zoo
## Registered S3 method overwritten by 'quantmod':
##   method            from
##   as.zoo.data.frame zoo
## Registered S3 methods overwritten by 'forecast':
##   method             from    
##   fitted.fracdiff    fracdiff
##   residuals.fracdiff fracdiff
library('tseries')
library(xts)
## Loading required package: zoo
## 
## Attaching package: 'zoo'
## The following objects are masked from 'package:base':
## 
##     as.Date, as.Date.numeric
library(Metrics)
## 
## Attaching package: 'Metrics'
## The following object is masked from 'package:forecast':
## 
##     accuracy
# Load bitcoin
library(quantmod)
## Loading required package: TTR
## Version 0.4-0 included new data defaults. See ?getSymbols.
loadSymbols(c('BTCUSD=X','AMD'))
## 'getSymbols' currently uses auto.assign=TRUE by default, but will
## use auto.assign=FALSE in 0.5-0. You will still be able to use
## 'loadSymbols' to automatically load data. getOption("getSymbols.env")
## and getOption("getSymbols.auto.assign") will still be checked for
## alternate defaults.
## 
## This message is shown once per session and may be disabled by setting 
## options("getSymbols.warning4.0"=FALSE). See ?getSymbols for details.
## [1] "BTCUSD=X" "AMD"
ls()
## [1] "AMD"      "BTCUSD=X"
plot(`BTCUSD=X`)

chartSeries(`BTCUSD=X`, subset='last 4 weeks')

chartSeries(`BTCUSD=X`, subset='last 8 weeks')

chartSeries(`BTCUSD=X`, subset='last 12 weeks')

addBBands()

addRSI()

addMACD()

##
my_daily_data <- `BTCUSD=X`
head(my_daily_data)
##            BTCUSD=X.Open BTCUSD=X.High BTCUSD=X.Low BTCUSD=X.Close
## 2010-07-19       0.08584       0.07723      0.09307        0.08080
## 2010-07-20       0.08080       0.07426      0.08181        0.07474
## 2010-07-21       0.07474       0.06634      0.07921        0.07921
## 2010-07-22       0.07921       0.05050      0.08181        0.05050
## 2010-07-23       0.05050       0.05050      0.06767        0.06262
## 2010-07-26       0.05050       0.05000      0.05600        0.05600
##            BTCUSD=X.Volume BTCUSD=X.Adjusted
## 2010-07-19            4900           0.08080
## 2010-07-20            2000           0.07474
## 2010-07-21            4200           0.07921
## 2010-07-22           12900           0.05050
## 2010-07-23           14100           0.06262
## 2010-07-26            4600           0.05600
names(my_daily_data) <- c("Open", "High", "Low", "Close", "Volume", "Adjusted")
names(my_daily_data)
## [1] "Open"     "High"     "Low"      "Close"    "Volume"   "Adjusted"
head(my_daily_data)
##               Open    High     Low   Close Volume Adjusted
## 2010-07-19 0.08584 0.07723 0.09307 0.08080   4900  0.08080
## 2010-07-20 0.08080 0.07426 0.08181 0.07474   2000  0.07474
## 2010-07-21 0.07474 0.06634 0.07921 0.07921   4200  0.07921
## 2010-07-22 0.07921 0.05050 0.08181 0.05050  12900  0.05050
## 2010-07-23 0.05050 0.05050 0.06767 0.06262  14100  0.06262
## 2010-07-26 0.05050 0.05000 0.05600 0.05600   4600  0.05600
##
class(my_daily_data)
## [1] "xts" "zoo"

R used to gather data from MT4 Terminal

Most of the parts here are from the Darwinex Algo Trading project.

https://blog.darwinex.com/download-tick-data-metatrader

In this example I’ll retrieve a history file frome one of data folders of MT4 Metatrader 4 is a platform mostly used by forex traders.

Reading a file from mt4

Preparation

# Disable scientific formatting
options(scipen=9999)

if (!require("pacman")) 
 install.packages("pacman")
## Loading required package: pacman
libs.vector <- c("anytime",
 "data.table",
 "xts", "zoo", "quantmod",
 "plotly",
 "microbenchmark")

pacman::p_load(char = libs.vector,
 install=TRUE, 
 update=FALSE)
EURUSD <- read.csv("EURUSD_TickData_2019-11-06.csv")
library(data.table)
if (!require("DT")) install.packages('DT')
## Loading required package: DT
library(DT)

datatable(EURUSD)
head(EURUSD)
##   time_milliseconds    bid    ask spread
## 1     1573055312000 1.1081 1.1095 0.0014
## 2     1573055392000 1.1080 1.1095 0.0015
## 3     1573055400000 1.1080 1.1094 0.0014
## 4     1573055401000 1.1080 1.1094 0.0014
## 5     1573055412000 1.1081 1.1095 0.0014
## 6     1573055456000 1.1080 1.1095 0.0015
names(EURUSD)
## [1] "time_milliseconds" "bid"               "ask"              
## [4] "spread"
dim(EURUSD)
## [1] 508   4
# Reread
EURUSD.M1.QUOTES.2019.11.06.dt <- fread("EURUSD_TickData_2019-11-06.csv", colClasses="numeric")
#anytime(EURUSD.M1.QUOTES.2019.11.06.dt$time_milliseconds / 1000, tz = "UTC")
str(EURUSD.M1.QUOTES.2019.11.06.dt)
## Classes 'data.table' and 'data.frame':   508 obs. of  4 variables:
##  $ time_milliseconds: num  1573055312000 1573055392000 1573055400000 1573055401000 1573055412000 ...
##  $ bid              : num  1.11 1.11 1.11 1.11 1.11 ...
##  $ ask              : num  1.11 1.11 1.11 1.11 1.11 ...
##  $ spread           : num  0.0014 0.0015 0.0014 0.0014 0.0014 0.0015 0.0014 0.0015 0.0014 0.0015 ...
##  - attr(*, ".internal.selfref")=<externalptr>
class(EURUSD.M1.QUOTES.2019.11.06.dt)
## [1] "data.table" "data.frame"
class(EURUSD.M1.QUOTES.2019.11.06.dt$time_milliseconds)
## [1] "numeric"
class(EURUSD.M1.QUOTES.2019.11.06.dt$ask)
## [1] "numeric"
class(EURUSD.M1.QUOTES.2019.11.06.dt$bid)
## [1] "numeric"
head(EURUSD.M1.QUOTES.2019.11.06.dt)
##    time_milliseconds    bid    ask spread
## 1:     1573055312000 1.1081 1.1095 0.0014
## 2:     1573055392000 1.1080 1.1095 0.0015
## 3:     1573055400000 1.1080 1.1094 0.0014
## 4:     1573055401000 1.1080 1.1094 0.0014
## 5:     1573055412000 1.1081 1.1095 0.0014
## 6:     1573055456000 1.1080 1.1095 0.0015

Data Type conversion using anytime

timestamp <- anytime(c(EURUSD.M1.QUOTES.2019.11.06.dt$time_milliseconds/1000), tz="UTC")
EURUSD.M1.QUOTES.2019.11.06.dt <- cbind(EURUSD.M1.QUOTES.2019.11.06.dt, timestamp)
names(EURUSD.M1.QUOTES.2019.11.06.dt)
## [1] "time_milliseconds" "bid"               "ask"              
## [4] "spread"            "timestamp"
head(EURUSD.M1.QUOTES.2019.11.06.dt$timestamp)
## [1] "2019-11-06 15:48:32 UTC" "2019-11-06 15:49:52 UTC"
## [3] "2019-11-06 15:50:00 UTC" "2019-11-06 15:50:01 UTC"
## [5] "2019-11-06 15:50:12 UTC" "2019-11-06 15:50:56 UTC"
datatable(EURUSD.M1.QUOTES.2019.11.06.dt)

Quantmod Scraping

library(quantmod)
getSymbols("EUR=X",src="yahoo",from="2019-07-01", to="2019-10-31")
## [1] "EUR=X"
getSymbols("GBP=X",src="yahoo",from="2019-07-01", to="2019-10-31")
## [1] "GBP=X"
# Adjusted Prices
eurAdj = unclass(`EUR=X`$`EUR=X.Adjusted`)

# Convert to EUR/USD
eurAdj = 1/eurAdj  

gbpAdj <- unclass(`GBP=X`$`GBP=X.Adjusted`)

# Convert to GBP/USD
gbpAdj <- 1/gbpAdj

# Extract EUR dates for plotting later.
eurDates = index(`EUR=X`)  

# Create merged data frame.
eurgbp_merged <- data.frame(eurAdj,gbpAdj)

Let’s check the data

datatable(eurgbp_merged)

Update the table

eurgbp_merged = data.frame(eurDates, eurgbp_merged)
colnames(eurgbp_merged) = c("Dates", "EURUSD", "GBPUSD")

datatable(eurgbp_merged)

Getting more details

#Mean
mean(eurgbp_merged$EURUSD)
## [1] 1.110691
# Standard Deviation
sd(eurgbp_merged$EURUSD)
## [1] 0.01027021

Plotting data

library(plotly)
plot_ly(name="EUR/USD Price", x = eurgbp_merged$Dates, y = as.numeric(eurgbp_merged$EURUSD), type="scatter", mode="lines") %>%
add_trace(name="EUR/USD Mean", y=(as.numeric(mean(eurgbp_merged$EURUSD))), mode="lines")

Additional Plot with SD

plot_ly(name="EUR/USD Price", x = eurgbp_merged$Dates, y = as.numeric(eurgbp_merged$EURUSD), type="scatter", mode="lines") %>%

add_trace(name="+1 S.D.", y=(as.numeric(mean(eurgbp_merged$EURUSD))+sd(eurgbp_merged$EURUSD)), mode="lines", line=list(dash="dot")) %>%

add_trace(name="-1 S.D.", y=(as.numeric(mean(eurgbp_merged$EURUSD))-sd(eurgbp_merged$EURUSD)), mode="lines", line=list(dash="dot")) %>%

add_trace(name="EUR/USD Mean", y=(as.numeric(mean(eurgbp_merged$EURUSD))), mode="lines")

Linear Regression on EURUSD and GBPUSD

# Perform linear regression on EUR/USD and GBP/USD
fit <- lm(EURUSD ~ GBPUSD, data=eurgbp_merged)

# Draw scatter plot with line of best fit
plot_ly(name="Scatter Plot", data=eurgbp_merged, y=~EURUSD, x=~GBPUSD, type="scatter", mode="markers") %>%

add_trace(name="Linear Regression", data=eurgbp_merged, x=~GBPUSD, y=fitted(fit), mode="lines")

Get Covariance

cov(eurgbp_merged$EURUSD, eurgbp_merged$GBPUSD)
## [1] 0.00005839436

Get the Correlation

cor(eurgbp_merged$EURUSD, eurgbp_merged$GBPUSD)
## [1] 0.2337197

Convert the data into XTS

Get data using History Center of MT4, this is an hourly chart example

quotes <- read.csv("EURJPY-60.csv", header=FALSE)
head(quotes)
##           V1    V2      V3      V4      V5      V6   V7
## 1 2019.09.09 21:00 118.386 118.396 118.352 118.387 1718
## 2 2019.09.09 21:30 118.388 118.455 118.380 118.424 2009
## 3 2019.09.09 22:00 118.423 118.483 118.423 118.446 1967
## 4 2019.09.09 22:30 118.447 118.458 118.417 118.442 1822
## 5 2019.09.09 23:00 118.440 118.501 118.404 118.464 1095
## 6 2019.09.09 23:30 118.467 118.491 118.435 118.472 1171
tail(quotes)
##              V1    V2      V3      V4      V5      V6   V7
## 2043 2019.11.07 10:00 120.684 120.708 120.597 120.667 4461
## 2044 2019.11.07 10:30 120.666 120.777 120.663 120.761 3472
## 2045 2019.11.07 11:00 120.761 120.817 120.711 120.788 3314
## 2046 2019.11.07 11:30 120.785 120.876 120.731 120.814 3430
## 2047 2019.11.07 12:00 120.815 120.990 120.807 120.959 3702
## 2048 2019.11.07 12:30 120.960 121.069 120.926 121.069 3550
library(quantmod)

x <- as.xts(quotes[,-(1:2)], as.Date(paste(quotes[,1]),format='%Y.%m.%d'))

colnames(x) <-c('Open','High','Low','Close','Volume')

y = last(x, "3 days")
colnames(y) <-c('Open','High','Low','Close','Volume')
head(x)
##               Open    High     Low   Close Volume
## 2019-09-09 118.386 118.396 118.352 118.387   1718
## 2019-09-09 118.388 118.455 118.380 118.424   2009
## 2019-09-09 118.423 118.483 118.423 118.446   1967
## 2019-09-09 118.447 118.458 118.417 118.442   1822
## 2019-09-09 118.440 118.501 118.404 118.464   1095
## 2019-09-09 118.467 118.491 118.435 118.472   1171
head(y)
##               Open    High     Low   Close Volume
## 2019-11-05 120.820 120.835 120.795 120.819   1772
## 2019-11-05 120.818 120.826 120.750 120.774   7280
## 2019-11-05 120.779 120.851 120.779 120.846   1686
## 2019-11-05 120.844 120.994 120.824 120.965   3661
## 2019-11-05 120.963 121.047 120.940 121.004   2777
## 2019-11-05 121.005 121.070 120.996 121.016   2328
# Draw Candlesticks from data
candleChart(last(y, "3 days"), theme = "white", TA = "addVo();addBBands();addCCI();addSMA();addRSI()")

datatable(x)

Now let’s try using the same SD+ and SD- and the Mean to have a greater decision on which direction the market is going

plot_ly(name="EUR/JPY Price", x = c(1:nrow(x)),  y = as.numeric(x$Close), type="scatter", mode="lines") %>%

add_trace(name="+1 S.D.", y=(as.numeric(mean(x$Close))+sd(x$Close)), mode="lines", line=list(dash="dot")) %>%

add_trace(name="-1 S.D.", y=(as.numeric(mean(x$Close))-sd(x$Close)), mode="lines", line=list(dash="dot")) %>%

add_trace(name="EUR/JPY Mean", y=(as.numeric(mean(x$Close))), mode="lines")

If you are into Automated Trading

Still this is risky, so management of account is a must. This is not financial advise.

I’ll just share this – from my perspective this system is a rule-based engine. Check the Link: https://web.facebook.com/Trading-Ops-103705947743249/

R Users Group Philippines Meetup - November 21, 2019

Presented by Rowen Remis R. Iral IT Engineer / Data Science Architect

http://wenup.wordpress.com

Further reading

## References
# https://www.mql5.com/en/forum/181249 - MT4R reference
# https://robjhyndman.com/hyndsight/forecast4/ - Forecast
# https://www.mql5.com/en/code/11112?source=metaeditor5_codebase - New Mt4R codebase
# https://www.mql5.com/en/articles/3486 - Deep Neural Network
# https://github.com/nopaixx/MaketsGA - Hands on Neural Network in Metatrader
# https://fann2mql.wordpress.com/documentation/tutorial/ - Fann2Mql
# https://www.mql5.com/en/articles/1574 - FANN2MQL NEURAL NETWORK TUTORIAL
# https://www.mql5.com/en/articles/1565 - USING NEURAL NETWORKS IN METATRADER, Fann2MQL
# https://github.com/magiciankartik/MetaTraderForecast - MetaTraderFOrecast
# http://rwanjohi.rbind.io/2018/04/05/time-series-forecasting-using-lstm-in-r/ - TimeSeries FOrecasting, LSTM in R
# https://github.com/rwanjohi/Time-series-forecasting-using-LSTM-in-R/blob/master/LSTM%20Time%20series%20forecasting.R - LSTM in R
# http://auto.tradingninja.com/how-to-connect-powerful-r-software-with-mt4-for-machine-learning/ - How To Connect Powerful R Software With MT4 For Machine Learning?
# https://blog.darwinex.com/automated-tick-data-collection-r-metatrader/ - Tick Data collection using R from MT4
# https://blog.darwinex.com/darwin-data-science-environment-setup/ - Darwin Data Science SEtup
# https://blog.darwinex.com/download-tick-data-metatrader/ - Download Tick Data mt4
# https://datascienceplus.com/time-series-analysis-using-arima-model-in-r/ - Time Series Analysis Using ARIMA Model In R
# https://towardsdatascience.com/time-series-forecasting-arima-models-7f221e9eee06 - Time series Forecasting — ARIMA models\
# https://people.duke.edu/~rnau/411arim.htm - ARIMA models for time series forecasting
# http://fx.doubledoji.com/neural-network-forex-trading-system/ - Neural Network Forex Trading System
# http://trade.doubledoji.com/how-to-convert-mt4-daily-and-4-hour-csv-file-into-r-xts-time-series/ - Convert mt4 data to xts R format
# https://github.com/micclly/mt4R/blob/new_mql4_support/MQL4/Experts/trendomat.mq4 - mt4R
# https://blog.darwinex.com/zeromq-interface-python-r-metatrader4/ - ZeroMQ Interface with Python / R / Metatrader 4
# https://blog.darwinex.com/darwin-time-series-data-r-mld2/ - XTS data processing
# https://blog.darwinex.com/darwin-time-series-data-r-mld2/ - darwin timeseries
# http://dirk.eddelbuettel.com/code/anytime.html - anytime function
# https://www.investopedia.com/terms/c/covariance.asp - Covariance 
# https://blog.darwinex.com/quantitative-modelling-algorithmic-traders-primer/ - Quantitative Modeling for Algorithmic Traders – Primer
# https://blog.darwinex.com/darwin-filters-alternative-markowitz-portfolio-theory/ - DARWIN Filters: A Practical Alternative to Markowitz Portfolio Theory
# https://www.octafx.com/education/article/currency-pairs-correlation-forex/ - Correlation in Forex
# https://www.quantmod.com/examples/charting/ - Quantmod
# https://plot.ly/r/candlestick-charts/ - CandleCharts with plotly
# https://rstudio.github.io/DT/ - DataTable, DT Package
# https://amunategui.github.io/wallstreet/index.html - Quantmod and Forecasting for Financial Markets
# https://rstudio-pubs-static.s3.amazonaws.com/260624_ca8ddf60e4c74d1993fca2167cae2bff.html - Automated Trading, Train, Test, Forecast
# https://rviews.rstudio.com/2017/03/17/quandl-and-forecasting/ - Quandl and Forecasting
# https://otexts.com/fpp2/the-forecast-package-in-r.html - Forecast Package in R
# https://www.r-bloggers.com/time-series-analysis-using-r-forecast-package/ - Time Series Analysis using R Forecast package
# https://www.forexfactory.com/showthread.php?t=260422 - Mt4R