Quantopian Practice Run¶

Quantopian is a crowd-sourced quantitative investment firm [inspiring] talented people from around the world to write investment algorithms. Quantopian provides capital, education, data, a research environment, and a development platform to algorithm authors (quants). Quantopian's community and backtester including price, volume, and corporate fundamentals data for all 8000+ US equities from 2002 to present is free for everyone to use. Everything you write is yours. Quantopian does not own your algorithms; you do. Your algorithms are kept secret. Quantopian only supports Python and has no plans to support another language at this time.

Live Trading: You can use your algorithm to live trade any of the US equities in the Quantopian database which includes any stocks and ETFs trading on the NYSE, NASDAQ, or AMEX. Futures are not yet available for live trading. Brokers currently supported are Interactive Brokers (IB) and Robinhood.

Competitions: You can compete to win cash prizes with new contests each month. One winning algorithm will win \$5000, 2nd gets \$1000, 3rd gets \$500. Plus 100 limited-edition t-shirts. Your algorithm must have low correlation to the overall market (low beta). It must have consistent, stable, and positive returns. It must avoid large drawdowns.

Capital Allocations: Quantopian evaluate your algorithms, and selected authors receive an offer to license their algorithm. If your algorithm receives an allocation, Quantopian will pay you a share of the returns that your algorithm earns on our capital. Their target is to pay you 10\% of the net profit on your algorithm's allocation. You will always own your intellectual property. At other firms, your algorithms become the property of the firm. With rare exceptions described in the Terms of Use, Quantopian doesn't look at your algorithm code. But they do look at some of the information that comes out of your backtest. Things like your performance, your risk metrics, how many securities you trade, how often you trade, and what the size of your positions are.

Interactive Brokers: Interactive Brokers accounts are designed for sophisticated traders and investors who regularly incur \$10+ in monthly commissions. If the minimum is not met, a monthly activity fee of \$10 less commissions is charged. This fee is waived the first three months and any month in which an account has more than \$100,000 in equity. If the average equity balance falls under \$2,000, the activity fee increases to \$20 less commissions. A minimum of \$10,000 is required to open an account. The following trades are commission-free: Global X ETFs, Infrastructure ETFs, O'Shares ETFs, Cambria ETFs, Eaton Vance NextShares Exchange-Traded Managed Funds, ETF Securities ETFs, ACSI ETFs. Exceptions (API, VWAP, DARK, IBKRATS, IBDARK, CSFB ALGO, CSFB PATH, JEFFALGO)

Robinhood: Robinhood provides free stock trading. Robinhood earns revenue by collecting interest on the cash and securities in Robinhood accounts, much like a bank collects interest on cash deposits. Robinhood does not support joint, custodial, trust, and IRA account types. Robinhood also does not currently have a feature to name a beneficiary for automatic transfer on death. In the event of a death, they will work with the executor of the estate to collect proper documentation and dissolve the account. In NYS, a Will needs to be probated if a decedent died with assets valued at \$30,000 or more. New York's probate cost can range anywhere from 2% to 7% of estate's value, but could be more under certain circumstances. Depending on the estate's value, executor's as well as attorneys' fees in NYS vary between 2.5\% to 5\% while court costs may range from \$215 to \$1,250.

Libraries¶

In [1]:
from scipy import stats, signal
import matplotlib.pyplot as pyplot
from scipy.interpolate import interp1d, spline
import numpy as np, pandas as pd, datetime

Constants¶

In [2]:
trading_year = int(np.ceil(365.25 * 5/7 - 9))
tax_pen = (0.35 - 0.15) # Capital Gains Maximum

Treasury Rates¶

Pull from Quandl here and then upload to Quantopian data file here. You should expect that your csv files could be lost at any time maintenance is performed.

In [3]:
Treasury_Yield = local_csv('USTREASURY-YIELD.csv', date_column = 'Date', use_date_column_as_index='True') / 100
Treasury_Yield[0:5]
Out[3]:
1 MO 3 MO 6 MO 1 YR 2 YR 3 YR 5 YR 7 YR 10 YR 20 YR 30 YR
Date
2017-08-04 00:00:00+00:00 0.0100 0.0108 0.0114 0.0123 0.0136 0.0151 0.0182 0.0208 0.0227 0.0261 0.0284
2017-08-03 00:00:00+00:00 0.0100 0.0108 0.0113 0.0122 0.0134 0.0149 0.0179 0.0205 0.0224 0.0256 0.0281
2017-08-02 00:00:00+00:00 0.0102 0.0108 0.0115 0.0124 0.0136 0.0152 0.0182 0.0208 0.0227 0.0260 0.0285
2017-08-01 00:00:00+00:00 0.0100 0.0108 0.0115 0.0122 0.0134 0.0150 0.0180 0.0207 0.0226 0.0261 0.0286
2017-07-31 00:00:00+00:00 0.0100 0.0107 0.0113 0.0123 0.0134 0.0151 0.0184 0.0211 0.0230 0.0266 0.0289

Securities¶

Aside from SPY and AGG benchmarks, using only Vanguard ETFs. Full backtest includes ETFs from Fidelity, iShares, Schwab, and State Street. Selection was based on a maximum 0.32 expense ratio and a minimum etfdb.com Expense Rating of B+. The 0.32 limit is the maximum expense ratio for a Vanguard ETF.

In [4]:
securities = [symbols(33650), symbols(33649), symbols(33652), symbols(44849), symbols(33651), symbols(22887),
              symbols(40108), symbols(40106), symbols(40111), symbols(35343), symbols(35344), symbols(35345),
              symbols(25898), symbols(25899), symbols(25900), symbols(25901), symbols(38987), symbols(38982), 
              symbols(25902), symbols(38985), symbols(25903), symbols(26667), symbols(34385), symbols(25904), 
              symbols(38984), symbols(27100), symbols(38988), symbols(38986), symbols(25905), symbols(25906), 
              symbols(28364), symbols(49785), symbols(40109), symbols(40103), symbols(40105), symbols(26668), 
              symbols(38983), symbols(26669), symbols(40337), symbols(25907), symbols(32521), symbols(40140), 
              symbols(40145), symbols(40144), symbols(40107), symbols(40104), symbols(40110), symbols(32522), 
              symbols(26670), symbols(27101), symbols(25908), symbols(38272), symbols(36486), symbols(49366),
              symbols(40139), symbols(22739), symbols(43529), symbols(25909), symbols(40141), symbols(40143), 
              symbols(40142), symbols(25910), symbols(25911), symbols(27102), symbols(44850), symbols(23408), 
              symbols(40785), symbols(32888), symbols(49784), symbols(33486), symbols(25485), symbols(8554)] 
benchmarks = securities[-3:]

Price History¶

Quantopian's price data starts on Jan 2, 2002.

In [5]:
start_date = pd.Timestamp('20020102')
end_date = pd.Timestamp(max(Treasury_Yield.index.date))
assets = get_pricing(securities, fields=['close_price'], start_date=start_date, end_date=end_date)

The market and the Treasury do not operate on the same exact calendar. For days the Market is open and the Treasury os closed, the Treasury Rates are forward filled with the last valid observation propagated forward to the next.

In [6]:
Treasury_Yield = Treasury_Yield.reindex(assets.major_axis).fillna(method='ffill')

Historic Returns¶

Annual returns.

In [7]:
assets['r_hist'] = np.nan
for date in assets.major_axis:
    index = assets.major_axis.get_loc(date)
    window_begin = assets.major_axis[index - (trading_year + 1)]
    window_end = assets.major_axis[index - 1]
    for etf in assets.minor_axis:
        sample = assets['close_price', window_begin:window_end, etf].fillna(method='ffill').dropna()
        if len(sample) >= trading_year:
            change = 1 + sample.pct_change()[1:]
            r_hist = change.cumprod().iloc[-1] - 1
            assets.set_value('r_hist', date, etf, r_hist)

Dynamic Return Projections¶

Trajectory of price and corresponding return based on continuously updated Linear Regression of entire price history.

In [8]:
assets['price_proj'], assets['r_proj'] = (np.nan, np.nan)
for date in assets.major_axis:
    index = assets.major_axis.get_loc(date)
    if index < trading_year:
        continue
    window_begin = assets.major_axis[0]
    window_end = assets.major_axis[index - 1]
    for etf in assets.minor_axis:
        sample = assets['close_price', window_begin:window_end, etf].fillna(method='ffill').dropna()
        if len(sample) >= trading_year:
            x = np.array(sample.index.to_julian_date())
            y = np.array(sample)
            m, b, r_value, p_value, std_err = stats.linregress(x, y)
            pred_date = date + pd.Timedelta(365, unit='d')
            price_reg = m * date.to_julian_date() + b
            price_proj = m * pred_date.to_julian_date() + b
            assets.set_value('price_proj', date, etf, price_proj)
            r_proj = (price_proj - price_reg) / price_reg
            assets.set_value('r_proj', date, etf, r_proj)

Dynamic Price Volatility¶

Probabilities and percentiles of each price based on continuously updated Empirical Distribution of entire price history. Empirical probabilities centered on the 50th percentile are then shifted onto the projected price.

In [9]:
assets['p_1yr'], assets['p_all'], assets['Q05_1yr'], assets['Q25_1yr']   = (np.nan, np.nan, np.nan, np.nan)
assets['Q50_1yr'], assets['Q75_1yr'], assets['Q95_1yr'], assets['Q05_all']  = (np.nan, np.nan, np.nan, np.nan)
assets['Q25_all'], assets['Q50_all'], assets['Q75_all'], assets['Q95_all'] = (np.nan, np.nan, np.nan, np.nan)
for date in assets.major_axis:
    index = assets.major_axis.get_loc(date)
    if index < trading_year:
        continue
    window_1yr = assets.major_axis[index - (trading_year + 1)]
    window_all = assets.major_axis[0]
    window_end = assets.major_axis[index - 1]
    for etf in assets.minor_axis:
        sample_1yr = assets['close_price', window_1yr:window_end, etf].fillna(method='ffill').dropna()
        sample_all = assets['close_price', window_all:window_end, etf].fillna(method='ffill').dropna()
        if len(sample_1yr) >= trading_year:
            x_1yr = sample_1yr.sort_values().unique()
            x_all = sample_all.sort_values().unique()
            frequency_1yr = sample_1yr.value_counts()
            frequency_all = sample_all.value_counts()
            probability_1yr = frequency_1yr / len(sample_1yr)
            probability_all = frequency_all / len(sample_all)
            ecdf_1yr = probability_1yr.sort_index().cumsum()
            ecdf_all = probability_all.sort_index().cumsum()
            F_1yr = interp1d(x_1yr, ecdf_1yr, 'linear', bounds_error=False, fill_value=(0, 1))
            F_all = interp1d(x_all, ecdf_all, 'linear', bounds_error=False, fill_value=(0, 1))
            price = assets['close_price', date, etf]
            assets.set_value('p_1yr', date, etf, F_1yr(price))
            assets.set_value('p_all', date, etf, F_all(price))
            Q_1yr = interp1d(ecdf_1yr, x_1yr, 'linear')
            Q_all = interp1d(ecdf_all, x_all, 'linear')
            assets.set_value('Q05_1yr', date, etf, Q_1yr(0.05))
            assets.set_value('Q05_all', date, etf, Q_all(0.05))
            assets.set_value('Q25_1yr', date, etf, Q_1yr(0.25))
            assets.set_value('Q25_all', date, etf, Q_all(0.25))
            assets.set_value('Q50_1yr', date, etf, Q_1yr(0.50))
            assets.set_value('Q50_all', date, etf, Q_all(0.50))
            assets.set_value('Q75_1yr', date, etf, Q_1yr(0.75))
            assets.set_value('Q75_all', date, etf, Q_all(0.75))
            assets.set_value('Q95_1yr', date, etf, Q_1yr(0.95))
            assets.set_value('Q95_all', date, etf, Q_all(0.95))

Return Required¶

Determine minimum required return for each day as the maximum of the mean historical returns of benchmark and the one-year treasury rate multiplied by the taqx penalty. FOr display puposes, the entire history is then smoothed using a Savitzky-Golay one-dimensional filter with a 253-day window and one degree polynomial.

In [10]:
assets['r_req'], assets['price_req'] = (np.nan, np.nan)
for date in assets.major_axis:
    r_treasury = Treasury_Yield['1 YR'][date]
    r_hist_marks = np.mean(assets['r_hist', date, benchmarks])
    r_req = max(r_treasury, r_hist_marks) * (1 + tax_pen)
    assets.set_value('r_req', date, assets.minor_axis, r_req)
    price_req = assets['close_price', date, assets.minor_axis] * (1 + r_req)
    assets.set_value('price_req', date, assets.minor_axis, price_req)
for etf in assets.minor_axis:
    savitzky_golay_r = signal.savgol_filter(assets['r_req',:,0].values, trading_year + 1, 1)
    assets.set_value('r_req', assets.major_axis, etf, savitzky_golay_r)
    savitzky_golay_price = signal.savgol_filter(assets['price_req',:,0].values, trading_year + 1, 1)
    assets.set_value('price_req', assets.major_axis, etf, savitzky_golay_price)
In [11]:
pyplot.plot(Treasury_Yield['1 YR'][assets.major_axis], label='1-Tear Treasury Rate', color='lightgreen')
pyplot.plot(assets['r_hist', assets.major_axis, symbols(25485)], label='AGG Annual Return', color='lightgrey')
pyplot.plot(assets['r_hist', assets.major_axis, symbols(8554)], label='SPY Annual Return', color='lightblue')
pyplot.plot(assets['r_hist', assets.major_axis, symbols(33486)], label='VEU Annual Return', color='pink')
pyplot.plot(assets['r_req', assets.major_axis, 0], label='Required Minimum Return', color='Black')
pyplot.title('Benchmarks vs Required')
pyplot.ylabel('Return')
pyplot.legend(loc=2)
pyplot.show()

Results Snapshot¶

In [12]:
assets[[0,4,6,8,10,1,3,16], (end_date - pd.Timedelta(7, unit='d')):end_date, symbols(8554)]
Out[12]:
close_price p_1yr Q05_1yr Q50_1yr Q95_1yr r_hist r_proj r_req
2017-07-28 00:00:00+00:00 246.90 0.981028 209.9065 225.51475 243.6420 0.164940 0.046705 0.173922
2017-07-31 00:00:00+00:00 246.74 0.972085 209.9065 225.53425 244.1305 0.162090 0.046700 0.174424
2017-08-01 00:00:00+00:00 247.31 0.994187 209.9065 225.61350 244.4905 0.159465 0.046707 0.174926
2017-08-02 00:00:00+00:00 247.44 0.999341 209.9065 226.27750 244.9410 0.163429 0.046714 0.175428
2017-08-03 00:00:00+00:00 246.97 0.975791 209.9065 226.87450 245.5140 0.171225 0.046721 0.175930
2017-08-04 00:00:00+00:00 247.40 0.992885 209.9065 226.88450 245.5925 0.165481 0.046728 0.176433

Plot Results and Decisions¶

In [13]:
for etf in securities:
    i = securities.index(etf)
    pyplot.figure(int(np.ceil((i + 1.0) / (2 ** 2))))
    pyplot.subplot(221 + i % (2 ** 2))
    price = assets['close_price', :, etf].fillna(method='ffill').dropna()
    hist = price.index
    r_req = assets['r_req', hist, etf]
    r_proj = assets['r_proj', hist, etf]
    price_proj = assets['price_proj', hist, etf]
    Q50_all = assets['Q50_all', hist, etf]
    Q05_shift = assets['Q05_all', hist, etf] + (price_proj - Q50_all)
    Q25_shift = assets['Q25_all', hist, etf] + (price_proj - Q50_all)
    Q75_shift = assets['Q75_all', hist, etf] + (price_proj - Q50_all)
    Q95_shift = assets['Q95_all', hist, etf] + (price_proj - Q50_all)
    price_proj =  price_proj.shift(periods=365, freq='d', axis=0)
    Q05_proj = Q05_shift.shift(periods=365, freq='d', axis=0)
    Q25_proj = Q25_shift.shift(periods=365, freq='d', axis=0)
    Q75_proj = Q75_shift.shift(periods=365, freq='d', axis=0)
    Q95_proj = Q95_shift.shift(periods=365, freq='d', axis=0)
    Q05_1yr = assets['Q05_1yr', hist, etf]
    Q25_1yr = assets['Q25_1yr', hist, etf]
    Q50_1yr = assets['Q50_1yr', hist, etf]
    Q75_1yr = assets['Q75_1yr', hist, etf]
    Q95_1yr = assets['Q95_1yr', hist, etf]
    invest = (r_proj >= r_req)
    divest = (r_proj < r_req)
    proj = price_proj.index
    fill = pyplot.fill_between
    fill(proj, Q05_proj, Q95_proj, label='Pred $\pm$ 0.45', color='lightblue', alpha=0.8, zorder=4)
    fill(proj, Q25_proj, Q75_proj, label='Pred $\pm$ 0.25', color='skyblue', alpha=0.5, zorder=5)
    pyplot.plot(price_proj, label='Prediction', color='steelblue', zorder=6)
    fill(hist, Q05_1yr, Q95_1yr, label='M $\pm$ 0.45', color='lightgrey', alpha=0.8, zorder=7)
    fill(hist, Q25_1yr, Q75_1yr, label='M $\pm$ 0.25', color='darkgrey', alpha=0.5, zorder=8)
    pyplot.plot(Q50_1yr, label='Median (M)', color='grey', zorder=9)
    pyplot.plot(price, label='Price', color='black', zorder=10)
    ymin, ymax = pyplot.ylim()
    fill(proj, ymin, ymax, where=proj > hist[-1], color='powderblue', alpha=0.5, zorder=1)
    fill(hist, ymin, ymax, where=invest, label='Invest', color='darkgreen', alpha=0.2, zorder=2)
    fill(hist, ymin, ymax, where=divest, label='Divest', color='firebrick', alpha=0.2, zorder=3)
    pyplot.title(etf.asset_name + ' (' + etf.symbol + ')')
    pyplot.ylabel('Price')
    pyplot.legend(loc=2).set_zorder(11)
    if (i + 1) % (2 ** 2) == 0:
        pyplot.show()

Backtest Results¶

The IDE (interactive development environment) is where you write your algorithm. It's also where you kick off backtests. Pressing "Build" will check your code for syntax errors and run a backtest right there in the IDE. Pressing "Full Backtest" runs a backtest that is saved and accessible for future analysis. Closing the browser will not stop the backtest from running. Quantopian runs in the cloud and it will continue to execute your backtest until it finishes running. If you want to stop the backtest, press the Cancel button. Before you can paper trade or live trade your algorithm, you must run a full backtest. Run your algorithm against IB's paper trading mode before deploying it to live trade on IB. Run your algorithm against Quantopian's paper trading mode before deploying it to live trade on Robinhood

In [15]:
result = get_backtest('598af7ca439e995462ce967a')
result.create_full_tear_sheet()

100% Time: 0:00:01|###########################################################|
Entire data start date: 2007-01-03
Entire data end date: 2007-12-31


Backtest Months: 11
Performance statistics Backtest
annual_return 0.01
cum_returns_final 0.01
annual_volatility 0.12
sharpe_ratio 0.18
calmar_ratio 0.17
stability_of_timeseries 0.06
max_drawdown -0.09
omega_ratio 1.04
sortino_ratio 0.25
skew -0.36
kurtosis 1.77
tail_ratio 0.75
common_sense_ratio 0.76
gross_leverage 0.44
information_ratio -0.03
alpha -0.02
beta 0.63
Worst drawdown periods net drawdown in % peak date valley date recovery date duration
0 8.54 2007-07-19 2007-08-16 2007-09-21 47
1 6.19 2007-10-31 2007-11-26 NaT NaN
2 3.06 2007-06-04 2007-06-26 2007-07-09 26
3 2.58 2007-10-12 2007-10-22 2007-10-26 11
4 1.07 2007-10-29 2007-10-30 2007-10-31 3 

[-0.015 -0.026]
Stress Events mean min max
Aug07 0.03% -1.93% 2.06%
Low Volatility Bull Market -0.02% -2.32% 1.34%
GFC Crash 0.05% -2.24% 2.30%
Top 10 long positions of all time max
IAU-26981 13.58%
VDE-26667 11.27%
VWO-27102 11.13%
VOX-26670 8.49%
VGK-27100 7.85%
XLE-19655 7.85%
VNQ-26669 7.38%
VPL-27101 7.25%
FEZ-26432 6.39%
VIG-28364 5.85%
Top 10 short positions of all time max
Top 10 positions of all time max
IAU-26981 13.58%
VDE-26667 11.27%
VWO-27102 11.13%
VOX-26670 8.49%
VGK-27100 7.85%
XLE-19655 7.85%
VNQ-26669 7.38%
VPL-27101 7.25%
FEZ-26432 6.39%
VIG-28364 5.85%
All positions ever held max
IAU-26981 13.58%
VDE-26667 11.27%
VWO-27102 11.13%
VOX-26670 8.49%
VGK-27100 7.85%
XLE-19655 7.85%
VNQ-26669 7.38%
VPL-27101 7.25%
FEZ-26432 6.39%
VIG-28364 5.85%

Getting Started With Quantopian¶

The Quantopian Tutorials are a good place to start. The Getting Started Tutorial covers the basics of the Quantopian API and walks you through writing your first algorithm in 11 short lessons that you can read, watch, or both. The Getting Started with Futures Tutorial demonstrates how to research a quantitative strategy using futures, and implement it in an algorithm. The Pipeline Tutorial teaches you how to build factors and filters to dynamically select and weight securities each day. The Algorithmic Trading Tutorial by Sentdex covers the quant workflow step by step, from finding and analyzing alpha factors to building and optimizing your algorithm.

The Quantopian Lecture Series¶

Once you are comfortable developing your own algorithm, check out the Quantopian Lecture Series to learn more about quantitative finance. The series includes lessons on linear regression, beta regression, pairs trading, momentum strategies, the dangers of overfitting, and much more. These lessons will help you design more advanced trading strategies.

Publishing HTML to RPubs¶

Using the R "markdown" package, run the following code to publish a new document:

install.packages("markdown") library("markdown") result <- rpubsUpload(title='Title', htmlFile='filepath/filename.html', method=getOption('rpubs.upload.method', 'auto')) browseURL(result$continueUrl)

Run the following code to update an existing document:

updateResult <- rpubsUpload(title="Title", htmlFile="filepath/filename.html", id=result$id) browseURL(updateResult$continueUrl)