A lookback on the S&P500
Produced in R Markdown using Python
Paul Jozefek
2020-12-31
2020 has been a great year for the stock market, following many great years. So how does this bull run compare to history? I examined returns data to find out.
Retrieve Data
> import numpy as np
+ import pandas as pd
+ import matplotlib.pyplot as plt
+ import seaborn as sns
+ import yfinance as yf
[*********************100%***********************] 1 of 1 completed Open High Low Close Adj Close Volume
Date
1927-12-30 17.660000 17.660000 17.660000 17.660000 17.660000 0
1928-01-03 17.760000 17.760000 17.760000 17.760000 17.760000 0
1928-01-04 17.719999 17.719999 17.719999 17.719999 17.719999 0
1928-01-05 17.549999 17.549999 17.549999 17.549999 17.549999 0
1928-01-06 17.660000 17.660000 17.660000 17.660000 17.660000 0 Open High ... Adj Close Volume
Date ...
2020-12-24 3694.030029 3703.820068 ... 3703.060059 1885090000
2020-12-28 3723.030029 3740.510010 ... 3735.360107 3527460000
2020-12-29 3750.010010 3756.120117 ... 3727.040039 3387030000
2020-12-30 3736.189941 3744.629883 ... 3732.040039 3145200000
2020-12-31 3733.270020 3760.199951 ... 3756.070068 3172510000
[5 rows x 6 columns]Price Charts
Linear Scale
> sns.set_style('darkgrid')
+ SPX.loc['2008':]['Adj Close'].plot(kind='line', rot=30);
+ plt.ylabel("Adj Close Price", labelpad=16);
+ plt.xlabel("Date", labelpad=16);
+ plt.title('LINEAR S&P 500 2008 to Dec 2020',
+ y=1.02, fontsize=20);
+ plt.show()
There have been some periods of short term losses, but from the bottom in 2008 to the end of 2020 the market has mostly moved higher.
> SPX['Adj Close'].plot(kind='line', rot=30);
+ plt.ylabel("Adj Close Price", labelpad=16);
+ plt.xlabel("Date", labelpad=16);
+ plt.title('LINEAR S&P 500 Dec 1927 to Dec 2020',
+ y=1.02, fontsize=20);
+ plt.show()
If we go back further it certainly appears to be a tremendous move.
However, the chart is difficult to interpret on a linear scale. A increase of \(5\) is very meaningful when the index is trading \(10\), but it’s barely noticeable when the index is trading at \(3,500\). On a logarithmic scale a move in the index from \(5\) to \(10\) would appear the same as a move from \(10\) to \(20\), from \(20\) to \(40\), and from \(1,500\) to \(3000\).
| Index | Log10 | Linear Difference | Log Difference |
|---|---|---|---|
| 5 | 0.7 | ||
| 10 | 1.0 | 5 | 0.3 |
| 20 | 1.3 | 10 | 0.3 |
| 40 | 1.6 | 20 | 0.3 |
| 1500 | 3.18 | ||
| 3000 | 3.48 | 1500 | 0.3 |
Logarithmic Scale
> from matplotlib.ticker import ScalarFormatter
+
+ ax = SPX['Adj Close'].plot(kind='line', rot=30, logy=True)
+ plt.ylabel("Adj Close Price", labelpad=18);
+ plt.xlabel("Date", labelpad=18);
+ plt.title('LOGARITHMIC S&P 500 Dec 1927 to Dec 2020',
+ y=1.02, fontsize=20);
+ for axis in [ax.yaxis]:
+ axis.set_major_formatter(ScalarFormatter())
+ ax.set_yticks([5,10,20,40,60,90,160,320,500
+ ,640,1000,1280,1750,2560,3500]);
+ plt.show()
Once it is displayed on a logarithmic scale (percentage changes are comparable) the recent bull market doesn’t look quite as dramatic.
On a linear scale the current period (bottom of 2008 to present) looks like the greatest in history, but on a logarithmic scale we can see that the bull market of the 80’s and 90’s was far greater (peaked in 1999).
> print(' 1980 to 2000 low: ',
+ SPX.loc['1980':'2000']['Adj Close'].min(),'\n',
+ '1980 to 2000 high: ',
+ SPX.loc['1980':'2000']['Adj Close'].max(),'\n',
+ '2008 to 2020 low: ',
+ SPX.loc['2008':'2020']['Adj Close'].min(),'\n',
+ '2008 to 2020 high: ',
+ SPX.loc['2008':'2020']['Adj Close'].max()) 1980 to 2000 low: 98.22000122070312
1980 to 2000 high: 1527.4599609375
2008 to 2020 low: 676.530029296875
2008 to 2020 high: 3756.070068359375> ax = SPX.loc['1980':]['Adj Close'].plot(kind='line',
+ rot=30, logy=True)
+ plt.ylabel("Adj Close Price", labelpad=18);
+ plt.xlabel("Date", labelpad=18);
+ plt.title('LOGARITHMIC S&P 500 1980 to Dec 2020',
+ y=1.02, fontsize=20);
+ for axis in [ax.yaxis]:
+ axis.set_major_formatter(ScalarFormatter())
+ ax.set_yticks([98,120,160,200,250,320,450,676,
+ 1000,1280,1528,2000,2560,3000,3740]);
+ plt.show()
Annual Returns
Open High ... Volume Annual Return
Date ...
2016-12-31 2251.610107 2253.580078 ... 2670900000 9.535023
2017-12-31 2689.149902 2692.120117 ... 2443490000 19.419966
2018-12-31 2498.939941 2509.239990 ... 3442870000 -6.237260
2019-12-31 3215.179932 3231.719971 ... 2893810000 28.878070
2020-12-31 3733.270020 3760.199951 ... 3172510000 16.258923
[5 rows x 7 columns]> import matplotlib.ticker as mtick
+
+ ax=df['Annual Return'].plot.bar(rot=90)
+ plt.ylabel("Return", labelpad=16);
+ plt.xlabel("Date", labelpad=16);
+ plt.xticks([]);
+ ax.yaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.title('Annual Returns', y=1.02, fontsize=20);
+ plt.show()
We haven’t had a major down year (drop of >10%) since 2008.
> ax=df['Annual Return'].plot.hist(rot=90, bins=30)
+ plt.xlabel("Percent Return", labelpad=16);
+ ax.xaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.title('Returns Distribution', y=1.02, fontsize=20);
+ plt.show()
The returns are positively skewed, with very few results below \(-20\%\).
Rolling Returns
Prior 10 Years
Adj Close Volume Annual Return 10yr Return
Date
1927-12-31 17.660000 0 NaN NaN
1928-12-31 24.350000 0 37.882223 NaN
1929-12-31 21.450001 0 -11.909649 NaN
1930-12-31 15.340000 0 -28.484850 NaN
1931-12-31 8.120000 0 -47.066494 NaN
1932-12-31 6.920000 0 -14.778323 NaN
1933-12-31 9.970000 0 44.075147 NaN
1934-12-31 9.500000 0 -4.714145 NaN
1935-12-31 13.430000 0 41.368424 NaN
1936-12-31 17.180000 0 27.922561 NaN
1937-12-31 10.550000 0 -38.591385 -40.260474> ax = df['10yr Return'].plot( rot=30)
+ ax.yaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("10yr Return", labelpad=16);
+ plt.title('Prior 10 Year Returns', y=1.02, fontsize=20);
+ plt.show()
We can see that stronger periods tend to be followed by weaker periods, but that’s not always the case. Results were positive in the 1940s and again in the 1950s, and very good in 1980s and 1990s.
Prior 20 Years
> ax = df['20yr Return'].plot(rot=30)
+ ax.yaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("20yr Return", labelpad=16);
+ plt.title('Prior 20 Year Returns', y=1.02, fontsize=20);
+ plt.show()
On a 20 year basis you can see the impact of consecutive strong decades, particularly the time period ending 1999.
Fed Funds Rate
The previous analysis doesn’t factor in the Federal Funds Rate, which differs dramatically over time and needs to be considered.
> start = datetime.datetime(1954, 7, 1)
+ end = datetime.datetime(2020, 11, 1)
+
+ fed = web.DataReader("FEDFUNDS", "fred", start, end)
+ fed.tail() FEDFUNDS
DATE
2020-07-01 0.09
2020-08-01 0.10
2020-09-01 0.09
2020-10-01 0.09
2020-11-01 0.09> fed.plot(rot=30)
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("Fed Funds Rate", labelpad=16);
+ plt.title('Fed Funds Rate', y=1.02, fontsize=20);
+ plt.show()
Rates were extremely high throughout the 1980s and they are extremely low now.
Real Returns
The Fed Funds rate can be considered the risk-free rate. If you earned \(10\%\) by keeping your money in the bank (or Treasury Bills/CDs) and that same year the stock market increased by \(12\%\), you wouldn’t be that excited about your excess return. You only made \(2\%\) by taking on the risk.
I chose to look at the market over time with Real Return representing your excess return.
FEDFUNDS
DATE
1954-12-31 1.008333
1955-12-31 1.785000
1956-12-31 2.728333
1957-12-31 3.105000
1958-12-31 1.572500> # Combine FED funds rate with annual returns
+ df3 = df.loc['1954':,['Adj Close','Annual Return']]
+ df4 = pd.concat([df3['Annual Return'],
+ fed_df['FEDFUNDS']],axis=1) Annual Return FEDFUNDS
Date
1980-12-31 25.773570 13.355833
1981-12-31 -9.730401 16.378333
1982-12-31 14.761319 12.258333
1983-12-31 17.271042 9.086667
1984-12-31 1.400602 10.225000
1985-12-31 26.333408 8.100833
1986-12-31 14.620409 6.805000
1987-12-31 2.027503 6.657500
1988-12-31 12.400841 7.568333
1989-12-31 27.250465 9.216667
1990-12-31 -6.559138 8.099167
1991-12-31 26.306703 5.687500
1992-12-31 4.464263 3.521667
1993-12-31 7.055156 3.022500
1994-12-31 -1.539291 4.201667
1995-12-31 34.110655 5.836667
1996-12-31 20.263666 5.298333
1997-12-31 31.008182 5.460000
1998-12-31 26.668589 5.353333
1999-12-31 19.526047 4.970000
2000-12-31 -10.139185 6.235833The Real Return was dramatically different in the 1980s relative to the actual return. For example, in 1981 the risk-free rate was about \(16\%\) and you would have lost almost \(10\%\) in the market, so your Real Return was about \(-26\%\).
Rolling Real Returns
Prior 10 Years
> #Create data frame with prior 10 year real returns
+
+ df5 = pd.DataFrame(columns=['Date', '10yr_Return',
+ '10yr_Fed','10yr_Real'])
+ c = 0
+
+ for i in range(1954,2012):
+ a = df4.loc[str(i):str(i+9)].index[-1]
+ b =((df4.loc[str(i):str(i+9)]['Annual Return']/100+1
+ ).cumprod()[-1]-1)*100
+ d = ((df4.loc[str(i):str(i+9)]['FEDFUNDS']/100+1
+ ).cumprod()[-1]-1)*100
+ df5.loc[c] = [a,b,d,b-d]
+ c+=1 10yr_Return 10yr_Fed 10yr_Real
Date
1980-12-31 47.325005 109.596328 -62.271323
1981-12-31 20.041147 133.058367 -113.017219
1982-12-31 19.135956 150.519008 -131.383052
1983-12-31 69.072258 151.346564 -82.274306
1984-12-31 143.932339 150.715368 -6.783029
1985-12-31 134.260997 156.109177 -21.848180
1986-12-31 125.358273 160.400216 -35.041943
1987-12-31 159.810732 163.163672 -3.352940
1988-12-31 188.960565 162.279802 26.680764
1989-12-31 227.404100 157.615363 69.788737
1990-12-31 143.238078 145.668928 -2.430851
1991-12-31 240.342706 123.101106 117.241600
1992-12-31 209.805172 105.737941 104.067231
1993-12-31 182.816973 94.300895 88.516077
1994-12-31 174.617301 83.683167 90.934133
1995-12-31 191.523096 79.835932 111.687163
1996-12-31 205.876036 77.299040 128.576996
1997-12-31 292.759424 75.308410 217.451014
1998-12-31 342.614855 71.698536 270.916319
1999-12-31 315.747036 65.022390 250.724646
2000-12-31 299.818310 62.177856 237.640454> ax = df5['10yr_Real'].plot( rot=30)
+ ax.yaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("10yr Real Returns", labelpad=16);
+ plt.title('Prior 10 Year Real Returns', y=1.02, fontsize=20);
+ plt.show()
> print(' 1990 previous 10yr real return:',
+ df5.loc['1990','10yr_Real'][0],'Percent','\n',
+ '1990 previous 10yr return:',
+ df.loc['1990','10yr Return'][0],'Percent') 1990 previous 10yr real return: -2.4308505452262636 Percent
1990 previous 10yr return: 143.23807791851047 Percent> print(' 1999 previous 10yr real return:',
+ df5.loc['1999','10yr_Real'][0],'Percent','\n',
+ '1999 previous 10yr return:',
+ df.loc['1999','10yr Return'][0],'Percent') 1999 previous 10yr real return: 250.72464631650706 Percent
1999 previous 10yr return: 315.7470360427802 Percent> print(' 2020 previous 10yr real return:',
+ df5.loc['2020','10yr_Real'][0],'Percent','\n',
+ '2020 previous 10yr return:',
+ df.loc['2020','10yr Return'][0],'Percent') 2020 previous 10yr real return: 192.14378675355334 Percent
2020 previous 10yr return: 198.66019088215415 PercentThe differences in actual return versus the adjusted return are quite dramatic. In the 10 years ending 1990 you would have made more money investing in a risk free asset then if you had invested in the market. However, if you looked at the market returns alone it would have seemed like a great decade to invest.
On an adjusted basis the most recent 10 years is much closer to the 10 years ending in 1999.
Prior 20 Years
> df6 = pd.DataFrame(columns=['Date', '20yr_Return',
+ '20yr_Fed','20yr_Real'])
+ c = 0
+
+ for i in range(1954,2002):
+ a = df4.loc[str(i):str(i+19)].index[-1]
+ b =((df4.loc[str(i):str(i+19)]['Annual Return']/100+1
+ ).cumprod()[-1]-1)*100
+ d = ((df4.loc[str(i):str(i+19)]['FEDFUNDS']/100+1
+ ).cumprod()[-1]-1)*100
+ df6.loc[c] = [a,b,d,b-d]
+ c+=1 20yr_Return 20yr_Fed 20yr_Real
Date
2016-12-31 202.242367 57.573727 144.668639
2017-12-31 175.507778 50.912280 124.595497
2018-12-31 103.936622 45.867706 58.068916
2019-12-31 119.893145 41.960577 77.932568
2020-12-31 184.490410 34.164712 150.325698> ax = df6['20yr_Real'].plot(rot=30)
+ ax.yaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("20yr Real Return", labelpad=16);
+ plt.title('Prior 20 Year Real Returns', y=1.02, fontsize=20);
+ plt.show()
> print(' 1999 previous 20yr real return:',
+ df6.loc['1999','20yr_Real'][0],'Percent','\n',
+ '1999 previous 20yr return:',
+ df.loc['1999','20yr Return'][0],'Percent') 1999 previous 20yr real return: 936.0498139545756 Percent
1999 previous 20yr return: 1261.172843031537 Percent> print(' 2020 previous 20yr real return:',
+ df6.loc['2020','20yr_Real'][0],'Percent','\n',
+ '2020 previous 20yr return:',
+ df.loc['2020','20yr Return'][0],'Percent') 2020 previous 20yr real return: 150.3256978576666 Percent
2020 previous 20yr return: 184.49040998974274 PercentOn a 20 year basis the tech bubble rally still eclipses the current period by quite a bit.
Sample Returns
On a rolling basis the losses of 2001 and 2002 are about to drop off and in another 8 years the Great Recession of 2008 will drop off.
Annual Return FEDFUNDS
Date
2001-12-31 -13.042693 3.887500
2002-12-31 -23.365964 1.666667
2003-12-31 26.380400 1.127500
2004-12-31 8.993452 1.349167
2005-12-31 3.001023 3.213333
2006-12-31 13.619432 4.964167
2007-12-31 3.529573 5.019167
2008-12-31 -38.485793 1.927500Even with modest returns over the next 8 years the Real Returns will spike higher.
As an example, I estimated real returns of \(5\%\) over the next 3 years which then drop to \(3\%\) in the 5 years after.
> x = [datetime.datetime(2021, 12, 31),
+ datetime.datetime(2022, 12, 31),
+ datetime.datetime(2023, 12, 31),
+ datetime.datetime(2024, 12, 31),
+ datetime.datetime(2025, 12, 31),
+ datetime.datetime(2026, 12, 31),
+ datetime.datetime(2027, 12, 31),
+ datetime.datetime(2028, 12, 31)] Annual Return FEDFUNDS
2024-12-31 5.0 2.0
2025-12-31 5.0 2.0
2026-12-31 5.0 2.0
2027-12-31 5.0 2.0
2028-12-31 5.0 2.0> df9 = pd.DataFrame(columns=['Date', '20yr_Return',
+ '20yr_Fed','20yr_Real'])
+ c = 0
+
+ for i in range(1954,2010):
+ a = df8.loc[str(i):str(i+19)].index[-1]
+ b =((df8.loc[str(i):str(i+19)]['Annual Return']/100+1
+ ).cumprod()[-1]-1)*100
+ d = ((df8.loc[str(i):str(i+19)]['FEDFUNDS']/100+1
+ ).cumprod()[-1]-1)*100
+ df9.loc[c] = [a,b,d,b-d]
+ c+=1 20yr_Return 20yr_Fed 20yr_Real
Date
2024-12-31 276.718470 26.417484 250.300986
2025-12-31 284.029579 24.931372 259.098206
2026-12-31 254.896210 21.403336 233.492875
2027-12-31 259.936789 17.913145 242.023644
2028-12-31 514.384298 17.997015 496.387282> ax = df9['20yr_Real'].plot(rot=30)
+ ax.yaxis.set_major_formatter(mtick.PercentFormatter())
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("20yr Real Returns", labelpad=16);
+ plt.title('Prior 20 Year Real Returns \n 2021 - 2028 Estimated ',
+ y=1.02, fontsize=20);
+ plt.tight_layout()
+ plt.show()
Although the 20 year Real Returns could move higher as the down years drop off, it will be difficult to approach 1999.
Valuations
The prior analysis only examines market returns and the Fed Funds rate.
I was able to retrieve valuation data from
https://www.macrotrends.net/2577/sp-500-pe-ratio-price-to-earnings-chart
value
date
2020-08-01 35.6266
2020-09-01 34.2290
2020-10-01 33.2820
2020-11-01 36.8614
2020-12-01 38.2297> ax = SPXpe.plot(kind='line', rot=30, logy=True)
+ plt.ylabel("PE value", labelpad=18);
+ plt.xlabel("Date", labelpad=18);
+ plt.title('LOGARITHMIC S&P 500 PE \n Dec 1927 to Dec 2020',
+ y=1.02, fontsize=20);
+ for axis in [ax.yaxis]:
+ axis.set_major_formatter(ScalarFormatter())
+ ax.set_yticks([10,20,40,60,80,120]);
+ plt.tight_layout()
+ plt.show()
> print(' 2001 Max PE: ',SPXpe.loc['2001'].max()[0],'\n',
+ '2008 Max PE: ',SPXpe.loc['2008'].max()[0],'\n',
+ '2009 Max PE: ',SPXpe.loc['2009'].max()[0],'\n',
+ '2020 Max PE: ',SPXpe.loc['2020'].max()[0]) 2001 Max PE: 46.4998
2008 Max PE: 65.1042
2009 Max PE: 122.4128
2020 Max PE: 38.2297The market valuation is very high relative to history. Valuations were only higher after the tech bubble and the Great Recession, when earnings dropped dramatically.
However,
These valuations are backwards looking. Forward valuations would be more relevant, but I could not find a publicly available source for estimates.
The current period has been impacted from Covid-19 (depressed earnings)
Business models have changed. The market is more weighted towards Technology, where the current trend is to focus on market share and revenue growth above profits. Investors are more accustomed to higher earnings valuations.
Conclusions
It seems prudent to be more cautious moving forward. 10 year excess returns and market valuations are very high relative to history.
There’s no good way to predict when we might see a correction. The market often moves both too high and too low before normalizing.
Other factors might influence the market. Federal Debt to GDP is at an all time high, which should also lead to caution.
> start = datetime.datetime(1966, 1, 1)
+ end = datetime.datetime(2020, 7, 1)
+
+ debt = web.DataReader("GFDEGDQ188S", "fred", start, end)
+ debt.head() GFDEGDQ188S
DATE
1966-01-01 40.33999
1966-04-01 39.26763
1966-07-01 39.62091
1966-10-01 39.51977
1967-01-01 39.20383> debt.plot(rot=30)
+ plt.xlabel("Date", labelpad=16);
+ plt.ylabel("Debt to GDP Percent", labelpad=16);
+ plt.title('Federal Debt to GDP Percent', y=1.02, fontsize=20);
+ plt.show()