When it comes to investing, there is certainly no shortage of opinions on the Internet. I myself was extremely influenced by The Millionaire Next Door and A Random Walk Down Wall Street, both essential reading for anyone trying to get rich, or die trying. However, I don't think any author was more influential on my developing investment attitude than the guy who draws the Dilbert cartoons was back in the late 90's.

Dilbert by Scott Adams
Pictured above: regression to the mean with pointy hair
Dilbert by Scott Adams

In the 2002 classic Dilbert and the Way of the Weasel, Scott Adams was kind enough to present a complete financial plan for the unsure investor, and better yet, he did so in less than one page. Adams' simple rules for success are as follows:

  1. Make a will.

  2. Pay off your credit cards.

  3. Get term life insurance if you have a family to support.

  4. Fund your 401k to the maximum.

  5. Fund your IRA to the maximum.

  6. Buy a house if you want to live in a house and can afford it. (emphasis mine)

  7. Put six months worth of expenses in a money-market account.

  8. Take whatever money is left over and invest 70% in a stock index fund and 30% in a bond fund through any discount broker and never touch it until retirement.

  9. If any of this confuses you, or you have something special going on (retirement, college planning, tax issues), hire a fee-based financial planner, not one who charges a percentage of your portfolio.

I've read dozens of books on finance in the 16 years since that was published, and I still find that to be some of the best advice on the subject I have ever encountered. In this article, we are going to consider points 4, 5 and 8. More specifically: which index fund should we invest in? And what should we expect to happen once we do?

The S&P 500

The Standard and Poor's 500 is perhaps the most famous of all stock indexes. The first iteration began in 1926, and the S&P 500 has existed in current form for over 50 years now. The S&P 500 is a weighted portfolio of 500 (technically 505) different companies, designed to offer a balanced perspective of all sectors of the American stock market.

The idea is this: rather than invest in a single stock, we can diversify into a bucket of stocks. When one industry is lagging (e.g. manufacturing) we can trust other industires to make up the difference (e.g. technology). Because of this, the S&P 500 is usually the benchmark of choice when discussing how well a given investment will do. Despite the best efforts of the professionals, they rarely beat the S&P 500 for very long, if ever.

Professionals rarely beat the S&P 500
Most "professional" investors can't beat this guy

The SPY, aka SPDR, aka "spider", is a stock index fund that simply follows the S&P 500. For this article we took the adjusted close price of SPY, from its inception on January 29, 1993 through the close of business on July 13, 2018 via the quandl service, and used that as the basis for the following analysis.

The Coming Crash?

As of the summer of 2018, some corners of the Internet have begun speculating that the stock market is overdue for another crash. Considering the crashes of 2001 and 2008, and the unprecedented growth we have experienced in recent years, this is not wholly without reason. Others say that thanks to the Fed's policy of quantitative easing, the next recession is nothing to worry about anyway. Which side is right? Who knows? Only time will tell.

Here's what I know for sure: we're about to walk through 25 years of SPY history, through two of the most impactful stock market crashes in history. We're going to look at the worst of the worst, and the best and the best, and try to understand the S&P 500 better than ever before. As far as the lawyers are concerned, nothing in this article should be considered investment advice. This is for educational purposes only.

That said, what is the data telling us here?

Economic Eras

The S&P 500 SPY index fund began trading on January 29, 1993. The world has changed a lot since then. For starters, we've had four different presidents in that time:

Economic Eras by President

The darker bands above mark the "lame duck" period, between the election, and the new president taking office. However, visualizations like this can be misleading. For example, if we blame George W. Bush for the crash of 2008, does this mean we credit Donald Trump for the continued spike following his election?

We could just as easily visualize this data like so:

Economic Eras by Popular TV Show

Does it make sense to blame "Friends" for the dot-com crash? Is it rational to credit "The Big Bang Theory" with the current boom? Does this prove, once and for all, that "American Idol" is terrible?

What if we ignore our own biases, and just look at the numbers?

Economic Eras of the S&P 500

If we ignore everything but the closing price of SPY, we can see that the past 25 years of the S&P 500 break into four distinct eras. To better understand the full story, let's consider each of these periods individually.

The 90's Boom

Coincidentally, roughly 2 months after the launch of "spider", Tim Berners-Lee released the source code of what was to become the World Wide Web. In 1994, Microsoft joined the S&P 500 for the first time. By 1996, Yahoo! had become dominate among the early web, and launched a high-profile IPO. By 1997 we had entered the dot-com bubble in full force. At the start of this era, the World Wide Web did not yet exist. By the end, it was hard to find a high schooler who hadn't already discovered the joy of getting free music from Napster.

The S&P 500 During the 90's Boom

It is worth remembering: although eBay and Amazon are members of the S&P 500 today, back then they were just upstarts on the NASDAQ, as were most tech IPOs of the time. On March 10, 2000, roughly two weeks before the end of the period pictured above, the dot-com bubble began to burst, and in the next 30 months, the NASDAQ would drop 78% from its peak.

If you purchased SPY on January 29, 1993, and sold it on March 24, 2000, you would have earned a total return on investment of 300.04% after 2,611 days (~7 years), an annualized return of 21.39%. In other words, every dollar invested in SPY at the start of this period was worth $3.00 at the end of this period.

The 00's Crash

As the 90's drew to a close, many proclaimed that Y2K would bring disaster. In retrospect, perhaps they were right.

The S&P 500 During the 00's Crash

About 18 months after the dot-com bubble burst, the September 11 attacks occurred, and thus began a series of vicious wars in the Middle East. As of the time of this publication, Operation Enduring Freedom remains an active military campaign, more than 17 years, and two presidents later. Approximately three months after 9/11, Enron filed for bankruptcy, which was the culmination of one of the largest corporate scandals in American history.

If you purchased SPY on March 24, 2000, and sold it on October 9, 2002, you would have lost a total ROI of -47.5% after 929 days (~2.5 years), an annualized return of -22.37%. In other words, every dollar invested in SPY at the start of this period was worth $0.52 at the end of this period. About half of your money, gone, after 2.5 years.

The Lost Decade

The Lost Decade was a phrase originally coined in regards to the Japanese economy, but if you will allow me some poetic license, I think it can apply to this 6.5 year period in the American economy too.

Following the 00's crash, the S&P 500 entered a period of solid recovery, peaking on October 9, 2007. Unfortunately, the subprime mortgage crisis resulted in the bankruptcy of S&P 500 member Lehman Brothers on September 15, 2008, which triggered the stock market crash of 2008-2009. This crash wiped out all the prior gains following the 00's Crash.

The S&P 500 During the Lost Decade

If you purchased SPY on October 9, 2002, and sold it on March 9, 2009, you would have lost a total ROI of -1.91% after 2,343 days (~6.5 years), an annualized return of -0.30%. In other words, every dollar invested in SPY at the start of this period was worth $0.98 at the end of this period.

The Current Boom

Following the stock market crash of 2008-2009, the Federal Reserve began a policy of quantitative easing. In layman's terms, our government began buying up unwanted assets in order to stimulate the economy and prevent things from crashing any further. This policy continued through the end of 2013. On December 6, 2015, the Federal Reserve increased the Federal funds rate for the first time since 2006, unofficially marking the end of the bailout crisis.

As of July 13, 2018, this boom has lasted 3,412 days (over 9 years), which is about 30% longer than the 90's boom which lasted 2,611 days. We are currently living through the single longest expansion in the history of the American economy.

The S&P 500 During the Current Boom

If you purchased SPY on March 9, 2009, and sold it on July 13, 2018, you would have earned a total ROI of 397.97% after 3,412 days (~9 years), an annualized return of 18.74%. In other words, every dollar invested in SPY at the start of this period was worth $3.98 at the end of this period.

SPY closed at $279.59 on July 13. As of the end of business on August 1, 2018 (the time of this publication) SPY closed at $280.86, still inching upwards into uncharted territory.

The Key Lesson

If you purchased SPY on January 29, 1993, and held on to it, through all the ups and downs detailed above, and sold it on July 13 2018, you would have earned a total ROI of 926.60% after 9,296 days (~25 years), an annualized return of 9.58%. In other words, every dollar invested in SPY at the start was worth $9.27 at the end of this period.

Of course, we're currently in the middle of a boom. What if we stopped at the bottom of the 2008-2009 crash instead?

If you purchased SPY on January 29, 1993, and sold at the bottom of the crash, on March 9, 2009, you would have earned a total ROI of 105.99% after 5,883 days (~16 years), an annualized return of 4.59%. In other words, every dollar invested in SPY at the start was worth $2.06 at the lowest point of the crash.

This is where we get the saying: "time in the market beats timing the market". Or if you prefer:

Warren Buffet Quote
From the master himself

Measuring Risk

What would happen if we ignored Warren Buffett's advice to be patient, bought a share of SPY, and sold it the next day? Obviously our result would depend on which day we picked. Considering the past 25 years of data, the best return we found after a single trading day was 14.52%; the worst return after a single day was -9.84%.

If you were asked to guess which day was the worst we could have picked, you would be right to assume it was somewhere during the biggest crash, in 2008. But when do you suppose the best day was? Would it surprise you to learn that the best day to pick was only 2 trading days before the worst?

Best and Worst Days to Buy SPY

When it comes to stocks, we measure risk in terms of volatility, which is just a fancy word for standard deviation. This looks at how much a group of observations diverge from their mean. Higher volatility means we are diverging further from the mean. This is true if we drop below the mean, and also true if we boom above it. In other words, just because a stock is encountering high volatility doesn't necessarily mean it will go down. It could just as easily go up.

This makes sense when we look at the chart from October 2008. Depending on the day, we might get significant gains, or devastating losses. In periods of lower volatility, this spread between best and worst is less extreme.

The Probability of Pain

Below we have visualized the SPY 1 day return for every day in our 25 years of data. In other words, we bought one share of SPY on January 29, 1993, and sold it on the next trading day, February 1, 1993. Then we bought a share on February 1, 1993 and sold it on February 2, 1993. We then repeated this for every single trading day over the past 25 years.

SPY ROI Distribution After 1 Day

This box plot (left) and histogram (right) show the same data. On one extreme is our worst day: -9.84%. On the other extreme is our best day: 14.52%. In between are the results of all other days. As we can see, these ROIs appear to be pretty normal. That is, we see the expected bell curve of any normal distribution. This allows us to think about things in terms of probability.

SPY ROI Distribution After 1 Day v Normal Distribution

In any normal distribution, we expect 95% of observations to fall within 2 standard deviations of the mean. In other words, 95% of the time we would expect to return between -2.26% and 2.35% after one day. Checking our data, we find that indeed 95.04% of our one day ROIs fall within this range.

More simply, let us assume we have the same chance of picking any particular day from our data.

SPY ROI Distribution After 1 Day - win or lose

The days in the red area lost money; those in the green area gained money. Ergo, we can calculate a "probability of losing money" by comparing the number of days that fall in the red, compared to the total number of all days. This gives us a 45.33% chance of losing money, assuming we buy a share of SPY on any given day, and sell it on the following day.

How much do we expect to earn after one day? For this we can use the expected value, which happens to be the same as the mean of all the above trials. We expect to earn 0.04% after holding a share of SPY for a single day.

So what would happen if we ignored Warren Buffett's advice to be patient, bought a share of SPY, and sold it the next day? On 45.33% of days we will lose. In the worst case we lose -9.84%, but usually (95% of the time) no more than -2.26%. On 54.67% of days we will win. In the best case we win 14.52%, but usually no more than 2.35%. And if we do this again and again and again, and average our results, we expect to earn 0.04% after any given day.

Does the S&P 500 suck?

An expected return of 0.04%, and a 45.33% chance of losing money, doesn't sound very good at all.

Is it time to sell our stocks and buy canned goods and AR-15's instead?

Slow your roll, dawg. What if we listen to Buffett instead?

Buddha Quote
From the master himself

Risk Over Time

What happens to our probability of losing money if we hold our investment a little longer? To find out, we repeated the above one day analysis for longer periods, purchasing one share of SPY on any given day, and selling it x days later, to determine what our ROI would have been.

The results look like this:

Probability of Losing Money with SPY over time

The lesson is clear: as our patience increases, our risk plummets. According to this, after 12 years we have a 0% probability of losing money. How is that even possible? Let's compare the worst 1 year period to the worst 12 year period to find out.

SPY worst 1 year period

As you would expect, the worst 1 year period occurred during the stock market crash of 2008-2009. If you bought SPY at the start of this period, and sold at the end, you lost -47.36% of your money. Every dollar invested became a measly $0.53.

SPY worst 12 year period

By comparison, 12 years is so long that it outlasts the crashes on both ends. If you bought SPY at the start of this period, and sold at the end, you earned 0.74% on your money. Every dollar invested became almost $1.01. Not bad for a worst case scenario. But what if we stop at the bottom of the crash?

SPY 12 year period ending at crash

If you bought SPY, and held for a period of 12 years that ended at the worst possible point of the 2008-2009 crash, you still earned 3.3% on your money. Every dollar invested became about $1.03. As we see, this is better than the true worst case scenario above (0.74%).

Probability of Losing Money with SPY over time

We can see the effects of these two stock market crashes in our probablity graph. Note the spikes to risk at 4 years and 10 years. These windows of time just happen to capture the worst of it. But what do these probabilities mean in the real world? Does this mean if we hold SPY for 12 years we will never, ever, ever lose money?

Anything is possible
When it comes to investments, never say never
"It's Always Sunny in Philadelphia", FX Productions

We can say with certainty, assuming you are willing to hold for 12 years, there has never been a period in the entire history of the SPY during which you would have lost money. Does this mean it could never happen? No. For example, I would assume a meteor strike coupled with some sort of alien invasion might hinder stock prices.

This is to say, anything is possible, but after 12 years, the risks of holding SPY become negligible.

The Time Value of Money

What happens to our expected ROI as we invest for longer periods?

Expected ROI from SPY over time

The longer we hold, the more we earn. Simple, right? Not so fast. We need to consider the time value of money.

Consider this: if we left $100 in a savings account earning 0.01% interest for infinity years, we would still end up with infinity dollars. So what? In the real world we care how long it takes to get paid. To this end, we will use the annualized return on investment to modify our above ROI values according to how long they take.

Expected Annualized ROI from SPY over time

As we can see above, the expected annualized ROI is about 11% for the first few years. For example, we expect $100 would grow by 11% to $111 in year one, and then by another 11% to $123.21 in year two, compounding. By converting all ROIs to annualized ROIs we get a nice apples-to-apples comparison to see what's really going on.

Why do annualized ROIs go down as we approach 10 years? The longer we must wait, the less impressive any ROI is on an annualized basis. The opposite is also true. Even a modest return, gained in a short time, inflates to an impressive annualized return. Compare the rates of change in both lines over time:

Expected Annualized ROI from SPY over time

Does this mean we should only invest for a few years? That depends on how much risk you are comfortable with.

Expected Annualized ROI and Probability of Losing Money in SPY over time

By the time we wait long enough for SPY to be a "sure thing", our ROI on an annualized basis dips a bit. This is the trade-off between risk and reward. Assuming you put your money in the SPY, and leave it there for 12 years, you can expect to earn 7.02% per year. Does this mean you will always earn 7.02% over all 12 years? Absolutely not.

We still have risk, just much less than before.

Visualizing Risk Over Time

If we buy and sell SPY after only one day, our worst day returns -9.84%, and our best returns 14.52%. On an annualized basis that's anywhere from -100% to 77,291,791.39%, anywhere from losing 100% of your starting dollar, to turning it into $772,917.91. What do these ridiculous numbers mean?

If you continued to gain (or lose) at these extreme rates for an entire year, this would be the result. Of course, we have no reason to expect the most extreme one day result would continue for an entire year, so why annualize it? Because we want to compare the range of expected ROIs for each time period, and need an apples-to-apples comparison to do so.

Expected Annualized ROI in SPY over time

Remember: when it comes to stocks, we measure risk in terms of volatility. How much does each individual observation differ from the mean? If we look at the 30 day results, we see quite a range, from a best of 1,186% to a worst of -98.81%. As we extend to a full year, we see our risk decrease, as this gap between best and worst shrinks.

Why didn't we include the one day results in the above graph? Because if we displayed 77,291,791.39% on the same scale as the above (~ 2 inches), we would need a graph taller than 8 Empire State buildings, stacked on top of each other, to show it. How risky is a single day in the stock market? Compared to having some patience, pretty damn risky. After all, the next crash might begin a split second after we invest, and this will always be true.

Expected Annualized ROI in SPY over time

As we increase our patience over 15 years, we decrease our risk. Our volatility slowly and steadily decreases. And starting in year 12 we see something magical happen: even the worst-case scenario drifts out of the red, and into the green.

Expected Annualized ROI in SPY over time

We can see our volatility clearly correlates to our probability of losing money, which is why we use it to measure risk. If we compare the distribution of returns after 1 year, to the distribution of returns after 15 years, we can see the differences in risk and reward clearly:

Expected Annualized ROI in SPY over time

The above visualization reveals a few interesting things. These distributions are no longer normal. Looking at the 1 year distribution, we see bumps around -35% and -20%, corresponding to our two stock market crashes. The 15 year distribution is also not normal, and although it doesn't reach the dazzling highs of the 1 year results, it never goes below an annualized return of 3.44%. Given this distibution, it returns more than 4.19% annualized in 94.98% of trials, returns more that 7% annualized in 27.9% of trials, and in the best case, returns 10.71%.

However we measure it, the lesson is clear:

Warren Buffett Quote
From the master himself

Dollar-Cost Averaging

Imagine: you setup a trading account, transfer in some money, and click on the "purchase" button. A moment later, your hearts sinks, as the next crash begins, immediately wiping away your hard earned cash. That sucks. Is there anything we can do to mitigate this risk?

Dollar-cost averaging provides an even more patient approach to investing. Say you wanted to invest $12,000. Rather than buying all at once, we break it into a monthly budget, and purchase $1,000 worth of SPY every month, no matter the cost. When the price goes down, and sooner or later it will, this gives us the chance to buy more at the cheaper price, before it goes up again. In theory, over time, it should work out to our advantage. Does it?

Buy-n-Sell versus DCA

The "Buy-n-Sell" strategy is the same one we've used so far- we purchase on a random day, hold for any given period of time, then sell everything at once. In this example, we're going to invest $12,000 in SPY, all at once, and hold it for a period of two years. As before, we will test on every two year period in our data, and look at the results in aggregate.

The "DCA" strategy (dollar-cost averaging) will be granted the same two year period, but behave a little differently. For the first year, each month we will purchase as many shares of SPY as $1,000 will afford us, until we have invested the full $12,000. In year two, we will sell 1/12th of our shares every month, exiting the market as gently as we entered it.

How do they compare?

Buy-n-Sell versus DCA after 2 years

With "Buy-n-Sell" 100% of our money is in the market for the entire two years. With "DCA" only a fraction of our money is in the market at any one time, except for the single month before we start selling. The difference is clear. Although "DCA" never reaches the same highs as "Buy-n-Sell", it also avoids the worst lows. Given the same amount of time to invest, we find less volatility, less risk, with the "DCA" strategy.

Item Buy-n-Sell strategy DCA strategy Difference
Best Return, Annualized 42.51% 19.04% -23.47%
Worst Return, Annualized -28.97% -18.52% 10.45%
Expected ROI, Annualized 11.17% 5.88% -5.29%
ROI, Annualized, Volatility 13.68% 7.91% -5.77%
Probability of Losing Money 20.52% 17.99% -2.53%

Avoid the temptation of the higher expected ROI. When we look at volatility, we can understand that the 5.88% offered by "DCA" is far less risky than the 11.17% offered by "Buy-n-Sell". And volatility is a much better indicator of risk than "probability of losing money", because that probability doesn't consider how much either strategy might lose.

Buy-n-Sell versus DCA Over Time

What happens when we compare "Buy-n-Sell" and "DCA" over periods longer than two years? To find out, we extended the "DCA" strategy to keep buying $1,000 of SPY every month until the final year of the trial, at which point it sells 1/12th of shares each month until they are all gone. "Buy-n-Sell" invests the same amount of money, over the same length of time, to make sure we are getting an apples-to-apples comparison.

Buy-n-Sell versus DCA, Expected ROI, annualized, over time

When we look at our expected ROI, the "DCA" strategy may appear disappointing at first. As we've seen before, as risk decreases, so too does our expected ROI. Why does this happen?

Consider this: the worst we can ever do on any investment is -100%, to lose everything. Going in the other direction, there is no upper limit; in the prior examples we saw some outliers as high as 77,291,791.39%. These outliers pull up the average, while ignoring the underlying risk they represent. As we decrease risk, and lower volatility, we lose more and more of these outliers, and our average suffers as a result.

Buy-n-Sell versus DCA, Probability of Losing Money, over time

Measured in terms of "Probability of Losing Money", we still see the effects of our two crashes. Despite the occasional advantages to "Buy-n-Sell", we still mitigate our risk with "DCA" when we consider the overall trend above.

Buy-n-Sell versus DCA, Volatility, over time

Measured in terms of volatility, the advantage of "DCA" is clear. Once again we see why volatility is the metric of choice when it comes to measuring the risk of an investment. And any expected ROI is meaningless, unless we also consider the underlying volatility, the risk that drives those returns.

Once again, the lesson is clear:

Warren Buffett Quote
From the master himself
"Star Wars", Walt Disney Company


The worst thing that can happen to a day trader is to win a silly amount of money after an outrageously lucky trade. No matter how many times the market may thoroughly kick their ass afterwards, they will remain forever convinced that they can do it again if only they work smarter. This is a tragic fallacy. After all, the entire premise of machine learning is that when it comes to making predictions no one is smarter than the data.

My friend Erik once shared what I think is the single best piece of advice ever given:

You can't control the outcome. All you can do is improve your odds.

When it comes to investing in the S&P 500, nothing could be truer. Despite the promises of countless "gurus", no one is 100% sure when the next crash is coming. We won't know for sure until we're already halfway through it. This is the nature of the beast. But by preparing to hold investments for the long term, and utilizing strategies like dollar-cost averaging, we can do a whole lot to improve our odds.

Here's a fun final thought for our friends on /r/wallstreetbets still pining for a lambo. Considering an entry level cost of about $200,000, and a "meager" 7.02% return, the SPY should turn your $10,000 investment into a Lamborghini in about 46 years. And if that sounds like it takes way too long to be worth it, don't worry, you'll most likely go broke far sooner than that.

Place your bets.


Did you find this article useful?

Connect with us on social media and let us know:

Get notified when DataJenius publishes:

* indicates required