Factor analysis has taken the professional consultant world by storm and we are slowly seeing this analysis being used more and more by sophisticated retail investors and investment advisors. And that’s great–factor analysis is a great tool. In fact, we discuss the use of the tool and how it is useful in a recent post called, Basic Factor Analysis: Simple Tools to Understand What Drives Performance.

But factor analysis is not an end-all, be-all. There are other considerations for evaluating strategies. One should use multiple tools–all the time–when analyzing the performance of a strategy/process.

We outline the pitfalls of factor analysis in one of our working papers on maximum drawdowns. The basic idea is that preservation is paramount in investing. This is fairly obvious, as an investor can’t compound $0. But if it is so obvious, why do most performance assessment discussions focus on statistics that do a poor job capturing tail risk?

Consider the concept of “alpha,” which is simply the intercept estimate from a linear regression with a bunch of ‘factors’ that are meant to control for risk exposures. As the story goes, if a manager has a positive alpha, after controlling for factor exposures, they’ve got a lot of skill. So should we just look for alpha in isolation from other considerations? No.

Where does this analysis go awry?

In Table 1 we highlight with a simple example why tail risk requires researcher attention. Table 1 shows a set of statistical measures included in many academic anomaly papers: average monthly returns, standard deviation of returns...