What Investors Want to See in a Data Provider's Backtest
The most effective reports are usually not the most impressive, but they adeptly lead the data buyer to activate their trial
Do we need to do our own backtest?
Earlier this year, I was having an internal mental debate about the value in using backtest reports to prove your worth as a data provider to the asset management industry. This seemingly straightforward question eventually led to (what I thought was a) throw-away LinkedIn post on the subject, and its currently my most commented on and seen post yet:
The fear is understandable as backtests are highly technical processes, and data providers assume the goal is to prove that their dataset has significant, uncorrelated alpha. So they obsessively curate and cherry-pick the trading strategies used and the outputs shown to "prove" outperformance of their data versus a benchmark. If not presented carefully this can, paradoxically, lead a data buyer to distrust that the data provider really knows if their data has any value for a particular use case. A provider might know every nuance of their data structures, but may be lost when it comes to putting together an investor-ready backtest package.
If it's truly an "alternative dataset" (a.k.a. almost anything besides traditional financial market data), a hedge fund will often just ask for a backtest report to confirm that the data has been seriously analyzed within the framework of their particular use case. Perhaps then it could be more easily onboarded into their trial queue. They may not even dive deep into the report or even care about the performance or Sharpe ratios. The methodology and queries run to get to those end results could instead be reused and updated for their needs. On the other hand, if a dataset hasn't ever been backtested by anyone, the informational value contained within it will not yet have been diluted at all. It could actually go both ways, some funds like to find new datasets that no one's ever tested (typically those that can afford to stay on the bleeding edge of alternative data, technology, or some specific niche), and some would prefer to be a fast follower after there's clear proof of traction.
Backtests aren't necessarily imperative
For many on the buyside, even the best-intentioned and most comprehensive backtests will be considered suspicious and as interested-party evidence: useful, but naturally viewed with skepticism. Data buyers usually know from experience how many assumptions need to be made and how much can go wrong in the process of creating these reports, including overfitting, lookahead bias, revisions in the underlying data, transaction costs, liquidity restraints, panelization and normalization, and anything that wouldn't hold up in live trading.
Done right, backtest reports can rapidly convince potential data buyers to convert to engaged trials. Done perfectly, they demonstrate the first places to look for the alpha in the dataset and the methodologies used to get there. Serious investors would always then test the data themselves, in their own environment, on their own universe, with their own market data, and using their own assumptions about liquidity, transaction costs and availability timing.
So why might a fund want to see results it will never take at face value? And how can a provider share those results without undermining its credibility?
Backtests force someone to take you seriously
Don't expect a backtest package to cause a data buyer lead to immediately buy the product. A well done report can help a buyer answer a much earlier question:
Is this dataset worth a serious look for what I'm trying to do?
That distinction changes what belongs in the report. A spectacular return chart does little to answer basic questions about coverage, timing, missing observations, or how much work it will take to use the data. Ideally the report leaves the buyer thinking the following:
validityBase helps data providers build credible backtests and independently verifiable live track records, making their data easier for investors to evaluate and trust.
