Honest Backtest Data: factor_store Audit and ADR 0008 Implementation
Critical Flaws in the Factor Store and the Truth of the Data
I began the task of building backtestable data for Phase 2a and managed to complete it in just one day. The biggest gain from this process was identifying three critical, hidden flaws in the factor_store through a comprehensive audit. To address these critical issues, I drafted ADR 0008 to finalize the design and successfully executed the 9-item checklist, from implementation to data backfilling and deployment. The related tasks were documented with commit 0248610d (design), implemented in 17b80c4e, and finalized in c7dd37b2 (documentation).
It was a painful but valuable realization that the backtests run so far were essentially invalid. The first defect discovered was the look-ahead bias in the momentum calculation. The _momentum() function was ignoring the as_of date and consistently utilizing the most recent closing prices. This meant that the factors at the time of rebalancing were being calculated using future data. Furthermore, when updating the universe data, the system simply overwrote the existing records. Consequently, the historical screening was severely biased, choosing only from stocks that have survived to this day. Stocks that had already been delisted did not even have OHLCV files remaining. Additionally, when appending data incrementally, corporate actions like stock splits or mergers were not applied retroactively, resulting in mixed split-adjustment standards within a single file. To make matters worse, there were issues like the synthetic bar problem, and the fact that the data storage had been down for 27 days since June 17 without anyone noticing due to the lack of quality gates.
Live Verification and Rebuilding Backtest Data
After clearly defining the problems, I verified the historical data by integrating the pykrx library with KRX credentials. Fortunately, I was able to query the universe at specific historical points without issues. Specifically, I confirmed that as of June 19, 2023, 43 out of 952 KOSPI stocks are now delisted. The historical prices of these delisted stocks were also successfully retrieved, and I found a way to fetch the OHLCV and market capitalization snapshots of all stocks in a single request per market.
Based on these findings, I went through the 9-item checklist one by one. I built the universe_history.parquet to append daily universe snapshots and created a unified query API that returns the correct universe for any given as_of date. In this process, I mapped a total of 7,021 symbols, including 4,160 delisted stocks. Next, I unified the split-adjustment standards and re-backfilled the entire historical OHLCV data for 2,976 stocks (including delisted ones) starting from June 19, 2023. The entire process completed in about 16 minutes with zero empty or failed records.
I also optimized the daily incremental collection. By reducing the process from 2,589 individual stock requests (which took over 9 minutes) to just 2 snapshot requests, the time was cut to seconds. I implemented automatic corporate action detection using rate of change back-calculation, triggering historical re-downloads only for the affected tickers. The reference dates for momentum, universe, and fundamentals were all strictly corrected to respect the as_of parameter. The backtester now liquidates delisted stocks at their last observed price and charges a round-trip transaction cost of 30bp for realism. Lastly, I backfilled the historical daily KRX data lineage and reserved a schema for joining DART quarterly financial reports using the actual publication date (rcept_dt).
Finally, I deployed 5 types of quality gates to validate freshness, universe continuity, calendar gaps, synthetic bars, and price sanity, designed to terminate with exit code 1 upon failure. The supervisor was also updated to refresh SKILL.md daily at 06:05 KST.
Backup Failures and the Outcome of Honest Data
During the post-implementation validation, an unexpected incident occurred. During the backup step at 21:43, the pg_dump process hung for 2.6 hours on a dead socket due to a brief NAS network interruption. Because launchd does not trigger a scheduled task if a previous run with the same label is still active, the critical morning data collection at 05:50 was on the verge of being skipped entirely. After obtaining user approval, I manually terminated the hanging pg_dump process and resumed the chain, allowing the subsequent collection steps to finish successfully. Setting up a backup timeout guard remains a task to be solved in a separate session.
Once the data was corrected, I verified that 109 currently delisted stocks were correctly included in the universe for as_of=2024-06-28, proving that the survival bias was successfully eliminated. Under this clean environment, I ran a backtest using a 6-month momentum strategy, selecting the top 20 stocks and rebalancing monthly from July 2024 to June 2026. The result was a harsh cumulative return of -62.7% and a monthly win rate of 33%.
Had the look-ahead bias remained, it would have yielded a highly attractive but fake return. The honest data laid bare the harsh reality that simple trend-following is ineffective for Korean micro-cap stocks. This is concrete proof that the data has become honest. I plan to use this solid foundation to test subsequent backlog hypotheses, starting with the oversold mean-reversion hypothesis.
For the next session, since the prerequisites of unified split-adjustments and calendar validation have been met, I plan to design the LEAN Korean data adapter based on the environment set up in the directory.
Today's Thoughts
Managing stock trading data is relatively simple when using Excel, but integrating and managing data from multiple brokers is not as straightforward as I thought. Handling delisted stocks separately is also a challenge... There are indeed many delisted stocks in Korea.