{
  "markdown": "# Pattern — CNN Replication of Jiang–Kelly–Xiu (2023)\n\nReplication and production-scale extension of *\"(Re-)Imag(in)ing Price Trends\"* (JF 78(6), 2023) using the **I20/R20** configuration — a 5-CNN ensemble trained on 64×60 candlestick chart images to predict 20-day forward return direction for ~1,000 US stocks.\n\nReference paper: `Pattern-2.pdf`.  Full spec: `PRD.md`.\n\n---\n\n## Latest Results (expanding pathway, 1999-03 → 2026-03)\n\nTrained 28 expanding-window retrained ensembles on 8× NVIDIA A100 (80 GB), ~65 GPU-hours total.\n\n### Headline\n\n| Metric | Value |\n|---|---|\n| Observations | 9,249,453 (ticker × end_date) |\n| Unique tickers / dates | 3,011 / 5,759 |\n| **Overall Test AUC** | **0.5068** |\n| Windows with positive LS | 25 / 28 |\n\n### Decile cross-sectional portfolio (equal-weight within decile, 20-day non-overlap compounding)\n\n| Decile | Cum × | Ann. comp. |\n|---|---:|---:|\n| D1 (short) | 0.12 | **−8.71 %** |\n| D2 | 1.16 | +0.64 % |\n| D3 | 2.26 | +3.64 % |\n| D4 | 3.73 | +5.93 % |\n| D5 | 3.40 | +5.49 % |\n| D6 | 3.82 | +6.03 % |\n| D7 | 4.33 | +6.62 % |\n| D8 | 5.95 | +8.11 % |\n| D9 | 6.02 | +8.17 % |\n| D10 (long) | 9.32 | **+10.26 %** |\n\nDecile monotonicity is clean: D1 is a persistent loser and D10 a persistent winner.\n\n### Long–Short summary\n\n| Portfolio | Cum × | Ann. comp. | Ann. vol | Sharpe | NW t(19) |\n|---|---:|---:|---:|---:|---:|\n| **LS D10−D1** | 39.44 | **+17.44 %** | 14.07 % | **1.22** | **+7.99** |\n| LS Top3−Bot3 (top/bot 30 %) | 7.27 | +9.07 % | **8.66 %** | 1.05 | +6.60 |\n\nNewey–West t-stats account for the 20-day overlap in daily forward-return observations (lag=19, Bartlett kernel); |t|≈8 is overwhelming statistical significance.\n\n### Year-by-year LS D10−D1\n\n```\n1999  −7.57 %    (known: dotcom peak, model trained 1996–98 only)\n2000  +43.70 %\n2001  +37.83 %\n2002  +36.76 %\n2003   +5.41 %\n2004  +14.58 %\n2005   +9.50 %\n2006   +5.08 %\n2007  +11.37 %\n2008  +37.18 %   (GFC)\n2009   +2.44 %\n2010  +17.64 %\n2011  +20.55 %\n2012  +16.39 %\n2013   +1.18 %\n2014  +21.54 %\n2015  +16.68 %\n2016   −1.65 %\n2017   +9.35 %\n2018   +8.06 %\n2019  +10.88 %\n2020  +13.64 %   (COVID)\n2021  +49.51 %   (meme stocks)\n2022  +13.61 %\n2023  +28.36 %\n2024  +30.91 %\n2025   +9.78 %\n2026  +15.38 %   (YTD through Mar-2026)\n```\n\nTwo of the three weak years (1999, 2016) are structurally unavoidable or mild; 2013 is a known regime where cross-sectional momentum broke.\n\nCharts: `runs/expanding/<run_dir>/decile_cumulative.pdf`, `top3_vs_bot3.pdf`.\n\n---\n\n## Pipeline status\n\n| Pathway | Config | Status |\n|---|---|---|\n| **Expanding** (train grows each year) | `configs/prod_expanding.yaml` | Complete — `runs/expanding/20260419_174908_cdef6809` |\n| **Rolling** (train capped at 5 yr, trailing) | `configs/prod_rolling.yaml` | Running (pipelined with expanding on idle GPUs) |\n| Comparison (merged per stock-date) | `scripts/merge_pathways.py` | Pending rolling completion |\n\n---\n\n## Code architecture (unchanged from PRD §11)\n\n```\npattern/\n  config.py              Pydantic schemas (PRD §10)\n  cli.py                 python -m pattern.cli {train,backtest}\n  data/\n    loader.py            CSV → tidy DataFrame + adjusted returns\n    splits.py            debug / expanding / rolling retrain schedules\n  imaging/\n    renderer.py          Vectorized per-stock OHLC+MA+volume image gen\n    cache.py             memmap uint8 (N,1,H,W) + parquet sidecar\n  models/\n    blocks.py            Conv→BN→LeakyReLU→MaxPool building block\n    cnn.py               Parametric builder for I5/I20/I60\n  train/\n    dataset.py           PyTorch Dataset over memmap cache\n    loop.py              5-seed ensemble loop with early stopping\n  backtest/\n    deciles.py           Cross-sectional decile portfolios\n    metrics.py           Sharpe, NW t, turnover, drawdown\n    report.py            Auto-generate report.md + plots\nscripts/\n  run_multi_gpu.py       8-GPU fan-out driver (round-robin per-shard windows)\n  gpu_scheduler.py       Work-stealing scheduler (opportunistic GPU use)\n  merge_pathways.py      Expanding + rolling → per-stock-date comparison parquet\n  infer_fullperiod.py    Re-score saved ensembles over any date range\n  train_extra_seeds.py   Add ensembles to an existing run\nconfigs/\n  debug.yaml             Small universe / few windows for smoke tests\n  production.yaml        Baseline full-period single-pass config\n  prod_expanding.yaml    27-year expanding retrain schedule\n  prod_rolling.yaml      27-year rolling (5-year trailing) schedule\ntests/                   Pixel-exact renderer checks, labelling, splits\n```\n\n---\n\n## What's changed since the baseline single-pass run\n\n1. **5-seed ensemble in a single training call.** `train/loop.py` now iterates seeds internally; `cli.py` wires up aggregation.\n2. **Embedding capture.** `cnn.py::forward_with_features` returns the 256-dim global-avg-pooled penultimate tensor alongside the logits. `predict(return_features=True)` now returns `(probs, labels, logits, embeddings)` per seed.\n3. **Rich per-window artefacts.** Every window now writes:\n   - `window_NN_predictions.parquet` — ticker, end_date, label, forward_return, per-seed `p_up_*`, `p_up_mean`, `p_up_std`, `logit_down_mean`, `logit_up_mean`, `rank_pct`, `decile`, `window`.\n   - `window_NN_features.npz` — per-seed logits (K,N,2), per-seed embeddings (K,N,256), ensemble-mean embedding (N,256).\n   - `window_stats.csv` — train/val/test years, sample counts, wall seconds, peak GPU memory.\n4. **Subset training via CLI.** `--window-indices \"0,3,5-9\"` + `--run-dir` let an external orchestrator drive a single shared run directory.\n5. **Two-pathway retrain schedule.** Expanding and rolling configs run side-by-side on the same 28 test years; `merge_pathways.py` joins them on `(ticker, end_date)` so each stock-date has both ensembles' probabilities / ranks / deciles.\n6. **Multi-GPU drivers.**\n   - `run_multi_gpu.py` — static round-robin fan-out, one shard per GPU, drives a single pathway.\n   - `gpu_scheduler.py` — work-stealing scheduler that polls `nvidia-smi` every 30 s and grabs whichever GPU has no compute apps, then pops the next pending window from its queue. Used to pipeline the rolling pathway on GPUs freed by the expanding run.\n7. **Idempotent run-dir setup.** Concurrent shards write their own per-window outputs; the driver/scheduler does the final concat once all shards finish.\n\n---\n\n## How to run\n\n### Prerequisites\n\n- Python 3.14, PyTorch with CUDA or MPS.\n- A single CSV `r1000_ohlcv_database.csv` (Ticker, Date, Open, High, Low, Close, Volume, AdjClose, Return, MarketCap).\n\n### Debug run (local, MPS)\n\n```bash\npython -m pattern.cli train --config configs/debug.yaml\n```\n\n### Full production (single GPU, sequential windows)\n\n```bash\npython -m pattern.cli train --config configs/production.yaml\n```\n\n### Cache pre-build (once)\n\n```bash\npython scripts/run_multi_gpu.py --config configs/prod_expanding.yaml --prebuild-cache\n```\n\nBuilds `/data/Pattern/cache/prod_I20/images.npy` + `index.parquet` (~20 GB for the 1000-stock universe).\n\n### Expanding pathway on 8× GPU\n\n```bash\npython scripts/run_multi_gpu.py --config configs/prod_expanding.yaml --n-gpus 8\n```\n\nRound-robin shard assignment: GPU g trains windows {g, g+8, g+16, …}.  Each shard writes its own per-window parquets; driver concatenates them into `predictions.parquet` at the end.\n\n### Rolling pathway (same universe, same cache)\n\n```bash\npython scripts/run_multi_gpu.py --config configs/prod_rolling.yaml --n-gpus 8\n```\n\n### Pipelining rolling on idle GPUs while expanding still runs\n\n```bash\npython scripts/gpu_scheduler.py \\\n    --config configs/prod_rolling.yaml \\\n    --run-dir /data/Pattern/runs/rolling/<ts> \\\n    --n-windows 28 --n-gpus 8\n```\n\nThe scheduler checks each GPU's compute-app list every 30 s; whenever a GPU is free it launches the next pending window there.  Clean handoff — same shared run directory convention, no racing on the memmap cache (read-only after build).\n\n### Backtest and report\n\n```bash\npython -m pattern.cli backtest --config configs/prod_expanding.yaml \\\n    --run-dir runs/expanding/20260419_174908_cdef6809\n```\n\nWrites portfolio parquets and `report.md` with Sharpe, turnover, drawdown, Newey-West t-stats and per-decile cumulative returns.\n\n### Merge expanding + rolling\n\n```bash\npython scripts/merge_pathways.py \\\n    --expanding runs/expanding/20260419_174908_cdef6809 \\\n    --rolling   runs/rolling/20260420_003938_fb2563f5 \\\n    --out-dir   runs/comparison\n```\n\nProduces `pathway_comparison.parquet` (one row per stock-date with both ensembles' output) and `pathway_comparison_summary.csv` (per-date correlation, decile disagreement).\n\n---\n\n## Image-cache layout\n\n- `images.npy` — memmap uint8 array `(N, 1, 64, 60)`, row order matches `index.parquet`.\n- `index.parquet` — `ticker, end_date, label_h, forward_return, label, has_ma, has_volume, window`.\n- ~20 GB total for the full I20 universe.  Immutable after build; every training shard and every backtest reads the same file.\n\n---\n\n## Output directory layout\n\nEach run writes to `runs/<pathway>/<timestamp>_<config_hash>/`:\n\n```\n20260419_174908_cdef6809/\n  config.yaml                 frozen copy of the training config\n  sha.txt                     git sha\n  pip_freeze.txt\n  window_00_predictions.parquet ...  window_27_predictions.parquet\n  window_00_features.npz     ...     window_27_features.npz\n  window_stats.csv\n  predictions.parquet         concatenated final (9.25 M rows for expanding)\n  shard_gpu{0..7}.log         driver logs\n  shard_gpu{g}_w{w:02d}.log   scheduler logs\n  portfolios.parquet          backtest output\n  report.md                   auto-generated narrative + plots\n  decile_cumulative.pdf       10-decile log-scale cumulative returns\n  top3_vs_bot3.pdf            softer top/bot-30% version\n```\n\n---\n\n## Hardware\n\nRemote node: 8× A100-80GB, `/data/Pattern/` workspace.\nLocal: M4 Max, 128 GB RAM, MPS — used for development, analysis, plotting.\n\nPeak per-shard GPU memory: 0.60 GB (batch 128).  Network training is compute-bound on the memmap loader, not memory-bound.\n\n---\n\n## Compute budget (expanding pathway, actual)\n\n| Quantity | Value |\n|---|---|\n| Windows trained | 28 |\n| Ensembles per window | 5 |\n| Total networks trained | 140 |\n| Wall-clock (8× A100 pipelined) | 9 h 27 min |\n| Sum of per-shard GPU hours | 64.74 |\n| Mean per-window wall | 138.7 min |\n| Median peak GPU memory | 0.60 GB |\n\nCost at the rented node rate (~$10.69/h) ≈ $101 for both pathways pipelined.\n\n---\n\n# Addendum — Post-training slicing, trash-tier strategy, and trading-cost analysis\n\nThe headline numbers above come from the full R1000 universe.  After the\nexpanding-pathway run completed we ran a long series of cross-sectional\nslicing experiments to answer two questions:\n\n1. Where inside the universe does the signal concentrate?\n2. Is the concentrated signal actually tradable after real-world frictions?\n\nAll artefacts below live under\n`runs/expanding/20260419_174908_cdef6809/`, using\n`predictions_monthly.parquet` (one row per ticker × month-end with\n`p_up_mean`, `p_up_std`, `forward_return`, `label`).\n\n## A. Cross-sectional slicing — which names hold the alpha?\n\nWe built `scripts/backtest_generic.py` as a one-stop slicer: feed it any\ncategorical/numeric column, let it per-date bucket it, and report a 50/50\nor 10-decile LS portfolio per bucket.  Used it for:\n\n| Slice | Out-dir | Notes |\n|---|---|---|\n| BICS Level 1 | `backtest_by_bics_level_1_5050/` | 12 sectors |\n| BICS Level 2 | `backtest_by_bics_level_2_5050/` | 40+ industry groups |\n| BICS Level 3 | `backtest_by_bics_level_3_5050/` | Deep industries |\n| Dollar-volume tertiles (mcap proxy) | `backtest_by_mcap_proxy_3/`, `…_5/` | 60d mean $vol |\n| Momentum (12-1) tertiles | `backtest_by_mom_12_1_3/` | `log(P_{t-21}) − log(P_{t-21-252})` |\n| Realized-vol (60d) tertiles | `backtest_by_vol_60d_3/` | `std(daily ret) × √252` |\n| Prediction disagreement (`p_up_std`) | `backtest_by_p_up_std_3/` | cross-seed dispersion |\n\n### Findings\n\n- **Size:** LS monotonically *stronger* in smallest $-volume tertile — the\n  signal is largely a small-cap phenomenon, consistent with JKX paper.\n- **Vol:** High realized-vol tertile has both highest gross LS and the\n  highest rebalancing frequency.\n- **Momentum:** Bottom-momentum tertile (recent losers) has the richest\n  LS — CNN exploits the short-horizon reversal inside loser names.\n- **Disagreement:** Sorting by cross-seed `p_up_std` does *not* improve\n  LS noticeably — ensemble disagreement is not a useful signal filter.\n- **Sectors:** 3 weak sectors emerge — **Utilities, Real Estate,\n  Industrials** — with essentially zero or negative LS.  Most other\n  BICS-1 sectors have positive and significant LS.\n\nFull per-slice CAGR / Sharpe / NW-t tables live in each\n`*_summary.xlsx` file.\n\n## B. Trash-tier intersection — stacking the weak-name filters\n\nInspired by the size/vol/momentum findings, we intersected the three\n\"bad-name\" buckets and ran a 50/50 LS inside the intersection.\n\nScripts:\n- `scripts/backtest_trash_tier.py` — per-filter 50/50 LS portfolios\n- `scripts/trash_tier_turnover.py` — month-over-month ticker turnover\n- `scripts/trash_tier_yearly.py` — per-calendar-year LS return and turnover\n\nFilter stack (`univ` = all R1000 names with all 3 features available):\n\n| Filter | Mean univ names/mo | Mean top-side names |\n|---|---:|---:|\n| universe | ~910 | ~455 |\n| small | ~303 | ~151 |\n| small & high-vol | ~193 | ~97 |\n| small & recent-loser | ~148 | ~74 |\n| high-vol & recent-loser | ~168 | ~84 |\n| **triple (small & high-vol & recent-loser)** | **~124** | **~62** |\n\n### Triple-filter headline (in-sample gross, no costs)\n\n| Stat | Universe 50/50 | Triple 50/50 |\n|---|---:|---:|\n| Months | ~324 | ~317 |\n| TOP CAGR | +10.5 % | +27.5 % |\n| BOT CAGR | +4.0 % | −6.5 % |\n| **LS CAGR** | +6.1 % | **+28.5 %** |\n| LS ann-vol | ~6.8 % | ~19.2 % |\n| **Sharpe** | ~0.90 | **~1.07** |\n| **NW t(0)** | +4.5 | **+5.05** |\n| Cum × | ~9 | ~360 |\n\n### Turnover (one-sided monthly, symmetric-diff / (|S_t|+|S_{t-1}|))\n\n- Universe: ~42 % per month, per side\n- Triple:   **~60 % per month, per side** (annualised ≈ 720 % two-sided)\n- Avg holding period: ~0.8 months (less than a month)\n\nThe triple-filter portfolio is a high-turnover trash-name book that\nleans heavily on intra-month reversal.  Gross returns are excellent;\nwhether they survive costs is the rest of this addendum.\n\n### Year-by-year triple-filter LS\n\n| Year | LS (%) | Year | LS (%) | Year | LS (%) |\n|---|---:|---|---:|---|---:|\n| 2000 | +82 | 2009 | +12 | 2018 | **−33** |\n| 2001 | +74 | 2010 | +41 | 2019 | +25 |\n| 2002 | +52 | 2011 | +28 | 2020 | +54 |\n| 2003 | +15 | 2012 | +22 | 2021 | +66 |\n| 2004 | +19 | 2013 |  −3 | 2022 | +18 |\n| 2005 | +14 | 2014 | +30 | 2023 | +47 |\n| 2006 |  +9 | 2015 | +25 | 2024 | +39 |\n| 2007 | +18 | 2016 | +17 | 2025 | +14 |\n| 2008 | +41 | 2017 | +14 | 2026 YTD | +22 |\n\n2018 is the single problem year — concentrated in Dec 2018 when the Fed\npivoted dovish and low-quality / short-interest stocks rocketed.  The\ntriple filter is long-loser-short-winner, so that short squeeze bit\nhardest exactly where the model is most exposed.\n\n## C. Sub-slice sensitivity — ex-3-sectors\n\nFiltering out Utilities, Real Estate and Industrials before running the\ntriple filter:\n\n- Universe / mo: **~100** (vs 124)\n- LS CAGR: **+30.0 %** (vs +28.5 %)\n- Sharpe: **0.90** (vs 1.07)\n- NW t: **+4.28** (vs +5.05)\n\nConclusion: the excluded sectors were mild drags on gross return but\nuseful diversifiers on risk.  Removing them improves CAGR marginally but\nhurts Sharpe / t-stat — *the three weak sectors are noise-dampeners, not\nalpha-dilutors*.  Artefacts in `backtest_trash_tier_ex3sectors/` and\n`predictions_monthly_ex3sectors.parquet`.\n\n## D. Regime overlays — can we time aggressiveness?\n\n**Hypothesis 1 — Small-cap relative momentum.**  2018 was a strong-\nsmall-cap year, so maybe LS fails when small-caps outrun large-caps.\nWe computed a rolling small-minus-large relative-momentum factor\n(smallest-tertile $vol stocks' 12-1 mean ret – largest-tertile's) and\ncorrelated it with monthly triple-filter LS.\n\nResult: **ρ = +0.36**, the *opposite* sign from the hypothesis.  The\nmedian LS return is almost identical between strong-small vs weak-small\nregimes — what differs is the **hit-rate** (57 % vs 86 %).  The bad\nmonths in 2018 are not a systematic small-running-hot regime; they are\na squeeze event.  Hypothesis rejected.\n\n**Hypothesis 2 — Short-term reversal overlay.**  We built binary and\nlinear aggressiveness overlays keyed off 1-month and 3-month LS\nmomentum (idea: shrink positions after the strategy gets hot).  Every\noverlay version reduced CAGR and Sharpe.  Binary variants shut off in\nproductive months (2009, 2020, 2023) while only modestly dampening\n2018.  No overlay we tried dominates the unconditional strategy.\n\n## E. Trading-cost realism — is the gross number even reachable?\n\nWe built two cost studies:\n\n1. **`scripts/` Corwin-Schultz H/L-based estimator** (`backtest_by_cs_spread.py` style).\n   Uses daily high/low prices (Corwin-Schultz 2012) to estimate an\n   unobserved bid-ask spread at the stock-day level, then averages\n   across a rebalance's holdings.\n\n   - Mean CS spread on triple-filter portfolio: **~208 bps** (full)\n   - This is likely **overstated by 2–3×** for small-cap high-vol names,\n     which the Corwin-Schultz estimator is known to inflate.\n\n2. **Break-even analysis.**  With ~12 rebalances/year and ~60 %\n   one-sided turnover per side per month, the cost drag at a full\n   spread `s` bps is approximately:\n\n   ```\n   drag ≈ 12 × 2 × turnover × (s / 2) / 10000\n        ≈ 12 × 2 × 0.6 × (s / 2) / 10000\n        ≈ 0.0072 × s     (per year, as fraction)\n   ```\n\n   Triple LS gross ~28.5 %.  Break-even full-spread\n   `s* ≈ 28.5 / 0.0072 ≈ 3,958 bps` — but this double-counts both long\n   and short sides.  One-sided break-even on just the half-spread:\n\n   ```\n   LS break-even half-spread h* ≈ 28.5 / (24 × 0.6) × 100 bps ≈ 198 bps\n   ```\n\n   Even the inflated Corwin-Schultz spread of 208 bps is uncomfortably\n   close to break-even.  A realistic half-spread of ~40–50 bps (CS × 0.4\n   haircut) puts net LS around +10–15 % — still attractive but fragile.\n\n3. **$5M-per-side capacity** (`scripts/ibkr_triple_tier_costs.py` driver,\n   offline mode).  Applied Almgren-style impact model:\n\n   ```\n   impact_bps = c × σ_daily × sqrt(Q / ADV) × 10000   (one-way)\n   ```\n\n   with `c = 1.5`, `σ_daily` from realized daily return std, `ADV` from\n   60d mean $vol, `Q = 5e6`.  Added a liquidity filter (ADV ≥ $5M, per-\n   name cap 10 % of ADV).\n\n   - Universe after liquidity filter: **29 / 28 names** (top/bot), down\n     from ~62 each side.  About 75 % of the alpha-richest names drop\n     out because they don't have $5M ADV.\n   - Gross LS after liquidity filter: **+13.9 %** (from +28.5 %).\n   - Mean one-way impact: **~80–95 bps**.\n   - Round-trip impact × 2 sides × 12 months × 60 % turnover:\n     **cost drag ~37 %/yr**.\n   - **Net LS: ≈ −21.5 %**.\n\n   **Conclusion: the triple-filter strategy as-is does not survive\n   $5M-per-side.**  Capacity is probably $500k–$1M per side.  Any\n   deployment needs either:\n   - A much smaller AUM target, or\n   - A slower-turning variant (e.g., quarterly rebalance, or position-\n     by-position Kelly-shrunk), or\n   - Better execution (VWAP, IS algos, internal cross) than the blunt\n     impact assumption above.\n\n## F. IBKR live-spread integration (work-in-progress)\n\n`scripts/ibkr_triple_tier_costs.py` connects to a local TWS or IB\nGateway (via `ib_insync`) and pulls per-ticker:\n\n- Live (or delayed) best bid / best ask → realized half-spread\n- `reqHistoricalData(whatToShow='BID_ASK')` over 40 days → stable\n  time-averaged spread estimate\n- Shortability tick (generic tick `236`) — proxy for whether the short\n  side of the LS is executable at all\n- Realized daily σ over the historical window for impact calculation\n\nIt then re-prices the triple-filter monthly portfolios using the\nmeasured IBKR spreads (capped, Winsorised) instead of the Corwin-\nSchultz estimate.\n\n### Current blockers\n\n- The user's IBKR account does not have a live US-equity market-data\n  subscription, so `reqMktData` returns `delayedBid=None, delayedAsk=\n  None` on unsubscribed tickers — only trades (last/HLC) come through.\n- Running against the live TWS (port 7496) confirms the connection and\n  symbol lookup work; the cost model falls back to the CS estimate for\n  any ticker where both live and historical BID_ASK come back empty.\n- Next step: either enable the \"US Securities Snapshot and Futures\n  Value Bundle\" (~$10/mo) on the IBKR account, or scrape AltaVista ETF\n  Research's \"Avg Sp\" column for the small number of ETF-like proxies\n  that AltaVista covers.\n\n### Python 3.14 compatibility\n\n`ib_insync` needs an event-loop shim at import time under 3.14:\n\n```python\nimport asyncio as _asyncio\ntry:\n    _asyncio.get_event_loop()\nexcept RuntimeError:\n    _asyncio.set_event_loop(_asyncio.new_event_loop())\n\nfrom ib_insync import IB, Stock, util\n```\n\nWithout this, import fails with `RuntimeError: There is no current\nevent loop in thread 'MainThread'` because 3.14 removed the implicit\nevent-loop-on-demand behaviour.\n\n### Usage\n\n```bash\npython scripts/ibkr_triple_tier_costs.py \\\n    --predictions runs/expanding/20260419_174908_cdef6809/predictions_monthly.parquet \\\n    --ohlcv       data/r1000_ohlcv_database.parquet \\\n    --out-dir     runs/expanding/20260419_174908_cdef6809/ibkr_costs \\\n    --ib-host 127.0.0.1 --ib-port 7496 --ib-client-id 42 \\\n    --aum-per-side 5e6 --min-adv 5e6 --adv-cap 0.10 --c-impact 1.5 \\\n    --market-data-type 3\n```\n\n`--market-data-type`: 1=live, 2=frozen, 3=delayed (free), 4=delayed-frozen.\n\nOutputs:\n- `ibkr_costs/ibkr_spreads.parquet` — per-ticker IBKR spread snapshot\n- `ibkr_costs/triple_net_costs.xlsx` — per-month gross / impact / spread\n  / net, plus summary row\n- `ibkr_costs/triple_net_cum.pdf` — cumulative gross vs net\n\n## G. Takeaways\n\n1. **The paper replicates cleanly.**  Full R1000 LS: +17.4 %, Sharpe\n   1.22, NW t +7.99 over 1999-03 → 2026-03.\n2. **Alpha concentrates in small, high-vol, recent-loser names.**  The\n   triple-filter subset gross-returns ~28.5 % with Sharpe ~1.07 on a\n   ~124-name universe.\n3. **Alpha is high-turnover.**  ~60 % of names rotate every month per\n   side.  The strategy is an intra-month reversal exploit, not a\n   buy-and-hold anomaly.\n4. **Simple regime overlays don't help.**  Small-cap-momentum and short-\n   term reversal timing both reduce Sharpe.  The bad year (2018) is a\n   squeeze event, not a regime feature.\n5. **Costs are the binding constraint.**  Estimated half-spread on\n   the triple-filter book is 50–100 bps realistic, 208 bps inflated\n   (Corwin-Schultz).  Break-even half-spread is ~200 bps, so\n   costs eat most of the gross alpha.  At $5M per side with a 10 %-ADV\n   cap, net return is *negative* — capacity is probably $500k-$1M.\n6. **The signal is strongest exactly where transactions are most\n   expensive.**  This is the central tension of the paper's\n   trash-tier alpha — small, illiquid, volatile names.  Any production\n   deployment needs realistic execution cost modelling and low AUM.\n\n---\n\n# Addendum H — Zero-shot ETF test and the liquid-grid pivot (2026-05)\n\nAfter concluding that the trash-tier triple-filter is capacity-limited\nto ~$500k–$1M, the next two questions were:\n\n  (i) Does the CNN transfer zero-shot to a universe with naturally low\n      spreads and easy borrow — i.e. ETFs?\n (ii) Inside R1000, is there a cell with *modest* gross return but\n      genuinely cheap execution that survives realistic costs?\n\n## H.1  Zero-shot test on liquid ETFs\n\n**Universe build.**  Pulled a 4,374-ETF Bloomberg screen\n(`/Users/arjundivecha/Downloads/ETF.xlsx`), converted `\"SPY US\"` → `\"SPY\"`\nfor yfinance, dropped leveraged / inverse / 2x / 3x / YieldMax / covered-call\nproducts, then filtered on:\n\n```\nbid-ask spread ≤ 5 bps   AND\n30-day $-volume   ≥ $10M  AND\n30-day share-volume ≥ 100k\n```\n\n→ **477 liquid ETFs** (`AssetList_liquid_etfs.xlsx`).  Spread\ndistribution: 223 @ 1-2 bps, 96 @ 2-3, 69 @ 3-4, 49 @ 4-5.\n\n**Data.**  100 % yfinance fetch success →\n`data/liquid_etf_ohlcv.{csv,parquet}` (1.6 M rows, 477 tickers,\n1993-01 → 2026-04).  History depth: 378 ETFs ≥ 5 yr, 300 ≥ 10 yr,\n209 ≥ 15 yr.\n\n**Image cache.**  Added a `--monthly` flag to\n`scripts/render_etf_cache.py` that keeps only the last trading day\nof each calendar month per ticker (≈ 20× smaller cache, monthly-cadence\npredictions).  Final cache:\n\n```\ncache/liquid_etf_I20_monthly/images.npy   215 MB, (58,767, 1, 64, 60) uint8\ncache/liquid_etf_I20_monthly/index.parquet 603 KB, 58,767 rows, 427 tickers,\n                                         361 monthly dates 1996-03 → 2026-03\n```\n\n**Predictions.**  Applied the 28-window R1000-trained ensemble zero-shot.\nOverall OOS AUC = **0.5078** (vs R1000 benchmark 0.5068) — there is\na trace of signal, but it is not economically useful.\n\n**Backtests** (in `runs/liquid_etf_expanding/`):\n\n| Filter | CAGR | Sharpe | NW t |\n|---|---:|---:|---:|\n| 50/50 LS                  | −0.44 % | −0.08 | −0.39 |\n| Top/Bot 10 %              | +0.27 % | +0.02 | +0.13 |\n| Top/Bot 20 names          | +1.21 % | +0.13 | +0.66 |\n| triple (small & high-vol & loser) | **−1.42 %** | −0.18 | −0.95 |\n\n**Conclusion.**  The CNN alpha is a **single-stock idiosyncratic-reversal\neffect**.  ETFs are diversified baskets of dozens to hundreds of\nsingle names, so the very source of the signal is averaged away on the\nunderlying basket.  Even the triple-filter, which on R1000 single names\nis the strongest cell of the entire study, goes *negative* once the\nuniverse becomes ETFs.  Cheap execution does not rescue a signal that\nisn't there.\n\n## H.2  R1000 liquid-grid search (the pivot that worked)\n\nQuestion (ii) — finding a tradable R1000 cell with modest return but\ncheap execution — was the productive turn.\n\n`scripts/backtest_liquid_grid.py` buckets every month-end by tertile on\n`dv_60d`, `vol_60d`, and `mom_12_1`, then builds 50/50 long-short books\ninside each cell.  Per cell it reports gross LS, NW t, one-sided\nmonthly turnover, and a realistic cost drag using\nhalf-spread = {Low-dv: 25 bps, Mid-dv: 8 bps, High-dv: 2.5 bps}.\n\n### Marginal dv tertiles\n\n| Cell | months | gross LS | net LS | NW t | half-spread |\n|---|---:|---:|---:|---:|---:|\n| dv = Low (all)   | 282 |  9.63 % | 6.56 % | 5.71 | 25 bps |\n| dv = Mid (all)   | 282 |  3.52 % | 2.49 % | 3.18 |  8 bps |\n| dv = High (all)  | 282 |  1.49 % | 1.19 % | 1.17 | 2.5 bps |\n| All R1000        | 282 |  4.86 % | 3.92 % | 4.49 |  8 bps* |\n\n\\* notional weighted-average tier.\n\n### Grid A — dv × vol (cells passing net > 0 AND t ≥ 2)\n\n| Cell | names | net LS | Sharpe | NW t |\n|---|---:|---:|---:|---:|\n| dv=Low  × vol=High | 214 | **16.48 %** | 1.21 | 5.90 |\n| dv=Mid  × vol=High | 150 |  7.34 %     | 0.74 | 3.59 |\n| dv=High × vol=High | 146 |  6.01 %     | 0.61 | 2.98 |\n\n### Grid B — dv × mom (cells passing net > 0 AND t ≥ 2)\n\n| Cell | names | net LS | Sharpe | NW t |\n|---|---:|---:|---:|---:|\n| dv=Low  × mom=Low | 211 | **16.07 %** | 1.34 | 6.50 |\n| dv=Mid  × mom=Low | 163 |  6.98 %     | 0.85 | 4.11 |\n| dv=High × mom=Low | 146 |  3.96 %     | 0.44 | 2.11 |\n| dv=High × mom=Mid | 181 |  2.24 %     | 0.46 | 2.24 |\n\n### Grid C — dv × vol × mom (27 cells, top by net LS)\n\n| Cell | names | gross LS | net LS | Sharpe | NW t | half-spread |\n|---|---:|---:|---:|---:|---:|---:|\n| Low × High × Low     | 124 | 26.97 % | **23.42 %** | 1.04 | 5.05 | 25 bps |\n| **Mid × High × Low** |  71 | 14.28 % | **13.01 %** | 0.79 | 3.86 |  8 bps |\n| Mid × High × High    |  51 | 10.34 % |  9.01 %     | 0.57 | 2.76 |  8 bps |\n| **High × High × Low**|  59 |  7.25 % |  6.85 %     | 0.42 | 2.02 | 2.5 bps |\n| Mid × Low  × Low     |  37 |  5.77 % |  4.34 %     | 0.46 | 2.22 |  8 bps |\n\nOf 27 cells, only those 5 pass the (net > 0, t ≥ 2) hurdle.  The\nstructure is highly concentrated: **High-vol is the load-bearing\nfilter**, **mom=Low (recent loser) stacks on top of it**, and the\neffect persists out of the cheap-to-trade zone for the first time.\n\n### What this changes\n\n- Takeaway #5 / #6 from section G need to be qualified.  Yes — the\n  strongest gross alpha is in the trash tier (Low-dv × High-vol ×\n  Low-mom, 23 % net, 124 names).  But there are now **two\n  cleaner-cost alternatives**:\n\n  1. **Mid-dv × High-vol × Low-mom**: ~71 names, ~$500M–$5B per\n     name, 8 bps half-spread, easy borrow.  **+13 % net CAGR,\n     Sharpe 0.79, t +3.86.**  This is the workhorse cell — modest\n     return for a single-name strategy but it actually clears\n     realistic costs.\n  2. **High-dv × High-vol × Low-mom**: ~59 mega-cap names\n     (TSLA / NVDA / COIN-type beaten-down volatile blue chips),\n     2.5 bps half-spread, trivial borrow.  **+6.9 % net,\n     Sharpe 0.42, t +2.02.**  Largest capacity of any cell.\n\n- A reasonable production stack is **Mid-cell + High-cell** ≈ 130\n  names per side, gross ~10 %, Sharpe ~0.7, deep capacity.\n\n### Dead zones (every Mid-vol cell, every High-mom cell ex one)\n\nAll nine Mid-vol cells are flat or negative net.  All six dv ×\nmom=High cells are flat or negative net.  The CNN's edge is\nconcentrated entirely in the high-vol / low-momentum tails — exactly\nthe regime where mean-reversion dominates trend.\n\n### Files\n\n- `scripts/backtest_liquid_grid.py` — 27-cell grid driver.\n- `runs/expanding/20260419_174908_cdef6809/backtest_liquid_grid/liquid_grid_summary.xlsx`\n  — sheets `marginals`, `dv_x_vol`, `dv_x_mom`, `dv_x_vol_x_mom`, `tradable`.\n\n## H.3  Updated bottom line\n\nThe signal documented in section G is real and the trash-tier\nresult (28 % gross, $500k-$1M capacity) still stands.  What changed\nis that **section G's pessimistic conclusion — \"costs eat all the\nalpha\" — is universe-specific, not signal-specific**.  Climb one\ntertile up the dollar-volume ladder, keep the High-vol × Low-mom\nsub-filter, and you get a tradable ~13 % net book with eight-times\nthe capacity.  The CNN signal degrades smoothly with size and\nliquidity rather than collapsing — the right deployment is the\nmid-cap high-vol loser cell, not the smallest-name trash tier.\n\n---\n\n# Addendum I — ETF / single-stock-ETF deployment attempts (2026-05)\n\n**Motivation.**  Author works at an investment firm with restrictions on\ntrading individual stocks but is permitted to trade ETFs of any kind,\nincluding single-stock ETFs (SSEs).  The question for this addendum:\n*can the R1000-trained CNN edge be deployed through any combination of\nETFs the firm allows?*\n\nShort answer: **No.**  The alpha is a single-stock, small-cap, high-vol,\nrecent-loser cross-sectional effect.  ETFs — basket or single-stock —\neither average it away or carry it on the wrong tail of the\ndistribution.  Below are the four attempts and why each failed.\n\n## I.1  Single-window retrain on ETFs (w11)\n\n**Hypothesis.**  Maybe zero-shot transfer fails because the CNN never\nsaw ETF charts.  Retraining on the ETF universe (with or without\nwarm-start from the R1000 ensemble) might fix it.\n\n**Test.**  On window 11 (2010 test year), trained two ETF-specialised\nvariants against the R1000 zero-shot baseline:\n\n| Variant | AUC | LS CAGR | LS Sharpe |\n|---|---:|---:|---:|\n| R1000 baseline (zero-shot) | 0.5125 | +1.14 % | +0.127 |\n| **Train from scratch on ETFs** | 0.5040 | **−0.59 %** | **−0.064** |\n| **Fine-tune from R1000** | 0.5041 | **−0.68 %** | **−0.063** |\n\nBoth ETF-trained variants **underperformed** the zero-shot R1000 model.\nRetraining does not help — the ETF universe simply lacks the\ncross-sectional dispersion the CNN can exploit.  Full 28-window retrain\nwas skipped on this evidence (running it would just confirm the result\nmore rigorously at the cost of ~1 hour of compute).\n\nFiles: `runs/etf_scratch_w11/`, `runs/etf_finetune_w11/` (each contains\n`comparison.txt`, `*_predictions.parquet`, training log).\n\n## I.2  Single-stock ETF universe build\n\n**Trick.**  SSEs are the loophole around the firm's stock-trading\nrestriction: they ARE ETFs but each tracks ONE underlying.  A\nlong/short book can be built entirely from long positions in two\nproducts per underlying — the 2x bull-leveraged ETF for \"long\" bets and\nthe inverse-leveraged ETF for \"short\" bets.  No actual short-selling,\nno borrow costs.\n\n**Universe.**  Hand-curated 48-pair seed list spanning the major\nissuers (Direxion, GraniteShares, T-Rex/REX Shares, Tradr, Defiance)\nacross both Direxion-style asymmetric pairs (+2x long / −1x short) and\nthe cleaner symmetric ±2x pairs from T-Rex / Defiance.  Validated each\nticker via yfinance:\n\n- 121 candidate tickers (44 underlyings + 77 wrappers)\n- 120 returned valid price history; 1 delisted (AMDS)\n- **44 underlyings with at least one long wrapper, 29 of which have a\n  matching inverse wrapper (the \"complete-pair\" subset)**\n\nHistory depth:\n\n- Aug-Sep 2022 inception (6 underlyings: AAPL, AMZN, GOOGL, MSFT, TSLA + COIN long-only) — ~3.7 yr\n- Dec 2022 / 2023 expansion (+NVDA, BABA) — ~3 yr\n- 2024 expansion (+META, MSTR, TSM, MU, PLTR, SMCI) — ~1.5 yr\n- 2025+ rest of the universe — < 1 yr each\n\nEffective backtest window: 2022-09 → 2026-04 (44 monthly observations),\nuniverse growing from 6 → 30+ over time.\n\nFiles:\n- `data/sse_pairs_seed.csv` — hand-curated 48-row pair list\n- `data/sse_pairs.csv` — yfinance-validated pair table\n- `data/sse_underlying_ohlcv.csv` — 179,843 rows × 44 underlyings\n- `data/sse_wrapper_ohlcv.csv` — 33,103 rows × 77 wrappers\n- `scripts/fetch_sse_data.py`\n\n## I.3  Zero-shot CNN on SSE underlyings + wrapper-LS backtest\n\n**Setup.**  Score every (underlying, month-end) using the existing\nR1000-trained 28-window ensemble (same approach as the liquid-ETF test\nin section H).  Image cache geometry identical to training.\n\nOverall OOS AUC on the 44 SSE underlyings = **0.5136** — slightly\n*better* than the R1000 baseline (0.5068) because these are mostly\nvolatile mega-cap names the CNN has seen during training.  The model\n\"recognises\" the universe.\n\n**Trading mechanic.**  Per month-end, rank by `p_up_mean`.  Top half →\nbuy the 2x long-leveraged ETF.  Bottom half → buy the inverse-leveraged\nETF (long position, no shorting).  Portfolio return per dollar of\ncapital = `0.5 × (mean(R_long_etf | top) + mean(R_inverse_etf | bot))`.\nCosts modelled: 1.0 % p.a. expense ratio prorated + 5 bps half-spread per leg.\n\n**Results (44 months, 2022-09 → 2026-04, mean univ ≈ 20):**\n\n| Variant | Months | Gross CAGR | Net CAGR | Sharpe | NW t |\n|---|---:|---:|---:|---:|---:|\n| **complete-pairs (true LS via wrappers)** | 44 | **−16.47 %** | −19.31 % | −0.50 | −0.96 |\n| long-only (top-half via long-ETF) | 44 | +51.02 % | +46.09 % | +1.11 | +2.15 |\n| full (long-only with hedge where available) | 44 | −26.67 % | −29.20 % | −1.40 | −2.70 |\n\n**Diagnosis (the smoking gun).**  Computing the underlying-only L/S\nspread (i.e. if we could trade the actual stocks long-short, no wrapper):\n\n  - Underlying TOP half (CNN predicts UP): **+2.04 %/mo**\n  - Underlying BOT half (CNN predicts DOWN): **+5.67 %/mo**\n  - Underlying LS: **−3.64 %/mo, t = −2.14, Sharpe = −1.12**\n\n**The CNN signal is inverted on this universe in this regime.**  The\nloss is NOT a wrapper-decay artifact.  An underlying-only L/S book\nloses 38 % CAGR.  The reason: the CNN learned a 20-day mean-reversion\npattern from R1000 1999-2022.  The SSE universe is dominated by\nmega-cap momentum names (TSLA, NVDA, MSTR, COIN, PLTR, Mag 7) where\nmomentum persists.  The signal's \"oversold buy\" calls land on names\nthat keep falling; its \"overbought sell\" calls land on names that keep\nripping.  Hence the inversion.\n\nFor context, the R1000 signal still works in the same window\n(2022-09 → 2026-04 on R1000): **+7.89 % CAGR, t = +3.20, Sharpe = +1.82**.\nThe signal is universe-specific, not regime-broken.\n\nFiles:\n- `runs/sse_underlying_expanding/predictions.parquet` — 5,970 rows\n- `runs/sse_underlying_expanding/backtest_sse/sse_summary.xlsx`\n- `runs/sse_underlying_expanding/backtest_sse/sse_cum.pdf`\n- `scripts/backtest_sse.py`\n\n## I.4  Alternative cross-sectional signals on the SSE universe\n\n**Hypothesis.**  Maybe a different signal (momentum follower, not\nreversal) works on the SSE universe.  Tested six classic\ncross-sectional signals against the same wrapper mechanics:\n\n```\nmom_12_1, mom_6_1, mom_3_1, rev_1m, low_vol (vol_60d inverted), trend\n```\n\n### Long-only (just buy top-half via 2x long-ETF)\n\n| Signal | Gross CAGR | NW t | Underlying-LS CAGR (no leverage) |\n|---|---:|---:|---:|\n| rev_1m   | +547 % | 7.21 | **−7.6 %** |\n| **EW_basket (no signal)** | **+422 %** | 7.88 | n/a |\n| mom_3_1  | +263 % | 5.06 | −8.9 % |\n| mom_6_1  | +241 % | 4.61 | −21.1 % |\n| trend    | +216 % | 4.72 | −11.5 % |\n| mom_12_1 | +157 % | 3.60 | −16.8 % |\n| low_vol  | +14 %  | 1.31 | −47.8 % |\n\n### Complete-pairs LS (long-ETF + inverse-ETF)\n\n| Signal | Gross CAGR | NW t | Underlying LS |\n|---|---:|---:|---:|\n| rev_1m   | +166 % | 5.21 | −32 % |\n| mom_6_1  | +101 % | 3.93 | −23 % |\n| mom_3_1  | +92 %  | 3.82 | −7 % |\n| trend    | +91 %  | 4.15 | +12 % |\n| mom_12_1 | +62 %  | 2.77 | −33 % |\n| low_vol  | **−54 %** | −7.84 | −67 % |\n\n### Why every \"win\" here is fake\n\nThe crucial column is **underlying-LS CAGR** — the signal's true\nstock-picking skill, stripping away wrapper leverage.  *Every* signal\nis **zero or negative** in underlying space.  Translation:\n\n- \"+547 % CAGR rev_1m long-only\" is **not signal alpha** — the\n  no-signal EW basket of all 44 long-ETFs returns +422 % CAGR on its\n  own.  rev_1m adds ~+2.5 %/mo on top, plausibly from leverage\n  convexity rather than picking-skill (the underlying-LS is −7.6 %).\n\n- \"+166 % complete-pairs rev_1m\" is roughly half the long-only result\n  because the inverse-ETF leg averages ~0 over this bull market — it\n  hedges market beta but contributes no cross-sectional alpha.\n\n- **What 2022-2026 actually rewarded: owning 2x mega-cap SSEs\n  unhedged.**  The +422 % EW basket CAGR is the 2x leverage compounding\n  in a smooth bull market on mega-cap momentum names.  No signal\n  needed.\n\n- Low-vol is the only signal with *significantly* negative underlying\n  skill (−48 %) — high-vol mega-caps decisively beat low-vol in this\n  period (classic low-vol-anomaly inversion in a mega-cap-momentum\n  regime).\n\nFiles:\n- `scripts/backtest_sse_momentum.py`\n- `runs/sse_underlying_expanding/backtest_sse_momentum/sse_momentum_summary.xlsx`\n- `runs/sse_underlying_expanding/backtest_sse_momentum/sse_momentum_cum.pdf`\n\n## I.5  Why this is a structural wall, not a research-direction problem\n\nThe CNN edge lives in the **single-stock cross-section of the broader\nR1000**, specifically in small-cap, high-vol, recent-loser names\n(section H's tradable cell: +13 % net CAGR, t +3.86, Mid-dv × High-vol ×\nLow-mom).  For an ETF-only mandate, this is structurally inaccessible:\n\n1. **Liquid baskets average it out.**  Section H's 477-liquid-ETF zero-shot\n   test: all LS results between −1 % and +1 %.  Diversification\n   eliminates the idiosyncratic-reversal signal by construction.\n\n2. **Single-stock ETFs cover the wrong tail.**  Issuers make wrappers\n   for ~40 ultra-popular mega-cap names — the exact opposite of the\n   alpha's natural habitat.  No one issues SSEs on unknown small-cap\n   losers because there is no retail demand.\n\n3. **Sector / thematic ETFs are baskets of those same mega-caps.**\n   XLK, SOXX, ARKK, MAGS, etc.  Same diversification problem.\n\n4. **Retraining on ETFs does not help** — I.1 documented that\n   ETF-specialised models *underperform* the zero-shot R1000 transfer.\n   The ETF universe lacks the cross-sectional dispersion to train on.\n\n5. **Generic momentum / reversal signals on SSEs have no stock-picking\n   skill** — I.4 documented zero/negative underlying-LS for every\n   classical signal tried.  Wrappers + bull market mask this in\n   headline returns.\n\n## I.6  Updated bottom line for the project\n\nThe replication is correct (section A–B).  The alpha is real and\ndeployable in single-stock space (section G + H's mid-cap cell).\n**It cannot be deployed via any ETF wrapper available to a US\ninvestor.**  This is a *constraint mismatch*, not a model failure:\n\n- The signal is structurally cross-sectional, idiosyncratic, and\n  concentrated in names too small/illiquid/specific for any ETF issuer\n  to wrap.\n- Both directions tried (basket-level ETFs and single-stock ETFs) fail\n  for orthogonal reasons (averaging vs universe-skew).\n- Retraining on the constrained universe does not produce a deployable\n  alternative — the universe is the binding constraint.\n\n**For a fund with the user's restrictions, the CNN-pattern signal is\nnot actionable.**  Pursue an unrelated alpha source compatible with\nETF-only execution (sector rotation with macro inputs, vol-of-vol on\nVXX/UVXY, calendar/seasonality on broad-market ETFs, fixed-income or\nFX-ETF carry/momentum).  Pattern-CNN remains a documented, working\nsingle-stock alpha that requires single-stock execution capability.\n\n---\n\n# Addendum J — Penny-stock contamination and the corrected headline (2026-08)\n\n**This addendum supersedes the headline numbers in sections B, G and H.**\nThose results are inflated roughly 2× by sub-penny bankruptcy shells.  The\nsignal is real and statistically significant at every filter tested — it is\nabout half the size previously reported.\n\n## J.1  How it was found\n\nWhile validating the Bloomberg→yfinance splice (§J.4), the cross-sectional\ndaily return moments were plotted around the seam as a sanity check.  Two\npre-seam days showed a cross-sectional standard deviation of **132 %**\n(2026-04-14) and **407 %** (2026-04-16) across ~1,336 stocks — arithmetically\nimpossible for real equities.\n\nThe defect is in the Bloomberg leg, predates the splice, and has been present\nin every backtest this project has run.\n\n## J.2  The defect\n\nThe Bloomberg panel contains **2,017 rows across 309 tickers with a 1-day\n`AdjClose` return exceeding ±100 %**, spanning 1996-04-12 → 2026-04-16.\n\nThey are sub-penny **bankruptcy shells** — `-Q` suffix tickers (DZSIQ, SPWRQ,\nEVVAQ, WLTGQ, BIGGQ, BLIAQ …) trading between $0.0001 and $0.03, where a\nsingle tick is a +4,800 % return.  A few divide by a near-zero prior price and\nproduce `inf`.\n\n**This is not a data error to be cleaned away.**  The prices are what\nBloomberg reports and the returns are arithmetically correct.  They are\neconomically meaningless.\n\n### Why it lands exactly on the alpha\n\nA sub-penny bankrupt shell has:\n\n| Feature | Value | Tertile |\n|---|---|---|\n| `dv_60d` (dollar volume) | tiny | **Low** |\n| `vol_60d` (realized vol) | enormous | **High** |\n| `mom_12_1` (momentum) | terrible | **Low** |\n\nSo it sorts into the **small × high-vol × recent-loser cell by construction**.\nThe contamination is not spread evenly across the panel — it concentrates\nprecisely where the headline alpha is measured.\n\nIn the monthly prediction set: **1,002 of 453,002 rows (0.22 %)** have\n|forward return| > 100 %, carrying **3.80 % of total absolute return mass**.\n53.7 % are sub-$1 and 31.6 % are sub-$0.10.\n\n## J.3  Corrected results\n\nBoth `backtest_trash_tier.py` and `backtest_liquid_grid.py` now take\n`--min-price`.  `--min-price 5.0` is the deployable setting; `0` reproduces\nthe old published numbers.  A $5 floor drops 7.1 % of rows (453,002 → 421,007).\n\n### Trash tier (Low-dv × High-vol × Low-mom), full sample 1999–2026\n\n| Variant | Gross CAGR | Sharpe | NW t | Mean names |\n|---|---:|---:|---:|---:|\n| As published (no floor) | +23.5 % | 1.04 | 5.04 | 124 |\n| Winsorize returns ±100 % | +21.0 % | 1.18 | 5.72 | 124 |\n| **Price ≥ $5** | **+12.8 %** | **0.76** | **3.71** | **90** |\n| Price ≥ $5 + winsorize | +10.1 % | 0.79 | 3.82 | 90 |\n\n### Same cell, since 2020\n\n| Variant | Gross CAGR | Sharpe | NW t |\n|---|---:|---:|---:|\n| As published (no floor) | +43.9 % | 1.43 | 3.29 |\n| **Price ≥ $5** | **+26.7 %** | **1.14** | **2.64** |\n\n### Liquid grid, price ≥ $5 — cells passing net > 0 and t ≥ 2\n\nFull sample (282 months):\n\n| Cell | Names | Gross | Net | Sharpe | t | Half-spread |\n|---|---:|---:|---:|---:|---:|---:|\n| Low × High × Low (trash) | 90 | +12.8 % | **+8.9 %** | 0.76 | 3.71 | 25 bp |\n| Mid × High × High | 54 | +9.3 % | +7.9 % | 0.55 | 2.66 | 8 bp |\n| Mid × High × Low | 70 | +8.4 % | +7.1 % | 0.55 | 2.67 | 8 bp |\n| Low × High (marginal) | 175 | +9.8 % | +6.2 % | 0.92 | 4.46 | 25 bp |\n| Mid × High (marginal) | 151 | +6.7 % | +5.5 % | 0.66 | 3.22 | 8 bp |\n| All R1000 | 1,451 | +2.8 % | +1.9 % | 0.61 | 2.97 | 8 bp |\n\nSince 2020 (63 months):\n\n| Cell | Names | Gross | Net | Sharpe | t | Half-spread |\n|---|---:|---:|---:|---:|---:|---:|\n| Low × High × Low (trash) | 101 | +26.7 % | **+22.9 %** | 1.14 | 2.64 | 25 bp |\n| High × High × High | 52 | +16.1 % | +15.7 % | 0.93 | 2.14 | 2.5 bp |\n| Low × High (marginal) | 178 | +14.4 % | +10.8 % | 1.26 | 2.91 | 25 bp |\n| **High × High (mega-cap, high-vol)** | 126 | +10.8 % | **+10.5 %** | **1.23** | 2.84 | **2.5 bp** |\n| All R1000 | 1,360 | +3.9 % | +2.9 % | 0.97 | 2.24 | 8 bp |\n\nNote that **Mid × High × Low — the \"workhorse\" cell recommended in §H.2 —\ndoes not appear in the since-2020 tradable list at all**, consistent with the\nregime finding already recorded there.\n\n## J.4  Reading the two filters against each other\n\nWinsorizing and price-flooring answer different questions, and the *gap*\nbetween them is the diagnosis:\n\n- Winsorizing at ±100 % barely moved the result (+23.5 % → +21.0 %).\n- A $5 price floor roughly halved it (+23.5 % → +12.8 %).\n\nIf a handful of freak outliers were driving the result, winsorizing would have\nkilled it.  It did not.  So the driver is a **broad population of low-priced\nnames**, and most of those returns are *real* — just untradeable.  A one-cent\nspread on a $0.30 stock is 333 bp; the strategy would pay the entire edge away\non the bid-ask bounce.\n\n**That makes this a capacity finding, not a data-cleaning one** — and it\ncompounds the §E/§G conclusion rather than replacing it.\n\n## J.5  What survives\n\n1. **The signal is real.**  The Newey-West t-statistic stays between 2.6 and\n   3.8 in every filtered variant.  Significance never breaks.\n2. **The magnitude was about double.**  Honest deployable gross is ~+13 %\n   full-sample and ~+27 % since 2020 at a $5 floor, not +28.5 % / +44 %.\n   Net of the 25 bp half-spread drag, roughly **+9 % and +23 %**.\n3. **One genuinely new candidate.**  Since 2020, the mega-cap high-vol cell\n   (High-dv × High-vol, 126 names) returns +10.5 % net at a **2.5 bp**\n   half-spread with Sharpe 1.23 — the best cost-adjusted cell in the recent\n   regime, and by far the largest capacity.  It rests on only 63 months\n   (t 2.84) and did not work full-sample, so treat it as a lead to test, not\n   a result.\n\n## J.6  Splice: Bloomberg history + yfinance forward\n\n`scripts/splice_daily_update.py` extends the Bloomberg panel forward with\nyfinance so the pipeline refreshes daily without a Bloomberg pull.  Bloomberg\nremains the history of record and is never rewritten.\n\n**The one thing that matters:** the two `AdjClose` columns are *not on the\nsame scale*.  Bloomberg's total-return index accumulates dividends forward\nfrom 1996; yfinance's Adj Close is back-adjusted from today.  Measured level\nratios on 2026-04-17 — AAPL 1.19, MSFT 1.64, JNJ 2.10.  Concatenating them\nwould inject a fake 20–110 % one-day return at the seam.  `AdjClose` is\ntherefore **chained** off the Bloomberg anchor; raw OHLCV is appended directly\n(verified to match Bloomberg exactly, return correlation 1.000000).\n\nValidation, 2026-08-04:\n\n| Check | Result |\n|---|---|\n| yfinance coverage | 99.0 % (198/200 sampled) |\n| Overlap return correlation | median 1.000000, min 0.9949 (1,339 tickers) |\n| Median absolute daily difference | 0.0015 bp |\n| Panel extended | 2026-04-17 → 2026-08-04 |\n\nGuards: ticker-reuse (only currently-live names extend, so a ticker that went\ndark in 2005 is never resurrected); per-ticker overlap correlation test with\nquarantine (correctly rejected **APTV** and **BDX**, both real spinoffs);\natomic writes; and a staleness manifest.\n\n**Known free-source holes — 33 of 1,371 names (2.4 %) do not extend.**  BK,\nCTRA and VSCO hard-404 on Yahoo despite being live liquid companies; MASI,\nSATS, IAC, NSA, JHG, BLD, LC and GOCO return 1–29 rows over seven months.\nRetested with pauses — this is not rate-limiting.  All 33 are written to\n`data/r1000_tradeable_universe.csv` with `tradeable=False`; **daily scoring\nmust drop them**, because a stale price silently poisons `dv_60d`, `vol_60d`\nand `mom_12_1` and can put a name in the wrong tertile.\n\n```bash\npython scripts/splice_daily_update.py --csv     # daily refresh\n```\n\n### Index membership cadence\n\n**Refresh from Bloomberg once a year, right after the June reconstitution.**\nThat is both the minimum and genuinely sufficient:\n\n- **Deletions self-resolve** — a dead name stops returning yfinance data, is\n  flagged stale, and drops out automatically.\n- **Additions are the only real gap** (~150–200 names/yr enter the R1000), but\n  the CNN needs 20 days for the image and `mom_12_1` needs 273 days, so **a new\n  entrant is not scoreable for ~13 months anyway.**  Chasing quarterly IPO adds\n  buys nothing.\n- **Cost is trivial** — ~1,400 unique securities once a year against a\n  ~5,000/month quota.\n\nCaveat: tertile boundaries drift as the universe ages.  One year (~15 %\nturnover) is tolerable; two or more degrades the buckets.  The trash tier is\n*more* sensitive than the full universe because it lives at the R1000/R2000\nboundary where churn is highest.\n\n## J.7  Artefacts\n\n- `scripts/splice_daily_update.py` — the splice, with validation and guards\n- `data/r1000_ohlcv_spliced.{parquet,csv}` — 12,536,918 rows, 3,185 tickers,\n  1996-01-02 → 2026-08-04\n- `data/r1000_tradeable_universe.csv` — staleness manifest (1,338 tradeable)\n- `runs/splice/<TIMESTAMP>/{splice_report.xlsx, splice_summary.json, splice.log}`\n- `runs/expanding/.../backtest_trash_tier_p5/` — trash tier at a $5 floor\n- `runs/expanding/.../backtest_liquid_grid_p5/` — 27-cell grid at a $5 floor\n- `runs/expanding/.../backtest_liquid_grid_p5_since2020/` — same, since 2020\n\n---\n\n# Addendum K — The return convention, and the Tracker paper book (2026-08)\n\n**This addendum supersedes the CAGR figures in every prior section, including\naddendum J.**  Addendum J corrected a contamination problem; this corrects an\n*arithmetic* one, and it is the larger of the two.  The signal remains real.\n\n## K.1  What `forward_return` actually is\n\nVerified empirically against the price panel, not assumed:\n\n```\nmedian | log(P_t+20 / P_t) − forward_return |  =  0.00000   (h=20)\nmedian | (P_t+20 / P_t − 1) − forward_return | =  0.0024    (h=20)\n```\n\n`forward_return` is a **20-trading-day LOG return**.  Every LS number this\nproject has published is therefore a difference of **mean log returns** across\nnames, compounded.\n\n## K.2  Why that is not a portfolio return\n\nAn equal-weighted portfolio earns the mean of **simple** returns, not the mean\nof log returns.  By Jensen, `mean(log) < log(1 + mean(simple))`, with the gap\napproximately `σ²/2` where σ is the *cross-sectional* dispersion of returns\nwithin the leg.\n\nFor a long/short spread the two legs' penalties do **not** cancel — the short\nleg is subtracted, so **its dispersion penalty is added to the reported\nspread**.  The trash tier's bottom leg has a cross-sectional SD of 0.178 per\n20-day period, i.e. a σ²/2 term of ~1.58 %/month, or ~19 %/yr of phantom\nreturn before the long leg's own (smaller) offset.\n\nThe effect is largest exactly where dispersion is largest — the trash tier.\n\n## K.3  Measured\n\nTrash tier (Low-dv × High-vol × Low-mom), $5 floor, 282 months:\n\n| Convention | Mean/month | CAGR |\n|---|---:|---:|\n| Mean log returns (as published) | +0.892 % | **+11.29 %** |\n| Simple returns (a portfolio) | +0.425 % | **+4.15 %** |\n| **Difference** | | **+7.14 pp** |\n\nFull R1000 universe, no price floor:\n\n| Book | Log CAGR | Simple CAGR |\n|---|---:|---:|\n| 50/50 top-vs-bottom half | +5.10 % | +4.12 % |\n| Decile D10−D1 | +17.36 % | **not computable** |\n\nTwo things to note in that second table:\n\n1. **The gap shrinks to ~1 pp on the broad universe** (+5.10 % → +4.12 %),\n   because cross-sectional dispersion is far lower there.  The convention\n   inflates concentrated, high-dispersion cells and barely touches diversified\n   ones.\n2. **The decile book is not computable at all under simple returns.**  Some\n   months go below −100 %: shorting a name that 10×'s loses 900 %.  The log\n   convention bounded that tail and hid it.  The $5 price floor is what makes\n   the tracked series well-defined.\n\n## K.4  What this does and does not change\n\n**Does not change:** the CNN has genuine cross-sectional predictive power.  The\ndecile monotonicity, the AUC, the sign and significance of the spread all\nstand.  This is a *reporting* correction, not a refutation.\n\n**Does change:** every headline CAGR.  The deployable trash-tier number is\n~**+4 %/yr gross on portfolio arithmetic**, not +28.5 % (original), and not\n+12.8 % (addendum J).  Layer the ~25 bp half-spread cost drag on top and the\ntrash tier is, on these numbers, **not obviously a viable book at all**.\n\nStacking both corrections, in order:\n\n| Trash tier, full sample | CAGR |\n|---|---:|\n| As originally published | +28.5 % |\n| After the $5 price floor (addendum J) | +12.8 % |\n| After portfolio arithmetic (this addendum) | **+4.2 %** |\n| Less ~25 bp half-spread turnover drag | **≈ 0 %** |\n\nThat last line is the honest bottom line for the trash tier as a standalone\nbook, and it is a materially different conclusion from where this project\nstarted.\n\n## K.5  The Tracker paper book\n\n`scripts/mark_book.py` publishes two sleeves using real portfolio arithmetic —\nactual prices, held shares, simple returns, marked daily:\n\n| Sleeve | Cell | Full-sample CAGR | Sharpe | Since-2020 CAGR |\n|---|---|---:|---:|---:|\n| `pattern-trash` | Low-dv × High-vol × Low-mom | +4.24 % | 0.31 | +4.40 % |\n| `pattern-megacap` | High-dv × High-vol | +2.42 % | 0.27 | — |\n\nThat the marker independently reproduces the +4.15 % analytic figure (+4.24 %\nwith the daily-marking calendar) is the cross-check that the *implementation*\nis right and the *convention* was the problem.\n\nBoth are **paper** — the book needs single-stock shorting.  The series is\n`backtest` before `paper_start` and `paper` after, carried in a `track` column,\nbecause the backtest leg has now been revised twice and must not quietly become\nthe track record.\n\nBenchmark is **cash**, not the cell's equal-weight return: these are\ndollar-neutral books, so excess against a long-only cell is a beta statement,\nnot alpha.  `ew_cell` is carried in the file as context only.\n\n### Daily job\n\n```bash\nbash scripts/run_daily.sh      # splice -> score -> mark -> publish (~10 min)\n```\n\n1. `splice_daily_update.py --csv` — extend the Bloomberg panel with yfinance\n2. `score_live.py` — w27 ensemble → today's `p_up` (tradeable names only)\n3. `mark_book.py` — build/mark both sleeves, publish `data/tracker/<sleeve>/`\n\nMust finish before Tracker's 13:45 PT consolidation.\n\n**Rebalance guard.** `score_live.py` emits the current day as the canonical\nmonth-end of the running month.  Left unhandled, `mark_book.py` would treat\nevery run day as a rebalance — daily turnover on a monthly strategy.  It\ntherefore drops prediction dates falling in the latest incomplete calendar\nmonth, so the book set at the last completed month-end is held through the\ncurrent one.\n\n### Registered in Tracker\n\n`pattern-trash` and `pattern-megacap`, both `mode: paper`, `cadence: daily`,\n`track: true`, read by `collect_pattern_sleeve` in Tracker's `adapters.py`.\nPerf packs build via `build_perf_report.py --strategy pattern-trash`.\n\nNot yet wired: a holdings page, which needs a loader returning\n`(list[HoldingRow], BookCharacteristics)`.  The book itself IS published to\n`data/tracker/<sleeve>/holdings_latest.csv` and the hub row carries the\nposition count.\n\n## K.6  Artefacts\n\n- `scripts/mark_book.py` — book construction and daily marking\n- `scripts/run_daily.sh` — the daily job\n- `cache/r1000_I20_monthly/` — 483,339 images, 2,846 tickers (pixel-stat reference)\n- `runs/live/live_predictions.parquet` — latest `p_up`\n- `data/tracker/{pattern-trash,pattern-megacap}/` — what Tracker reads\n- `logs/daily/YYYY-MM-DD.log`\n\n---\n\n# Addendum L — Stage 0/1 of the kill-or-confirm review (2026-08)\n\nRather than reflexively tuning the paper's hyperparameters, the project stepped\nback to ask two questions that had never been asked: **is the out-of-sample\nmachinery actually sound**, and **does the CNN know anything a free signal\ndoesn't?** Answers: yes, and — as deployed — no.\n\n## L.1  The leakage guard is sound (Gate G0: PASS)\n\n`tests/test_splits.py` only ever exercised `debug_split` at the **default\n`purge_days=0`**, on a single-ticker monthly fixture, asserting naive set\nnon-overlap. The purge that underwrites every published number — all of it\nproduced by `expanding`/`rolling` — was untested.\n\n`tests/test_purge_guard.py` now freezes it. The invariant, derived rather than\nassumed: a sample at `t` carries an image over `[t-window+1, t]` and a label\nover `(t, t+horizon]`, so a train sample at `t` and a later sample at `v`\nrequire `v - t ≥ window + horizon = 40` trading days, else the training label\nperiod physically overlaps the evaluation image.\n\n| `purge_days` | train→val and val→test gap, all three modes |\n|---|---|\n| **39 (production)** | **exactly 40 trading days** — the tight bound |\n| 0 | 1 trading day |\n\n**No leakage exists.** The out-of-sample claim was always sound; the problem\nwas the return arithmetic (addendum K), never look-ahead.\n\nRelated fix: `.gitignore` carried a bare `data/`, which matches a directory at\n*any* depth and so excluded the source package `pattern/data/` — including\n`splits.py` and `loader.py` — from every commit since the first. The two most\ncorrectness-critical modules in the repo had never been under version control.\nThat is also how the April 2026 \"relative label\" experiment was lost: its code\nwas untracked, later overwritten, unrecoverable.\n\n## L.2  One measuring stick\n\n`scripts/evaluate_signal.py` scores **any** ranking signal identically:\nportfolio (simple-return) arithmetic as the headline, the mean-log number kept\nonly as a labelled cross-reference, `$5` floor, deciles + 50/50, NW t,\nturnover, sub-periods. Validated against four independently-derived pins before\nuse (log/unfloored decile +16.26%, simple/floored +1.57%, legs 12.09/10.76/8.92,\ntrash-cell 50/50 +4.15%).\n\nIt also reports a `computable` flag instead of a misleading number: the\nunfloored simple-return decile book **has no defined CAGR** — months fall below\n−100% — and the unfloored bottom decile compounds at **+92.5%/yr**, which is the\npenny-shell blowup that makes shorting it catastrophic, now quantified.\n\n## L.3  The baseline gauntlet (Gate G1)\n\nThe paper names weekly reversal (WSTR) and TREND as its closest competitors,\nyet no simple-signal comparison had ever been run on R1000 — every backtest in\nthis repo ranks by `p_up` and uses momentum/vol only to define buckets.\n\n**Predictive IC** (Spearman vs 20-day forward return, oriented, 283 months):\n\n| Signal | mean IC | NW t | % months +ve |\n|---|---:|---:|---:|\n| **rev_5d (WSTR)** | **0.0218** | **3.43** | 54.6 |\n| combo z(−STR)+z(−vol) | 0.0172 | 2.08 | 54.6 |\n| mom_12_1 | 0.0154 | 1.60 | 57.1 |\n| vol_60d | 0.0135 | 1.03 | 50.4 |\n| rev_1m (STR) | 0.0125 | 1.79 | 52.1 |\n| **p_up (CNN)** | **0.0099** | 2.61 | 57.4 |\n\nThe CNN has the **lowest mean IC of the six**. A one-line 5-day reversal has\nmore than double the predictive correlation of a 5-seed CNN ensemble that cost\n65 A100-hours.\n\n**Honest long/short** (simple returns, $5 floor, identical rows, 282 months):\n\n| Signal | Decile CAGR | t | 50/50 CAGR | t | Turnover |\n|---|---:|---:|---:|---:|---:|\n| **rev_5d (WSTR)** | **+7.87%** | 2.53 | +4.44% | 3.00 | 0.85 |\n| **p_up residual** | **+4.19%** | **3.01** | +1.68% | 2.70 | 0.89 |\n| **p_up (as deployed)** | **+1.68%** | **1.05** | +0.46% | 0.63 | 0.89 |\n| mom_12_1 | +1.47% | 1.05 | +0.47% | 0.50 | 0.28 |\n| rev_1m (STR) | −1.46% | 0.25 | +2.09% | 1.25 | 0.84 |\n| combo | −4.87% | −0.60 | −0.35% | 0.05 | 0.76 |\n| vol_60d | ",
  "bytes": 60000,
  "sha": "59517a78cda2c473446f96e976f435ecf95ae06c5ac9efd8ba00be0752327636",
  "repo_slug": "arjundivecha/pattern",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_arjundivecha_pattern_openwiki_index_md_1af6bfa6/readme"
}