{
  "markdown": "# ASADO — Country Data Collection & Research Platform\n\nASADO assembles macro / governance / risk / climate / trade / market data for a **34-country\nuniverse** from **~38 sources** and stores it in a **hybrid DuckDB + Neo4j** stack with a raw\nwarehouse, a canonical normalized feature layer, bilateral trade/banking/portfolio networks, a\n**daily-frequency extension**, prediction-market & event surfaces, a **global commodity surface**,\nand country-state vector embeddings.\n\nIt runs on **two cadences**, both self-generating (no hand-maintained spreadsheets in the loop):\n\n- **Monthly** (`scripts/monthly_update.py`) — collects everything, rebuilds the warehouse + graph.\n- **Daily** (`scripts/daily_update.py`) — the same factor pipeline on a daily metronome, for the two\n  domains that move daily: **T2 prices** (Bloomberg-direct) and **GDELT news** (GKG-direct). Econ has\n  no daily factors and is skipped.\n\n**Returns are the outcome source of truth.** Country and factor-portfolio returns live in dedicated\nreturn surfaces; macro, commodity, prediction-market, GDELT, Bloomberg, and graph layers are\n*explanatory context* unless explicitly joined back to returns.\n\n> Live, per-variable detail is auto-generated each rebuild — see **[`docs/factor_reference.md`](docs/factor_reference.md)**\n> and the latest **[`docs/DATABASE_AUDIT_2026_06_09.md`](docs/DATABASE_AUDIT_2026_06_09.md)**.\n\n---\n\n## Quick start\n\n```bash\ncd ASADO\nsource venv/bin/activate\n\npython scripts/monthly_update.py        # full monthly: collect → DuckDB → normalize → daily → Neo4j\npython scripts/daily_update.py          # daily metronome: T2 + GDELT factors & returns\n```\n\n### Cold start for a new agent (read this first)\n\nIf you just picked up this repo, do these five things in order before touching code:\n\n1. **Read the docs in this order:** `README.md` (this file), `CLAUDE.md`\n   (coding conventions), `AGENTS.md` (learned facts), `docs/README.md`\n   (documentation index: canonical specs vs. generated reports vs. historical\n   snapshots), and `llmchat.md` (recent session history).\n2. **Activate the venv:** `source venv/bin/activate`. Bloomberg-only scripts run\n   under the OpusBloomberg conda env; everything else uses this venv.\n3. **Check services:** `brew services list | grep neo4j` must show `started` for\n   any graph operation. Bloomberg Terminal must be logged in on the Parallels\n   Windows VM for live Bloomberg pulls.\n4. **Run the smoke tests:** `python -m pytest tests/loop/test_harness_pit.py -q`\n   should pass. For a full health check:\n   `python scripts/setup_duckdb.py --check && python scripts/setup_neo4j.py --check`.\n5. **Verify the loop DB exists:** `Data/loop/asado_loop.duckdb` is where loop\n   signals, returns, and harness results live. It is gitignored but essential.\n   If it's missing, run `scripts/loop/loop_daily_job.py` (or the full\n   `daily_update.py`) after the main DB is built.\n\n**Never create persistent tables in `Data/asado.duckdb`** — `setup_duckdb.py`\ndeletes and recreates it. Loop data lives in `Data/loop/asado_loop.duckdb` and\nparquet intermediates under `Data/work/loop/`. For the full source-of-truth map,\nsee `docs/README.md`.\n\n| Command | What it does | Runtime |\n|---|---|---|\n| `monthly_update.py` | All collectors, both Bloomberg pulls, warehouse + normalized + daily panels + optimizer returns + Neo4j + embeddings + schema | ~12–18 min |\n| `daily_update.py` | Daily Bloomberg pull → T2 master/normalize/returns; GDELT normalize/returns | ~12–15 min |\n\n**Prerequisites:** Bloomberg Terminal logged in on the Parallels Windows VM (for the live pulls);\nNeo4j running (`brew services start neo4j`). Logs land in `Data/logs/`.\n\n**Automated morning run (2026-06-10):** launchd job `com.arjundivecha.asado-daily`\nruns `scripts/run_asado_daily.sh` weekdays at 07:30. It auto-starts the Parallels\nVM + bbcomm, runs a live Bloomberg data-pull preflight (iMessages once and\nretries every 20 min until 11:00 if the Terminal needs a login), then runs\n`daily_update.py --resume` with one auto-retry. `daily_update.py` is now\n**fail-fast + resumable**: completed stages checkpoint to\n`Data/logs/daily_update_progress_YYYY_MM_DD.json`, a failed stage aborts the\nrest (no downstream stages on stale data), and `--resume` continues from the\nfailed stage without redoing the Bloomberg pull. The 07:00 NightWatch digest\n(`A Working/NightWatch/`) reports the result each morning.\n\n### Common options\n```bash\npython scripts/monthly_update.py --skip-bloomberg     # no Terminal — reuse existing Bloomberg data\npython scripts/monthly_update.py --db-only            # rebuild DBs from existing panels (no collection)\npython scripts/monthly_update.py --commodity-only     # commodity collector + DB/daily/schema refresh\npython scripts/monthly_update.py --skip-neo4j         # skip graph + embeddings\npython scripts/daily_update.py --skip-bloomberg       # reuse existing daily Bloomberg pull\npython scripts/daily_update.py --t2-only              # T2 chain only (no GDELT / DB / graph)\n```\n\n---\n\n## Architecture: two cadences, one warehouse\n\n### Monthly pipeline (`monthly_update.py`)\n1. **Collectors** — free sources (external, extended, IMF, bilateral, macrostructure, World Bank\n   commodities) + **two live Bloomberg pulls**:\n   - `collect_bloomberg.py` — the 28 Bloomberg **econ/market** variables (bonds, CDS, OIS, breakevens, WIRP, ECFC, PMI, M2, ETF passive flows).\n   - `collect_t2_bloomberg.py` — the **T2 country index/price/MCAP/commodity/FX** workbook, pulled **direct from Bloomberg** (replaces the old hand-saved Excel dump).\n2. **T2 master + benchmarks** (`build_t2_master.py`, `build_benchmark_rets.py`) — raw factors, returns, P2P.\n3. **GDELT ingest** (`gdelt_ingest/`) — fetches new GKG days from `data.gdeltproject.org`, builds the monthly + daily signal workbooks in-repo.\n4. **Econ workbook** (`build_econ_panel.py`) — Econ.xlsx from the warehouse.\n5. **Normalize + optimizers** — T2 / GDELT / Econ `_CS`/`_TS` factors and top-20 factor-return portfolios; ingested to `factor_returns`.\n6. **Warehouse** — `setup_duckdb.py` (raw union), `build_normalized_panel.py` (normalized + `feature_panel`), `build_daily_panels.py` (daily tables), prediction markets, event log.\n7. **Graph + docs** — `setup_neo4j.py`, `build_embeddings.py`, `build_schema_registry.py`, `build_factor_reference.py`.\n\n### Daily pipeline (`daily_update.py`)\nThe monthly factor flow at a **daily metronome** — 1/5/20/60/120-**day** returns instead of\n1/3/6/9/12-month — for the two fast-moving domains. T2 prices come **direct from Bloomberg**\n(`collect_t2_bloomberg.py --daily`, calendar-day grid); GDELT signals come from the GKG ingest.\nOutputs land in `Data/work/t2_daily/` and `Data/work/gdelt_daily/`, then load into the DuckDB daily\ntables. Designed to run unattended on a schedule with the Bloomberg connection kept alive.\n\n### Self-contained work dirs\nAll intermediate artifacts are internalized under `Data/work/{t2,gdelt,econ,t2_daily,gdelt_daily}` —\nASADO generates everything it needs from Bloomberg + the free APIs + GKG; there are no\nhand-maintained Excel files in the pipeline.\n\n---\n\n## Warehouse contents (current shape)\n\nDuckDB at `Data/asado.duckdb` (~3.5 GB). **37 objects** (33 tables + 4 views) across five layers.\nTidy schema everywhere: `(date, country, value, variable, source)`.\n\n### Layer 1 — collected sources (feed `unified_panel`)\n~38 sources. Largest contributors (base variables): **t2** (111), **t2_raw** (53), **gdelt** (93),\n**bloomberg** (28), plus the macro block — IMF (×7 datasets), BIS (×5), OECD (×5), ECB FX, World\nBank, FRED, GPR, EPU, ND-GAIN, FAOSTAT, ILOSTAT, UNDP, OFAC, demographics. (Full per-source\nbreakdown in the audit doc.)\n\n### Layer 2 — query surfaces (country-keyed)\n| Surface | Rows | Variables | Note |\n|---|---:|---:|---|\n| `unified_panel` (view) | ~12.1M | ~426 base | raw cross-source union |\n| `normalized_panel` | ~0.8M | ~294 | `_CS` (cross-sectional) + `_TS` (time-series) z-scores |\n| `feature_panel` (view) | ~3.3M | **~720** | primary query-facing union (raw + normalized) |\n\n### Layer 3 — daily extension (fresh through the latest run)\n| Table | Rows | Vars |\n|---|---:|---:|\n| `t2_factors_daily` | ~35.6M | 111 |\n| `t2_levels_daily` | ~15.3M | 48 |\n| `gdelt_factors_daily` | ~10.2M | 75 |\n| `gdelt_raw_daily` | ~967K | 45-col (249-country bridge) |\n| `daily_calendar` | ~328K | per-country trading days |\n\n### Layer 4 — returns (outcome source of truth)\n| Table | Factors | Sources |\n|---|---:|---|\n| `factor_returns` (monthly) | ~390 | `t2_optimizer`, `gdelt_optimizer`, `econ_optimizer` |\n| `factor_returns_daily` | ~180 | `t2_optimizer_daily`, `gdelt_optimizer_daily` |\n| `factor_top20_membership` | ~393 | sparse country membership per factor bucket |\n| `country_factor_attribution` (view) | — | `membership ⨝ returns` = weight × factor_return |\n\n### Layer 5 — knowledge graph (Neo4j)\n~**1,174 nodes** (Factor, Country ×34, CentralBank ×31, DataSource, CrisisEvent, SanctionsProgram,\nCommodity ×4) and ~**30K edges** (`HAS_FACTOR_EXPOSURE`, `TRADES_WITH`, `HAS_BANKING_EXPOSURE_TO`,\n`HOLDS_PORTFOLIO`, `HAS_CRISIS_HISTORY`, `HAS_CENTRAL_BANK`, `SUBJECT_TO`, `EXPORT_EXPOSED_TO`,\n`DATA_AVAILABLE_FROM`). 34-dim cosine `countryStateIndex` on `Country.state_embedding`.\n\n### Global commodity surface (NOT country-tiled)\nCommodities are **global series** — one value per date, not tied to a country. They live in:\n- `commodity_panel` (view) — `WB_CMDTY_<SERIES>_<FEATURE>` for 87 series × 7 features (level, MOM, YOY, 3M/12M return, 12M vol, 36M z), date-keyed.\n- `wb_commodity_prices` / `wb_commodity_indices` / `wb_commodity_features` / `wb_commodity_meta` — canonical Pink Sheet tables.\n\n> Earlier builds *broadcast* commodities identically across all 34 countries into `feature_panel`\n> (`wb_commodity_factor_panel`). That tiling is **deprecated** — it inflated the variable count and\n> produced degenerate cross-sectional variants. Join `commodity_panel` to returns **on `date`**.\n\n### Style-factor benchmark (Ken French — NOT country-tiled)\n`ff_factors` holds the Fama-French **5 factors + momentum + RF** (`Mkt_RF, SMB, HML, RMW, CMA,\nRF, WML`) for **8 FF regions** (US, Developed, Developed_ex_US, North_America, Europe, Japan,\nAsia_Pacific_ex_Japan, Emerging), **monthly + daily**, USD, values in **percent**. US history\nruns to 1926/1963, the developed regions to 1990, Emerging (monthly only) to 1989.\n\nThese are a **benchmark/explanatory** surface, not a return source — the question they answer is\n\"is a signal's P&L genuine alpha, or just repackaged value/size/momentum/quality/market beta?\"\nLike commodities, they are **isolated** (region-keyed, 8 series) and **never broadcast** to the\n34 countries or unioned into `feature_panel`/`unified_panel`. The country→region link lives in\n`config/ff_region_map.json` and is applied **on the fly** at regression time. Built by\n`scripts/collect_ff_factors.py`; the spanning tool is `scripts/harness/ff_spanning.py` (below).\n\n### Auxiliary\n`bilateral_portfolio_matrix` (reporter–counterparty ownership), `predmkt_*` (Kalshi/Polymarket\nsnapshots + spillovers + composites), `event_log` (curated dated events), `variable_meta`\n(structural metadata), `country_reference`.\n\n---\n\n## Returns source of truth\n\nAnchor performance / winner-loser / attribution questions on these surfaces:\n\n- **Country returns (T2, 34 countries):** monthly = `feature_panel` rows `source='t2'`, variables `1MRet/3MRet/6MRet/9MRet/12MRet`; daily = `t2_factors_daily`, `1DRet/5DRet/20DRet/60DRet/120DRet`. The `gdelt`-source return rows are bit-exact aliases of the T2 returns (the optimizer's dependent variable), not a second source.\n- **Factor portfolio returns:** monthly `factor_returns` (`econ_/t2_/gdelt_optimizer`); daily `factor_returns_daily` (`t2_/gdelt_optimizer_daily`). These are top-20%-of-countries portfolio returns, not raw factor levels.\n- **Attribution:** `country_factor_attribution` = `factor_top20_membership ⨝ factor_returns` → `weight × factor_return` per (country, factor, period).\n\n**Cycle guardrail:** optimizer outputs (`factor_returns*`, `factor_top20_membership`,\n`country_factor_attribution`) are never unioned into `feature_panel`/`unified_panel` — those are the\ninputs the optimizer consumes. `scripts/qa/validate_returns_first.py` enforces this.\n\n---\n\n## Data resilience\n\nEvery collector is safe to re-run: load existing panel → fetch fresh → **source-level merge** (a\nfailed source keeps its prior data) → timestamped backup before overwrite → run-history JSON →\ndelta report. One source failing never breaks a run. The DB is fully rebuildable from the panels\n(`--db-only`); rebuilds from an empty file shed any orphan tables.\n\n---\n\n## Variable families (reference)\n\n- **Equity factors (T2, Bloomberg-direct):** `t2_master`/`t2_raw` — Best/Trailing PE, Earnings Yield, ROE, EPS, MCAP, 120MA Signal, RSI14, vol, momentum (1/3/6/9/12M returns + spreads), P2P.\n- **External macro (`external_factors`, 7 src):** EPU, GPR (+Global GPR/Threat/Act), BIS Credit-GDP Gap, BIS Property, BIS REER, OECD CLI, World Bank governance/macro/demographics/reserves/climate.\n- **Extended macro (`extended_factors`, 12 src):** BIS policy/debt-service rates, OECD BCI/CCI, ECB FX (24 pairs), ND-GAIN (score/vulnerability/readiness), ILOSTAT, UNDP HDI/IHDI/GDI/GII, OFAC, FAOSTAT, FRED (VIX, UST 2Y/10Y, curve, USD index, HY OAS), EIA.\n- **IMF (`imf_factors`, 7 datasets):** CPI + YoY, WEO projections (→2031), BOP aggregates, money-market/discount/bond/T-bill rates, FX, employment, trade-in-goods.\n- **Macrostructure (`macrostructure_factors`):** IMF FSI bank fragility, WB QPSD debt structure, OECD institutional depth, central-bank footprint (`MS_CentralBank_*`), reserve adequacy, swap-line access, policy backstop, investor-base fragility.\n- **Bloomberg econ (`bloomberg_factors`, 28):** sovereign bonds (2/5/10/30Y), CDS 5Y, breakeven 10Y, OIS 10Y + Z-spread, WIRP implied rate, M2 YoY, PMI mfg/svcs, ECFC GDP/CPI consensus, debt/GDP, MIPD, yield-curve slope, and the ETF passive-flow family (`MS_*`).\n- **GDELT (`gdelt_panel`, 93):** salient news signals — attention (fast/slow/shock/trend ±z), country news risk/sentiment/attention (raw + normalized, ±CS/TS). *(The 1,300+ deep theme/GCAM fields were retired.)*\n- **Commodities (`commodity_panel`, 87 global series):** Pink Sheet prices + indices with level/MOM/YOY/3M/12M-return/vol/z features. Global, joined to returns on date.\n\n---\n\n## Country coverage notes\n- 34-country T2 universe is always fully present. **9 extra countries** (Austria, Belgium, Finland, Greece, Ireland, New Zealand, Norway, Portugal, Russia) leak into the panels from multi-country feeds (ECB FX / OECD / IMF).\n- Sparse-by-source: EPU ~21/34, OECD CLI/BCI/CCI ~20–22, BIS policy rates 26, CDS 15, breakevens 6.\n- Annual/lagged sources sit a year+ back by nature (UNDP HDI, ND-GAIN → 2023; FAOSTAT, IMF BOP, portfolio ownership → 2024). **`eia` is dead at 2019-12** (broken collector — fix or drop).\n- Forecast data is intentional: `imf_weo` → 2031, `demographics_dip` → 2100.\n\n---\n\n## Database access\n\n### Python bridge (`scripts/db_bridge.py`)\n```python\nfrom scripts.db_bridge import AsadoDB\nwith AsadoDB() as db:\n    df       = db.query_panel(\"SELECT * FROM feature_panel WHERE country='Brazil' LIMIT 10\")\n    records  = db.query_graph(\"MATCH (c:Country)-[:HAS_CRISIS_HISTORY]->(e) RETURN c.t2_name, e.name\")\n    profile  = db.country_profile(\"Turkey\")           # factors + all graph relationships\n    snapshot = db.factor_snapshot(\"BIS_Credit_GDP_Gap\")\n    ff       = db.ff_factor_series(country=\"Brazil\", frequency=\"monthly\", wide=True)  # → Emerging FF bundle\n    similar  = db.query_graph(\"\"\"MATCH (c:Country {t2_name:'Turkey'})\n        CALL db.index.vector.queryNodes('countryStateIndex',6,c.state_embedding)\n        YIELD node,score WHERE node<>c RETURN node.t2_name AS country, score ORDER BY score DESC\"\"\")\n```\n\n### Neo4j\n`brew services start neo4j` · bolt://localhost:7687 · web http://localhost:7474 · creds `neo4j`/`mythos2026`.\n\n### MCP server — query ASADO from Claude Desktop\n`scripts/asado_mcp_server.py` is a stdio MCP server exposing a read-only tool surface. Register it in\n`~/Library/Application Support/Claude/claude_desktop_config.json` under `mcpServers` using ASADO's\nvenv Python, then relaunch Claude Desktop:\n\n```json\n{ \"mcpServers\": { \"asado\": {\n    \"command\": \"/Users/arjundivecha/Dropbox/AAA Backup/A Working/ASADO/venv/bin/python\",\n    \"args\": [\"/Users/arjundivecha/Dropbox/AAA Backup/A Working/ASADO/scripts/asado_mcp_server.py\"],\n    \"env\": { \"ANTHROPIC_API_KEY\": \"sk-ant-...\" } } } }\n```\n\nKey tools: `ask_asado` (NL Q&A — needs `ANTHROPIC_API_KEY`), `get_schema_summary`, `run_duckdb_sql`,\n`run_neo4j_cypher`, `get_country_profile`, `event_window` / `events_in_window`, `daily_factor_series`,\n`country_returns`, `factor_return_series`, `country_factor_attribution`, `return_leaders`,\n`predmkt_snapshot` / `country_signal_now` / `event_market_set`, `commodity_price_series`. DuckDB is\nopened read-only; treat Cypher as read-only by convention. (ChatGPT Desktop isn't supported — it\nneeds a remote SSE endpoint; this server is stdio.)\n\nTroubleshooting: `nc -z localhost 7687` (Neo4j up?); `./venv/bin/python scripts/build_schema_registry.py`\n(refresh schema cache); check `~/Library/Logs/Claude/mcp*.log` if the server fails to start.\n\n---\n\n## Alpha-Hunting Loop (Layer 1 + validation infrastructure)\n\nThe research loop that sits on top of the warehouse (spec: `PRD_Alpha_Hunting_Loop.md`). All loop-owned\nstate lives in a **separate DuckDB** — `Data/loop/asado_loop.duckdb` — so monthly rebuilds of\n`asado.duckdb` can never destroy it. The main DB is attached read-only as the `asado` schema.\n\n### Nightly job (launchd, 06:45)\n`scripts/loop/loop_daily_job.py` (`com.arjundivecha.asado-loop-daily`) runs 33 steps, in order\n(each in its own subprocess; one failure never stops the rest, but any failure exits non-zero):\n\n1. `collect_news_bridge.py` — portfolio holdings + 800-ticker ETF closes from the News repo\n   (`portfolio_holdings_daily`, `portfolio_summary_daily`, `etf_prices_daily`, `etf_t2_map`).\n2. `ledgers.py --mark` — auto-marks open theses from T2 daily returns; closes on invalidation/expiry.\n3. `build_country_returns.py` — refreshes `country_returns_monthly` (canonical marking surface).\n4. `build_tot_shares.py` — Comtrade commodity trade shares for D1 (slow-moving, fetch-failure-safe).\n5. `build_graph_features.py` — Neo4j edges × T2 returns → `graph_features_daily`\n   (trade/banking neighbor-return gaps, holder stress, two-hop propagation; 2008→present).\n6. `build_forward_calendar.py` — curated forward catalysts (CB decisions, elections, index reviews).\n7-8. `collect_foreign_flows_bbg.py` + `collect_foreign_flows.py` — Bloomberg exchange-sourced foreign\n   equity flows (KR/TW/TH/PH/ID) + NSDL India → `foreign_flows_daily`.\n9-10. `collect_sovereign_daily_bbg.py` + `load_sovereign_daily.py` — daily 5Y + 1Y CDS (20/18\n   countries) + direct-pull 10Y + 2Y yields (32/27) → `sovereign_daily` (the D4 v2 cross-asset\n   legs) + `sovereign_signals` (CDS curve slope 5Y−1Y — **inversion = imminent-distress\n   pricing** — and the daily 2s10s govt slope, both with 252d z-scores).\n11. `build_valuation_block.py` — month-end CAPE/PB/DY/EY/ERP + 10y percentiles → `valuation_monthly`.\n12. `collect_weo_vintages.py` — IMF WEO vintage surface → `weo_vintages` / `weo_revisions` (D3 input).\n13-14. `collect_etf_flows_bbg.py` + `load_etf_flows.py` — country-ETF shares-out/NAV/AUM →\n   `etf_flows` + `etf_flow_signals` (positioning/crowding layer).\n15-16. `collect_consensus_bbg.py` + `load_consensus.py` — Bloomberg ECFC consensus GDP/CPI forecast\n   history → `consensus_daily` + `consensus_revisions`.\n17. `collect_cot.py` — CFTC COT speculator positioning, 12 commodity futures (keyless Socrata).\n18-19. `collect_market_implied_bbg.py` + `load_market_implied.py` — **market-implied stress layer**:\n   FX ATM implied vol in THREE tenors (1W/1M/3M — the 1W−3M term slope inverts when stress is\n   priced NOW), 25-delta risk reversals + butterflies, and 3M forward/NDF-implied carry for 25 T2\n   currency pairs (30 countries, incl. the HKD/SAR peg surfaces and forward-only Denmark), the\n   VIX/VIX3M/MOVE/HY-IG-OAS/DXY/BBDXY dashboard, and CL/CO/HG/GC/NG 1st+2nd futures generics\n   (2006+) → `market_implied_daily` + `market_implied_signals` (252d z-scores, VIX term-structure\n   ratio, vol term slope, carry z, curve shape). RR is sign-normalized: **positive = options\n   premium on local-currency depreciation**; carry is sign-normalized: **positive = local rates\n   above USD**. See `docs/MARKET_IMPLIED_EXTENSION_STATUS.md` +\n   `docs/BBG_SKILL_ENHANCEMENTS_2026_06_12.md`.\n20-21. `collect_sov_ratings_bql.py` + `load_sov_ratings.py` — **sovereign rating history via BQL**\n   (ASADO's first BQL collector: `//blp/bqlsvc` + EXCEL unlock + the `issuerof()` hop, CDS-contract\n   anchors): S&P/Moody's/Fitch monthly back to 2015, 33 countries, 21-pt numeric scale →\n   `sov_ratings_monthly` + the dated `sov_rating_changes` event table (128 events).\n22-23. `collect_eco_surprise_bbg.py` + `load_eco_surprise.py` — **economic surprise layer**:\n   ACTUAL_RELEASE vs BN_SURVEY_MEDIAN on ECO release tickers (CPI YoY 31 countries, unemployment,\n   GDP, Markit PMI) → `eco_surprise_monthly` + `eco_surprise_signals` (per-print surprise z,\n   growth/inflation surprise composites).\n24. `build_graph_features_pit.py` — **point-in-time graph features** (see \"The graph machine\"\n   below): PIT trade/bank/twohop/holder gaps + Katz, hub-amplified and trade-bloc features from\n   the stored edge vintages → `graph_features_pit_daily`.\n25. `build_similarity_features.py` — **fundamental-twins map**: month-end factor vectors →\n   top-5 cosine twins → twin-convergence gaps → `similarity_features_daily` + `similarity_twins`.\n26. `build_leadlag_features.py` — **lead-lag network**: monthly lag-1 cross-correlation edges →\n   leader-gap features → `leadlag_features_daily` + `leadlag_edges`.\n27. `build_combiner.py` — **walk-forward ridge combiner** → `combiner_scores` (monthly, tested\n   DEAD, kept for the record) + `combiner_scores_daily` (the live prediction surface).\n28. `write_graph_discoveries.py` — pushes `SIMILAR_TO` / `LEADS` edges + combiner ranks into\n   Neo4j for MCP/browser exploration.\n29. `build_dislocations.py` — detectors **D1 D2 D3 D4 D5 D7 D8 D9 D10** → `dislocation_daily` +\n   the daily brief at `Data/dislocations/brief_YYYY_MM_DD.md` (the Layer 2 reading list; includes\n   forward-calendar, market-implied stress, sovereign-curves/ratings/surprises, foreign-flow,\n   ETF-positioning and COT context sections). D10 (A10, live 2026-06-11) fires on\n   FX-options-vs-equity conflicts: options stress unpriced by equity, or equity stress unconfirmed\n   by options. D6 stays blocked until predmkt history accumulates.\n29b. `build_triptych_scan.py` — **the Triptych prior layer** (2026-07-02, ASADO-native port of\n   the Triptych visual tool's analytics kernel): exhaustive sweep of every `t2_raw` factor +\n   the warehouse variables declared in `config/triptych_scan.yaml` × 34 countries ×\n   3 normalizations (raw / expanding-z / cross-country-z) × 2 return modes (absolute/relative)\n   × 6 horizons, in BOTH threshold modes — **pit** (point-in-time expanding deciles, no\n   lookahead → the prior surface, PRD 7.3 confidence) and **full** (full-sample descriptive,\n   confidence hard-zeroed). → `triptych_scan` + `triptych_review_queue` tables, `triptych_priors`\n   view, parquets in `Data/loop/`. Kernel: `scripts/loop/triptych_kernel.py` (line-verified\n   parity with the tool's core.js; PIT canaries in `tests/loop/test_triptych.py`). Surfaces:\n   cockpit \"Triptych\" desk tab + country-letter priors, Fable packet block, MCP tools\n   `triptych_link` / `triptych_prior_snapshot` / `triptych_queue`. Deep links into the visual\n   tool (https://triptych-one.vercel.app) via `scripts/triptych_tool_link.py`. ~10 s, all cores.\n30. `build_evidence_packs.py` — freezes GDELT headlines for tonight's fired dislocations.\n31. `ledgers.py --rebuild` — folds the JSONL ledgers into loop-DB tables.\n32. `calibration_report.py` — regenerates the current-month calibration report (PARTIAL-stamped\n   until ≥ 10 closed theses).\n33. `build_jst_risk_report.py` — dated JST long-cycle tail-risk report (xlsx + PDF) in\n   `Data/loop/risk_reports/`: per-country current drawdown → JST 1870-2020 bucket → forward\n   real-equity tail (the once-in-a-century p10 the modern sample can't see). Read-only; DM names\n   are in-scope, EM names carry a DM-analogy label.\n\nA second launchd job (`com.arjundivecha.asado-predmkt-daily`, 06:30) runs\n`scripts/predmkt_daily_job.py`: restore-from-archive → collect prediction markets → re-archive to\n`Data/loop/predmkt_archive/` (rebuild-proof). The curated registry (`config/predmkt_curated.yaml`)\nwas expanded to 152 live markets on 2026-06-11; `scripts/loop/discover_predmkt_candidates.py`\nsweeps Kalshi + Polymarket for new candidates to curate.\n\n### Validation discipline (the skeptic)\n- **Hypothesis ledger** (`ledgers/hypothesis_ledger.jsonl`, git-tracked): mechanism written **before**\n  results; every registration counts a trial against its family for the deflated Sharpe.\n- **Harness** (`scripts/harness/evaluate_signal.py`, MCP `evaluate_signal`): PIT embargo inside the\n  harness, rank IC + Newey–West t, top-7 vs EW with costs, sub-periods, deflated Sharpe vs family\n  trial count. v2.1 (2026-06-12) adds a 1d/5d/21d hold-period grid, breakeven cost (bps), and a\n  5 bps cost case for daily runs — see \"The cost / holding-period model\" below.\n  Forward-return variables (`1MRet` etc.) are hard-blacklisted as signals.\n  Results → `Data/loop/harness_runs/` + `harness_results` table; verdicts auto-attach to the ledger.\n- **Thesis ledger** (`ledgers/thesis_ledger.jsonl`): frozen entry thesis + probability + invalidation\n  level; auto-marked daily; Brier calibration accumulates as theses resolve. Paper by default.\n- PIT unit tests: `tests/loop/test_harness_pit.py` (alignment, lookahead canaries, NW-t, DSR).\n\n### Analysis tools (added 2026-06-12)\n- **Batch sweep runner** (`scripts/harness/sweep_signals.py`): reads a YAML spec\n  (`config/sweeps/*.yaml`), pre-registers each signal as a hypothesis (mechanism text is\n  mandatory), runs the harness, and writes a sweep summary (JSON + xlsx) to\n  `Data/loop/harness_runs/sweeps/`. Every entry is a trial charged against its family —\n  re-specs and flipped re-registrations included. First family swept:\n  `bbg_skill_2026_06` (15 trials across the six new Bloomberg layers; see\n  `config/sweeps/new_bbg_layers_2026_06.yaml` and `..._round2_...yaml`).\n  The 2026-06-12 systematic pass added 18 more trials over the previously\n  untested loop families (valuation block, ETF flows/short interest, graph\n  spillovers, consensus revision momentum, FX vol level — specs in\n  `config/sweeps/`). Honest scoreboard: graph spillover family produced four\n  WEAK verdicts with the strongest ICs in the warehouse so far\n  (banking-claims neighbor gap nw_t 5.7, two-hop trade gap 4.5 — but graph\n  edges are current-weights, not PIT, so promotion requires a vintage-edge\n  robustness pass); CPI consensus revision momentum WEAK (nw_t 2.3, same\n  reflation direction as the flipped inflation-surprise result); ETF flow\n  momentum significant *against* its registered direction (contrarian,\n  nw_t −2.2, would need a fresh flipped registration to claim); valuation\n  percentiles and short interest DEAD cross-sectionally.\n- **Event-study engine** (`scripts/loop/event_study.py`): cross-event CARs on T2 country\n  returns around discrete events with market-adjusted abnormal returns, bootstrap CIs,\n  next_day/next_month anchoring, and PDF/xlsx/JSON outputs to `Data/loop/event_studies/`.\n  Presets: `rating_downgrade`, `rating_upgrade`, `cds_inversion`, `growth_hot`,\n  `growth_cold`, `inflation_hot`, `dislocation` (filterable by detector), `event_log`\n  (curated registry), or arbitrary `--events-sql`. Verified findings (2026-06-12):\n  sovereign rating downgrades → −0.7% abnormal return in 5 trading days (t≈−2.0,\n  73 events since 2008), drifting to −2.1% at 63 days; CDS 1Y/5Y curve inversions →\n  −4.5% at 63 days (t≈−2.4, 41 events) — the 1Y CDS layer's payoff is event-conditional,\n  not cross-sectional; rating *upgrades* also drift −2.4% at 63 days (t≈−2.2, hit rate\n  0.31, n=45 — read cautiously); hot/cold growth surprises show no event drift. The\n  `dislocation` preset correctly returns INSUFFICIENT_EVENTS until detector history\n  accumulates a forward window (D1–D10 rows begin 2026-06-09).\n- **Daily portfolio backtests** were already in harness v2 (`backtest_daily`); the harness\n  now also scales its coverage gate and top-N proportionally when a sub-universe is\n  declared (e.g. the 26-country FX-options universe).\n- **Ken French style-spanning** (`scripts/harness/ff_spanning.py`, added 2026-06-17):\n  regresses any return series on a regional Fama-French factor model\n  (`capm`/`ff3`/`carhart`/`ff5`/`ff5_mom`) and reports the **alpha with a Newey–West HAC\n  t-stat**, every factor beta, and R². The complement to IC/Sharpe: a signal whose raw\n  Sharpe looks good but whose spanning alpha is insignificant is a known style tilt, not\n  new alpha. Country→region via `config/ff_region_map.json` (Brazil→Emerging, Germany→\n  Europe, U.S.→US, …). CLI: `python scripts/harness/ff_spanning.py --country Brazil\n  --model ff5_mom`; programmatic: `style_spanning(ls_returns, country=\"Brazil\")`. Data is\n  the isolated `ff_factors` table (8 FF regions, monthly+daily). Use `subtract_rf=False`\n  for a self-financing long-short P&L, `True` for a long-only total return.\n\nMCP tools added to the server: `country_news` (live GDELT DOC 2.0 headlines), `register_hypothesis`,\n`evaluate_signal`, `open_thesis`.\n\nMonthly vintage snapshots: `scripts/snapshot_vintages.py` → `Data/vintages/{YYYY_MM}/` (wired into\n`monthly_update.py`).\n\n### The graph machine (added 2026-06-12)\n\nThe connection-finding layer built after the graph spillover family produced the strongest\nresults in the first systematic pass. Five new builders (all in the nightly job, steps 24-28):\n\n- **Point-in-time edges** (`scripts/loop/collect_pit_edges.py`, run monthly via\n  `monthly_update.py` step 4b): the FULL historical archive of bilateral weights — trade\n  (IMF IMTS, 27 annual vintages 1999-2025), banking (BIS LBS, 108 quarterly vintages),\n  portfolio holders (IMF PIP, 25 annual vintages) — each applied only after a conservative\n  publication lag (trade +4m, bank +4m, holder +9m) → `graph_edge_vintages`. This removed\n  the v1 \"current-weights\" lookahead caveat.\n- **PIT graph features** (`build_graph_features_pit.py`): PIT versions of all seven v1\n  features plus three new graph algorithms — Katz 3-hop propagation (decay 0.5), PageRank\n  hub amplification, spectral trade blocs (k=4) → `graph_features_pit_daily` (GRAPHP_*).\n- **Fundamental twins** (`build_similarity_features.py`): each month-end, countries become\n  vectors of ~41 normalized fundamental factors (NO return/technical factors); top-5 cosine\n  twins applied to the next month → `similarity_features_daily` + `similarity_twins`.\n- **Lead-lag network** (`build_leadlag_features.py`): monthly re-estimated lag-1\n  cross-correlation network (corr ≥ 0.15, ≥ 150 overlapping days; U.S. complex dominates as\n  leader, exactly as timezone mechanics predict) → `leadlag_features_daily` + `leadlag_edges`.\n- **Ridge combiner** (`build_combiner.py`): walk-forward ridge (expanding window, January\n  refits, 60m burn-in) over the harness survivors → `combiner_scores` (monthly) +\n  `combiner_scores_daily` (daily, 6 market-derived features, next-5d target).\n- **Neo4j write-back** (`write_graph_discoveries.py`): latest twins → `SIMILAR_TO` edges\n  (170), lead-lag → `LEADS` edges (192), combiner ranks → `Country.combiner_score/rank`.\n\n**Scoreboard from the 2026-06-12 registration sweeps** (all charged trials, specs in\n`config/sweeps/`): the **PIT re-test validated the whole graph family** — twohop t=4.4,\nKatz t=4.25, trade-gap-21d t=4.1, hub t=4.0, bloc t=3.2, and the banking gap at IC 0.034 /\nt=4.5 on the 17 stable-coverage countries since 2008 (current-weight vs PIT feature\ncorrelation 0.988: the v1 result was structure, not lookahead). New families:\nfundamental twins IC 0.028 / t=5.6; lead-lag 5d gap IC 0.057 / t=8.5 on the 16 structural\nfollower countries (1d version t=6.4 — both heavily timezone-channel, costs bite at this\nturnover, DSR strongly negative everywhere). The **monthly combiner tested DEAD** (IC 0.017,\nt=1.1 — month-end sampling throws away the days-to-weeks horizon where the components live);\nthe **daily combiner is the strongest signal in the ledger: IC 0.057, NW-t 10.7** on the 29\nfully-covered countries since 2006 (verdict WEAK only because the deflated Sharpe charges the\nwhole family's trial count, and because the component list was itself selected in-sample this\nmonth — the honest read is \"ceiling, verify forward\"). Conditioned event studies: the\ndowngrade drift is an **EM phenomenon** (EM −0.9%@5d t=−2.0, −2.8%@63d; DM flat) and\nsplits by regime (high-VIX downgrades hit immediately, low-VIX drift slowly).\n\n### The cost / holding-period model (harness v2.1, 2026-06-12)\n\nThe answer to \"do any daily signals survive costs?\". `evaluate_signal.py` v2.1 adds, for every\ndaily run: a **hold-period grid** (the same daily ranks re-costed at 1d / 5d / 21d tranched\nholds, `hold_period_grid` in the result JSON), a **breakeven cost** (`breakeven_cost_bps_ls` =\nthe one-way bps at which mean net LS return crosses zero given the strategy's own turnover),\nand a **5 bps cost case** (liquid-futures / DM-ETF execution). Verdict gates are unchanged —\nstill keyed to the registered hold at net-25bps; the grid is a design diagnostic. All 29\nverdicted daily hypotheses were re-measured in place (same hypothesis IDs, zero new trials).\n\n**The quantified answer** (`Data/loop/harness_runs/cost_model_summary_2026_06_12.xlsx`):\nnothing survives 25 bps one-way — that conclusion stands. But at **10 bps**, 4 signals clear\nnet LS Sharpe > 0.3: the daily ridge combiner (breakeven 14.2 bps, net Sharpe 0.99 @10bps /\n2.15 @5bps at a 1-day hold), the PIT banking-claims neighbor gap (13.1 bps breakeven), and the\nfundamental-twins + bank-gap 63d variants (~14 bps). At **5 bps** twelve signals clear the bar.\nTwo structural findings: the strong daily signals **decay fast** — gross Sharpe falls faster\nwith longer holds than turnover savings compensate, so 1-day holds win net for the top family\n(slower 21d holds only rescue slow signals like SOV_2S10S, breakeven 15.8 bps); and the\nimplementation channel decides everything — these are futures/cheap-DM-ETF strategies, not\nEM-ETF strategies. The graph family's economics: breakevens cluster at 8-14 bps, i.e. real but\nthin edges that only an efficient execution stack can monetize.\n\n**Known caveats:** v1 `graph_features_daily` still uses *current* Neo4j edge weights (kept for\ncontinuity; the PIT table is the analytical surface); the headline DSR for daily signals is\nstill computed at net-25bps on the registered hold — conservative by construction (see the\nhold-period grid for the 5/10 bps economics); GDELT DOC API rate-limits aggressively — `country_news` fails loudly and recovers\nwhen the block lifts; D10 peg-currency rows (Hong Kong, Saudi Arabia) carry a peg note because\nz-scores off a near-zero vol baseline run hot — read them as peg-risk repricing, not magnitude.\nBloomberg quota usage for the loop's nightly pulls is logged append-only to\n`Data/work/loop/bbg_quota_log.csv`.\n\n---\n\n## Configuration & dependencies\n\n- **Country mapping:** `config/country_mapping.json` (34-country ISO/OECD/BIS/WB/EPU/GPR codes).\n- **Event registry:** `config/event_log_seed.yaml`.\n- **API keys** (env or `/Users/arjundivecha/Dropbox/AAA Backup/.env.txt`): `FRED_API_KEY`, `EIA_API_KEY`. Missing keys → that source skips gracefully.\n- **Python:** `pandas numpy requests openpyxl pyarrow wbgapi sdmx1 xlrd duckdb neo4j scikit-learn mcp[cli] cvxpy` (see `requirements.txt`).\n- **Bloomberg:** separate OpusBloomberg conda env (`/Users/arjundivecha/Dropbox/AAA Backup/A Working/OpusBloomberg/.venv`, has `blpapi`). Connection: macOS → TCP 8194 → Parallels Windows VM → bbcomm → Terminal (VM IP auto-detected).\n\n---\n\n## Architecture notes\n- IMF: SDMX 3.0 REST at `api.imf.org` (no key); WEO carries forecasts to 2031; IMTS bilateral trade + BIS LBS banking feed the graph edges.\n- BIS via SDMX (`data.bis.org`); OECD via `sdmx.oecd.org`; ECB FX via per-currency SDMX; ND-GAIN/UNDP/FAOSTAT via bulk download.\n- Bloomberg: both the econ pull (`collect_bloomberg.py`) and the T2 pull (`collect_t2_bloomberg.py`) run live `blpapi`. The T2 pull batches by field (Excel-BDH-style) and supports a `--daily` calendar-day mode (matches Excel `BDH(...Per=D)` via `calendar_fill`). Derived signals (MIPD from CDS, Z-spread, curve slope) computed in-collector.\n- GDELT: ingested from `data.gdeltproject.org` GKG v2 into in-repo panels + monthly/daily workbooks; the deep theme/GCAM layer is retired in favor of the salient signal set.\n- Commodities: global Pink Sheet series; the country-broadcast (`wb_commodity_factor_panel`) is deprecated in favor of the global `commodity_panel`.\n- Embeddings: PCA-compressed z-scored factor values on `Country` nodes with a cosine vector index.\n\n---\n\n## Skipped sources\n| Source | Why skipped | Covered by |\n|---|---|---|\n| UN Comtrade | API registration broken | IMF IMTS bilateral trade, IMF ITG |\n| ACLED | requires API key + registration | GPR index, GDELT signals |\n| IPU Parline | 403 Forbidden | OFAC, WGI governance |\n| WITS | 403 Forbidden | IMF trade, WB trade openness, BIS REER |\n\n---\n\n## Reference docs\n- `docs/factor_reference.md` — auto-generated, every table/source/variable + the full graph map.\n- `docs/DATABASE_AUDIT_2026_06_09.md` — latest full warehouse audit.\n- `docs/DAILY_PIPELINE_REPORT_2026_06_09.md` — daily pipeline build + canonical validation.\n- `CLAUDE.md` — agent/dev guidance and conventions.\n",
  "bytes": 38049,
  "sha": "e7f82830e7096de600a5877bb6fb540e5c5ecf4028aee262df377b89d3c487cc",
  "repo_slug": "arjundivecha/asado",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_arjundivecha_asado_openwiki_index_md_c5472882/readme"
}