{
  "markdown": "# vidgrid\n\n[![PyPI](https://img.shields.io/pypi/v/vidgrid?style=flat-square&color=1f4fd1)](https://pypi.org/project/vidgrid/)\n[![Python](https://img.shields.io/pypi/pyversions/vidgrid?style=flat-square)](https://pypi.org/project/vidgrid/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-black.svg?style=flat-square)](LICENSE)\n[![Downloads](https://img.shields.io/pypi/dm/vidgrid?style=flat-square&color=e03127)](https://pypi.org/project/vidgrid/)\n[![Hosted](https://img.shields.io/badge/hosted-vidgrid.site-e03127?style=flat-square)](https://vidgrid.site)\n\n> Convert video clips into annotated image grids for vision LLM analysis.\n> **One cell = one second, by default.**\n\n![vidgrid example — a 3×3 grid generated from \"Me at the zoo\" with numbered cells, top-right timestamps, and real auto-captions burned in](docs/img/hero.jpg)\n\nLLMs can't watch video, but they can analyze a single image. `vidgrid` samples\none frame per second from a video, tiles them into a numbered storyboard with\ntimestamps, and optionally sends the result to Claude, GPT, or Gemini with a\nprompt. The result is something close to \"my LLM just watched a video\" for\nthe cost of a handful of image uploads.\n\n**Don't want to install?** Use the hosted version at\n[vidgrid.site](https://vidgrid.site) — drop a file, get the grid in the\nbrowser. 3 free renders, $5 lifetime after that. Free for ever on the CLI.\n\n## The model\n\n**One cell = one second, by default.** The auto-picker chooses the smallest\ngrid (biggest, most-legible cells) whose board count stays under\n`--max-boards` (default 10). When that's not enough for a long clip, it\nbumps the grid up; as a last resort, it reduces the sampling rate. Override\nwith `--fps` and `--max-boards` for full control.\n\n- Grid size determines how many seconds fit in one photo\n- Default sampling is 1fps; drops below 1fps only when needed to stay under the max-boards cap\n- Videos over 5 minutes are rejected (chop them up first)\n\n| Grid | Cells | Seconds per photo | Best for |\n|---|---|---|---|\n| `2x2` | 4 | 4 | Very short clips (2–4s) |\n| `3x3` | **9** | **9** | **Default — best overall readability** |\n| `4x4` | 16 | 16 | More compact, cells get smaller |\n| `5x5` | 25 | 25 | Experimental — cells small, LLM accuracy drops |\n\n**Quality degrades with bigger grids.** Cells shrink, detail is lost, and the\nLLM has a harder time reading fine content like text or UI elements. Stick\nwith 3×3 unless you specifically need to pack more seconds into one photo.\n5×5 exists mostly as a \"let me see what happens\" option.\n\n## How many photos a video produces\n\nAt 1fps sampling, the board count at each grid size:\n\n| Video length | 2×2 | 3×3 | 4×4 | 5×5 |\n|---|---|---|---|---|\n| 3s    | **1** (partial) | 1 (partial) | 1 (partial) | 1 (partial) |\n| 9s    | **3** | 1 | 1 (partial) | 1 (partial) |\n| 25s   | **7** | 3 | 2 | 1 |\n| 60s   | 15 | **7** | 4 | 3 |\n| 186s (3 min) | 47 | 21 | **12** | 8 |\n| 300s (5 min, cap) | 75 | 34 | **19**¹ | 12 |\n\n**Bold** = what `auto` picks — the smallest grid (biggest cells) that\nkeeps the board count under `--max-boards` (default 10).\n\n¹ At the 5-min cap, even 4×4 exceeds 10 boards at 1fps, so auto drops\nthe sampling rate (≈1 cell per 1.9s) to land at the 10-board limit. Use\n`--fps 1.0 --max-boards 20` to preserve 1fps and accept more boards.\n\nMost vision LLMs accept ~10–20 images per request, so auto's default\nceiling of 10 keeps a full video inside a single model call.\n\n## Install\n\n```bash\npip install vidgrid                         # core renderer only\npip install vidgrid[transcribe]             # + faster-whisper for --transcribe\npip install vidgrid[anthropic]              # + Claude support via --ask\npip install vidgrid[llm]                    # + Claude + GPT + Gemini\npip install vidgrid[all]                    # everything\n```\n\nRequires Python 3.9+ and `ffmpeg` on your `PATH`.\n\n## Quick start\n\n```bash\n# 1. Auto-pick grid and sampling rate — smallest grid that fits in 10 boards\nvidgrid clip.mp4 -o grid.png\n\n# 2. Force a specific grid\nvidgrid clip.mp4 -o grid.png --grid 4x4\n\n# 3. Force a sampling rate — 0.5fps = 1 cell every 2 seconds\nvidgrid long-clip.mp4 -o grid.png --fps 0.5\n\n# 4. Raise the max-boards ceiling (default 10) if you want more boards\nvidgrid lecture.mp4 -o grid.png --max-boards 20\n\n# 5. Render + auto-transcribe + send to Claude in one call\nvidgrid lecture.mp4 --transcribe --ask \"bullet-point summary\"\n\n# 6. Use existing Whisper captions, burn them onto the grid\nvidgrid interview.mp4 -o grid.png --captions whisper.json --burn-captions\n\n# 7. Let the CLI fall back to python -m if the console script isn't on PATH\npython3 -m vidgrid clip.mp4 -o grid.png\n```\n\n## Three things you can do with it\n\n### 1. Summarize a talk without watching it\n\n```bash\nvidgrid \"team-meeting.mp4\" \\\n  --transcribe \\\n  --ask \"list the decisions made and who owns each\" \\\n  --model claude-opus-4-7\n```\n\nvidgrid samples one frame per second, runs Whisper on the audio, sends the\ngrid + transcript to Claude, and prints the answer. The model correlates\nframes and words via the burned-in timestamps.\n\n### 2. Find a specific moment in a screen recording\n\n```bash\nvidgrid bug-repro.mp4 --grid 3x3 \\\n  --ask \"at which numbered frame does the error dialog appear?\" \\\n  --model gpt-5\n```\n\nBecause cells are globally numbered (1, 2, 3...) and tagged with timestamps,\nthe model can point you at the exact moment. No scrubbing.\n\n### 3. Rank a pile of stock footage\n\n```bash\nfor clip in broll/*.mp4; do\n  vidgrid \"$clip\" -o \"grids/$(basename $clip .mp4).png\"\ndone\n```\n\nSend the PNGs to Claude in a single request and ask it to rank or reject\nclips against your shot list. This is the workflow vidgrid was built for.\n\n## Portrait vs landscape\n\nvidgrid keeps the grid shape square (N×N) regardless of source orientation\nand preserves the source aspect inside each cell. Landscape sources produce\nwide boards; portrait sources produce tall boards. Cells are never cropped.\n\n## Two-layer captions (default)\n\nThe default mode gives the LLM **two correlated inputs**: the rendered grid\nimage AND the Whisper transcript as separate text. The model correlates them\nvia the timestamps printed on each cell.\n\nThis beats burning captions into the image because:\n\n1. Frames keep their pixels for actual content\n2. Text is higher fidelity as tokens than as baked-in pixels\n3. The grid stays clean and shareable\n\nAdd `--burn-captions` if you want a self-contained image (useful for sharing\nor offline analysis).\n\n## Caption file formats\n\nvidgrid reads and writes three caption formats. The `--captions` flag\nauto-detects from the file extension. The `--transcript-format` flag\ncontrols what `--transcribe` writes.\n\n| Format | Extension | Size (36 words) | When to use |\n|---|---|---|---|\n| `json` | `.json` | ~4.8 KB | Remotion pipelines, tools that need word confidence |\n| `srt` | `.srt` | ~1.4 KB | Video editors, universal subtitle format |\n| `txt` | `.txt` | ~0.4 KB | Smallest, grep-friendly, trivial to parse |\n\n**JSON** (default, Remotion-compatible):\n```json\n[\n  {\"text\": \"hello\", \"startMs\": 0, \"endMs\": 500, \"timestampMs\": 0, \"confidence\": 0.98},\n  ...\n]\n```\n\n**SRT** (SubRip subtitles):\n```\n1\n00:00:00,000 --> 00:00:00,500\nhello\n\n2\n00:00:00,500 --> 00:00:01,000\nworld\n```\n\n**TXT** (plain timestamped text, one word per line):\n```\n0.00 hello\n0.50 world\n```\n\nUse any format as input, output, or both. You can mix — read an `.srt` and\nwrite a `.txt` with `--captions foo.srt --transcript-format txt`.\n\n## Python API\n\n```python\nfrom vidgrid import render\n\nstoryboard = render(\n    input_path=\"interview.mp4\",\n    output_path=\"grid.png\",\n    grid=\"3x3\",  # or \"2x2\", \"4x4\", \"5x5\", or None for auto\n    transcribe=True,\n)\n\nprint(storyboard.board_paths)        # ['grid-1.png', 'grid-2.png', ...]\nprint(storyboard.transcript_path)    # 'grid-transcript.json'\nprint(storyboard.all_samples)        # list[Sample] with timestamps\n```\n\nModules: `vidgrid.probe`, `vidgrid.sample`, `vidgrid.compose`,\n`vidgrid.captions`, `vidgrid.llm`, `vidgrid.presets`.\n\n## Output structure\n\n**Single-board run:**\n```\ngrid.png                 # the storyboard\ngrid.json                # sidecar: timestamps, layout, source info\ngrid-transcript.json     # only if --transcribe or --captions was used\n```\n\n**Multi-board run:**\n```\ngrid-1.png, grid-2.png, grid-3.png, ...\ngrid.json                # index covering all boards + global cell numbering\ngrid-transcript.json\n```\n\nCells are numbered **globally** across boards. A 3-board run has cells 1–27\nso the LLM can reference any frame without ambiguity.\n\n## Limits and caveats\n\n- **5-minute hard cap on video length.** Longer videos are rejected. Chop\n  them up with `ffmpeg -ss START -t 300 input.mp4 chunk.mp4`.\n- **No scene detection.** v1 samples strictly 1 frame per second, uniform.\n  No dedupe, no shifting — the spacing is always exactly 1 second.\n- **Variable-framerate videos** may have sub-frame seek drift (≤1 frame),\n  which is acceptable at 1fps sampling.\n- **Bigger grids hurt legibility.** A 5×5 grid has cells ~300px wide; fine\n  for people and objects, marginal for dense text or code. Stick with 3×3.\n- **LLM integration** uses the official SDKs (anthropic, openai, google-genai)\n  and won't be installed unless you request them as extras.\n\n## Prior art\n\n- [IG-VLM](https://arxiv.org/abs/2403.18406) — research paper proving the grid trick works\n- [llm-video-frames](https://github.com/simonw/llm-video-frames) — Simon Willison's per-frame approach\n- [vcsi](https://github.com/amietn/vcsi) — contact sheets without LLMs\n- [byjlw/video-analyzer](https://github.com/byjlw/video-analyzer) — whisper + sequential frames\n\nvidgrid's differentiator: **1 cell = 1 second, numbered cells, simple CLI,\nmulti-provider LLM integration in one package**.\n\n## License\n\nMIT. The bundled Source Sans 3 font is licensed under [SIL OFL 1.1](vidgrid/assets/fonts/LICENSE-SourceSans3.md).\n",
  "bytes": 9873,
  "sha": "66f39b2f9179783aa253ab2874b446cc43f848aa3e5ec1473c477c4bd2f24384",
  "repo_slug": "pawvej/vidgrid",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_pawvej_vidgrid_2024aa37/readme"
}