Watchthrough

I built Watchthrough because an agent needs different views of a video at different moments. A transcript helps it follow the argument. A few frames establish the scene. A transition, cursor movement or animation may need every frame in a short interval.

Start broad, then look closely

Watchthrough joins speech with timestamped frames and contact sheets. The agent can ask for an overview, a precise moment or a short range, then increase the detail only where the question requires it. Frame timestamps and speech timing remain distinct, so a nearby caption is not mistaken for an exact description of that frame.

watchthrough prepare demo.mp4 --out demo-study --defer-transcript
watchthrough inspect demo-study overview
watchthrough inspect demo-study 00:10..00:11 --every 1f

The implementation seeks to the requested region instead of decoding the entire video for every question. It reuses inspected frames when only the contact-sheet layout changes. Transcription can run locally, and visual inspection can start before speech processing is needed.

Example

watchthrough inspect ANALYSIS 03:18..03:22 --every 800ms --cells 6 --width 1280 --sheet-format png

Watchthrough contact sheet with six weather-model video frames, each paired with its frame timestamp and a separately timed speech excerpt.

Six frames from 03:18-03:22 of WeatherNext 3, Google DeepMind. Unaltered Watchthrough output, with separate frame and speech times.

Keep what the investigation discovered

A useful inspection should survive the temporary files around it. I added a durable library for source notes, transcripts and selected visual evidence, with source identity recorded alongside them. An agent can return to the evidence behind an earlier finding rather than rely on a detached summary.

I use Watchthrough every day. It is especially useful for questions about interfaces in motion, where a screenshot leaves out the thing you are trying to understand.

View on GitHub