Aaron Helwig

Software engineer building AI products
Personal projects

Watchthrough

Video an agent can skim or step through

Contents
  1. Read before looking
  2. Start broad, then look closely
  3. Keep what the investigation discovered

I built Watchthrough because an agent needs different views of a video at different moments. A transcript helps it follow the argument. A few frames establish the scene. A transition, cursor movement or animation may need every frame in a short interval.

Read before looking

Most questions start with words, so the transcript comes first. Before decoding a single full frame, the agent can also ask where the picture actually changes: one low-resolution pass over the whole video returns ranked candidates. They are hints for where to look, not observations, and they often point at moments nobody is talking about.

watchthrough inspect plunge events
  1. [Music throughout]
  2. Have you ever wondered what it would look like to fly into a black hole?
  3. So, what is happening here?
  4. This is a simulation of a flight into a supermassive black hole surrounded by a hot, glowing disk of gas.
  5. The thin inner circle is called the photon ring.
  6. This oval, centered on the camera’s direction of travel, shows the entire simulated sky.
  7. Camera position and orientation relative to the black hole.
  8. For the camera, time slows compared to that of a faraway observer.
  9. The camera is aiming at the photon ring’s far side.
  10. The camera’s speed causes light sources directly ahead to brighten greatly.
  11. The camera begins its 10-minute plunge to the event horizon.
  12. The camera hits the event horizon … now.
speech
picture changes
Real Watchthrough output for NASA Goddard’s black hole plunge visualization: its transcript, then the speech cues and twenty ranked visual-change candidates on one timeline.

Start broad, then look closely

Watchthrough joins speech with timestamped frames and contact sheets. The agent can ask for an overview, a precise moment or a short range, then increase the detail only where the question requires it. Frame timestamps and speech timing remain distinct, so a nearby caption is not mistaken for an exact description of that frame.

When exactly does the camera cross the event horizon?

watchthrough inspect plunge 02:09.6..02:10.1 --every 1f

02:09.830 is the first frame showing “NOW.”, the same moment the narrator says it.

Unaltered Watchthrough output on “Simulation’s Plunge Into a Black Hole: Explained” by NASA Goddard: a whole-video overview, a minute-long range, then every frame of half a second.

The implementation seeks to the requested region instead of decoding the entire video for every question. It reuses inspected frames when only the contact-sheet layout changes. Transcription can run locally, and visual inspection can start before speech processing is needed.

Keep what the investigation discovered

A useful inspection should survive the temporary files around it. I added a durable library for source notes, transcripts and selected visual evidence, with source identity recorded alongside them. An agent can return to the evidence behind an earlier finding rather than rely on a detached summary.

I use Watchthrough every day. It is especially useful for questions about interfaces in motion, where a screenshot leaves out the thing you are trying to understand.

Personal projects