Sub-moments: agent vs lookalike

London film · local page, built 2026-09-26 13:52 · evidence only — nothing here picks an answer.

What this asks

Big moments are split into sub-moments so that when a moment gets 20 seconds, the seconds land on different things. Today an agent makes them.
Two ideas were raised: (1) use the lookalike engine inside each moment to make or inform sub-moments; (2) a separate "visual themes" bucket across the trip. Only idea 1 is tested here.
Four moments: three big walks and Borough Market as a small case.

The two engines

Default engine (what made the left column). An agent (Opus 5.5) reads the day: capture time, GPS site, a description of every clip, and contact sheets of the pictures. It groups moments and splits big ones (> ~15 videos or ~45 min) into sub-moments, over videos. It writes a name for each sub-moment — no "why" for sub-moments (the moment has one).
Lookalike engine (right column). Pictures only: SigLIP2 google/siglip2-base-patch16-224 vectors of each clip's keyframes. No clock, no GPS, no descriptions. Each item = mean of its keyframe vectors (a still also pools its Live Photo clip), mean-centred, unit length; each item linked to its 15 most similar; Louvain community detection cuts that graph. It writes no text for its groups.
The trip-wide lookalike run (lookalike_groups.json, build b2): resolution 1.0479 tuned toward 15 moments, k 15, seed 0, 15 groups, 1311 of 1312 items grouped (1 with no vector). Its own caution: "LOOKS ONLY (SigLIP2 keyframes; no time, GPS, text or voice), Louvain not Leiden. A proposal to compare against moments.json, never a moment list."
This experiment calls the engine's own functions (item_vectors, similarity_graph, groups, tune from lookalike.py, corpus from daysheet.py) — nothing copied — but feeds them one moment's items only. So the centring is on that moment's own average. k = 15 and seed = 0 (the engine's constants). Two runs per view:
- Tuned: resolution searched (engine's tune()) until the group count equals the agent's sub-moment count. Matched on purpose, so the two sides split into the same number of pieces.
- Natural: the engine's untuned default, resolution 1.0 — however many groups that gives.

How to read it

Each moment has two switches: View (videos only — what the agent split — or videos + photos) and Lookalike run (tuned or natural). The label next to them always says which view you are looking at.
Left = the agent's sub-moments, right = lookalike groups. Thumbnails in time order, time under each. The coloured number on a thumbnail is its place on the other side (left thumbs show their lookalike group, right thumbs their sub-moment).
In the photos view, a photo's sub-moment is the agent window its time falls in (the agent only listed videos). Photos have a dashed edge.
Strip: every item in time order, equal width. Top row coloured by sub-moment, bottom row by lookalike group. Colours on the two rows are independent — look at where colour changes, not at which colour.
ARI (adjusted Rand index): how alike two groupings are. 1 = identical, 0 = no more alike than chance.
Contiguous = the group is one unbroken stretch of the moment's items in time order. k runs = it is split into k separate stretches.
Things to look for: does each side put different things in different pieces? Does lookalike merge the same view seen at two different times? Does it split one continuous scene?

Rotation

8 videos in these moments were sideways: IMG_8459, IMG_8487, IMG_8751, IMG_8757, IMG_8763, IMG_8775, IMG_8797, IMG_9457. The file's rotation flag (and orient.json) says portrait, but the scene is landscape.
The existing lookalike and moment sheets show them sideways — checked by eye: Photos' poster, the sheet's extra video frames and the store keyframes all agree with the wrong flag. So "rotate by orient.json" would not fix them; here they are turned by an eye-checked list (scratch/submoment-exp/rot_fix.json). Photos (580 across the four moments) were all upright.
The same sideways keyframes are what the SigLIP vectors were computed from, for these clips.
Elsewhere in the trip (videos with a rotation flag only, not checked in full): IMG_8690, IMG_9076, IMG_5610 also look sideways on the existing sheets.

All four at a glance

moment	videos	agent sub-moments	videos only				videos + photos
moment	videos	agent sub-moments	ARI tuned	lookalike contiguous	ARI natural	lookalike contiguous	ARI tuned	lookalike contiguous	ARI natural	lookalike contiguous
guided-walk-st-pauls-to-palace	79	12	0.39	2 / 12	0.38	1 / 5	0.22 (videos 0.30)	0 / 12	0.20	0 / 8
arrival-walk-to-tower-bridge	54	10	0.32	2 / 10	0.34	0 / 4	0.25 (videos 0.30)	0 / 10	0.21	0 / 7
afternoon-wander-seven-dials	41	6	0.08	0 / 6	0.08	0 / 3	0.19 (videos 0.06)	0 / 6	0.15	0 / 5
borough-market	25	4	0.02	0 / 4	0.13	1 / 3	0.01 (videos 0.05)	0 / 4	0.01	0 / 4

Every agent sub-moment is contiguous in both views (by construction: it is a time window).