day-summ — a day labelled by camera, heart rate and a local LLM
The laptop camera labels minutes as focus, phone or fatigue, a Whoop strap streams heart rate over BLE, and a local model writes the evening verdict. Fully offline: frames are never stored, the database holds only per-minute numbers.
Whoop shows heart rate, sleep and recovery but says nothing about what I was actually doing in those minutes. Screen-time trackers know the app but not whether I was looking at the screen at all. I wanted one picture of the day where physiology and behaviour sit on the same time axis — without handing anything to a cloud.
The most useful signal came not from the camera but from its absence. No face in frame while BLE heart rate keeps ticking means I am still nearby — a short break. Heart rate gone too means I left the room. A single `hr_avg is not None` check replaced a whole class of heuristics. Also: MediaPipe hands you blendshapes (blink, jaw open, gaze down) as ready numbers, so PERCLOS and blink rate take a couple of lines and no model training.
For battery reasons the camera shoots in bursts: ~10 seconds every 5 minutes. A twenty-minute absence turned into four one-minute specks on the timeline, because only minutes with actual frames were labelled. The fix was sample-and-hold: each burst holds its category until the next one, but capped — otherwise a closed laptop lid would smear across the whole evening.
One day on one axis
I already had body data: Whoop honestly logs heart rate, sleep and recovery. What it never answers is "and what was I doing at that moment". Screen-time trackers answer the exact opposite: they know the app, but not whether I was looking at the screen or drifting into my phone next to the laptop.
day-summ merges both streams into one evening report: a minute-by-minute timeline, fatigue windows, heart rate in focus versus heart rate on the phone — plus a written verdict produced by a model running on the same laptop.
How a minute gets its label
Frames are not analysed one by one: everything collapses into a per-minute record (presence ratio, gaze-on-screen ratio, phone-in-frame ratio, blinks, PERCLOS, average heart rate), and then the minute gets exactly one category from a pure function, no magic:
def minute_category(rec, cfg, workout_spans):
if not rec.data_ok:
return "nodata"
if overlaps_workout(rec, workout_spans):
return "workout"
if rec.presence_ratio < cfg.break_presence_max:
# no face: heart rate still coming → nearby, gone → left the desk
return "break" if rec.hr_avg is not None else "left"
if rec.phone_detected_ratio > cfg.phone_minute_ratio:
return "phone"
if (rec.presence_ratio >= cfg.focus_presence
and rec.gaze_on_screen_ratio >= cfg.focus_gaze
and rec.phone_detected_ratio < cfg.focus_phone_max):
return "focus"
if rec.gaze_on_screen_ratio < cfg.away_gaze_max:
return "away"
return "idle"Face and gaze come from MediaPipe Face Landmarker: it returns blendshapes (eyeBlinkLeft, jawOpen, eyeLookDown, browDown) and a head transformation matrix that yields pitch and yaw. The same source gives blinks per minute and PERCLOS — the share of time with closed eyes, a classic drowsiness metric.
A phone in frame is caught by YOLO11-nano via the COCO class cell phone, and only on every tenth frame, because the detector is heavier than the face model.
Break or gone — heart rate decides
The nicest find in the project. The camera sees the same thing — an empty frame — in two very different situations: I leaned back in the chair to think, or I left the room. BLE tells them apart: with Broadcast Heart Rate enabled, the Whoop strap advertises the standard Heart Rate Service, and while I am within a couple of metres the beats keep arriving. Heart rate gone means I left, not that I turned away.
Battery: bursts instead of continuous capture
Keeping the camera on all day means a warm laptop and a few hours of battery gone. So by default it wakes for 10 seconds every 5 minutes (sample_duration_sec / sample_interval_sec in the config), while BLE heart rate stays connected the whole time — it costs almost nothing. On macOS the lid is checked through ioreg (AppleClamshellState): closed lid, no camera burst at all.
The model writes the text, the code computes the numbers
The verdict on top of the report is generated by qwen3:8b through a local ollama. The principle matters: the model does not compute, it narrates. Every fact (focus minutes, fatigue windows, heart rate per category, recovery and sleep from the Whoop API) is calculated in code and handed over as ready JSON, and the prompt explicitly says "do not recompute these".
If ollama isn't running the report doesn't break: it falls back to a template summary built from f-strings. qwen3 also has a reasoning mode — the <think>…</think> block is stripped with a regex so only the clean text reaches the report.
uv run daysumm calibrate # once: thresholds tuned to your desk
uv run daysumm start # morning: collection + live dashboard on localhost:9683
uv run daysumm report # evening: build and open the HTML reportPrivacy as architecture, not as a promise
You can't build this kind of thing "almost privately". Either frames are never stored, or one day you discover a folder holding six months of your own face. That's why there is no code path where a frame reaches the disk.