A real-time collaborative whiteboard where multiple people sketch together with a built-in AI chat sidebar that can "see" the canvas and respond to it. Instead of a separate summarization step, the AI
You draw/type on the tldraw canvas as normal.
The app listens directly to tldraw's document store (editor.store.listen) instead of polling on a clock — so it reacts the moment something changes.
It waits for a short pause (400ms, DEBOUNCE_MS in AutoSummarizer.jsx) after the last edit before acting. A freehand stroke updates the store continuously while you drag, so this debounce is what stops it from firing mid-stroke — it fires once, right after you lift the pen, which in practice feels closer to instant than a fixed interval ever could.
The canvas is exported to a PNG in-browser (editor.toImage) and sent to the Gemini API with a prompt that asks for { confident, summary } JSON.
If Gemini isn't confident yet (board's empty, mid-stroke, or the marks are still too ambiguous to mean anything), the sidebar keeps showing whatever summary is already there — it only overwrites the summary once Gemini is confident. A status indicator ("Reading board" / "Still watching" / "Idle" / "Error") shows what's happening without disturbing the text itself.
This intentionally skips any custom shape/handwriting recognition — the vision model reads the board directly, which is both less code and far more robust for a hackathon timeframe than hand-rolled pattern matching.