AgentInternals

# AgentInternals Live, turn-by-turn visualizer for **GitHub Copilot CLI** sessions. Helps developers understand the impact of prompt, context, skill, and MCP-tool changes on their inner-loop iterations. A small Fastify backend tails the active CLI's log file (and optionally receives webhook events), normalizes them into turn / tool-call / token aggregates, and pushes everything to a React UI in real time over a WebSocket.  --- ## Capabilities at a glance - **Auto-attach to a running Copilot CLI session.** When a single live session is detected the UI attaches automatically. When multiple sessions are running, the picker surfaces all of them with cwd / pid / live-vs-historical badges so you choose which one to follow. - **Load and analyze any past session.** Every Copilot CLI session left on disk (under `~/.copilot/`) is discoverable from the picker and can be opened read-only — full timeline, turn details, anomalies, and flow graph, exactly as if it were live. - **Built-in turn analyzers / anomaly detection.** Each turn is scanned for behaviours worth noting and tagged with one or more flags — `fail`, `retry`, `loop`, `fanout`, `compaction-loss`, `thrash` — visible on the timeline, the turn detail, and the flow graph. Hovering an anomaly tells you *why* it was flagged (extracted error text, the earlier failure that triggered the retry, the sibling fan-out tools, the roundtrip count, etc.). - **Compare two turns within a session.** From the turn timeline click **Compare turns…** to diff every metric (tokens in / out, premium requests, tool counts, durations, errors) between any two turns. Δ values are colored by direction and shown as both an absolute number and a percent change. Invaluable when you're tuning a prompt, a skill, or an MCP tool to push down token usage, premium-request count, or roundtrip count and want fast, quantified before/after evidence. - **Compare two whole sessions.** The **Compare** view in the top "View" menu pits any two sessions head-to-head — side-by-side aggregates with per-metric Δ, plus a per-tool change table that highlights tools that appeared / disappeared between the two runs. - **Cumulative insights across all your local sessions.** The **Insights** view rolls up the most recent N sessions on the machine into a summary strip + bar charts (tools/turn, total turns, input tokens, premium reqs) + a sortable session table — useful for spotting trends as you iterate. - **Whole-session flow graph.** Renders the agent's trajectory as a directed graph (`prompt → assistant → tool → assistant → …`), with anomaly-flag chips that filter which turns are visible and a one-click "Open in live →" jump-back to the matching turn. Per-turn flow graphs are also embedded in the turn detail with a synced tool-call sidebar. - **Live in-progress feedback.** A sticky banner shows the current turn's elapsed time, tool count, and token count. With the optional CLI hook bridge installed (one click from the UI), pulsing chips show **each tool that is currently running** before it finishes. - **One-click CLI hook install.** Writes a single `.github/hooks/agentinternals.json` bridge file per repo so Copilot's `sessionStart` / `userPromptSubmitted` / `pre|postToolUse` / `errorOccurred` events flow into AgentInternals. Cleanly uninstalls and restores any backed-up file. The bridge has a 2-second timeout so the CLI is never blocked when AgentInternals isn't running. - **Dark and light themes.** Toggle from the ⋯ menu in the connection bar. The flow graph, mini-map, controls, and chip palettes all have explicit overrides for both themes. Theme choice is persisted. - **Privacy-aware.** When the CLI's redaction has stripped the prompt or assistant text, the UI shows an inline callout instead of guessing — and falls back to the per-session `events.jsonl` to recover the full text when available. - **Filterable timeline.** Free-text search across prompts / responses / tool args, plus a filter popover for tool name and "errors only." Tool chips are colored by category (file / shell / search / web / agent / vcs / ide / other). - **Skill invocation tracking.** Each turn surfaces which built-in skills were loaded and invoked, with a tooltip breakdown by source (personal / project / plugin / custom). - **Resizable layout & persisted preferences.** Drag the divider between timeline and detail. View choice, pin-to-latest, theme, and split-ratio all survive reload. - **Keyboard navigation.** `j` / `↓` next turn, `k` / `↑` previous, `g` first, `G` last, `Esc` clear (ignored while typing in inputs). - **Toast notifications.** Attach / detach / reconnect / errors surface in the corner so nothing important is silently dropped. See `plan.md` for the full design. This repo currently implements **Phase 0** (log-tail spike), **Phase 1** (collector backend), **Phase 2** (Mode A standalone UI, with all polish stages), **Phase 3** (hook auto-install), **Phase 4** (compare & insights), and a **Flow Graph** visualization built on top of the per-session `events.jsonl` stream. --- ## Requirements - **Node.js >= 22** (uses the built-in `node:sqlite` module) - Windows / macOS / Linux ## Quickstart ```powershell $env:NODE_NO_WARNINGS = "1" # silence node:sqlite experimental warning npm install npm run build:ui # build the React UI into packages/ui/dist # Start the collector + UI on http://127.0.0.1:4174 npm run dev ``` Open the URL it prints. If exactly one Copilot CLI session is live, it auto-attaches; otherwise enable **Auto-attach** in the UI and start (or pick) a CLI session. ### Other entry points ```powershell # Phase 0 spike: tail the active CLI log and print every event to the console npx tsx packages/cli/src/spike.ts --list # list discovered live sessions npx tsx packages/cli/src/spike.ts --from-start # replay a session from start # Vite dev server for the UI (proxies API/WS to the collector at :4174) npm run dev:ui ``` --- ## What the UI shows - **App header** — *"AgentInternals — Your Window to inner workings of coding agents."* - **Connection bar** — status badge with countdown when reconnecting; **view tabs** (Live / Compare / **Flow** / Insights); Attach picker (with **Attach (full session)** option to load history from turn 0), Detach, **Show full session** button to reload history without detaching, Auto-attach toggle, hooks installer, theme toggle. - **Live view** — session header + in-progress banner + filterable turn timeline + turn detail (the original real-time experience). - **Compare view (sessions)** — pick any two indexed sessions; shows side-by-side aggregates, per-aggregate Δ (B − A) colored by direction, and a per-tool change table (delta + tools-only-in-A / -in-B). - **Compare view (turns within one session)** — from any turn detail click **Diff** and pick a second turn; the comparator diffs every metric, tool counts, and tool calls between the two turns of the same session. - **Flow view** — whole-session graph of every turn, rendered with [`@xyflow/react`](https://reactflow.dev/). Anomaly-flag chips (`fail`, `retry`, `loop`, `fanout`, `compaction-loss`, `thrash`) filter which turns are visible. Each turn block has an "Open in live →" button to jump back into the Live view at that turn. See [Flow graph](#flow-graph) below. - **Insights view** — analyze the most recent N sessions in one shot. Summary strip + 4 bar charts (tools/turn, total turns, input tokens, premium reqs) + a sortable session table. All pure read. - **Session header** — session id, cwd, turn / tool / token counts, premium-request total, top tools, mini sparklines for cumulative tokens and premium requests, log file path, copy-to-clipboard buttons. - **In-progress banner** — sticky panel with live elapsed timer, tool count, token count, and (if CLI hooks are configured) pulsing chips for each tool currently running. - **Turn timeline** — every turn with prompt preview, status, tool chips colored by category (file / shell / search / web / agent / vcs / ide / other), wall-clock duration, errors badge. Filter buttons are labelled (**Files modified**, **Search**, **Shell**, …) and have hover-tip explanations. Filter by text, by tool, errors-only. Keyboard nav (also documented in a hover tooltip on the timeline header): `j` / `↓` next, `k` / `↑` prev, `g` first, `G` last, `Esc` clear; ignored while typing in inputs. - **Turn detail** — privacy callout when prompts/responses aren't logged, model I/O stats, wall vs. model time, all tool calls grouped by category. The **User Prompt** and **Agent Response** sections (full text from `events.jsonl`) sit at the top, followed by a collapsible **Flow Graph** of just this turn. Every section has a hover-tip explaining what it shows. - **Theme** — dark by default, with a one-click light theme. Flow graph nodes, controls, and minimap have explicit palettes for both themes. - **Toasts** for attach / detach / reconnect / errors. - **Persisted prefs** for view, pin-to-latest, theme, and the resizable timeline-vs-detail split. --- ## Architecture ``` packages/ shared-types/ # TS types shared between collector and UI collector/ # Log tail + DB read + normalization + Fastify REST/WS server cli/ # `spike` and `server` entry points ui/ # React + Vite + Tailwind UI (served from /) ``` ### Data sources The collector currently reads: 1. **`~/.copilot/logs/process-<startMs>-<pid>.log`** — primary live stream. Tailed with byte-offset bookmarks; rotation-aware. Only `cli.telemetry`-channel records are ingested (the CLI also re-emits everything on `cli.tool_call` / `cli.restricted_telemetry` — those are duplicates). 2. **`~/.copilot/session-store.db`** — read-only WAL-aware open via `node:sqlite`. Used to enrich the in-memory summary with the session's `cwd` (which the CLI doesn't write to the log). 3. **`~/.copilot/session-state/<sessionId>/events.jsonl`** — per-session structured event stream. Source of the **prompt / agent response text** shown in turn details and of the **Flow Graph**. Watched for changes; the UI re-fetches on every appended line. 4. **`POST /hook/:event`** — optional. If you wire up the included CLI hooks (one-click from the UI), the collector receives `preToolUse` / `postToolUse` etc. as documented at https://docs.github.com/en/copilot/reference/hooks-configuration, and the UI shows live "running tool" chips with elapsed time before the tool finishes. Session-state folder watching and full config-snapshot capture (other instructions, skills, MCP) are planned for the next phase. ### Important behaviors learned from real CLI logs - A single user prompt triggers many `assistant_turn_start`/`end` roundtrips. A user-turn closes on the **next** `user_message`, not on `assistant_turn_end`. - `tool_call_executed` can appear on multiple channels for the same call. Dedupe by `tool_call_id`. - The CLI emits `cost_usd` in **premium-request units**, not USD. We sum it as-is and label it "premium reqs" everywhere. - `cwd` is **not** in the log file; it lives in `session-store.db`. The collector seeds the in-memory summary from the DB when attaching. - There is no `tool_call_started` event. Per-tool live "running" state requires the hook receiver. --- ## Endpoints REST: - `GET /api/health` — `{ ok, attachedSessionId?, autoAttach }` - `GET /api/sessions` — all sessions discovered on disk - `GET /api/sessions/live` — sessions whose log file is being actively written by a live process - `GET /api/sessions/:id` — summary + aggregates - `GET /api/sessions/:id/turns` — full turn list - `POST /api/attach { sessionId?, fromStart? }` — attach to a specific session, or to the only live one if omitted - `POST /api/detach` — idempotent - `POST /api/auto-attach/start { intervalMs?, onMultiple?, sessionId?, fromStart? }` - `POST /api/auto-attach/stop` - `POST /hook/:event` — sink for Copilot CLI lifecycle hooks (e.g., `/hook/preToolUse`) - `GET /api/hooks/status?cwd=…` — whether the AgentInternals hook bridge is installed at `<cwd>/.github/hooks/` - `POST /api/hooks/install` — write the bridge file (body: `{ cwd?, port?, force? }`) - `POST /api/hooks/uninstall` — remove the bridge file (body: `{ cwd?, force? }`) - `GET /api/analysis/:id` — re-parse a session's log on demand and return aggregates + turns + topTools (cached by mtime) - `GET /api/compare?a=<sid>&b=<sid>` — analyze two sessions in parallel and return per-aggregate / per-tool deltas - `GET /api/insights?limit=N` — analyze the most recent N sessions and return per-session aggregates + a merged summary - `GET /api/sessions/:id/messages` — full prompt + assistant-response text for every turn, parsed from `events.jsonl` - `GET /api/sessions/:id/flow` — the per-session **FlowGraph** (`{ nodes, edges, turnCount }`) used by the Flow view and the per-turn flow embed WebSocket: `ws://127.0.0.1:4174/ws`. Messages match `LiveEvent` from `@agentinternals/shared-types`: `hello`, `session.updated`, `session.context`, `turn.started`, `turn.tool_call`, `turn.updated`, `turn.completed`, `log.raw`, `hook.received`, `running_tools.updated`, `attached`, `detached`, `auto_attach.waiting`, `auto_attach.state`. --- ## Flow graph The Flow tab and the per-turn collapsible Flow Graph in turn details visualize the agent's trajectory as a directed graph. Each event in `events.jsonl` becomes a node: | Kind | Source event | Notes | |------|--------------|-------| | `prompt` | `user.message` | Starts a new turn. | | `assistant` | `assistant.message` | One per model roundtrip; carries reasoning text + intentions. | | `tool` | `tool.execution_start` / `…_complete` paired by `toolCallId` | Status `pending` → `ok` / `fail`; result preview from `result.content` / `detailedContent`. | | `hook` | `hook.start` / `hook.end` | `decision: block / deny` is treated as a failure. | | `compaction` | `session.compaction_complete` | Tool calls still in flight at this point are flagged `compaction-loss`. | | `plan_change` | `session.plan_changed` | | | `model_change` | `session.model_change` | | | `abort` / `error` | `abort` / `session.error` | Always rendered as failures with `failReason`. | Edges: - **`next`** — chronological sequence between meaningful nodes. - **`requested_by`** — assistant.message → tool node (animated). - **`parallel`** — sibling tools requested by the same assistant message (dashed). ### Anomaly flags Every node carries `flags?: FlowFlag[]` *and* `flagDetails?: { [flag]: string }`. The detail strings are surfaced in the node's hover tooltip alongside `failReason` so you can see *why* something was flagged: | Flag | Meaning | |------|---------| | `fail` | Tool / hook / abort / error reported `success: false`. Tooltip includes the extracted error message (`errorMessage` / `error.message` / `result.error` / `result.detailedContent` / `result.content`, in that priority). | | `retry` | A later same-name tool call was issued in the same turn after a previous failure; tooltip references the earlier failure reason. | | `loop` | Two or more calls to the same tool with effectively identical arguments in one turn — likely an unproductive cycle. | | `fanout` | A single assistant.message requested more than one tool in parallel. | | `compaction-loss` | The call was still in flight when conversation history was compacted — the assistant may have lost the original request context. | | `thrash` | A turn made ≥ 8 model roundtrips. The flag is attached to the last assistant node of the turn. | ### Controls - Top-right **+ / − / ⛶** buttons zoom in / out / fit-view. - Mouse wheel + pinch-to-zoom on the canvas. - Mini-map in the bottom-right (panable, zoomable; node colour matches kind, red stroke = failure). - Each node has a multi-line hover tooltip with label, summary, failure reason, and per-flag explanation. ### Implementation notes - Extractor: `packages/collector/src/sources/flowGraph.ts`. Streams the JSONL file in one pass, builds nodes / edges / flags, and emits a `FlowGraph` matching `@agentinternals/shared-types`. - Per-turn slicing + parallel-aware layout: `packages/ui/src/lib/flowLayout.ts` (pure helpers, unit-tested). - Renderer: `packages/ui/src/components/TurnFlow.tsx` (per-turn embed) and `FlowView.tsx` (whole-session view). - When attached mid-session the EventStore starts at turn 0 but `events.jsonl` already contains earlier turns; both `messages` and `flow` use a tail-anchor (`Math.max(0, file.length - displayTurns.length)`) so visible turn N maps to the right turn in the JSONL. --- ## Wiring CLI hooks (optional) Copilot CLI hooks (see [docs](https://docs.github.com/en/copilot/reference/hooks-configuration)) are **shell commands** loaded from `<cwd>/.github/hooks/*.json`. AgentInternals ships its own bridge file that simply pipes the stdin payload to `POST /hook/<event>` via `curl` (or PowerShell's `Invoke-RestMethod`), then exits successfully so the agent isn't blocked. ### One-click install (recommended) When attached to a session, click the **Install Hooks** / **Uninstall Hooks** toggle in the connection toolbar (a single button whose label and tooltip reflect the current state). It writes (or updates) a single file: ``` <sessionCwd>/.github/hooks/agentinternals.json ``` The file is marked with an `_agentinternals` block so it can be cleanly **uninstalled** later from the same UI. If a foreign `agentinternals.json` already exists, you'll be asked to confirm before it's backed up to `agentinternals.json.bak.<timestamp>` and replaced. Uninstall restores the backup if one is present, otherwise deletes the file. The bridge wires up all six documented hook events: `sessionStart`, `sessionEnd`, `userPromptSubmitted`, `preToolUse`, `postToolUse`, `errorOccurred`. ### Manual install (if you don't want to use the UI) `POST /api/hooks/install` with body `{ "cwd": "<your-repo>", "port": 4174 }` produces the same file. `GET /api/hooks/status?cwd=...` reports the current state. `POST /api/hooks/uninstall` removes it. ### Notes - Hooks are **per-cwd**. Install once per repo you launch `copilot` in. - The bridge runs `curl -m 2` / `Invoke-RestMethod -TimeoutSec 2`, so the CLI is never blocked more than ~2 s if AgentInternals isn't running. - Other hook files in `.github/hooks/` are untouched — Copilot CLI loads all of them. --- ## Development ```powershell # Run all tests (collector + UI lib — 115 currently) $env:NODE_NO_WARNINGS = "1" npm test # Build UI npm run build:ui # Restart the dev server after backend changes; just rebuild + hard-refresh after UI changes. ``` ### Project conventions - All cross-package types live in `@agentinternals/shared-types`. - Storage is 0-based for turn indices; the UI displays 1-based. - The HTTP layer is a thin adapter on top of `Collector` so a future VS Code extension can host the collector in-process and bridge events via `postMessage` instead of WebSocket. --- ## Status | Phase | Status | |------:|--------| | 0 — Spike | ✅ | | 1 — Collector backend | ✅ | | 2 — Mode A UI | ✅ | | 3 — Hook auto-install | ✅ | | 4 — Compare & insights | ✅ (pure-read; tagging deferred) | | Flow Graph (per-turn + whole-session) | ✅ | | Turn-vs-turn diff in same session | ✅ | | 5 — VS Code extension | ⏳ | | 6 — Tauri / OTel / export | ⏳ |