audient

# audient An experimental **audio perception layer for agents**: continuous-listening, event-gated, multi-layer classification, and dynamic memory. LLMs handle text natively. They handle audio when it's speech, via transcription or audio-native models. They don't hear the rest — a bird singing outside while you read, a glass breaking two rooms away, a smoke-alarm ringing. audient gives an agent ears to detect and classify audio events using specialized models and reasoning. Under the hood it's an **MCP server** (stdio + HTTP at `/mcp`) plus a **FastAPI REST + WS + SSE adapter** on `127.0.0.1:8089`. Drive it from an MCP-capable agent (Claude Desktop, `fast-agent`), access the knowledge through the HTTP/SSE endpoints, or run the bundled [`frontend/`](frontend/) UI to watch the loop in real time and close it in a browser.  For the story behind the design, the biology parallel, and a walkthrough of the growth loop with screenshots: [Let the AI Out: audient — an ambient audio perception layer that grows](https://es617.dev/2026/07/07/audient-ears/). For a 90-second demo of the memory going from empty → auto-recognizing in real time: [demo on YouTube](https://www.youtube.com/watch?v=6ofyj-2gQ8c). --- ## What audio actually needs An agent that only handles audio-as-speech misses most of what audio carries. Useful things it can't reach today: - **Ambient awareness** — what's happening around the agent. Music in another room. Footsteps in the hall. The HVAC kicking on. - **Event detection** — alarms, doorbells, a phone ringing two floors down, glass breaking, a baby crying. - **Voice beyond the words** — stress in a voice on a call, tone shifts, who's actually speaking, the background of a meeting. - **Anomaly detection** — equipment that just changed pitch. A pet that sounds different. Construction next door that wasn't there yesterday. There ARE specialized models for each of these. The problem isn't models, it's orchestration: today you decide which one to run on which clip and stitch the answers together yourself. The agent isn't in the loop. What this project aims at: an agent with the audio channel always on. It decides what's worth attending to, recognizes what it's heard before, escalates the rest to whatever specialist fits, and learns the things it didn't know. The longer view: an agent that can deploy new specialists on its own — and eventually train them. ## Four stages ``` continuous audio │ ▼ ┌─────────────────────────────────────┐ │ Gate │ │ energy / novelty / VAD │ └──────────────┬──────────────────────┘ ▼ ┌─────────────────────────────────────┐ │ Fingerprint │ │ CLAP embedding + symbolic features │ └──────────────┬──────────────────────┘ ▼ ┌─────────────────────────────────────┐ │ Memory (top-k cosine) │ └──┬──────────────────────────────┬───┘ │ strong match │ weak / miss │ ▼ │ ┌─────────────────────────────┐ │ │ Agent │ │ │ AST coarse (+ spectral) │ │ │ → Specialist if AST hints │ │ │ at a known domain │ │ │ (e.g. Whisper, BirdNET) │ │ └─┬───────────────────────┬───┘ │ │ recognized │ nothing fits ▼ ▼ ▼ ┌──────────────────────────┐ ┌──────────────┐ │ Record event │ │ Unhandled │ │ (extends memory) │ └──────┬───────┘ └──────────────────────────┘ │ labeled ▼ (Memory) ``` The pipeline coarsely mirrors the way human hearing works. 1. **Gate** — `detect_events`, `vad`, energy/novelty/spectral segmentation. Trims the continuous signal to candidate regions. Most of the audio never reaches the rest of the system. *(Hearing: cochlea + brainstem amplify onsets and transients, attenuate steady background.)* 2. **Fingerprint** — each candidate gets a CLAP embedding (dense, comparable) plus a panel of symbolic features (spectral centroid, peaks with prominence, harmonics, noise floor, mains-hum, clipping, temporal stability). Embedding is for retrieval; symbolic features are what the agent reads and reasons about in plain English. *(Hearing: primary auditory cortex turns the signal into a representation downstream areas can use.)* 3. **Recognize** — sqlite-vec top-k cosine over learned concepts. Strong match against a calibrated concept → labeled server-side. On a miss, the agent reaches for AST (527-label AudioSet classifier) as broad routing, sometimes alongside spectral features for disambiguation. AST is coarse — the agent chains a narrow specialist (e.g. Whisper for speech, BirdNET for bird vocs; the set is arbitrary) when AST hints at a known domain. *(Hearing: specialized regions — speech, music, environmental sound — see the fingerprint in parallel, each with its own memory; "recognize" is what lights up where.)* 4. **Unhandled** — if nothing recognizes it, the event lands in `list_unhandled` with whatever the system thinks it heard. Someone (an agent or a person) labels it once via `label_unhandled`; that label becomes a concept; next time the memory check matches. *(Hearing: you don't recognize it; you look at the source or ask. Once you have a label, you recognize it next time.)* The agent always *perceives* — every event flows out as it happens, whichever stage produced it. What varies is who's *classifying*: the server for strong memory matches (cheap, no LLM call), the agent for everything else. Stage 1 also has a **fast lane for speech.** Silero VAD runs in parallel with energy/novelty segmentation; when it fires, the region skips CLAP → memory → specialist routing and goes straight to Whisper. This treats VAD as a frozen, always-on "speech" concept — the analog of recognizing a familiar voice before parsing the words. If Whisper returns a transcript, the event is recorded as speech directly. If Whisper returns nothing usable (silence, or non-speech audio that fooled VAD — music, animal calls, sustained tones), the region falls through to the same ambient classifier path everything else takes; no audio is silently dropped. ## Three ways to reason about a sound **The agent reasons over symbolic representations, never raw audio.** Three layers are exposed for it to reason across: - **The signal.** `summarize`, `spectrum`, `stats` return plain numbers — RMS, SNR, noise floor, spectral centroid, rolloff, bandwidth, flatness, top spectral peaks with prominence in dB, harmonic ratio, zero-crossing rate, mains-hum harmonics, kurtosis, clipping, and a temporal-stability block. The agent reads these as JSON and reasons about them the way an audio engineer would. - **The embedding.** The agent never sees the CLAP vector itself. It sees the *similarity score* against everything in concept memory — `query_memory` returns the top-k matches with cosine similarity, threshold, and recognizer kind. - **The labels.** Specialists return structured labels. Whisper transcripts. BirdNET species with confidence. AST's coarse guess across 527 categories. The agent hands a sound off via `invoke_specialist` and reads back the structured answer alongside the specialist's plain-English contract for interpretation. The agent can disagree with any single layer — override an embedding similarity if the labels suggest divergent categories, override a specialist if the signal features don't fit. No single layer is authoritative. --- ## Install ### From PyPI (usual path) Requires Python 3.12 or 3.13 on macOS or Linux. The wheel bundles both the server and the browser UI, so one install gets you everything. ```bash pip install audient export ANTHROPIC_API_KEY=sk-... # only for --classifier=internal (default) audient --no-stdio ``` Then open <http://127.0.0.1:8089/> and grant microphone permission. First run downloads ~600MB of model weights from Hugging Face; expect 20–40s of warm-up on cold cache. Skip the API key by running the server without the built-in classifier (memory-first fast path still works for promoted concepts; new sounds land in the unhandled bucket for later labeling): ```bash audient --no-stdio --classifier=external ``` ### From source (dev workflow) Clone + `make demo` for the two-process dev setup with Vite hot reload and the server side-by-side. ```bash git clone https://github.com/es617/audient cd audient (cd server && uv sync) (cd frontend && pnpm install) export ANTHROPIC_API_KEY=sk-... make demo ``` Server on `:8089`, Vite dev on `:5173`. Open <http://localhost:5173/> for the hot-reloading UI. For server-only smoke tests, headless mic mode, or the run-mode matrix, see [`server/README.md`](server/README.md#run-modes); for the per-surface frontend walkthrough, see [`frontend/README.md`](frontend/README.md). ## Driving audient from Claude Code (or any stdio MCP client) The default entry point (`uv run audient`) speaks stdio MCP, which is what Claude Code and most MCP hosts consume. Add it to your client config: ```json { "mcpServers": { "audient": { "command": "uv", "args": ["--directory", "/absolute/path/to/audient/server", "run", "audient"] } } } ``` The agent gets the full MCP tool surface — `load_audio`, `summarize`, `spectrum`, `detect_events`, `vad`, `invoke_specialist`, `query_memory`, `record_event`, `teach`, `list_unhandled`, `label_unhandled`, etc. Tool calls land server-side and modify the SQLite concept memory. A couple of caveats worth knowing: - **Claude Code does not surface MCP server notifications.** The protocol supports `ResourceUpdatedNotification` (and audient fires them on new events / candidates), but Claude Code's UI doesn't render them as in-conversation messages. To see "what happened since I last asked," have the agent call `list_events` or `list_stream_candidates`. - **The bundled frontend still works in parallel.** The FastAPI / SSE adapter on `127.0.0.1:8089` is part of the same process. So anything Claude Code writes (a new event, a corrected label, a taught concept) shows up live in the frontend timeline the moment it lands. The two clients drive the same server; only Claude Code is blind to live push. ## Known limitations audient is experimental. It's been lightly evaluated on recorded data during development; no formal benchmark suite or public-dataset numbers yet. Other limitations: - **Single-source assumption.** The pipeline treats each detected region as one event. Overlapping sounds — a bird call while music plays, a doorbell mid-conversation — are classified as whichever signal dominates the embedding or wins the VAD fast lane. The non-dominant event is lost. Source separation isn't wired up. - **macOS-first.** Everything has been tested on macOS. The dependencies (torch, transformers, faster-whisper, sounddevice/portaudio) all have Linux wheels, so Linux should work with minimal fuss; I just haven't verified. Windows needs a Windows branch added to the BirdNET dep gate in `pyproject.toml` and probably a look at the mic-source path; nothing about the models themselves is Windows-incompatible. The WebSocket `/ingest` endpoint and the bundled frontend's browser-mic capture are platform-agnostic, so a Windows user can drive audient through a browser today — the backend side is the friction. - **Classifier prompt tuned for Anthropic.** The in-process classifier runs through [fast-agent](https://github.com/Evalstate/fast-agent), so any supported provider (Anthropic, OpenAI, Gemini, Ollama-local) will execute, but the prompt was written and validated against Claude models. Switching providers may need prompt adjustments. ## Acknowledgements audient stands on a lot of other people's work. The load-bearing pieces: - **MCP + agent infra** — [FastMCP](https://github.com/jlowin/fastmcp) (Jeremiah Lowin), [fast-agent](https://github.com/Evalstate/fast-agent) (Shaun Smith / Evalstate). - **Models** — [LAION-CLAP](https://github.com/LAION-AI/CLAP) for audio embeddings; [AST](https://github.com/YuanGongND/ast) (Yuan Gong et al) as the AudioSet coarse classifier; [Whisper](https://github.com/openai/whisper) (OpenAI) via [faster-whisper](https://github.com/SYSTRAN/faster-whisper); [BirdNET](https://github.com/kahst/birdnetlib) (Cornell Lab of Ornithology) via birdnetlib; [Silero VAD](https://github.com/snakers4/silero-vad). - **Model hosting** — [Hugging Face](https://huggingface.co/) + `transformers` for CLAP loading. - **Storage** — [sqlite-vec](https://github.com/asg017/sqlite-vec) (Alex Garcia). - **Audio infra** — librosa, soundfile, sounddevice. - **UI** ([`frontend/`](frontend/)) — React, Vite, Tailwind, TanStack Query, shadcn/ui, Zustand. ## License Apache License 2.0 — see [LICENSE](LICENSE).