chat-with-audio

# Chat with Audio [](https://github.com/learningtour/chat-with-audio/actions/workflows/ci.yml) [](LICENSE)  _Part of **Agentic Production**._ Chat-driven audio enhancement: talk to your local Claude (Desktop or Claude Code) about a recording and let the toolkit analyze and improve it. A local A/B viewer lets you compare original and result in perfect sync — hearing *and* seeing. ``` Claude (chat) ── MCP (stdio) ──> Python orchestration ──> C++ DSP core (pybind11) │ gate · compressor · limiter · EQ ├─> AI denoise (DeepFilterNet, optional) ├─> analysis (LUFS, SNR, hum, clipping, spectrum) └─> sessions ──> A/B viewer (http://127.0.0.1:8471) ``` ## What can you ask? - **"Analyze this file: /path/to/recording.wav"** — metrics, scores and issues. - **"Make this sound better"** — auto-improve: the tool detects speech/music and picks a chain by itself (highpass, hum notches, noise reduction, gate, EQ, compression, loudness), with an explanation for every step. - **"Reduce the noise"** — noise reduction only (spectral gating or DeepFilterNet AI). - **"Bring the level up without clipping"** — loudness normalization (BS.1770) with a true-peak limiter. - **"Cut 3 dB around 300 Hz and compress lightly"** — an explicit chain via `apply_chain`. - **"Put the speech at −6 and balance the music"** — `refine_audio`: segments speech/music/silence and runs a measure-and-adjust loop until the targets are right to the decibel. AI denoising is applied only when the speech SNR is low AND Whisper confirms intelligibility doesn't drop; the report includes the measurement history, the decisions and a final word-retention check. - **"Fix it only where something is wrong"** — `smart_edit`: AI finds problem regions on the timeline — mains hum that comes and goes, noise that rises temporarily (AC, traffic), clusters of clipping, a passing low-frequency rumble — and applies a targeted mini-chain per region with crossfades. Everything outside the regions stays bit-for-bit untouched; the region map shows up as a timeline in the viewer. The surgical counterpart to improve_audio. - **"Save this as a preset" / "Do this like my podcast preset"** — `save_recipe`, `apply_recipe` and `list_recipes`: keep the chain of a session that sounded right as a named recipe, reuse it on new files, and share it — a recipe is a small JSON file, and apply_recipe also accepts a path to someone else's recipe. Curated presets ship built in. - **"Make it even better, take your time"** — `optimize_audio`: runs multiple pipeline variants (EQ, leveler, compressor, ClearVoice dereverberation) and lets the best one win on an objective score: Whisper word retention/confidence plus target deviation. The ranking comes back in the chat. - **"Transcribe this"** — `transcribe_audio` (Whisper, [asr] extra). - **"Fix the clips and clicks"** — `repair_audio`: declip (waveform reconstruction) and declick; improve_audio applies declip automatically when clipping is detected. - **"Make it sound like this reference"** — `match_reference`: 1/3-octave match EQ + loudness match; keeps episodes/recording days consistent. - **"Split the stems" / "vocals up 3 dB" / "make a karaoke version"** — `separate_stems` and `rebalance_music` (Demucs, [stems] extra). - **"Do the whole folder"** — `improve_folder`: batch processing (improve/refine/optimize). - **"Show me what changed"** — `view_audio`: a perceptual panel (auditory-scale spectrograms + difference map + level curves) that the AI inspects itself to judge what is audible. - **"This sounds good / this sounds bad"** — `rate_audio`: train your own taste model; analyze_audio then scores new audio against your taste. - **"Set this up in Audition"** — `export_to_audition`: stems + a .sesx multitrack session, opened directly in Adobe Audition. - **"Is this broadcast-proof?"** — `check_compliance`: pass/fail report against EBU R128, ATSC A/85, Netflix 2.0 & 5.1 (dialogue-gated, on 5.1 detected on the center channel; LFE excluded from loudness), Apple Podcasts, Spotify, YouTube or ACX audiobook, including the technical QC gates (clipping, dropouts, dead channel, anti-phase, ITU-downmix peak) and ADM BWF (Dolby Atmos) recognition. - **"Master this for European broadcast, 48 kHz 24-bit"** — `master_for`: masters to the spec (dialogue-gated specs steer on the detected speech), re-verifies, and exports a delivery file with high-quality SRC and bit-depth control. The compliance report shows up as a panel in the viewer. - **"Polish the dialogue"** — the film-post steps: `breath_control` (dim breaths, never cut), `deplosive` (p/b-pops fixed on the pop only) and `duck_music` (music beds ride down under the speech level) — bundled in the built-in `dialogue-polish` recipe. - **"Give me the region map as markers"** — `export_markers`: what the AI found lands as Audition marker CSV / Audacity labels in your DAW. - **"Fill the gaps with room tone"** — `fill_room_tone`: digital holes (dropouts, edit gaps) get filled with the file's own ambience — the dialogue editor's classic. - **"There's a chair squeak at 12.3"** — `spectral_repair`: RX-style spectral painting; the damaged time-frequency patch is repainted from its context while the programme underneath runs straight through. - **"Duck the music under the speech"** — `duck_music(mode="stems")`: real sidechain ducking for music playing *under* dialogue (Demucs separation, vocals-driven gain, no pumping). - **"QC this file" / "QC this folder"** — `qc_report` (one printable QC sheet) and `qc_folder` (a whole directory audited into one index table with verdicts). - **"Sync these recorders"** — `sync_tracks`: the 32-track recorder. Files from different devices (lav, boom, field recorder, camera, phone) are aligned on the audio itself — sample-accurate GCC-PHAT, confidence per track, optional clock-drift correction — and come out as aligned WAVs plus an Audition session. A/B in the viewer: unsynced echo soup vs the synced sum. - **"Open the viewer"** / **"What exactly changed?"** — A/B comparison; Claude reads the same session data the viewer shows. ## Documentation Extensive docs live in [`docs/`](docs/README.md): [getting started](docs/getting-started.md), the full [tool reference](docs/tools.md), a [workflows cookbook](docs/workflows.md) (podcast, broadcast delivery, film dialogue, restoration, QC), [delivery compliance](docs/compliance.md), [smart regions](docs/smart-regions.md), [recipes](docs/recipes.md) and the [architecture](docs/architecture.md). ## Installation (macOS) Requires: [uv](https://docs.astral.sh/uv/), ffmpeg (`brew install ffmpeg`), Xcode Command Line Tools. Python 3.11 is fetched by uv itself. ```bash cd chat-with-audio uv sync --all-extras # builds the C++ core and installs everything (incl. AI denoise) uv run pytest # 109 tests uv run python scripts/mcp_smoke.py # MCP smoke test ``` `uv sync` (without `--all-extras`) installs the basics without torch/DeepFilterNet; the tool then falls back to spectral gating automatically. ## Registering with Claude (and Codex) - **Claude Code**: lives in `.mcp.json` in the project folder (works automatically in this folder). - **Claude Desktop**: entry `chat-with-audio` in `~/Library/Application Support/Claude/claude_desktop_config.json`. Restart Claude Desktop after installing; the tools appear under "chat-with-audio". - **Codex CLI/app**: registered as a global MCP server via `codex mcp add chat-with-audio -- <uv-path> run --directory <project-folder> chat-with-audio-mcp` (verify with `codex mcp list`). Same 30 tools, same sessions and viewer. Note: run `uv sync --all-extras` first, otherwise the first server start may time out while building/downloading. ## The viewer `open_viewer` (or `uv run ait viewer`) starts it at <http://127.0.0.1:8471>. Space = play, **a/b = switch between original and processed**, **r = residual** (you hear exactly what the processing changed — ideal for artifact checking) while everything keeps playing in sync. Click the waveform to seek. Change the port with the `AIT_VIEWER_PORT` environment variable. Under the waveforms sits a timeline: a content lane (speech/music/silence) and, for smart_edit sessions, an interventions lane showing exactly where which problem was treated — click a region to jump there and use **r** to hear what was removed. Sessions live in `~/AudioImprove/sessions/` (override: `AIT_SESSIONS_DIR`), each with the original, result, analyses, chain + rationale, timeline, waveforms and spectrograms. ## Recipes A recipe is a saved processing chain as a small JSON file — reusable and shareable. Built-in presets (distilled from real sessions) ship with the package; your own live in `~/AudioImprove/recipes/` (override: `AIT_RECIPES_DIR`). Say "save this as my podcast preset" after a session that sounded right, apply it to new files with "do this like my podcast preset", and share the JSON file with anyone — `apply_recipe` also takes a file path. Steps are validated before anything runs. ## Windows 1. Install [uv](https://docs.astral.sh/uv/), ffmpeg (`winget install ffmpeg`) and the **Visual Studio Build Tools** (C++ workload) for the native DSP core. Everything works without the Build Tools too, via the pure-Python fallback — in that case don't remove the C++ build step, it simply fails soft. 2. `uv sync --all-extras` in the project folder (DeepFilterNet ships win_amd64 wheels). 3. Register in `%APPDATA%\Claude\claude_desktop_config.json` with the full path to `uv.exe` and the project folder (same shape as the `.mcp.json` here). ## Architecture | Layer | Location | Role | |---|---|---| | C++ DSP core | `cpp/` | biquad EQ (RBJ), noise gate, soft-knee compressor, look-ahead brickwall limiter; exposed via pybind11 as `chat_with_audio._dsp` | | DSP dispatch | `src/chat_with_audio/dsp/` | native ↔ scipy fallback, spectral gating (`spectral_nr.py`), DeepFilterNet (`ai_nr.py`) | | Analysis | `analysis.py` | LUFS/LRA (pyloudnorm), true peak, SNR, noise floor, hum, clipping, spectrum, scores + issues | | Decision logic | `improve.py` | speech/music detection, rules → chain + rationale | | Segmentation | `segments.py` | speech/music/silence timeline (level Otsu + speech modulation) | | Smart regions | `regions.py` | windowed problem detectors (hum/noise/clip/boom) + per-region mini-chains with crossfades | | Recipes | `recipes.py` | saved chains as shareable JSON; built-in presets + `~/AudioImprove/recipes/` | | Compliance | `compliance.py` | delivery specs (EBU/ATSC/Netflix/streaming/ACX) + pass/fail checker | | Dialogue suite | `dsp/dialogue.py` | breath control, plosive repair, music-bed ducking | | DAW markers | `markers.py` | region map → Audition CSV / Audacity labels / JSON | | Refinement loop | `refine.py` | iterative measure → adjust (speech peak, balance, pause floor), Whisper-guarded | | Optimization | `optimize.py` | variant contest, scored on intelligibility + targets | | Intelligibility | `asr.py` | Whisper transcription + word retention ([asr] extra) | | Dereverberation | `dsp/dereverb.py` | ClearVoice MossFormer2 48 kHz, speech segments only ([enhance] extra) | | Chain | `chain.py` | step registry (incl. leveler, smart_denoise), loudness normalization | | MCP server | `server.py` | 30 tools over stdio (FastMCP) | | Viewer | `viewer/` | stdlib http.server + Web Audio A/B player | Loudness targets: speech −16 LUFS / TP −1.5 dBTP, music −14 LUFS / TP −1.0 dBTP. ## Version pins (deliberate) - **Python 3.11** and **numpy < 2.0**: DeepFilterNet 0.5.x only ships wheels up to cp311 and requires numpy 1.x. - **torch/torchaudio < 2.9**: DeepFilterNet imports `torchaudio.backend`, which was removed in torchaudio 2.9. After changing the C++ code: `uv sync --reinstall-package chat-with-audio`.