Content
# MIR Algorithm Evaluation and Automatic Improvement Platform
Please note that this repository serves as an experimental environment, including the possibility of Vibe coding, and is not intended for practical use.
## What can actually be done?
- **AI Proposals:** The [`algos/`](algos/) directory contains specifications for 10 music signal processing algorithms (such as ONDE, PZSTD, ECHOE, FAZE) proposed by AI.
- **AI Implementations:** The [`src/detectors/`](src/detectors/) directory includes implementations of the 10 algorithms proposed by AI.
- Among the proposed algorithms, only **PZSTD** and **ONDE/ONDE-FIVE** are practically usable.
- Detailed results and analysis of this endeavor are published in the [Zenn article "Does AI Dream of Discovering New Algorithms?"](https://zenn.dev/romot/articles/11edc86fb6ab11).
## 1. Project Overview
This project is a platform designed to support the performance evaluation, comparison, and improvement of Music Information Retrieval (MIR) algorithms (such as note detection, onset detection, pitch estimation) using AI.
**Core Features:**
* **Evaluation:** Based on the `mir_eval` library, it calculates objective evaluation metrics (such as Precision, Recall, F-measure).
* **Data Processing:** Loads audio files and reference labels (in CSV format, etc.) necessary for evaluation.
* **Detector Interface:** Provides a common interface for easily adding and evaluating new detection algorithms (`src/detectors/`).
* **Parameter Optimization:** Allows exploration of optimal parameter settings for detectors through grid search (`src/evaluation/grid_search/`).
**Main Components:**
* **Core MIR Functions (`src/`):** Provides basic functionalities such as evaluation, data processing, and detectors. These can be used independently from other components.
* **Integrated CLI (`src/cli/mirai.py`):** Allows direct use of core functionalities (standalone evaluation) and operations such as evaluation, grid search, and automatic improvement loop execution via the MCP server from a single command-line interface.
* **MCP Server (`src/cli/mcp_server.py` and `src/mcp_server_logic/`):** An interface for exposing core functionalities and AI integration features (LLM calls) externally. It complies with the Model Context Protocol (MCP) and handles asynchronous job management, **server-centric state management** (DB, improvement session history, cycle states, etc.), and integration with AI agents (e.g., Claude Desktop).
**AI-Driven Automatic Improvement System (Experimental Feature):**
This project includes an **experimental feature** that automates the improvement cycle of MIR algorithms using the MCP server and the `mirai improve` command. The `mirai improve start` command manages and executes a series of steps such as evaluation, analysis, hypothesis generation, code improvement, and parameter optimization in a **CLI host-driven** manner. This is an attempt to explore the potential for algorithm improvement by directly calling the LLM API, and stability or quality of results is not guaranteed.
**For detailed setup, usage, and troubleshooting of the AI-driven automatic improvement system, please refer to [`MCP_README.md`](MCP_README.md).**
## 2. Directory Structure (Main Parts)
```
.
├── src/ # Source code for MIR-related functionalities and server logic
│ ├── detectors/ # Detection algorithm implementations (.py)
│ ├── evaluation/ # Evaluation-related modules
│ │ └── grid_search/ # Grid search related
│ ├── data_generation/ # Synthetic data generation (optional)
│ ├── utils/ # Common utilities (paths, exceptions, JSON, Logging etc.)
│ ├── mcp_server_logic/ # Core logic of the MCP server
│ │ ├── core.py # Configuration loading, cleanup, etc.
│ │ ├── db_utils.py # Asynchronous DB operations
│ │ ├── job_manager.py # Asynchronous job management
│ │ ├── session_manager.py # Improvement session management (including history recording)
│ │ ├── llm_tools.py # LLM prompt generation tools (analysis, improvement, hypothesis, strategy proposal, evaluation)
│ │ ├── evaluation_tools.py # Evaluation/grid search execution tools (server-side)
│ │ ├── code_tools.py # Code retrieval/saving tools (including Git integration)
│ │ ├── schemas.py # Pydantic schema definitions
│ │ └── prompts/ # LLM prompt templates (.j2)
│ └── cli/ # Command-line interface scripts
│ ├── mirai.py # ★ Integrated CLI tool (evaluation, grid search, improvement) - implements MCP host functionality
│ ├── llm_client.py # ★ LLM API call client (e.g., Anthropic Claude)
│ ├── mcp_server.py # MCP server startup script
│ └── setup_claude_integration.py # (optional) Claude Desktop integration setup
├── tests/ # Test code
├── datasets/ # Data files (paths specified in configuration files or .env)
├── output/ # Output base directory for server-based/standalone outputs (configurable)
├── .mcp_server_data/ # Default working directory for the MCP server (DB, improvement code, etc. Configurable)
├── pyproject.toml # Project configuration, dependencies (using uv)
├── config.yaml # Project-wide configuration file
├── requirements-lock.txt # (Recommended) Generated dependency lock file
├── .env.example # Template for environment variable settings
├── Dockerfile # For building Docker images (using uv)
├── MCP_README.md # Detailed README for MCP server/AutoImprover
└── README.md # This file
```
## 3. Setup (Using `uv`)
This project recommends using `uv`, a fast Python package installer and resolver.
1. **Clone the repository:**
```bash
git clone <repository_url>
cd <repository_name>
```
2. **Install `uv` (if not already installed):**
```bash
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (see https://github.com/astral-sh/uv for details)
# pip install uv
```
3. **Create and activate a virtual environment:**
```bash
# Create a virtual environment (e.g., .venv)
uv venv .venv
# Activate the virtual environment
source .venv/bin/activate # Linux/macOS
# .venv\\Scripts\\activate # Windows (Cmd/Powershell)
```
Please add `.venv/` to `.gitignore`.
4. **Install dependencies:**
Dependencies will be installed based on `pyproject.toml`. **Includes `aiosqlite` for asynchronous DB access, web server `FastAPI`, `uvicorn`, LLM integration `httpx`, `jinja2`, and MCP protocol `mcp`, among others.**
* **(Recommended) If using a lock file:**
Generate the recommended lock file (`requirements-lock.txt`, etc.) in advance.
```bash
# Compile all dependencies into the lock file (initial or update)
uv pip compile pyproject.toml --all-extras -o requirements-lock.txt
# Sync the environment based on the lock file
uv pip sync requirements-lock.txt
```
Please commit the generated lock file (`requirements-lock.txt`) to Git.
* **If not using a lock file:**
Install directly from `pyproject.toml`.
```bash
# Install dependencies in editable mode (-e), including dev, numba, crepe options
uv pip install -e .[dev,numba,crepe]
```
Adjust the necessary options (`numba`, `crepe`) as needed.
5. **Set environment variables:**
Create a `.env` file by copying `.env.example` and configure the necessary settings (especially **LLM API key (e.g., `ANTHROPIC_API_KEY`)**, workspace path `MIREX_WORKSPACE`, output path `MIREX_OUTPUT_BASE`, etc.). Do not commit the `.env` file to Git.
```bash
cp .env.example .env
# Edit with nano .env or vim .env, etc.
```
Environment variables will override values in `config.yaml`. For details, refer to `MCP_README.md` or the Docstring of the `load_config` function in `src/mcp_server_logic/core.py`.
6. **Generate synthetic data (required for evaluation):**
```bash
python -m src.data_generation.generate_all
```
This will generate evaluation audio and labels under `datasets/synthesized/` (or the path specified in `config.yaml` or environment variables).
## 4. Basic Usage
This project is primarily used through the integrated CLI tool `mirai`.
```bash
python -m src.cli.mirai --help
```
### 4.1. Standalone Evaluation (`mirai evaluate run`)
Directly performs evaluation of a specified detector on a single audio file and reference file without starting the MCP server.
```bash
python -m src.cli.mirai evaluate run \\
--detector YourDetectorClassName \\
--audio path/to/audio.wav \\
--ref path/to/reference.csv \\
--output results/standalone_eval \\
--params '{"threshold": 0.6}' \\
--save-plot
```
* `--detector`: Class name of the detector to evaluate.
* `--audio`, `--ref`: Paths to the audio and reference files to evaluate.
* `--output`: (Optional) Directory to save results (JSON or plots).
* `--params`: (Optional) Specify detector parameters as a JSON string.
* `--save-plot`: (Optional) Save the plot of the results.
For more details, refer to `python -m src.cli.mirai evaluate run --help`.
### 4.2. Evaluation and Grid Search via Server (`mirai evaluate run --server`, `mirai grid-search run --server`)
To perform evaluation or parameter grid search on the entire dataset, use the MCP server. This allows processing to run in the background and manage multiple requests.
**Note:** To use this method, you must **start the MCP server (`src/cli/mcp_server.py`) in advance**. For instructions on starting the server, refer to [`MCP_README.md`](MCP_README.md).
**Executing Evaluation:**
```bash
# Assuming the MCP server is running in a separate terminal
python -m src.cli.mirai evaluate run \\
--server http://localhost:5002 \\
--detector YourDetectorClassName \\
--dataset your_dataset_name \\
--version optional_version_tag # Version of the code to evaluate (latest if omitted)
```
* The command requests an evaluation job from the MCP server and waits for completion.
* For more details, refer to `python -m src.cli.mirai evaluate run --help`.
**Executing Grid Search:**
```bash
# Assuming the MCP server is running in a separate terminal
# 1. Prepare a grid configuration file (e.g., grid_config.yaml)
# 2. Execute the command
python -m src.cli.mirai grid-search run \\
--server http://localhost:5002 \\
--config path/to/grid_config.yaml
```
* The command requests a grid search job from the MCP server and waits for completion.
* For more details, refer to `python -m src.cli.mirai grid-search run --help`.
### 4.3. (Experimental) AI-Driven Automatic Improvement (`mirai improve start --server`)
**Note:** This is an experimental feature. For setup, detailed usage, and internal logic, please refer to **[`MCP_README.md`](MCP_README.md)**.
Using the MCP server, this command directly calls the LLM API from the CLI host to automatically execute cycles of evaluation, analysis, hypothesis generation, code improvement, and parameter optimization.
```bash
# Assuming the MCP server is running in a separate terminal
# and the LLM API key (ANTHROPIC_API_KEY) is set in .env or environment variables
python -m src.cli.mirai improve start \
--server http://localhost:5002 \
--detector YourDetectorClassName \
--dataset your_dataset_name \
--max-cycles 10 # Maximum number of cycles to execute
```
* The `improve start` command adopts a CLI host-driven design compliant with the MCP architecture:
1. Retrieves prompts from the server
2. Directly calls the LLM API on the host side (API key managed on the client side)
3. Executes actions based on results (code improvement, evaluation execution, etc.)
4. Records the results of actions as session history
* User confirmation is required for code changes, ensuring safe application after reviewing the changes.
* For more details, refer to `python -m src.cli.mirai improve start --help`.
## 5. Synthetic Dataset Details
### 5.1. Overview
For algorithm evaluation, synthetic audio files (`.wav`) with various acoustic features and corresponding ground truth label files (`.csv`) are generated.
### 5.2. Label Format (`.csv`)
Each label file is in CSV format and contains the following three columns (no header row):
1. **Onset (seconds):** Start time of the note
2. **Offset (seconds):** End time of the note
3. **Frequency (Hz):** Fundamental frequency (f0) of the note. Sounds without pitch (such as percussion) or silent intervals are represented as `0.0`.
Polyphony is recorded as temporally overlapping notes, each on a separate line.
### 5.3. Annotation Guidelines
* **Onset/Offset:** The time when the sound is perceptibly started/ended.
* **Frequency:** The fundamental frequency (f0) during the note duration. `0.0` indicates no pitch.
* **Evaluation Tolerance:** The default values of `mir_eval` (Onset: 50ms, Pitch: 50 cents, Offset: max(50ms, 20% duration)) apply. These can be modified in `config.yaml`.
### 5.4. List of Generated Files
Includes diverse test cases. For details, refer to the code in `src/data_generation/synthesizer.py`. (Sine waves, harmonics, dynamics, pitch modulation, noise, reverb, chords, polyphony, percussion, etc.)
## 6. Notes and Known Issues
* **Performance:** Large-scale evaluations, grid searches, and AI improvement loops may take time.
* **MCP Server Dependency:** Features using the `--server` option of the `mirai` CLI require the MCP server to be running. Standalone evaluations do not require the server.
* **Path Validation:** Paths passed from the client to the server are validated to ensure they are within the allowed range set by the server configuration.
* **(Experimental Feature):** The AI-driven automatic improvement system is under development and may exhibit unexpected behavior due to LLM responses or strategy decision logic. Please refer to `MCP_README.md` if using this feature.
## 7. Developer Information
### 7.1. Code Style and Formatting
This project uses the following tools to maintain a consistent code style:
* **black**: An automatic formatter for Python code
* **isort**: Organizes and optimizes import statements
* **pre-commit**: Automatically checks and formats code before commits
All of these are included in the development dependencies (`[dev]`).
```bash
# Install development dependencies
uv pip install -e ".[dev]"
# Manually run black
black src/
# Manually run isort
isort src/
```
### 7.2. Setting Up Pre-commit Hooks
Setting up pre-commit hooks will automatically check and fix code style before commits:
```bash
# Install pre-commit
uv pip install pre-commit
# Install pre-commit hooks
pre-commit install
```
The configuration is written in the `.pre-commit-config.yaml` file and includes the following checks:
* Removing trailing whitespace
* Fixing newline at the end of files
* Validating YAML files
* Checking for additions of large files
* Formatting Python code with black
* Organizing import statements with isort
To manually run checks on all files:
```bash
pre-commit run --all-files
```
To run a specific tool only:
```bash
pre-commit run black --all-files
```
## 8. Running with Docker
You can build and run a container image using the `Dockerfile`.
```bash
# Build
docker build -t mirai_app .
# Run (e.g., start server - read environment variables from .env)
# Set API keys and other settings in the .env file
docker run -it --rm -p 5002:5002 --env-file .env -v $(pwd)/output:/app/output -v $(pwd)/datasets:/app/datasets mirai_app python -m src.cli.mcp_server --host 0.0.0.0
# (Adjust volume mounts as necessary)
```
