researchFlowAgent

# ResearchFlow Agent > Project directory and file responsibility description: [docs/project-structure.md](docs/project-structure.md) A traceable research workflow agent for papers and technical documents. The system can ingest PDFs, retrieve page-level evidence, reorder candidate segments, query experimental metrics, record tool trajectories, and reuse the same research capabilities through LangGraph, HTTP API, and MCP. ## Current Tech Stack - DeepSeek: Dialogue, intent routing, Function Calling, and comprehensive answers. - Alibaba `text-embedding-v4`: Document and query vectors, fixed 1024 dimensions. - Alibaba `qwen3-rerank`: Reorder top 20 to top 6 after vector recall. - LlamaIndex: `Document`, `SentenceSplitter`, `VectorStoreIndex`, and Qdrant node management. - RapidOCR + PDFium: Supplement table structure and numerical values when ordinary text extraction misses table cells. - Qdrant Local: Vector disk persistence, no Docker required. - LangGraph: Research workflow orchestration; FastAPI: HTTP/SSE interface. - SQLite: Local development metadata; PostgreSQL reserved for deployment. ## RAG Data Flow ```text PDF → Extract text by page → Detect table pages and execute RapidOCR, generate dataset/model key line summaries → LlamaIndex SentenceSplitter (800 tokens / overlap 120) → text-embedding-v4 document vector (1024d) → Qdrant Local persistence User query → text-embedding-v4 query vector (query + research retrieval instruction) → Qdrant metadata filter → Top 20 → qwen3-rerank → Top 6 → DeepSeek comprehensive answer → Citation Check ``` Each paper citation retains `paper_id`, paper title, page number, `chunk_id`, evidence segment, and score. When Rerank is unavailable, the system degrades to vector sorting and records `rerank_fallback` in the tool trajectory. ## Local Launch (Conda) ```powershell cd C:\Users\lenovo\Documents\agent开发\researchflow-agent conda activate researchflow python -m pip install -e ".[dev]" python -m alembic upgrade head python -m uvicorn app.main:app --app-dir backend --reload ``` Open another terminal with activated Conda: ```powershell cd C:\Users\lenovo\Documents\agent开发\researchflow-agent\frontend npm install npm run dev ``` Open `http://localhost:5173`, API documentation at `http://localhost:8000/docs`. ## Model Configuration Create local configuration from template: ```powershell Copy-Item .env.example .env ``` `.env` is ignored by Git. Real keys can only be placed in `.env`, not in `.env.example`, source code, screenshots, or commit records. Formal local retrieval configuration: ```dotenv LLM_MODE=openai_compatible CHAT_API_KEY= CHAT_BASE_URL=https://api.deepseek.com CHAT_MODEL=deepseek-v4-flash CHAT_THINKING=disabled EMBEDDING_MODE=dashscope DASHSCOPE_API_KEY= EMBEDDING_MODEL=text-embedding-v4 EMBEDDING_DIMENSIONS=1024 RERANK_MODE=dashscope RERANK_MODEL=qwen3-rerank RETRIEVAL_CANDIDATES=20 RERANK_TOP_N=6 VECTOR_STORE_MODE=qdrant_local QDRANT_PATH=./data/qdrant ``` Production operation only uses DeepSeek and Dashscope. Automated testing injects offline test doubles through `backend/tests/doubles.py`, without reading real keys, accessing external APIs, or incurring costs. When model keys are missing, the application still starts and returns a degraded health status; upload and rebuild index interfaces explicitly return `503 embedding_not_configured`. ## Index Version and Rebuild Index `profile_id` is generated by Provider, model, dimensions, chunk size, overlap, splitter, and table OCR configuration. Different profiles use different Qdrant collections to avoid mixing test indexes, old model indexes, and 1024-dimensional production vectors. After upgrading, old test indexes are displayed as "Index expired": 1. Start the backend and frontend. 2. Click the "Rebuild paper index" button on the left paper list. 3. Wait for the status to become "Retrievable". 4. Only papers with the current profile as `ready` can join the Q&A. The system first writes to the new collection and new chunk; only after all succeed does it switch the paper's active profile. At startup, it only detects expired status and does not automatically call paid APIs. ## Core Interfaces | Method | Path | Action | |---|---|---| | GET | `/api/v1/health` | Model, database, and vector service status | | GET | `/api/v1/index/status` | Current model, dimensions, profile, vector count, and paper index statistics | | POST | `/api/v1/papers` | Upload PDF, return paper and ingestion task ID | | GET | `/api/v1/papers` | Query paper and index status | | POST | `/api/v1/papers/{id}/reindex` | Asynchronous rebuild of single paper index | | GET | `/api/v1/ingestion-jobs/{id}` | Query ingestion or rebuild progress | | POST | `/api/v1/chat` | Complete research answer | | POST | `/api/v1/chat/stream` | SSE streaming research answer | | POST | `/api/v1/papers/{id}/metrics/import` | Import experimental metrics CSV | | GET | `/api/v1/conversations/{id}` | Session and tool trajectory | | POST | `/mcp` | MCP Streamable HTTP | ## Data Migration ```powershell python -m alembic current python -m alembic upgrade head ``` Migration `0002_embedding_index_profile` adds paper index status, active profile, index time, chunk profile, and ingestion task type and details. Existing PDFs and database records are retained. ## Quality Check ```powershell python -m ruff check backend python -m mypy backend/app python -m pytest backend/tests cd frontend npx tsc -b --pretty false npm run lint npm test npm run build ``` Real Dashscope calls are not included in the default CI; default tests pass through dependency injection of offline doubles and mock verification of document/query parameter differences, batch processing, dimension verification, Rerank, degradation, profile changes, Qdrant Local persistence, and rebuild API. ## MCP HTTP client connects to `http://localhost:8000/mcp`. Stdio entry is: ```powershell researchflow-mcp ``` MCP tools include `search_papers`, `ask_paper`, `query_experiment_metrics`, and `get_citations`. ## Learning Route Start from [docs/learning/README.md](docs/learning/README.md). It is recommended to understand configuration and Provider, PDF ingestion, LlamaIndex nodes, Qdrant filter, Rerank degradation, LangGraph orchestration, and citation verification in sequence.