document-field-agent

# Document Workbench A batch document OCR and field extraction workbench for daily office work. It can recognize text from PDFs, images, and scanned documents, extract structured results by user-specified fields, and export them as Excel, Markdown, CSV, HTML, or plain text. Typical applications include lesson recording, contract accounting, form archiving, file organization, and bill/proof information extraction. This project is based on [andyhuo520/ppocrv6-studio](https://github.com/andyhuo520/ppocrv6-studio) secondary development. The original project provides an excellent local OCR workbench, model switching, and evaluation foundation around PP-OCRv6. This version extends it to a "batch document field extraction workbench". Thanks to the original author for their open-source work. ## Core Capabilities - Batch upload images, PDFs, or folders. - Automatic PDF pagination: one PDF generates one task, and each page is a record. - Task-level history record: history homepage displays tasks, and entering a task shows all pages. - Field configuration: manually fill in fields or read field names from an Excel template. - Field extraction: extract specified fields from OCR full text and output structured JSON results. - Low confidence prompt: fields with confidence below the threshold are marked red for easy review. - Excel export: with fields, export as "one row per page, fields as columns". - No field mode: export as TXT / Markdown / CSV / HTML. - Local priority: local OCR + local small model field extraction, suitable for private document processing. - Optional cloud high precision: connect to PaddleOCR-VL-1.6 and DashScope qwen-plus. - Optional Agent scheduling: built-in PilotDeck Skill / MCP adapter layer, allowing Agent to schedule batch processing, export, and review reports. ## Technical Route ```text File / Folder / PDF -> PDF rendering and pagination -> PP-OCRv6 local OCR or cloud PaddleOCR-VL parsing -> OCR full text / HTML text normalization -> Field candidate recall and structure judgment -> MiniCPM5-1B local small model field value decision -> Confidence verification -> Excel / Markdown / CSV / HTML / TXT export ``` Local field extraction uses by default: - `openbmb/MiniCPM5-1B-GGUF` - Default quantization file: `MiniCPM5-1B-Q4_K_M.gguf` - Running mode: `llama.cpp / llama-server` CPU sidecar Local models are not directly "hard-reading images" but make field decisions after OCR, candidate recall, and table/layout structure judgment. This allows using smaller models to complete more stable field extraction. ## Project Structure  | Path | Description | | --- | --- | | `webapp/server.py` | FastAPI backend, including OCR inference, task/page/field storage, PDF rendering, and export interface | | `webapp/static/` | Single-page front-end UI | | `pilotdeck/` | PilotDeck Skill, MCP/CLI adapter layer, and examples | | `scripts/download_models.sh` | Download PP-OCRv6 ONNX models | | `scripts/setup_minicpm5_sidecar.py` | Download MiniCPM5-1B GGUF and llama.cpp CPU sidecar | | `assets/` | Example images, screenshots, and evaluation materials | | `bench_local_v2.py` | OmniDocBench local evaluation script | Runtime data will be written to `webapp/data/`, and model files will be written to `models/`, `ppocrv6_onnx/`, `tools/llama/` and other directories. These directories are not committed to Git by default. ## Environment Requirements - Python 3.10+ - Windows - Recommended memory 8GB+ - Local field extraction requires about 700MB of MiniCPM5-1B GGUF model file - PDF support depends on `pypdfium2` Verification status: - Windows: verified OCR, task processing, field extraction, Excel export, and PilotDeck MCP call process. - macOS / Linux: current branch not fully verified. Theoretically, FastAPI, ONNX Runtime, and llama.cpp can run across platforms, but startup scripts, model downloads, MCP calls, and local sidecars still require real machine confirmation. - OCR capability comes from PP-OCRv6, and local OCR main process has been verified. ## Quick Start ### 1. Clone Project ```bash git clone https://github.com/BHD110/document-field-agent.git cd document-field-agent ``` ### 2. Create Virtual Environment and Install Dependencies Windows PowerShell: ```powershell python -m venv .venv .\.venv\Scripts\Activate.ps1 pip install -r requirements-webapp.txt ``` macOS / Linux: ```bash python -m venv .venv source .venv/bin/activate pip install -r requirements-webapp.txt ``` ### 3. Download PP-OCRv6 Models ```bash bash scripts/download_models.sh all ``` The script will download Tiny / Small / Medium ONNX models. Model files will not be included in the Git repository. ### 4. Prepare Local Field Extraction Model ```bash python scripts/setup_minicpm5_sidecar.py ``` The script will download: - `models/minicpm5-1b-q4_k_m.gguf` - `tools/llama/llama-server` or `tools/llama/llama-server.exe` ### 5. Start MiniCPM5 Sidecar Windows PowerShell: ```powershell .\tools\llama\llama-server.exe ` -m .\models\minicpm5-1b-q4_k_m.gguf ` --host 127.0.0.1 ` --port 11435 ` -c 4096 ` -t 8 ` --chat-template chatml ``` macOS / Linux: ```bash ./tools/llama/llama-server \ -m ./models/minicpm5-1b-q4_k_m.gguf \ --host 127.0.0.1 \ --port 11435 \ -c 4096 \ -t 8 \ --chat-template chatml ``` If you need to change the port, you can set it when starting the backend: ```bash DOC_WORKBENCH_LOCAL_LLM_URL=http://127.0.0.1:11436/v1/chat/completions python webapp/server.py ``` ### 6. Start Document Workbench ```bash python webapp/server.py ``` Open the browser: ```text http://127.0.0.1:8766 ``` If you need to change the port: ```bash DOC_WORKBENCH_PORT=8767 python webapp/server.py ``` ## Usage 1. Upload PDFs, images, or folders on the homepage. 2. Enter the fields to extract, such as name, unit, course name, contract number, date, amount, etc. 3. You can also upload an Excel template, and the system will read the first row of the first sheet as field names. 4. Choose local mode or cloud mode. 5. After processing, enter the result page to view the full text and field results. 6. Low-confidence fields will be marked red for easy manual review. 7. Click export to generate Excel or other format files. ## Cloud Mode Cloud mode is an optional capability for higher-precision document parsing and field extraction. Supported environment variables: ```bash DASHSCOPE_API_KEY=... PADDLEOCR_VL_TOKEN=... ``` You can also generate a locally run obfuscated blob using `scripts/make_secret_blob.py`, but this is just a deployment obfuscation scheme, not a real key security boundary. Please do not submit any API Key, token, `.env`, `secrets.blob`, or running logs to the repository. ## MCP / Agent Access The project has a built-in lightweight MCP Server to expose the "document field extraction workbench" to external Agent calls. It does not depend on a specific Agent client; as long as the client supports MCP, it can access. MCP tools include: - `create_document_task` - `get_task_status` - `list_task_pages` - `export_task_excel` - `export_task_report` After starting the document workbench, any MCP Client can call the local service through this MCP Server to achieve an automated process of "creating a task from a local path -> querying status -> exporting Excel -> generating a review report".  The above figure demonstrates the MCP call effect in PilotDeck: the user provides a local image path and fields to extract, and the Agent creates a document task, reads the processing status, summarizes field results, and exports Excel and review reports through MCP. The MCP Server itself is universal and can also access any Agent that supports MCP. ## API Overview | Interface | Description | | --- | --- | | `GET /health` | Health check | | `GET /info` | Current model, configuration, and workbench information | | `POST /tasks` | Create a task from an uploaded file | | `GET /tasks` | Get historical tasks | | `GET /tasks/{id}` | Get task details | | `GET /tasks/{id}/pages` | Get task pages | | `GET /tasks/{id}/export?fmt=xlsx\|txt\|md\|csv\|html` | Export task results | | `POST /templates/fields` | Read field names from an Excel template | | `POST /agent/tasks/from-path` | Agent-specific: create a task from a local path | | `GET /agent/tasks/{id}/summary` | Agent-specific: task summary | | `GET /agent/tasks/{id}/report?fmt=md\|json` | Agent-specific: review report | The old `/ocr` interface is still retained for compatibility with single-image OCR calls. ## Development and Verification Compilation check: ```bash python -m compileall webapp bench_local_v2.py gen_result_vis.py run_apple_vision.py scripts/make_secret_blob.py scripts/setup_minicpm5_sidecar.py pilotdeck/mcp_server.py ``` Common smoke test: - Upload a single image and generate 1 task and 1 page record. - Upload a multi-page PDF and confirm that the history page only shows 1 task. - Use an Excel template to read fields. - Export Excel with fields. - Export TXT / Markdown / CSV / HTML without fields. - Low-confidence fields are marked red on the result page. - MCP Server can list tools and create tasks. ## Enterprise Cooperation / Customized Services If your team has a large amount of repetitive data recognition, system entry, knowledge base question answering, data crawling, content generation, customer service, or operation process, you can further develop based on this project or existing AI automation capabilities. Currently, general product delivery is supported, and customized services are also accepted. The directions that can be covered include: - Document recognition, field extraction, and automatic entry system - Enterprise knowledge base, knowledge base question answering, and internal data retrieval - AI customer service, operation assistant, and content generation workflow - Content platform automation, such as Xiaohongshu - Live broadcast monitoring, data crawling, and abnormal reminder - Wind data analysis and industry data processing - Medical AI, law firm data entry, and other vertical scenarios Enterprises or teams with similar needs can scan the QR code to add WeChat for communication.  ## Open Source and Acknowledgments This project is based on [andyhuo520/ppocrv6-studio](https://github.com/andyhuo520/ppocrv6-studio) secondary development, thanks to the original author for their open-source contribution to PP-OCRv6 local workbench, evaluation process, and real scene examples. Thanks to the following open-source projects and models: - [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) - [ONNX Runtime](https://onnxruntime.ai/) - [llama.cpp](https://github.com/ggml-org/llama.cpp) - [OpenBMB MiniCPM](https://github.com/OpenBMB/MiniCPM) - [FastAPI](https://fastapi.tiangolo.com/) ## License This project uses the MIT License, see [LICENSE](LICENSE) for details. The use of third-party models, datasets, and cloud APIs must comply with their respective licenses and service terms.