EasySourceFlow

EasySourceFlow Pass long texts and videos to your Agent, let it summarize first, and then decide whether to read or watch. Regardless of whether you're using Feishu, WeChat, or other communication tools, simply send the webpage, public account article, video link, or file to your Agent. The Agent will call EasySourceFlow to extract content, generate structured summaries, and send the results directly back to the current conversation, helping you quickly determine if the content is worth reading in-depth. > **In addition:** The local service also supportsilibili or YouTube content, allowing you to choose to download the complete video or just the audio. [Quick Start](#quick-start [Agent Integration](#agent-integration) · [Supported Sources](#supported-sources) · [Privacy and Security](#privacy-and-security) · [](#full-documentation)   [  ## Why Do EasySourceFlow? Agents are good at calling tools, but they shouldn't duplicate efforts in web scraping, subtitle judgment, audio transcription, prompt management, and file writing. EasySourceFlow converges these processes into a recoverable and inspectable local service. - **One-time access, shared by multiple Agents**: Codex, OpenClaw, and other MCP clients call the same set of tools. - **Results delivered directly by the service**: The Agent returns the final Markdown as-is, without secondary summarization or rewriting. - **Prioritize reliable sources**: Videos prioritize platform subtitles, and local ASR is used only when subtitles are missing, with the source marked in the results. - **Tasks won't disappear due to temporary disconnections**: SQLite persists tasks, cache, retries, and recovery states. - **Data stays local**: The service listens to `127.0.0.1` by default, and output, database, and credentials are managed locally by the user. ## Main Capabilities - Web console for submitting links or local files, viewing tasks, results, favorites, and full-text search. - Web console for downloading Bilibili/YouTube videos or audio, with progress, cancellation, retries, and local file delivery; this capability is not open to Agents. - Bilibili subtitle extraction, login state import, local ASR fallback, and resource package output. - YouTube Chrome login state import, manual/auto subtitle priority, local ASR fallback, and diagnosable errors. - WeChat public account and ordinary webpage content extraction, with browser fallback. - OpenAI-compatible model access, with pre-set common services, Fast/Pro models, and independent credentials. - All cloud models share a set of summary rules and Markdown templates that can be edited in the Web console. - Videos automatically use Pro and generate clickable timelines corresponding to core points. - Favorites also copy Markdown, original subtitles, transcription, and source metadata. - Cache-aware models, prompts, and pipeline versions, with support for forced reprocessing. - Local backup, log rotation, cleanup preview, and macOS LaunchAgent auto-start. ## Supported Sources | Source | Processing Method | Notes | | --- | --- | --- | | Ordinary Webpage | Content and metadata extraction | News, tutorials, blogs, etc. | | WeChat Public Account | Content node, lazy-loaded images, browser fallback | Only processes | | Bilibili | Platform subtitles prioritized, local ASR fallback | Supports QR code login state import | | YouTube | Manual subtitles prioritized, automatic subtitles secondary, local ASR fallback | Supports importing login state from local Chrome | | Local Files | PDF, DOCX, EPUB, TXT, Markdown, subtitles, HTML | Content submitted by browser or Agent | ## Quick Start ### Environment Requirements - Python 3.10 or higher - macOS is the primary supported platform - Video processing recommends installing `ffmpeg`, `yt-dlp`, and at least one Whisper backend - YouTube Deno 2.3+ (recommended) or Node+, with matching `yt-dlp-ejs` installation - Cloud summarization requires an OpenAI-compatible model API Key ### Installation ```bash git clone <repository-url> EasySourceFlow cd EasySourceFlow python3 -m venv .venv .venv/bin/pip install -e ".[dev]" cp .env.example .env ``` Start the service and open the console: ```bash scripts/easysourceflow start scripts/easysourceflow open ``` Default address: [http://127.0.0.1:8765/](http://127.0.0.1:8765/). For first-time use, it's recommended to complete model configuration, general summary and Agent integration checks in "Maintenance". Before processing YouTube, log in to in Chrome, then go to "Maintenance → Account and Authorization" to access the Chrome login state. After integration, YouTube tasks directly read the current login state of the selected Chrome configuration file, avoiding the use of expired snapshots due to browser Cookie rotation; local filtering files are only used for status checking and compatibility fallback. To save audio/video, open the sidebar "Audio/Video Download", paste a single Bilibili or YouTube link, and choose video quality or audio format. Downloaded files are saved in `EASYSOURCEFLOW_DATA_DIR/media-downloads/`, without writing to the summary result database or exposing it as an MCP/Agent tool. Common commands: ```bash scripts/easysourceflow status scripts/easysourceflow health scripts/easysourceflow restart scripts/easysourceflow logs scripts/easysourceflow regression scripts/easysourceflow youtube-regression --force-refresh ``` ### macOS Auto-Start ```bash scripts/easysourceflow install-launchd scripts/easysourceflow launchd-status ``` The default running copy is located in `~/.local/share/easysourceflow/launchd`, avoiding direct dependency on user document directories. ## Agent Integration MCP clients start adapters in stdio mode: ```json { "mcpServers": { "easysourceflow": { "command": "<PROJECT_ROOT>/.venv/bin/easysourceflow-mcp", "env": { "EASYSOURCEFLOW_BASE_URL": "http://127.0.0.1:8765" } } } } ``` Actual project paths should only be written to local Agent configurations and not submitted to Git. Install the official Skill: ```bash scripts/easysourceflow install-skill "$AGENT_WORKSPACE" ``` The Skill requires the Agent to: 1. Submit links using asynchronous tasks and continuously query the same `job_id`. 2. Deliver `result.summary_markdown` as-is, without secondary summarization. 3. Videos default to using Pro and retain subtitle or ASR source markers. 4. When the user replies with "Favorite", favorite the most recent successful result. Audio/video downloads are exclusive to the local Web and not in the MCP tool list; Agents cannot trigger downloads. See the [Agent Integration Guide](docs/AGENT_INTEGRATION.md) and [MCP API](docs/MCP_API.md) for the complete tool contract. ## Working Principle ```mermaidchart LR A[Agent / Web] -->|MCP or HTTP| B[easysourceflowd] W[Web Audio/Video Download] --> J[Controlled yt-dlp / FFmpeg] J --> K[Local media-downloads] B --> C[Webpage and Video Extraction] C --> D{Has reliable subtitles?} D -->|Yes| E[Platform Subtitles] D -->|No| F[Local ASR] E --> G[General Prompts + Model] F --> G G --> H[Markdown and Resource Package] H --> I[Result Database / Favorites / Search] ``` The underlying extraction, transcription, model calls, and file writing are all completed by the service, while the Agent is only responsible for selecting tools and delivering results. ## Output and Favorites Successful tasks are written to `EASYSOURCEFLOW_OUTPUT_DIR`. Video resource packages usually contain: - `summary.md` - `metadata.json` and source information - Platform subtitles or local transcription - Timestamped full text - Core point timeline The Web result page renders Markdown and provides directory, copy, download, and favorite operations. Favorites also copy the complete resource package and prevent duplicate favoriting. ## General Summary Prompts Prompts in the Web console are unified and effective for DeepSeek, OpenAI, Qwen, Kimi, GLM, and other compatible services. The default template includes: - Only based on source content, without supplementing external facts - Subtitle and transcription integrity check - One-sentence conclusion, core points, detailed notes, and quotable excerpts - Action items, sedimentation suggestions, recommended tags, and quality checks Titles, authors methods, subtitle status, source types, and text are automatically appended by the program. After changing prompts, new tasks will not mistakenly reuse old cache.ive fallback does not call models, so it does not use this prompt. ## Privacy and Security - API Keys, Cookies, and local paths are only placed in `.env` or local configurations ignored by Git. - Agent configurations do not need to hold model API Keys. - Notifications and logs do not contain source content, complete subtitles, Cookies,. - Local files are uploaded through content and do not allow server-side arbitrary path reading. - Cleanup defaults to dry-run, and deletion is only executed after confirmation. - CI runs tests, Ruff, and Gitleaks historical scans with desensitized output. Please execute the [GitHub Release Checklist](docs/RELEASE_CHECKLIST.md) before public release. Security issues, see [SECURITY.md](SECURITY.md). ## Development and Verification ```bash PYTHONPATH=src .venv/bin/python -m compileall -q src tests PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v .venv/bin/ruff check src tests scripts/easysourceflow regression ``` Tests cover extraction, YouTube/Bilibili subtitle selection, ASR quality, task recovery, cache, output, HTTP, MCP, Web interaction, and sensitive information protection. Real platform regression does not enter regular CI. ## Known Limitations - Currently primarily targets local macOS operation and launchd deployment. - YouTube may still require Cookies, PO Tokens, or updated `yt-dlp` based on video, account, or client requirements; EasySourceFlow does not bypass platform permissions. - Audio/video downloads only handle single Bilibili/YouTube links, not playlists, DRM, paid, or unauthorized content. - Local ASR speed and quality depend on the selected backend and model. - Currently does not provide Obsidian automatic writing, NotebookLM, RAG, or vector database. ## Full Documentation - [Release Notes](CHANGELOG.md) - [Configuration Instructions](docs/CONFIGURATION.md) - [Agent Integration Guide](docs/AGENT_INTEGRATION.md) - [MCP API](docs/MCP_API.md) - [Architecture Design](docs/ARCHITECTURE.md) - [Deployment Instructions](docs/DEPLOYMENT.md) - [Operations Manual](docs/OPERATIONS.md) - [Security and Privacy](docs/SECURITY_PRIVACY.md) - [Test Plan](docs/TEST_PLAN.md) - [YouTube Regression Samples](docs/YOUTUBE_REGRESSION_SAMPLES.md) - [Roadmap](docs/ROADMAP.md) ## Contribution and License Welcome to submit issues and improvements through Issue and Pull Request. Please read the [Contribution Guide](CONTRIBUTING.md) first. EasySourceFlow is licensed under the [MIT License](LICENSE).