FNewsCrawler

# FNewsCrawler [](https://www.python.org/downloads/) [](https://fastapi.tiangolo.com/) [](https://modelcontextprotocol.io/) [](LICENSE) **Professional Financial Information MCP Service** - Providing real-time and accurate financial information data for large models ## 🚀 Project Introduction FNewsCrawler is a financial information MCP (Model Context Protocol) service specifically designed for large models. It obtains real-time information from major financial websites through efficient crawler technology, providing accurate and timely financial data support for AI models. ### 🎯 Core Features - **🤖 MCP Protocol Compatible**: Fully compatible with Model Context Protocol, seamlessly integrated into various AI applications - **📈 Real-time Financial Data**: Supports obtaining the latest information from mainstream financial platforms such as iFinD - **🏗️ Enterprise-level Architecture**: Employs a singleton browser management and context sharing mechanism - **🔐 Intelligent Login System**: Supports QR code login, automatically maintains login status - **⚡ High-performance Concurrency**: Based on asynchronous architecture, supports high concurrent access - **🛡️ Production-level Stability**: Complete error handling, automatic retry, and health check mechanisms - **📊 Visual Management**: Provides a Web interface for tool management and system monitoring  ## 🏛️ System Architecture ``` FNewsCrawler ├── 🧠 MCP Service Layer # Provides standardized interfaces for large models ├── 🕷️ Crawler Engine Layer # Multi-site financial data collection ├── 🌐 Browser Management Layer # Singleton browser instance management ├── 🔄 Context Sharing Layer # Domain-level session state management ├── 🔐 Login Authentication Layer # QR code login and status maintenance └── 💾 Data Storage Layer # Redis caching and state persistence ``` ## 📦 Quick Start ### Environment Requirements - Python 3.12+ - Redis Server - Chrome/Chromium Browser ### Installation and Deployment #### 🐳 Docker Deployment (Recommended) **Method 1: Deploy from project source code** ```bash # Clone the project git clone https://github.com/noimank/FNewsCrawler.git cd FNewsCrawler # Build and start the service docker-compose up -d # View service status docker-compose ps # View logs docker-compose logs -f ``` **Method 2: Quick Deployment and Startup** ```bash #Pull the image from dockerhub and start it in the background docker run --name fnewscrawler -p 8480:8480 -d noimankdocker/fnewscrawler:latest # View logs docker logs fnewscrawler -f ``` > 📚 **Detailed Documentation**: See [Docker Deployment Guide](docker/README.md) for complete deployment options, configuration instructions, and troubleshooting methods. #### 📦 Local Development Testing 1. **Clone the project** ```bash git clone https://github.com/noimank/FNewsCrawler.git cd FNewsCrawler ``` 2. **Install dependencies** ```bash uv sync uv run playwright install chromium ``` 3. **Configure environment** ```bash # Create a .env file, configure environment variables, it is recommended to turn off headless mode cp .env.example .env ``` 4. **Start Redis service** ```bash # Ubuntu/Debian sudo systemctl start redis-server # macOS brew services start redis # Windows # Download and start Redis service ``` 5. **Start the service** ```bash # Start the Web service uv run python main.py ``` ### MCP Client Integration MCP access protocol supports sse and http, modify through environment variable MCP_SERVER_TYPE mcp service address: http://localhost:8480/mcp/mcp-server cherryStudio configuration: ``` { "mcpServers": { "fnewscrawler": { "url": "http://localhost:8480/mcp/mcp-server", "type": "streamableHttp" } } } ``` ### get Method Call Calling endpoint: http://localhost:8480/api/mcp/call_tool/{tool_name}?xxxx Parameters: - tool_name: Tool name - xxxx: Tool parameters Return: - Tool execution result, json format Example: ``` GET http://localhost:8480/api/mcp/call_tool/get_industry_stock_funds_flow_tool?industry_name=bank ``` If the parameter type is list, it needs to be separated by commas Example: ``` GET http://localhost:8480/api/mcp/call_tool/news_crawl_batch?urls=http://example.com,http://example2.com ``` Additional supported functions: Call any akshare function Calling endpoint: http://localhost:8480/api/mcp/call_akshare/{fun_name}?xxxx Parameters: - fun_name: Function name - xxxx: Function parameters Supports processing the returned results: - duplicate_key: Deduplication field - drop_columns: Delete fields, multiple fields are separated by commas - return_type: Return type, markdown or json, default markdown - filter_condition: Filter condition string, similar to sql syntax, implemented using pandas' query statement, reference: https://gairuo.com/p/pandas-query - limit: Limit the number of returned data, no limit by default - sort_by: Sort field, specify which column to sort by - ascending: Sort order, true for ascending, false for descending, default true Return: - Function execution result, json format, the result format is as follows: ```json { "success": true, "message": "Called tool stock_zh_a_gbjg_em successfully", "data": { "result": xxxxxFunction execution result, markdown format is a string, json format is list[str]xxxx } } ``` Example: ``` GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?symbol=603392.SH&return_type=json GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?duplicate_key=Change Date&drop_columns=Restricted Shares,Reason for Change&return_type=json&symbol=603392.SH GET http://localhost:8480/api/mcp/call_akshare/news_trade_notify_dividend_baidu?return_type=json&date=20240409&filter_condition=Exchange == "SZ" # Sorting and limiting examples GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&sort_by=Date&ascending=true&limit=10 GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&sort_by=Close&ascending=false&limit=5&return_type=json GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&filter_condition=Close > 10&sort_by=Volume&ascending=false&limit=3&drop_columns=Change Factors ``` Error example: ``` GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?symbol="603392.SH"&return_type=json ``` **Note:** - Do not add quotes to parameters, single or double quotes, except for filter_condition - The sort_by field must be the actual column name in the returned data - limit must be a positive integer, ascending supports true/false, 1/0, yes/no, etc. > 📚 **Detailed Documentation**: See [GET API Documentation](docs/get_api.md) for complete interface descriptions, parameter configurations, and usage examples. ## 🛠️ Feature Development Roadmap ### 🔄 Completed - [x] iFinD news information query interface - [x] MCP protocol standard support - [x] iFinD industry fund flow query interface - [x] iFinD concept fund flow query interface - [x] General news query interface - [x] iFinD financial news, macroeconomic, industrial news, international finance, financial market, company news, regional economy, financial commentary, financial figure query interface - [x] iFinD A-share stock semantic screening tool - [x] iFinD margin trading information query tool - [x] iFinD historical fund flow query tool - [x] Eastmoney industry historical fund flow query tool - [x] Eastmoney industry stock historical fund flow query tool - [x] Eastmoney market fund flow query tool - [x] Eastmoney stock block trade daily details query tool - [x] Eastmoney stock dragon and tiger list query tool - [x] tushare data source support - [x] Stock technical indicator query interface - Moving Average - Bollinger Bands - KDJ - MACD - RSI - VWMA - ATR - [x] Stock chip and win rate query interface (tushare) - [x] Stock chip distribution query interface (akshare) - [x] Access akshare data source - [x] Provide get endpoint to call any akshare function - [x] Northbound/Southbound fund flow ### 📈 Coming Soon - [ ] Dragon and Tiger List data interface - Institutional transaction details - Business department ranking analysis - [ ] Market center interface - Real-time market push - Order book data analysis tips: Most interfaces, if not made into mcp interfaces, can be implemented by calling akshare functions through the get method. ### 🎯 Long-term Plan - [ ] Expand more financial data sources - [ ] Provide complete API documentation - [ ] Advanced data analysis tools The goal is to build a comprehensive financial information MCP service, providing real-time and accurate financial information data for large models. ## 🔧 Development Guide ### Add a New Financial Data Source 1. **Create a crawler module** ```bash mkdir fnewscrawler/spiders/new_site_name touch fnewscrawler/spiders/new_site_name/__init__.py touch fnewscrawler/spiders/new_site_name/crawl.py touch fnewscrawler/spiders/new_site_name/login.py # If login is required ``` 2. **Implement the login class** (if needed) ```python from fnewscrawler.core.qr_login_base import QRLoginBase class NewSiteLogin(QRLoginBase): async def get_qr_code(self, qr_type: str) -> Tuple[bool, str]: # Implement QR code acquisition logic pass async def verify_login_success(self) -> bool: # Implement login verification logic pass ``` 3. **Create an MCP tool** ```python # fnewscrawler/mcp/new_site_name/crawl.py from fnewscrawler.mcp import mcp_server @mcp_server.tool() async def new_site_query(query: str): """New site query tool description""" # Implement query logic pass ``` 4. **Register the tool** ```python # fnewscrawler/mcp/new_site_name/__init__.py import fnewscrawler.mcp.new_site_name.crawl ``` ### Core Design Principles - **Singleton Pattern**: BrowserManager and ContextManager adopt the singleton pattern to ensure efficient resource utilization - **Domain Isolation**: Each domain maintains an independent browser context to avoid session conflicts - **Asynchronous First**: Comprehensive use of asynchronous programming to improve concurrent performance - **State Persistence**: Login status and session information are automatically saved to Redis - **Health Check**: Regularly check the health status of browsers and contexts, and automatically recover from exceptions ## 📊 System Monitoring Visit `http://localhost:8480/monitor` to view system status: - 🌐 **Browser Status**: Instance health, version information, resource usage - 🔄 **Context Management**: Active sessions, idle time, usage statistics - 🛠️ **MCP Tools**: Tool status, enable/disable management - 📈 **System Logs**: Real-time log viewing and filtering ## 🔧 Tool Management Visit `http://localhost:8480/mcp` for tool management: - ✅ **Enable/Disable Tools**: Dynamically control tool availability - 📝 **View Tool Details**: View tool descriptions and parameters - 🔄 **Batch Operations**: Support batch enabling/disabling of tools - 📊 **Usage Statistics**: Tool call counts and status monitoring ## 🤝 Contribution Guide We welcome community contributions! See [Development Documentation](docs/) for details: - [Crawler Development Guide](docs/spider_development.md) - [MCP Tool Development Guide](docs/mcp_tool_development.md) ## 📄 License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## 🙏 Acknowledgements - [FastMCP](https://github.com/jlowin/fastmcp) - MCP protocol implementation - [Playwright](https://playwright.dev/) - Browser automation - [FastAPI](https://fastapi.tiangolo.com/) - Web framework - [Redis](https://redis.io/) - Data storage --- **Providing professional financial data support for large models** 🚀 If you have any questions or suggestions, please submit an [Issue](https://github.com/noimank/FNewsCrawler/issues) or [Pull Request](https://github.com/noimank/FNewsCrawler/pulls).