Content
# NewsPy
A NewsNow news aggregation and MCP service project implemented in Python.
## Technology Stack
- **Package Management**: uv (High-performance Python package manager written in Rust)
- **MCP Service**: fastmcp v3.2.4 (Model Context Protocol SDK)
- **Web Scraper**: httpx + BeautifulSoup4/lxml
- **Database**: SurrealDB (Supports embedded file storage and remote server)
- **RSS Support**: feedparser-rs
- **Python Version**: 3.12+
## Project Structure
```
newspy/
├── newspy/
│ ├── sources/ # Data source collectors
│ │ ├── __init__.py # Data source registry
│ │ ├── base.py # Basic crawler class
│ │ ├── ithome.py # IT Home
│ │ ├── baidu.py # Baidu Hot Search
│ │ ├── bilibili.py # Bilibili Hot Search/Popular Videos
│ │ ├── github.py # GitHub Trending
│ │ ├── rss_source.py # RSS data source
│ │ └── ...
│ ├── database/ # Database operations
│ │ ├── __init__.py
│ │ ├── cache.py # SurrealDB cache management
│ │ └── models.py # Data models
│ ├── settings.py # Centralized configuration file
│ ├── cli.py # CLI entry point
│ ├── mcp_tool.py # MCP server
│ └── cron.py # Scheduled collection script
├── scripts/ # Auxiliary scripts
│ ├── mcp_client.py # MCP client test
│ └── db_manage.py # Database management
├── pyproject.toml # Project configuration (supports command-line entry points)
├── uv.lock # Dependency lock file
└── CRON.md # Detailed instructions for scheduled tasks
```
## Quick Start
```bash
# Install dependencies
uv sync
# View all available commands
uv run newspy --help
uv run newspy-mcp --help
uv run newspy-cron --help
```
## Command-Line Tools
The project provides three command-line entry points:
### 1. newspy - Testing and database management tool
```bash
# List all data sources
uv run newspy list
# Test a single data source
uv run newspy test ithome -v
# Test all data sources
uv run newspy test-all
# Randomly retrieve news
uv run newspy random -n 10
# Database operations
uv run newspy db --list # List all cache
uv run newspy db -s ithome # Query specified source cache
uv run newspy db --clear # Clear database
```
### 2. newspy-mcp - MCP server
```bash
# Start MCP server (stdio mode)
uv run newspy-mcp
# MCP tools:
# - get_cached_news: Read latest news from database cache
# - get_latest_news: Get latest news from specified data source
# - list_all_sources: List all available data sources
```
### 3. newspy-cron - Scheduled collection
```bash
# Collect all data sources
uv run newspy-cron
# Collect specified data sources
uv run newspy-cron -s ithome baidu github
# Set crontab (execute every 15 minutes)
crontab -e
# Add:*/15 * * * * cd /path/to/newspy && uv run newspy-cron >> /var/log/newspy-cron.log 2>&1
```
See [CRON.md](CRON.md) for detailed instructions.
## Configuration System
Newspy uses `settings.py` for centralized configuration (similar to Django style), and all configuration items support environment variable overrides.
### Environment Variable Configuration
```bash
# Proxy configuration (supports http/https/socks5)
export NEWSPY_PROXY="http://127.0.0.1:7890"
# Or use system proxy
export HTTP_PROXY="http://127.0.0.1:7890"
export HTTPS_PROXY="http://127.0.0.1:7890"
# Request timeout (seconds)
export NEWSPY_TIMEOUT=15
# Database connection path
## Local file storage
export NEWSPY_DB_URL="file:///data/newspy"
## Websocket service
export NEWSPY_DB_URL="ws://localhost:8000"
# Disable specific data sources (comma-separated)
export NEWSPY_DISABLED="hackernews,github"
# Maximum number of news per data source (0 means no limit)
export NEWSPY_MAX_NEWS=50
# Log level
export NEWSPY_LOG_LEVEL=INFO
```
### Main Configuration Items
| Configuration Item | Environment Variable | Default Value | Description |
| --- | --- | --- | --- |
| `PROXY` | `NEWSPY_PROXY` | `None` | HTTP/SOCKS5 proxy |
| `REQUEST_TIMEOUT` | `NEWSPY_TIMEOUT` | `10` | Request timeout (seconds) |
| `DATABASE_URL` | `NEWSPY_DB_URL` | `file://~/.cache/newspy` | Database connection |
| `CACHE_TTL` | `NEWSPY_CACHE_TTL` | `1800000` | Cache expiration time (milliseconds) |
| `CRON_CONCURRENCY` | `NEWSPY_CONCURRENCY` | `4` | Collection concurrency |
| `LOG_LEVEL` | `NEWSPY_LOG_LEVEL` | `INFO` | Log level |
| `DISABLED_SOURCES` | `NEWSPY_DISABLED` | `[]` | Disabled data sources |
| `MAX_NEWS_PER_SOURCE` | `NEWSPY_MAX_NEWS` | `0` | Maximum news per source |
## Data Sources
The project supports 20+ data sources, including:
### Technology
- **ithome** - IT Home
- **kr36** - 36Kr
- **github** - GitHub Trending
- **juejin** - Juejin
- **solidot** - Solidot
- **devto** - Dev.to
### News
- **baidu** - Baidu Hot Search
- **toutiao** - Toutiao
- **ifeng** - Phoenix News
- **thepaper** - The Paper
- **cankaoxiaoxi** - Reference News
- **sputniknewscn** - Sputnik News
### Finance
- **wallstreetcn** - Wall Street Insights
- **gelonghui** - Gelonghui
- **xueqiu** - Xueqiu
- **jin10** - Jin10 Data
### Community
- **nowcoder** - Nowcoder
### Video
- **bilibili** - Bilibili Hot Search/Popular Videos
- **kuaishou** - Kuaishou
- **douyin** - Douyin
### Others
- **cls** - CLS
- **ghxi** - GHXI
- **tencent** - Tencent News
- **rss_source** - RSS data source (supports customization)
## Database
Newspy uses SurrealDB as the database:
- **Embedded mode**: Default file storage (`file://~/.cache/newspy`)
- **Server mode**: Supports connecting to remote SurrealDB server
- **Data format**: JSON document storage, supports complex queries
### Database Management
```bash
# View database status
uv run newspy db --list
# Query specific data source
uv run newspy db -s ithome
# Clear database
uv run newspy db --clear
# Or manually delete database file
rm -rf ~/.cache/newspy
```
## Add New Data Source
1. Create a new file in `newspy/sources/` directory
2. Inherit `BaseCrawler` class and implement `crawl()` method
3. Register data source in `newspy/sources/__init__.py`
Example:
```python
# newspy/sources/example.py
from typing import Any
from .base import BaseCrawler, NewsItem
class ExampleCrawler(BaseCrawler):
async def crawl(self) -> list[NewsItem]:
# Implement collection logic
items = []
# ... collection code ...
return items
def create_crawler(source_id: str, config: dict[str, Any]) -> BaseCrawler:
return ExampleCrawler(source_id, config)
```
Then register in `__init__.py`:
```python
from .example import create_crawler as example_factory
def init_all_sources():
register_source("example", example_factory)
# ... other data sources
```
## Development Guide
### Install Development Dependencies
```bash
uv sync --group dev
```
### Code Style
```bash
# Use ruff for code checking and formatting
uv run ruff check .
uv run ruff format .
```
### Type Checking
```bash
uv run mypy newspy
```
### Testing
```bash
uv run pytest
```
## Troubleshooting
### Common Issues
1. **Collection failure**: Check network connection, some data sources may require proxy
2. **Database error**: Ensure no multiple processes write to database simultaneously
3. **Dependency issue**: Run `uv sync` to reinstall dependencies
4. **Proxy issue**: Set `Newspy_PROXY` environment variable
### Log Debugging
```bash
# Set detailed log
export NEWSPY_LOG_LEVEL=DEBUG
# View cron log
tail -f /var/log/newspy-cron.log
```
## Performance Optimization
- **Concurrency**: Adjust `Newspy_CONCURRENCY` according to server performance (recommended 4~16)
- **Collection frequency**: Adjust according to data source update frequency (hot list 5~15 minutes, news 30~60 minutes)
- **Database**: Default SurrealDB embedded mode suitable for single process, high concurrency scenario recommends using SurrealDB Server
## License
This project is licensed under the MIT License.
