Content
# Trafilatura MCP Server
This repository contains a Model Context Protocol (MCP) server that provides a tool-based interface to the [Trafilatura](https://trafilatura.readthedocs.io/) library, a powerful tool for web scraping. It is designed for use with MCP-compatible clients, allowing developers and models to extract main content and metadata from web pages programmatically.
## Features
- **Web Scraping:** Utilizes Trafilatura to extract the main text content from a given URL.
- **Metadata Extraction:** Retrieves metadata such as title, author, date, and more.
- **Configurable Extraction:** Options to include or exclude comments and tables from the output.
- **Simple Tool:** Exposes a single, easy-to-use `fetch_and_extract` tool.
- **Asynchronous:** Built with an asynchronous architecture for efficient I/O operations.
- **MCP Standard:** Communicates over standard I/O, making it compatible with various MCP clients.
## Prerequisites
Before running the server, you need to have Python 3.12+ and `uv` installed. You will also need Node.js and `npx` to run the MCP Inspector tool for testing.
## Installation
1. **Clone the repository.**
```bash
git clone <repository-url>
cd trafilatura_mcp
```
2. **Create a virtual environment and install the required dependencies:**
```bash
# Create a virtual environment
uv venv
# Activate the virtual environment
source .venv/bin/activate
# Install the dependencies
uv sync
```
## Running and Testing the Server
The MCP server is a command-line application that communicates over standard I/O. To use it, a client (like an IDE, a coding agent, or an inspector tool) must launch the server process.
### Running for Diagnostics
You can run the script directly from your terminal to see if it starts without errors. This is a quick way to validate your Python environment and the script's basic syntax.
```bash
python3 trafilatura_mcp.py
```
The server will start and wait for input, but you won't be able to interact with it directly from your terminal.
### Testing with MCP Inspector
The recommended way to test the server interactively is with **MCP Inspector**. It provides an interactive shell for sending requests to your server.
1. **Launch the Inspector:**
You can run the inspector without a permanent installation using `npx`. The inspector will launch your MCP server script for you. From your project directory, run:
```bash
npx @modelcontextprotocol/inspector uv run -- python3 trafilatura_mcp.py
```
2. **Interact with the Server:**
Once the inspector starts, you can connect to the server and use commands like `list_tools` and `call_tool`.
**Example session:**
```bash
# List the available tool
> list_tools
# Call the 'fetch_and_extract' tool with a URL
> call_tool fetch_and_extract '''{"url": "https://www.theguardian.com/us-news/2025/sep/28/mass-shootings-north-carolina-texas-new-orleans"}'''
```
## Configuration
This server does not require any external API keys or configuration files.
## Usage with an MCP Client (VS Code Example)
You can connect to this server from any standard MCP client. Here’s how to do it in a VS Code environment that supports MCP:
1. **Configure Your MCP Client:**
In your IDE's MCP client settings (e.g., in `mcp.json` for VS Code), configure a new MCP server that points to the script.
**Example `mcp.json` entry:**
```json
{
"servers": {
"trafilatura_scraper": {
"command": "uv",
"args": [
"run",
"python3",
"trafilatura_mcp.py"
],
"cwd": "/path/to/your/project/trafilatura_mcp"
}
}
}
```
*Note: Replace `/path/to/your/project/trafilatura_mcp` with the absolute path to the project directory.*
2. **Use the Tool:**
Once connected, you can use the exposed tool in your chat or agent interactions. For example, to extract content from a news article, you could send the following structured tool call:
```json
{
"tool": "fetch_and_extract",
"arguments": {
"url": "https://apnews.com/article/elon-musk-x-twitter-hate-speech-antisemitism-0d35c5a69fd5c6183b729f7f3c87064a",
"include_comments": false,
"include_tables": true
}
}
```
The server will fetch the URL, extract the main content and metadata, and return it as a JSON object.
