Content
<div align="center">
<!-- Logo placeholder — replace with actual logo when available -->
<!-- <img src="docs/screenshots/logo.png" alt="CrawlSEO" width="80" /> -->
# CrawlSEO
### Open-source SEO monitoring for founders, not SEO specialists
Google Search Console + Site Crawler + Core Web Vitals + MCP Server — all in one self-hosted dashboard. Free forever.
[](https://github.com/crawlseo/crawlseo/stargazers)
[](LICENSE)
[](CONTRIBUTING.md)
[](docker-compose.yml)
</div>
---
<p align="center">
<img src="docs/screenshots/dashboard.png" alt="CrawlSEO Dashboard" width="800" />
</p>
## Why CrawlSEO?
| | CrawlSEO | OpenSEO | Ahrefs | Semrush | Moz |
|---|:---:|:---:|:---:|:---:|:---:|
| **Price** | **Free** | $10/mo | €119/mo | $139/mo | $49/mo |
| **Self-hosted** | ✅ | ✅ | ❌ | ❌ | ❌ |
| **GSC integration** | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Site crawler** | ✅ (2000 pages) | ✅ | ✅ | ✅ | ✅ |
| **Core Web Vitals** | ✅ | ❌ | ❌ | ✅ | ❌ |
| **MCP Server** | ✅ (10 tools) | ✅ (24 tools) | ❌ | ❌ | ❌ |
| **AI agent ready** | ✅ | ✅ | ❌ | ❌ | ❌ |
| **Keyword Research** | ✅ (BYOK) | ✅ | ✅ | ✅ | ✅ |
| **Backlinks** | ✅ (BYOK) | ✅ | ✅ | ✅ | ✅ |
| **Open source** | ✅ MIT | ✅ | ❌ | ❌ | ❌ |
| **Your data stays yours** | ✅ | ✅ | ❌ | ❌ | ❌ |
> **BYOK** = Bring Your Own Key. Keyword research and backlink data use DataForSEO (optional). Google Autocomplete suggestions work as a free fallback.
## Features
### 🔍 GSC Analytics
Keywords, pages, clicks, impressions, position tracking with 28-day comparison and delta indicators.
<p align="center">
<img src="docs/screenshots/keywords.png" alt="Keywords — GSC Analytics" width="800" />
<br />
<em>Top keywords with position badges, clicks, impressions, and CTR</em>
</p>
### 🕷️ Site Crawler
Crawl up to 2,000 pages with concurrent fetching. Health score, 16 issue types, content scoring, and remediation guidance.
<p align="center">
<img src="docs/screenshots/audit.png" alt="Crawl / Audit" width="800" />
<br />
<em>Crawl results with health score, issue breakdown, and per-page audit data</em>
</p>
### 🤖 MCP Server — AI Agent Integration
10 tools for Claude Code, Claude Desktop, and Cursor. Query your SEO data, run crawls, and find opportunities without leaving the terminal.
<p align="center">
<img src="docs/screenshots/mcp.png" alt="AI & MCP" width="800" />
<br />
<em>MCP setup page with connection config, setup guides, and available tools</em>
</p>
### More features
| | Feature | Description |
|---|---|---|
| ⚡ | **Core Web Vitals** | LCP, CLS, INP, TTFB via PageSpeed Insights with mobile/desktop comparison |
| 🔑 | **Keyword Research** | DataForSEO-powered keyword ideas with volume, difficulty, CPC. Free Google Autocomplete fallback |
| 🔗 | **Backlinks** | Backlink profile, referring domains, anchor text, dofollow/nofollow analysis |
| 📊 | **Rank Tracking** | Historical position snapshots with saved keywords and notes |
| 💡 | **SEO Opportunities** | Striking distance keywords, low CTR, content decay, cannibalization detection |
| 🔔 | **Alerts** | Traffic drops, position changes, new 404s, vitals degradation — via email, Slack, Telegram, webhook |
| 📥 | **CSV Export** | Export keywords and pages data for offline analysis |
| 🌗 | **Dark / Light theme** | Atomize PRO design system with smooth theme toggle |
## Quick Start
```bash
git clone https://github.com/crawlseo/crawlseo.git
cd crawlseo
cp .env.example .env.local
# Add your Google OAuth credentials to .env.local
docker compose up -d db
npm install
npx prisma migrate dev --name init
npm run dev
```
Open [http://localhost:3000](http://localhost:3000), sign in with Google, and add your first site.
<details>
<summary>🔑 Getting Google OAuth credentials</summary>
1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a project (or select existing)
3. Enable the **Google Search Console API**
4. Go to **Credentials** → **Create Credentials** → **OAuth 2.0 Client ID**
5. Application type: **Web application**
6. Authorized redirect URI: `http://localhost:3000/api/auth/callback/google`
7. Copy Client ID and Client Secret to `.env.local`
Required scopes: `openid`, `email`, `profile`, `https://www.googleapis.com/auth/webmasters.readonly`
</details>
## MCP Server — AI Agent Integration
CrawlSEO includes a Model Context Protocol server so AI agents can query your SEO data directly.
Add to your Claude Code settings (`.claude/settings.json`):
```json
{
"mcpServers": {
"crawlseo": {
"command": "npx",
"args": ["tsx", "mcp/server.ts"],
"cwd": "/path/to/crawlseo"
}
}
}
```
**10 tools available:**
| Category | Tools |
|---|---|
| **Sites** | `list_sites`, `get_site_overview` |
| **Keywords & Pages** | `get_keywords`, `get_pages`, `get_traffic` |
| **Crawl & Audit** | `run_crawl`, `get_crawl_status`, `get_crawl_issues` |
| **Performance** | `get_vitals`, `get_opportunities` |
Works with Claude Code, Claude Desktop, and Cursor. See [`mcp/README.md`](mcp/README.md) for full setup guide.
## Tech Stack
| Layer | Technology |
|---|---|
| **Framework** | [Next.js 16](https://nextjs.org/) (App Router) |
| **Language** | [TypeScript](https://www.typescriptlang.org/) |
| **Database** | [PostgreSQL](https://www.postgresql.org/) |
| **ORM** | [Prisma](https://www.prisma.io/) |
| **Auth** | [NextAuth.js v5](https://authjs.dev/) |
| **UI** | [shadcn/ui](https://ui.shadcn.com/) + [Tailwind CSS v4](https://tailwindcss.com/) |
| **Charts** | [Recharts](https://recharts.org/) |
| **Icons** | [Lucide React](https://lucide.dev/) |
| **MCP** | [@modelcontextprotocol/sdk](https://modelcontextprotocol.io/) |
| **Deployment** | Docker Compose |
## Self-Hosting
### Docker Compose (recommended)
```bash
git clone https://github.com/crawlseo/crawlseo.git
cd crawlseo
cp .env.example .env.local
# Edit .env.local with your credentials
docker compose up --build
```
The full stack (PostgreSQL + Next.js) starts in ~2 minutes. Migrations run automatically.
### Manual
```bash
# Prerequisites: Node.js 20+, PostgreSQL
npm install
cp .env.example .env.local
# Configure .env.local
npx prisma migrate deploy
npm run build
npm start
```
## Environment Variables
| Variable | Required | Description |
|---|---|---|
| `DATABASE_URL` | Yes | PostgreSQL connection string |
| `NEXTAUTH_SECRET` | Yes | Session encryption key (`openssl rand -hex 32`) |
| `GOOGLE_CLIENT_ID` | Yes | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | Yes | Google OAuth client secret |
| `NEXTAUTH_URL` | No | Base URL (auto-detected in most environments) |
## Contributing
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
```bash
# Fork the repo, then:
git checkout -b feature/your-feature
# Make your changes
git commit -m "feat: add your feature"
git push origin feature/your-feature
# Open a Pull Request
```
## License
MIT License — see [LICENSE](LICENSE) for details.
---
<div align="center">
Built by [Brandson Digital](https://brandson.digital) · Created by [Mike](https://m1ke.digital)
Self-hosted SEO tools should be free. Your data should be yours.
</div>
