Content
# TikTok Scraper MCP
Local Go MCP server for scraping public TikTok metadata through a cheap HTTP-first fetcher, deterministic fixture tests, and an optional browser fallback.
> GitHub description: Local Go MCP server for public TikTok metadata scraping with HTTP-first extraction, fixture-tested contracts, and optional browser fallback.
## Status
This project is an MVP for local MCP usage. It is not a hosted scraping service, does not require Docker, and does not use TikTok login cookies.
Current live behavior:
- Direct video URL metadata works through `public_http` with bounded retries.
- Public profile metadata works through `public_http` when TikTok returns hydration payload.
- Browser mode is available as a fallback, but it is slower and should not be the default for direct video.
- Profile video lists are not claimed as solved yet. TikTok exposes them through signed `api/post/item_list` requests, so the current contract returns structured `TARGET_DRIFT` when the list is unavailable.
TikTok changes its web payloads frequently. Fixture tests are the deterministic contract; live smoke tests are opt-in.
## Features
- MCP `stdio` server.
- Typed tools powered by `github.com/skosovsky/toolsy`.
- Public data only, no TikTok login.
- Metadata-only output, no media downloads.
- Direct video URL scraping.
- Public profile metadata scraping.
- Structured error items instead of silent partial failures.
- In-memory cursor cache for large MCP responses.
- Fetcher interface with three modes:
- `public_http`
- `browser`
- `fixture`
- Raw fixture extraction and deterministic tests.
## Requirements
- Go `1.26.3` or newer compatible with the module.
- `jq` for JSON fixture validation.
- Google Chrome only if you use `TIKTOK_MCP_FETCHER_MODE=browser`.
## Install
Install the MCP server binary:
```bash
go install github.com/skosovsky/tiktok-scraper/cmd/tiktok-scraper-mcp@latest
```
Make sure your Go binary directory is on `PATH`:
```bash
export PATH="$(go env GOPATH)/bin:$PATH"
```
For local development, clone the repository and run the verification commands from the project root.
## MCP Client Configuration
Use an absolute path to the binary. Example:
```json
{
"mcpServers": {
"tiktok-scraper": {
"command": "/absolute/path/to/tiktok-scraper-mcp",
"env": {
"TIKTOK_MCP_LOG_LEVEL": "info",
"TIKTOK_MCP_FETCHER_MODE": "public_http",
"TIKTOK_MCP_MAX_RETRIES": "3"
}
}
}
}
```
The server speaks MCP over `stdio`. Logs are written to `stderr`; `stdout` is reserved for MCP frames.
## Tools
### `tiktok_scrape`
Scrapes supported TikTok inputs.
Example arguments:
```json
{
"videoUrls": [
"https://www.tiktok.com/@tiktok/video/7651659857640033566"
]
}
```
Profile example:
```json
{
"profiles": ["tiktok"],
"limit": 5
}
```
### `tiktok_get_more`
Reads the next page from the in-memory cursor cache when `tiktok_scrape` returns `hasMore=true`.
### `tiktok_validate_input`
Validates input without scraping.
## Manual MCP Smoke
List tools:
```bash
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | ./tiktok-scraper-mcp
```
Call `tiktok_scrape`:
```bash
printf '%s\n' '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"tiktok_scrape","arguments":{"videoUrls":["https://www.tiktok.com/@tiktok/video/7651659857640033566"]}}}' | ./tiktok-scraper-mcp
```
## Fetcher Modes
| Mode | Env value | Purpose | Notes |
| --- | --- | --- | --- |
| Public HTTP | `public_http` | Default live mode for direct video and profile metadata | Cheapest mode. Uses Go HTTP, public HTML payloads, extraction fallbacks, and bounded retries. |
| Browser | `browser` | Fallback and discovery mode | Uses local headless Chrome through `chromedp`. Slower and more expensive. Requires Google Chrome. |
| Fixture | `fixture` | Deterministic tests and local contract checks | Reads from `fixtures/` without network. |
Default:
```bash
TIKTOK_MCP_FETCHER_MODE=public_http
```
Browser fallback:
```bash
TIKTOK_MCP_FETCHER_MODE=browser
TIKTOK_MCP_HTTP_TIMEOUT=45s
TIKTOK_MCP_FETCH_CONCURRENCY=1
TIKTOK_MCP_PER_HOST_CONCURRENCY=1
```
Fixture mode:
```bash
TIKTOK_MCP_FETCHER_MODE=fixture
```
## Configuration
The server is configured through environment variables.
Important settings:
| Variable | Default | Description |
| --- | --- | --- |
| `TIKTOK_MCP_FETCHER_MODE` | `public_http` | `public_http`, `browser`, or `fixture`. |
| `TIKTOK_MCP_HTTP_TIMEOUT` | `30s` | Fetch timeout. |
| `TIKTOK_MCP_MAX_RETRIES` | `3` | Bounded retry count for transient fetch errors, retryable classifications, and target drift. |
| `TIKTOK_MCP_DEFAULT_LIMIT` | `20` | Default item limit. |
| `TIKTOK_MCP_MAX_LIMIT` | `100` | Hard limit. |
| `TIKTOK_MCP_RESPONSE_ITEM_LIMIT` | `50` | Inline response item limit before cursor pagination. |
| `TIKTOK_MCP_CURSOR_TTL` | `30m` | In-memory cursor TTL. |
| `TIKTOK_MCP_USER_AGENT` | Chrome-like UA | User-Agent for live fetchers. |
| `TIKTOK_MCP_ACCEPT_LANGUAGE` | `en-US,en;q=0.9` | Accept-Language header. |
Examples:
- `.env.example`
- `.env.live.example`
- `../docs/tiktok-scraper/configuration.md`
## Testing
Deterministic local checks:
```bash
go test -count=1 ./...
jq empty fixtures/expected/*.json fixtures/json/*.json fixtures/meta/*.json
go build ./cmd/tiktok-scraper-mcp
```
Fast live direct-video smoke:
```bash
go test -count=20 -tags=public_network ./internal/scraper -run TestPublicNetworkHTTPVideoScrapeSmoke -v
```
Profile metadata smoke:
```bash
go test -count=10 -tags=public_network ./internal/scraper -run TestPublicNetworkProfileScrapeSmoke -v
```
Browser fallback smoke:
```bash
go test -count=1 -tags=public_network ./internal/scraper -run 'TestPublicNetwork(BrowserVideoScrapeSmoke|BrowserProfileScrapeSmoke)' -v
```
Live tests are opt-in and may change as TikTok changes its public web surface.
## Documentation
Deep documentation lives in the workspace-level docs folder:
- `../docs/tiktok-scraper/README.md`
- `../docs/tiktok-scraper/problem-solving-log.md`
- `../docs/tiktok-scraper/scraping-strategy.md`
- `../docs/tiktok-scraper/discovery-notes.md`
- `../docs/tiktok-scraper/testing-strategy.md`
- `../docs/scraping.md`
## Boundaries
This project intentionally does not provide:
- TikTok login support.
- CAPTCHA solving.
- Proxy/session pool management.
- Media downloads.
- Remote MCP transport.
- Docker packaging.
- Persistent storage.
- Guaranteed profile video pagination.
## License
MIT. See `LICENSE`.
