canyougrab-api

# CanYouGrab API Domain availability lookup API with subscription billing, built on FastAPI + DNS (Unbound resolver). **Live services:** - API: `https://api.canyougrab.it` - Developer portal: `https://portal.canyougrab.it` - Auth: `https://auth.canyougrab.it` (Auth0 custom domain) ## Architecture Overview ``` ┌──────────────────────────────────────────────────────────────────┐ │ Developer Portal │ │ (Zudoku/React on portal.canyougrab.it) │ │ Usage Dashboard · API Keys · Pricing · API Reference (OAS) │ └──────────────────┬───────────────────────────────────────────────┘ │ Auth0 JWT (portal) / Bearer API key (API) ▼ ┌──────────────────────────────────────────────────────────────────┐ │ FastAPI Backend (v5.0.0) │ │ api.canyougrab.it:8000 │ │ │ │ ┌─────────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │ │ │ app.py │ │ keys.py │ │billing.py│ │ auth.py │ │ │ │ /check/bulk │ │ /keys │ │/billing │ │ API key + │ │ │ │ /usage │ │ CRUD │ │/stripe │ │ JWT auth │ │ │ │ │ │ rotate │ │ webhook │ │ │ │ │ └──────┬──────┘ └──────────┘ └────┬─────┘ └─────────────┘ │ │ │ │ │ └─────────┼────────────────────────────┼───────────────────────────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Valkey (Redis) │ │ Stripe API │ │ Job queue + │ │ Subscriptions │ │ Rate limiting │ │ Webhooks │ └────────┬────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ ┌──────────────────────────────┐ │ RQ Worker │──DNS──▶ │ Unbound Resolver │ │ (worker.py) │ │ (dedicated droplet) │ │ ThreadPool(10) │ │ NS queries via VPC │ │ via RQ queue │ └──────────────────────────────┘ └─────────────────┘ ┌──────────────────────────────┐ ───────────────────▶│ PostgreSQL │ (auth, usage, │ API keys + Usage logs │ billing only) └──────────────────────────────┘ ``` ## Directory Structure ``` zuplo/ ├── backend/ # Python FastAPI backend │ ├── app.py # Main API: /check/bulk, /usage, /health │ ├── auth.py # API key auth (SHA-256) + Auth0 JWT auth (RS256) │ ├── billing.py # Stripe checkout, portal, webhooks, card-on-file, usage details │ ├── keys.py # API key CRUD: create, list, rotate, revoke (+ Turnstile) │ ├── antifraud.py # Anti-fraud: Turnstile, device fingerprints, risk scoring │ ├── email_utils.py # Email normalization + disposable email detection │ ├── dns_client.py # DNS-based domain availability checking via Unbound │ ├── queries.py # PostgreSQL queries: usage tracking, auth, billing │ ├── valkey_client.py # Redis/Valkey job queue client (RQ-backed) │ ├── rq_tasks.py # RQ task function: process_domain_job() │ ├── worker.py # RQ worker process (ThreadPoolExecutor per job) │ ├── migrations/ # SQL migrations │ │ └── 001_free_tier_antifraud.sql │ └── requirements.txt # Python dependencies ├── portal/ # Developer portal (Zuplo + Zudoku) │ ├── config/ │ │ ├── routes.oas.json # OpenAPI 3.1 spec (public API documentation) │ │ └── policies.json # Zuplo policies (empty — all routing is direct) │ ├── docs/ # Zudoku documentation portal │ │ ├── src/ │ │ │ ├── config.ts # API_BASE, Turnstile site key, Stripe PK │ │ │ ├── UsageDashboard.tsx # Usage + billing dashboard component │ │ │ ├── PricingPage.tsx # Plan selection + Stripe checkout │ │ │ ├── PricingPlans.tsx # Pricing card grid component │ │ │ └── CardSetupPage.tsx # Stripe Elements card-on-file for Free+ │ │ ├── public/ # Static assets (logos, banners, CSS overrides) │ │ ├── zudoku.config.tsx # Portal config: theme, nav, Auth0, API key mgmt │ │ └── package.json # Frontend dependencies (React 19, Zudoku) │ ├── package.json # Workspace root (Zuplo v6, TypeScript v5) │ └── README.md # Zuplo boilerplate (not project-specific) ├── mcp-server/ # MCP package for ChatGPT, Claude, and remote MCP clients │ ├── pyproject.toml # MCP package metadata + version │ ├── server.json # MCP registry/server metadata │ ├── uv.lock # Locked MCP runtime dependencies │ └── src/canyougrab_mcp/ │ ├── __init__.py │ └── server.py # stdio + streamable-http MCP entrypoint ├── .github/workflows/ │ ├── deploy.yml # Production deploy (on tag push v*) │ └── deploy-dev.yml # Dev deploy (on push to dev branch) ├── .claude/ │ └── launch.json # Local dev server config (port 9200) └── package.json # Root workspace config ``` ## API Endpoints ### Public API (API key auth: `Authorization: Bearer cyg_...`) | Method | Path | Description | |--------|------|-------------| | `POST` | `/api/check/bulk` | Check up to 100 domains. Long-polls until results ready (30s max). | | `GET` | `/api/account/usage` | Usage summary for the authenticated consumer. | | `GET` | `/api/account/quota-check` | Lightweight monthly + per-minute quota check. | | `GET` | `/health` | Health check (no auth). | ### Portal API (Auth0 JWT auth) | Method | Path | Description | |--------|------|-------------| | `POST` | `/api/keys` | Create new API key. | | `GET` | `/api/keys` | List user's API keys. | | `POST` | `/api/keys/{id}/rotate` | Rotate key (revoke old, create new). | | `DELETE` | `/api/keys/{id}` | Revoke (soft-delete) a key. | | `POST` | `/api/billing/checkout` | Create Stripe Checkout session. | | `POST` | `/api/billing/portal` | Create Stripe Customer Portal session. | | `POST` | `/api/billing/setup-card` | Create SetupIntent for Free+ card-on-file. | | `POST` | `/api/billing/confirm-free-plus` | Verify card fingerprint and upgrade to Free+. | | `GET` | `/api/billing/card-status` | Check if user has a card on file. | | `GET` | `/api/billing/usage/detailed` | Per-key usage breakdown for portal dashboard. | | `POST` | `/api/antifraud/turnstile/verify` | Verify Cloudflare Turnstile token. | | `POST` | `/api/antifraud/device/register` | Register device fingerprint (Fingerprint Pro). | | `GET` | `/api/antifraud/risk` | Get risk assessment for authenticated user. | | `POST` | `/api/antifraud/assess-signup` | Run multi-signal risk assessment at signup. | ### Internal / Webhook | Method | Path | Description | |--------|------|-------------| | `POST` | `/api/account/usage/detailed` | Multi-consumer usage breakdown. | | `POST` | `/api/stripe/webhook` | Stripe webhook receiver (signature-verified). | ## Core Request Flow ### Domain Availability Check ``` Client FastAPI Valkey Worker Unbound │ │ │ │ │ │ POST /api/check/bulk │ │ │ │ │ { domains: [...] } │ │ │ │ │─────────────────────────▶│ │ │ │ │ │── validate key ──────────│ │ │ │ │── check minute rate ────▶│ INCR ratelimit:id:min │ │ │ │── check monthly quota ──▶│ (PostgreSQL) │ │ │ │── record usage ─────────▶│ (PostgreSQL) │ │ │ │── create_job() ─────────▶│ HSET job:{uuid} │ │ │ │ │ RQ enqueue (with retry) │ │ │ │ │ │ │ │ │ │◀── RQ Worker dequeues ───│ │ │ │ │ │ │ │ │ │──── claim_job() ────────▶│ │ │ │ │ │── ThreadPool(10) │ │ │ │ │── check_domain_dns() │ │ │ │── NS query ───────▶│ │ │ │ │◀── NOERROR/NXDOMAIN│ │ │ │◀── complete_job() ───────│ │ │ │ │ │ │ │ │◀─ poll get_job_status() ─│ │ │ │ │ (0.3s interval, 30s │ │ │ │ │ max timeout) │ │ │ │◀─────────────────────────│ │ │ │ │ { results: [...] } │ │ │ │ ``` ### Billing / Subscription Flow ``` User → Pricing Page → Select Plan → Auth0 login → POST /api/billing/checkout (JWT auth) → Find/create Stripe customer (linked by auth0_sub metadata) → Stripe Checkout Session created → redirect to Stripe → User pays → Stripe fires webhook → POST /api/stripe/webhook (HMAC-SHA256 verified) → checkout.session.completed → fetch subscription → get price ID → Map price to plan → UPDATE api_keys SET plan, lookups_limit ``` ## Subscription Plans | Plan | Monthly Lookups | Per-Minute Rate Limit | Domains/Request | Price | |------|----------------|----------------------|-----------------|-------| | Free | 500 | 30/min | 30 | $0 | | Free+ | 10,000 | 100/min | 100 | $0 (card on file) | | Basic | 20,000 | 300/min | 100 | $10/mo | | Pro | 50,000 | 1,000/min | 100 | $20/mo | | Business | 300,000 | 3,000/min | 100 | $30/mo | ## Authentication ### API Key Auth (public API consumers) - Format: `Authorization: Bearer cyg_<token>` - Keys are SHA-256 hashed in the `api_keys` table (plaintext never stored) - Key prefix (`cyg_XXXXXXXX...`) is stored for display in the portal - Full key returned **only once** at creation time - Soft-delete on revocation (`revoked_at` timestamp, not physically deleted) ### JWT Auth (portal/dashboard) - Auth0 tenant: `dev-mqe5tavp6dr62e7u` - Custom domain: `auth.canyougrab.it` - Algorithm: RS256 with JWKS validation (1-hour cache) - Audience: `https://api.canyougrab.it` - Social logins: Google, Apple (Sign in with Apple) ## Domain Availability via DNS Domain availability is checked by querying a dedicated [Unbound recursive DNS resolver](https://github.com/ericismaking/canyougrab-unbound) running on a separate DigitalOcean droplet. The worker sends NS record queries over the VPC private network and interprets the response: | DNS Response | `available` | Meaning | |---|---|---| | NOERROR + NS records | `false` | Domain is registered and delegated | | NXDOMAIN | `true` | Domain not in zone — probably available | | NoAnswer (NOERROR, no NS) | `false` | Registered but parked/undelegated | | SERVFAIL / Timeout | `null` | Ambiguous — check failed | The Unbound resolver caches aggressively (7 days for registered domains, 5 minutes for NXDOMAIN) and queries TLD authoritative servers directly, avoiding public resolver rate limits. ## Database Schema PostgreSQL is used for authentication, usage tracking, and billing — not for domain lookups. ### PostgreSQL Tables | Table | Purpose | Key Columns | |-------|---------|-------------| | `api_keys` | User API keys | `id`, `user_sub`, `email_normalized`, `key_hash`, `key_prefix`, `plan`, `lookups_limit`, `revoked_at` | | `usage_log_daily` | Daily usage aggregates | `consumer`, `lookups`, `recorded_at` (DATE, unique per consumer+day) | | `usage_log_minute` | Per-minute usage aggregates | `consumer`, `lookups`, `minute_start` (TIMESTAMP, unique per consumer+minute) | | `card_fingerprints` | One free account per card | `user_sub`, `stripe_fingerprint` (unique pair) | | `device_fingerprints` | Device-based multi-account detection | `user_sub`, `visitor_id` (Fingerprint Pro) | | `account_risk` | Composite risk scoring | `user_sub` (unique), `risk_score`, `risk_signals` (JSONB) | ### Valkey (Redis) Data Structures | Key Pattern | Type | Purpose | TTL | |-------------|------|---------|-----| | `job:{uuid}` | Hash | Job status, domains, results | 1 hour | | `rq:queue:canyougrab-jobs` | List | RQ job queue (managed by RQ) | — | | `rq:job:{uuid}` | Hash | RQ job metadata (retries, status) | Configurable | | `ratelimit:{consumer}:{YYYYMMDDHHmm}` | String | Per-minute rate limit counter (INCR) | 60s | ## Infrastructure ### Servers | Environment | Domain | Purpose | |-------------|--------|---------| | Production | `api.canyougrab.it` | FastAPI + Worker | | Dev | `dev.canyougrab.it` | FastAPI + Worker | | Portal | `portal.canyougrab.it` | Zudoku static site | | Auth | `auth.canyougrab.it` | Auth0 custom domain | ### External Services | Service | Purpose | |---------|---------| | **DigitalOcean** | Droplets (API servers, Unbound resolver), Managed PostgreSQL, Managed Valkey | | **Cloudflare** | DNS, CDN, SSL for canyougrab.it zone, Turnstile bot prevention | | **Auth0** | User authentication, social login (Google + Apple), JWT issuance | | **Stripe** | Subscription billing, checkout, customer portal, webhooks | | **GitHub Actions** | CI/CD deployment pipelines | ## Deployment Pipelines ### Production Deploy **Trigger:** Push a git tag matching `v*` (e.g., `v1.0.5`) ``` git tag v1.0.5 && git push origin v1.0.5 ``` **Pipeline** (`.github/workflows/deploy.yml`): 1. GitHub Actions triggers on `v*` tag push 2. SSH into production server (`DEPLOY_HOST` secret) using `DEPLOY_SSH_KEY` 3. Bootstraps the target ref on the server (`git fetch` + `git checkout <tag>`) 4. Runs the repo-managed deploy script: `/opt/canyougrab-repo/scripts/deploy-host.sh` ### Dev Deploy **Trigger:** Push to the `dev` branch ``` git push origin dev ``` **Pipeline** (`.github/workflows/deploy-dev.yml`): 1. GitHub Actions triggers on push to `dev` 2. SSH into dev server (`DEV_DEPLOY_HOST` secret) using same `DEPLOY_SSH_KEY` 3. Bootstraps the `dev` ref on the server (`git fetch` + `git checkout dev`) 4. Runs the repo-managed deploy script: `/opt/canyougrab-repo/scripts/deploy-host.sh` ### Repo-Managed Host Deploy Script Both pipelines use a small inline SSH bootstrap to update `/opt/canyougrab-repo` to the target ref, then call `scripts/deploy-host.sh` from that checked-out revision. The repo-managed deploy script handles: - `pip install -r requirements.txt` for backend dependencies - `rsync` of backend, portal, and MCP source trees into their runtime directories - Reinstall or refresh the `mcp-server` package/runtime when the host also runs `canyougrab-mcp.service` - Restart of FastAPI (uvicorn), worker, and MCP services via systemd If `/mcp` is served by the same host as the API, backend-only deploys are not enough. The MCP service must be updated and restarted during the same deploy or the OAuth metadata and live MCP behavior can drift apart. An existing `/opt/deploy.sh` can still be kept as a manual compatibility shim if desired, but the automated pipeline should treat the repo copy of `scripts/deploy-host.sh` as the source of truth. ### Branching Strategy | Branch | Purpose | Deploys To | |--------|---------|------------| | `main` | Production releases | Tagged → `api.canyougrab.it` | | `dev` | Development/staging | Auto → `dev.canyougrab.it` | | Feature branches | In-progress work | No auto-deploy | ## Environment Variables ### Backend (required on servers) ```bash # DNS Resolver (Unbound on dedicated droplet) DNS_RESOLVER_HOSTNAME=unbound.canyougrab.internal # VPC internal hostname (resolved via socket.gethostbyname) DNS_RESOLVER_PORT=53 DNS_QUERY_TIMEOUT=5.0 # Per-query timeout in seconds # PostgreSQL (DigitalOcean Managed Database — auth, usage, billing only) POSTGRES_HOST= # DB cluster hostname POSTGRES_PORT=5432 POSTGRES_DB=canyougrab POSTGRES_USER=canyougrab POSTGRES_PASSWORD= # DB password POSTGRES_SSLMODE=require # Valkey / Redis (DigitalOcean Managed Database) VALKEY_HOST= # Valkey cluster hostname VALKEY_PORT=25061 VALKEY_USERNAME=default VALKEY_PASSWORD= # Valkey password # Auth0 AUTH0_DOMAIN=dev-mqe5tavp6dr62e7u.us.auth0.com AUTH0_AUDIENCE=https://api.canyougrab.it # Stripe STRIPE_SECRET_KEY= # sk_live_... (prod) or sk_test_... (dev) STRIPE_WEBHOOK_SECRET= # whsec_... (per-environment) STRIPE_PRICE_BASIC= # price_... (live price ID for Basic plan) STRIPE_PRICE_PRO= # price_... (live price ID for Pro plan) STRIPE_PRICE_BUSINESS= # price_... (live price ID for Business plan) # Cloudflare Turnstile (bot prevention on key creation) TURNSTILE_SECRET_KEY= # 0x4AAAA... (from Cloudflare dashboard) # Portal PORTAL_URL=https://portal.canyougrab.it # Worker BATCH_CONCURRENCY=10 # Thread pool size for domain checks VALKEY_QUEUE_NAME=canyougrab-jobs # RQ queue name (default: canyougrab-jobs) # Monitoring (optional — only if monitoring stack is installed) SLACK_ALERTS_WEBHOOK_URL= # Slack incoming webhook for Alertmanager RQ_METRICS_PORT=9122 # Prometheus metrics exporter port # Auto-scaler (optional — only on API host with autoscaler enabled) DO_API_TOKEN= # DigitalOcean API token DO_WORKER_SNAPSHOT_ID= # Snapshot ID for new worker droplets AUTOSCALER_MIN_WORKERS=1 AUTOSCALER_MAX_WORKERS=5 AUTOSCALER_SCALE_UP_THRESHOLD=50 AUTOSCALER_SCALE_DOWN_IDLE_MINUTES=10 ``` ### GitHub Actions Secrets | Secret | Purpose | |--------|---------| | `DEPLOY_HOST` | Production server IP | | `DEV_DEPLOY_HOST` | Dev server IP | | `DEPLOY_SSH_KEY` | SSH private key for both servers | ## Local Development ### Backend ```bash cd backend pip install -r requirements.txt # Start API server uvicorn app:app --reload --port 8000 # Start worker (separate terminal) python worker.py ``` Requires `DNS_RESOLVER_HOST` pointing to an Unbound instance (or use `8.8.8.8` for basic testing), plus PostgreSQL and Valkey/Redis (local or tunneled to managed instances). ### Portal ```bash cd portal/docs npm install npm run dev # Starts Zudoku dev server (via zuplo) ``` Or use the configured launch server: ```bash # From repo root npx zuplo dev # Starts on port 9200 ``` Portal dev server hardcodes `API_BASE` to `https://api.canyougrab.it` in `portal/docs/src/config.ts`. To develop against a local backend, change this to `http://localhost:8000`. ## SEO: .it TLD Geo-Targeting Requirements Because canyougrab.it uses a `.it` country-code TLD (Italy), all user-facing HTML (portal, docs) must include signals that the content targets English speakers: 1. **`<html lang="en">`** on every page. 2. **Hreflang tags** in `<head>`: ```html <link rel="alternate" hreflang="en-US" href="{page_url}" /> <link rel="alternate" hreflang="en" href="{page_url}" /> <link rel="alternate" hreflang="x-default" href="{page_url}" /> ``` 3. **Canonical tag** — Self-referencing `<link rel="canonical" href="{page_url}" />`. 4. **All content in English** — No Italian text on any page. This applies to the portal (`portal.canyougrab.it`), API docs, and any other publicly rendered HTML. ## Key Design Decisions - **Live DNS lookups**: Domain availability is checked via NS queries to a dedicated [Unbound recursive resolver](https://github.com/ericismaking/canyougrab-unbound). This gives real-time results (no 24-hour zone file lag), works for any TLD automatically, and avoids the operational complexity of daily batch zone file loading. Unbound caches aggressively (7 days for registered domains) so repeated queries are ~1ms. - **Nullable availability**: DNS has more failure modes than a database lookup. When Unbound returns SERVFAIL or times out, the API returns `available: null` instead of a potentially dangerous false positive. Consumers should treat `null` as "could not determine." - **Job queue for bulk checks**: The bulk endpoint uses RQ (Redis Queue) backed by Valkey to dispatch jobs with automatic retries (2 retries, 5s/30s backoff) and failed-job tracking. Workers parallelize DNS queries across a thread pool. - **Long-polling**: The bulk check endpoint holds the HTTP connection open (polling Valkey every 0.3s, up to 45s) rather than requiring clients to implement polling. Job results are stored in custom Valkey hashes (not RQ's built-in result storage) so the poll logic is independent of the queue framework. - **API keys with SHA-256 hashing**: Keys are hashed before storage (like passwords), so a database breach doesn't expose raw keys. - **Stripe metadata linking**: Stripe customers are linked to Auth0 users via `auth0_sub` metadata on the Stripe customer object, avoiding a separate mapping table. - **Usage double-tracking**: Both daily (`usage_log_daily`) and per-minute (`usage_log_minute`) usage are recorded for monthly quota and per-minute rate limiting respectively. - **No Zuplo gateway policies**: The project originally used Zuplo as an API gateway but migrated to direct FastAPI (commit `3abef70`). The Zuplo portal/docs framework is still used for the developer portal. ## Processes on Servers ### Core services (every host) 1. **FastAPI** (uvicorn): Serves all HTTP endpoints on port 8000 2. **RQ Worker** (worker.py): Processes domain check jobs from the RQ queue 3. **MCP server** (if `/mcp` is hosted on this server): Serves remote MCP clients ### Monitoring stack (API host, optional) 4. **Prometheus** (localhost:9090): Scrapes metrics, evaluates alert rules 5. **Alertmanager** (localhost:9093): Routes alerts to Slack 6. **redis_exporter** (localhost:9121): Exports Valkey metrics to Prometheus 7. **RQ metrics exporter** (localhost:9122): Exports queue depth, worker count, failed jobs 8. **Grafana** (localhost:3000, nginx-proxied): Dashboards for queue health 9. **Auto-scaler** (optional): Scales worker droplets up/down via DO API Install the monitoring stack with: `sudo bash scripts/setup-monitoring.sh` All active services on the host must be managed via systemd and restarted during deployments. All automated deploys should route through `scripts/deploy-host.sh`, which is responsible for restarting whichever of these services are installed on the host.