subio-mcp

<div align="center"> # Subio Scrape **Read what people are actually posting — in a browser that is never signed in.** An MCP server that opens a browser on your own machine, reads the public posts a signed-out visitor is shown on X, LinkedIn and Hacker News, and hands them back as structured data your agent can reason about. [Install](INSTALL.md) · [Tools](#tools) · [What signed-out access buys you](#what-signed-out-access-actually-buys-you) · [Blocks](#blocks-and-what-to-do-about-them) · [Security](SECURITY.md) </div> --- ## What this is Ask your agent *"what have @naval and @paulg been posting about onboarding this week?"* and it opens a real browser window, reads those two timelines, and answers with the posts, their engagement counts and their links. Everything happens locally. There is no API key, no account, no cloud service and no data leaving your machine except the page requests themselves. **It never signs in.** Session cookies are deleted from its browser profile before every run. It reads exactly what a logged-out visitor is shown — nothing behind a login, nothing behind a challenge. It does not solve CAPTCHAs, does not disguise that it is automated, and does not retry a refusal. That constraint is the point, and it shapes what the tool can and cannot do. The sections below are honest about both. ## Quick start ```bash claude mcp add subio-scrape -- npx -y github:alijancb/subio-mcp ``` Then ask, in plain language: > собери, что @naval и @paulg писали за неделю про онбординг Using a different agent — Codex, Cursor, Windsurf, Cline, Zed, VS Code? See **[INSTALL.md](INSTALL.md)**, which has a one-page guide per client that your agent can follow for you. For Claude Code there is also a plugin, which adds a `/scrape` command and a workflow skill on top of the tools: ```bash git clone https://github.com/alijancb/subio-mcp.git ~/subio-mcp && cd ~/subio-mcp && npm install ``` ```bash claude plugin marketplace add ~/subio-mcp && claude plugin install subio-scrape@subio ``` ### Requirements - **Node.js 20 or newer.** - **A browser.** It uses Chrome or Edge if you have one. If not, run `npx playwright install chromium` once. - **A desktop session.** The window has to be visible — see [below](#what-signed-out-access-actually-buys-you) for why. This does not run on a headless server. ## Tools | Tool | What it reads | | --- | --- | | `x_account_posts` | Posts from named X accounts, with exact view / like / repost / reply counts. `match` filters them by keyword or hashtag | | `x_account_profile` | Bio, followers, post count for one account | | `linkedin_page` | A public company or person page, from its JSON-LD | | `linkedin_company_posts` | A company's posts, when LinkedIn serves them signed out | | `hackernews_search` | Hacker News stories and comments on a topic | | `browser_status` | Which browser will be driven, from where, how fast, and what happens on a block | | `browser_close` | Close the window early | Every tool returns the same item shape, so results from four sources merge into one report: ```json { "source": "x", "externalId": "1960...", "url": "https://x.com/naval/status/1960...", "author": "naval", "text": "…", "publishedAt": "2026-08-27T02:22:11.000Z", "engagement": { "views": 8499257, "likes": 14021, "replies": 689, "reposts": 1367 }, "via": "@naval" } ``` ### Results tell you what actually happened Collector tools answer with `{ items, notes, partial }`, and **`notes` is not decoration**. A source that was blocked, walled or misspelled returns zero items — and so does a genuinely quiet week. Without the notes, an agent reports the second for both: ``` notes: [ "@naval: 2 items", "@paulg: 2 items", "@definitely-not-real: FAILED — not-found: X sent @definitely-not-real to its signup page instead of a profile. Either the handle does not exist, or X declined to serve it signed out." ] partial: true ``` ## What signed-out access actually buys you Worth reading before you plan anything around this. - **X serves account timelines to a signed-out browser, and serves search to nobody.** `x.com/search`, hashtags and explore all redirect to a login wall. There is deliberately no search tool here, because there is no signed-out search. A topic on X is answered by reading the accounts that discuss it and filtering with `match`. - **X answers a headless browser with a bare 403 and an empty document.** The window is visible by default for that reason, and not as a preference. `SUBIO_MCP_HEADLESS=1` exists and mostly does not work. - **A missing X handle is not a 404.** X answers 200 and redirects into its signup flow, which reads exactly like an empty timeline. The final URL is what gives it away, and is checked. - **LinkedIn gives the least and takes it away fastest.** A public company or person page renders, usually with JSON-LD on it. After a handful of pages from one client it becomes a sign-in wall — served with a 200 status and carrying an `og:title` of its own, so a wall parses as a page named "Sign Up | LinkedIn" unless you check for the JSON-LD. Expect to be walled within a session. - **Hacker News has a real public API** and is the one source with no ceiling. - **Reddit was removed rather than worked around.** It closed `/*.json` to clients without a session, and its public RSS feed carries neither score nor comment count, so what remained could not be ranked against anything else. ## Blocks, and what to do about them **Nothing here retries a refusal.** A 403, a 429, a challenge or a sign-in wall is the site answering, and asking again is how being throttled becomes being banned. The first block also ends the whole run: the accounts that were not read are listed in the notes, with an instruction not to retry them now. If you are getting blocked, what helps, in order: 1. **Read fewer accounts per run.** This is the whole answer most of the time. 2. **Widen the gap** — `SUBIO_MCP_MIN_INTERVAL_MS=10000`. Reads are already jittered by ±25% so they do not arrive on a metronome. 3. **Wait.** A signed-out quota refills on its own, and nothing you change locally makes it refill faster. What does **not** help is a fresh browser per account. Blocks are counted against your IP address, and a new browser profile does not change your address — a hundred brand-new fingerprints from one address is a stronger bot signal than one returning visitor. `SUBIO_MCP_ISOLATE=1` exists if you want to try anyway: it opens a throwaway browser on a throwaway profile per account and deletes it afterwards. It costs a browser launch per account, and it resets the cookie-scoped guest quota — which is working around a limit rather than respecting one, and is the opposite of every other decision here. Off by default. ## One window, one tab A scan does not open a browser per account. The server keeps a single browser with a single tab and drives every account, company and page through it, one load at a time — lighter on your machine than a tab per source, and less conspicuous to the sites being read. The window closes itself five minutes after the last read, and immediately if the client goes away, so nothing is ever left running with nobody driving it. ## Configuration Everything has a working default. Set these only if you need to. | Variable | Default | | | --- | --- | --- | | `SUBIO_MCP_PROFILE` | `~/.subio-mcp/browser-profile` | Browser profile, kept apart from your own Chrome | | `SUBIO_MCP_CACHE` | `~/.subio-mcp/cache` | Read cache | | `SUBIO_MCP_CACHE_TTL_MS` | `1800000` | How long a read counts as fresh | | `SUBIO_MCP_MIN_INTERVAL_MS` | `4000` | Minimum gap between two loads of one host | | `SUBIO_MCP_IDLE_MS` | `300000` | Close the window after this long idle. `0` never | | `SUBIO_MCP_HEADLESS` | unset | `1` runs without a window. X will refuse | | `SUBIO_MCP_BROWSER` | unset | Force `chrome` or `msedge`, or leave it to pick | | `SUBIO_MCP_ISOLATE` | unset | `1` opens a throwaway browser per account. Read [above](#blocks-and-what-to-do-about-them) first | Session cookies (`auth_token`, `ct0`, `li_at`, …) are deleted from the profile before every run, so the promise on the tin holds even if you once opened that profile by hand and signed in. ## Development ```bash npm install ``` ```bash npm run check && npm run build && node scripts/probe.mjs ``` `scripts/probe.mjs` starts the built server over stdio the way a client does and exercises the tools that need no browser. It catches a broken build, a bad tool schema, or anything accidentally written to stdout — all of which look like a silent hang from inside an agent. `scripts/live.mjs` is the manual counterpart that reads the real sites; it is not in CI, because CI should not scrape. Where things live: | Path | | | --- | --- | | `src/server.ts` | The tool surface | | `src/browser/session.ts` | One window, pacing, signing out, idle close | | `src/collect/` | One module per source | | `src/parse/x.ts` | Everything known about X's markup — the only file to edit when X changes | | `src/parse/linkedin.ts` | The same for LinkedIn, parsed as HTML in node | | `skills/`, `commands/` | The Claude Code plugin's workflow | The X parser runs inside the page. The LinkedIn parser deliberately does not: LinkedIn redirects on entry, and anything evaluated in the document loses its execution context when it does. It reads the HTML as a string instead, which makes it immune to that rather than lucky about it — and testable from a fixture. Contributions welcome: see [CONTRIBUTING.md](CONTRIBUTING.md). ## History This repository previously held a different Subio MCP server — one that brought a live Subio workspace into MCP clients over the Subio API. It was replaced rather than extended, because a local signed-out scraper and an API client share a name and nothing else. That work is preserved on the [`legacy/subio-api-server`](https://github.com/alijancb/subio-mcp/tree/legacy/subio-api-server) branch and under the `v0.1.0` tag. ## License [MIT](LICENSE).