social-media-toolkit

# Tool List Convert public links from TikTok, Xiaohongshu, Bilibili, and YouTube into text, metadata, videos, covers, images, and public comments; YouTube also supports MD/SRT/JSON timelines with timestamps that link back to the original video. Provides: - Python SDK - `socialkit` CLI - MCP Server Current version: `0.3.0`. ## One-Line Installation Requires Node.js 18+ (with `npm` / `npx`) on the machine: ```bash npx -y github:JNHFlow21/social-media-toolkit ``` Use directly after installation: ```bash socialkit doctor ``` This command will: 1. Use the existing `uv` on the machine; if not, install `uv` via [Astral's official installer](https://docs.astral.sh/uv/getting-started/installation/). 2. Install Python packages and `yt-dlp-ejs` in an isolated `uv tool` environment without polluting business projects. 3. Install `socialkit` and `social-media-toolkit-mcp` commands. 4. No need to clone the repository, no need for Agent Switch, and no creation of project `.env`. Re-run the same command to overwrite and update the installation. Uninstall: ```bash uv tool uninstall social-media-toolkit ``` If the current terminal cannot find `socialkit` after installation, reopen the terminal. The installer will add the `uv tool` command directory to the `PATH` of subsequent terminals. <details> <summary>Manual Installation by Developers</summary> ```bash git clone https://github.com/JNHFlow21/social-media-toolkit.git cd social-media-toolkit uv sync uv run socialkit doctor ``` </details> ## Standalone Operation: No Dependency on Agent Switch This is a standard Python open-source project. Any user can directly install and run it **without Agent Switch and without relying on the author's local workspace, Skill, or private configuration**. - Metadata reading, public media downloading, public comments, and Bilibili/YouTube text extraction with native subtitles do not require Volcano API Key. - GetNote is an optional first text source; if not installed or logged in, it will automatically continue with platform-native subtitles or Volcano ASR. - Only when entering Volcano Cloud ASR is the standard process environment variable `VOLCENGINE_ASR_API_KEY` required. - Agent Switch is just an optional secret-manager adaptation on the maintainer's machine; the code always prioritizes reading standard environment variables, and it can run normally even if Agent Switch is not found. - The project does not read the repository `.env` and does not depend on any machine-specific paths. ## Two Enhancement Capabilities with Optional Configuration ### 1. GetNote: Prioritized Acquisition of Existing Text ```bash npm install -g @getnote/cli getnote auth login ``` - Official documentation: [@getnote/cli](https://www.npmjs.com/package/@getnote/cli) - Source code: [iswalle/getnote-cli](https://github.com/iswalle/getnote-cli) - Cost: GetNote CLI itself is open-source, but its OpenAPI/Skill currently requires a Brain Member. - Credentials: managed by GetNote itself, do not copy to this project. If GetNote is not installed, logged in, or processed failed, the tool will continue to check platform-native subtitles; if the video still has no subtitles, it will enter Volcano Cloud ASR. ### Execution Rules by Default Calling `text`, enabling text `capture`, or requiring a complete link test means executing the currently configured text link: GetNote → platform-native subtitles → Volcano Cloud ASR. The tool will not pop up a secondary authorization confirmation. - GetNote may save the link to the user's own GetNote account. - Volcano ASR may consume resource packages or generate pay-per-use fees. - The execution result must explain which path was actually hit and whether Volcano ASR was called. - If you only want to read metadata and not trigger GetNote/ASR, use `inspect`. ### 2. Volcano Engine Cloud ASR: The Only Speech-to-Text Service This project only recognizes one secret name: ```text VOLCENGINE_ASR_API_KEY ``` - Interface documentation: [Large Model Audio File Recognition Express API](https://www.volcengine.com/docs/6561/1631584?lang=en) - Product page: [Baidu Voice Recognition](https://www.volcengine.com/product/asr) - Cost: Cloud services may generate pay-per-use fees or consume resource packages; whether there is a trial quota depends on the current display of Volcano Engine Console. - Usage in this project: `volc.bigasr.auc_turbo`, consistent with the Volcano Cloud transcription path of `cloud-transcript` Skill. Please inject through the operating system, MCP client, or Agent's secret manager, **do not create project `.env`**. Ordinary shell users can put the Key into the current process environment through hidden input, which does not require Agent Switch and will not write the Key into command history: ```bash read -s VOLCENGINE_ASR_API_KEY export VOLCENGINE_ASR_API_KEY socialkit doctor ``` After use, you can execute `unset VOLCENGINE_ASR_API_KEY`. MCP users should inject the same-named variable through the client's secret store or secure environment. If the machine already uses Agent Switch, you can also choose to write it through hidden input; this is just an optional integration: ```bash read -s VOLCENGINE_ASR_API_KEY printf %s "$VOLCENGINE_ASR_API_KEY" | agent-switch secret set --stdin VOLCENGINE_ASR_API_KEY unset VOLCENGINE_ASR_API_KEY ``` Cloud ASR will directly return specific errors if it fails. **It will not switch to local Whisper, nor will it switch to other cloud vendors, nor will it let users continue waiting without prompts.** ## Supported Features | Feature | Effect | Requirement | Possibly Paid | |---|---|---|---| | Unified Metadata | Title, author, release time, interaction metrics, media address | Python dependency; YouTube requires `yt-dlp` | No | | Get Text | GetNote → native subtitles → Volcano Cloud ASR | GetNote; videos without subtitles require `VOLCENGINE_ASR_API_KEY` and `ffmpeg` | GetNote membership, Volcano ASR possibly paid | | YouTube Timeline Transcript | Human subtitles cue → automatic subtitles cue → Volcano Cloud ASR; output MD/SRT/JSON | `yt-dlp`; videos without subtitles require `VOLCENGINE_ASR_API_KEY` and `ffmpeg` | No subtitles, Volcano ASR possibly paid | | Download Video | Download complete video and generate SHA-256 list | TikTok/Xiaohongshu use public CDN; Bilibili/YouTube require `yt-dlp` and `ffmpeg` | Tool itself free | | Download Cover/Images | Save cover and image text | Public link | Tool itself free | | Get Comments | Get primary comment samples returned by TikTok's public interface | No login or Cookie required | No | | Complete Data Package | Merge metadata, text, comments, and download on demand | Depends on enabled capabilities | Depends on GetNote/Volcano ASR | | Environment Check | Check dependencies, login status, and secret name | None | No | ### Platform Matrix | Platform | Metadata | Text | Video | Cover/Images | Public Comments | |---|---:|---:|---:|---:|---:| | TikTok | ✅ | GetNote → Volcano ASR | ✅ | ✅, including public image collection | ✅ Requestable 1–100 primary comment samples; source station may return less | | Xiaohongshu | ✅ | GetNote → text content / Volcano ASR | ✅ | ✅ | — | | Bilibili | ✅ | GetNote → native subtitles → Volcano ASR | ✅ | ✅ | — | | YouTube | ✅ | GetNote → human subtitles → automatic subtitles → Volcano ASR | ✅ | ✅ | — | The `likes` / `recent` sorting of comments only applies to the publicly returned samples of the interface, not representing the global ranking of the platform's entire comment. `--limit`, MCP `limit`, and bundle `comment_limit` accept `1..100`, indicating how many are returned at most. The source station gives how much, and returns how much: request 20, the source station gives 19, and returns 19, without turning the page, supplementing. ## Unique Text Processing Path ```mermaid flowchart LR U["Public Link"] --> G["GetNote Original Content"] G -->|"No available text"| N["Platform Native Subtitles"] N -->|"Video still has no subtitles"| V["Volcano Engine Cloud ASR"] V -->|"Failed"| E["Directly Return Failure Reason"] ``` No paths: - Local Whisper / Local ASR - Other cloud ASR providers - OCR / Vision model bottoming - LLM cleaning or rewriting - Browser, CDP, Playwright, login state crawling - Automatically generating `script.md` / `info.json` The text result will not download persistent media. The audio required for Volcano transcription exists only in a temporary directory: download remote media, convert it to single-channel 16kHz MP3 with `ffmpeg`, call the cloud, and then delete the temporary directory. ### YouTube Timeline Transcript The ordinary `text` pursues "getting readable canonical text", so GetNote can prioritize hitting. The timeline mode pursues "each sentence can return to the original video", so it is another deterministic link: ```text YouTube human subtitles cue → YouTube automatic subtitles cue → Volcano Cloud ASR utterance/word timeline ``` GetNote text without timestamps will not truncate the timeline mode. This mode requires explicit output directory, only persist the requested transcript file; the video/audio used for ASR is always in a temporary directory and deleted after the call. Default output: ```text youtube-<video-id>-transcript.md youtube-<video-id>-transcript.srt youtube-<video-id>-transcript.timeline.json ``` JSON saves normalized `segments`, and Volcano's response contains word-level boundaries and also saves redacted `words`. The return list will clearly record provider, route, timing precision, segment count, verification hash, and whether the temporary media is deleted, without saving the original response of the cloud or YouTube's temporary signed media URL. ## CLI ### Check Installation Status ```bash socialkit doctor ``` Output only includes: - Dependency installation - GetNote login status - `VOLCENGINE_ASR_API_KEY` configuration - Official configuration link - Missing items No secret values are output. ### Parse Metadata, No Download ```bash socialkit inspect "SHARE_URL" ``` ### Get Text ```bash socialkit text "SHARE_URL" ``` ### Get YouTube Timeline Transcript ```bash socialkit text "YOUTUBE_URL" \ --timed \ --output "/absolute/path/to/transcripts" \ --outputs md,srt,json ``` `--timed` currently only accepts a single YouTube video URL; even if the link has a playlist parameter, it will not crawl the entire playlist. ### Get TikTok Public Comments ```bash socialkit comments "DOUYIN_URL" --sort likes --limit 10 socialkit comments "DOUYIN_URL" --sort likes --limit 20 socialkit comments "DOUYIN_URL" --sort likes --limit 50 socialkit comments "DOUYIN_URL" --sort recent --limit 100 ``` ### Explicitly Download Media ```bash socialkit download "SHARE_URL" \ --include video,cover,images \ --output "/absolute/path/to/output" ``` ### Generate Complete Data Package ```bash socialkit capture "SHARE_URL" \ --comments \ --output "/absolute/path/to/output" ``` Do not pass `--output` to avoid persistent media download. ## Python SDK ```python from social_media_toolkit import SocialMediaToolkit toolkit = SocialMediaToolkit() metadata = toolkit.inspect("SHARE_URL") text = toolkit.get_text("SHARE_URL") timed = toolkit.get_text( "YOUTUBE_URL", timed=True, output_dir="/absolute/path/to/transcripts", outputs="md,srt,json", ) comments = toolkit.get_comments("DOUYIN_URL", sort_by="likes", limit=10) bundle = toolkit.capture( "SHARE_URL", include_text=True, include_comments=False, ) ``` No `asr_provider`, `asr_model`, or local fallback parameters to avoid multiple behaviors for the same link. ## MCP Server Start: ```bash social-media-toolkit-mcp ``` stdio MCP example: ```json { "mcpServers": { "social-media-toolkit": { "command": "social-media-toolkit-mcp" } } } ``` Do not write secrets directly into this JSON. Inject through the client's secret store or secure process environment. If the MCP client does not inherit the `PATH` of the shell, run `uv tool dir --bin`, and then replace the `command` with the absolute path of `social-media-toolkit-mcp` in that directory. Only keep six MCP Tools: | MCP Tool | Effect | |---|---| | `social_inspect` | Return unified `PostBundle`, no download, no transcription | | `social_get_text` | Default execution of canonical text path; `timed=true` writes YouTube MD/SRT/JSON timeline transcript | | `social_get_comments` | Get publicly supported comment samples | | `social_download` | Explicitly download media and return verification list | | `social_capture_bundle` | Merge data and download on demand | | `social_doctor` | Check dependencies and configuration, only show secret name | MCP, CLI, and Python SDK all call the same `SocialMediaToolkit`, and there is no second compatible scheduler. ## PostBundle ```json { "schema_version": "1.0", "source": {}, "post": {}, "author": {}, "media": { "videos": [], "covers": [], "images": [], "audio": [] }, "metrics": {}, "content": {}, "comments": {}, "provenance": {} } ``` ## Development and Verification ```bash uv sync uv run python -m unittest discover -s tests uv run python -m compileall social_media_toolkit social_post_extractor_mcp uv build git diff --check ``` Tests must use synthetic fixtures, and no Cookies, Tokens, real user data, or private content should be submitted. ## Boundary - Only process public URLs that users have access to, without bypassing access control. - The platform may change the public page or interface; return the source and reason for failure, do not forge success. - When downloading, saving, and re-releasing content, users must comply with platform terms, copyright, and local laws. - Automatic upload and automatic release belong to another product with account side effects, not in this toolkit. Architecture and capability boundary: - [docs/architecture.md](docs/architecture.md) - [docs/capabilities.md](docs/capabilities.md) - [CHANGELOG.md](CHANGELOG.md) ## License [Apache-2.0](LICENSE)