Content
# SciTeX Scholar (`scitex-scholar`)
<p align="center">
<a href="https://scitex.ai">
<img src="docs/scitex-logo-blue-cropped.png" alt="SciTeX" width="400">
</a>
</p>
<p align="center"><b>Scientific paper search, enrichment, PDF download, and library management for reproducible research.</b></p>
<p align="center">
<a href="https://scitex-scholar.readthedocs.io/">Full Documentation</a> · <code>uv pip install scitex-scholar[all]</code>
</p>
<!-- scitex-badges:start -->
<p align="center">
<a href="https://pypi.org/project/scitex-scholar/"><img src="https://img.shields.io/pypi/v/scitex-scholar?label=pypi" alt="pypi"></a>
<a href="https://pypi.org/project/scitex-scholar/"><img src="https://img.shields.io/pypi/pyversions/scitex-scholar?label=python" alt="python"></a>
<a href="https://github.com/ywatanabe1989/scitex-scholar/actions/workflows/rtd-sphinx-build-on-ubuntu-latest.yml"><img src="https://img.shields.io/github/actions/workflow/status/ywatanabe1989/scitex-scholar/rtd-sphinx-build-on-ubuntu-latest.yml?branch=develop&label=docs" alt="docs"></a>
</p>
<p align="center">
<a href="https://github.com/ywatanabe1989/scitex-scholar/actions/workflows/pytest-matrix-on-ubuntu-py3-11-3-12-3-13.yml"><img src="https://img.shields.io/github/actions/workflow/status/ywatanabe1989/scitex-scholar/pytest-matrix-on-ubuntu-py3-11-3-12-3-13.yml?branch=develop&label=tests" alt="tests"></a>
<a href="https://codecov.io/gh/ywatanabe1989/scitex-scholar"><img src="https://img.shields.io/codecov/c/github/ywatanabe1989/scitex-scholar/develop?label=cov" alt="cov"></a>
<a href="https://www.gnu.org/licenses/agpl-3.0"><img src="https://img.shields.io/badge/license-AGPL_v3-blue.svg" alt="License: AGPL v3"></a>
</p>
<!-- scitex-badges:end -->
---
## Problem and Solution
| # | Problem | Solution |
|---|---------|----------|
| 1 | **Literature search is balkanized** -- CrossRef / OpenAlex / Semantic Scholar / arXiv / PubMed each have different APIs, rate limits, auth | **Unified search** -- `scitex scholar search "topic"` federates across all, deduplicates by DOI, returns ranked results |
| 2 | **BibTeX from the wild is missing abstracts / DOIs / impact factors** -- manuscript prep wastes hours | **`scitex scholar bibtex` enrichment** -- one call resolves DOIs, fetches abstracts, adds impact factors, normalizes formatting |
| 3 | **Paywalled PDFs require institutional login per journal** -- manual login-download-rename is the bottleneck | **Browser-automation + OAuth** -- persistent Chrome profile with stealth; `scitex scholar fetch 10.1038/...` grabs the PDF end-to-end |
## Problem
Literature management spans many tools and APIs: searching databases, resolving DOIs, downloading PDFs through institutional access, enriching BibTeX metadata, and keeping a reproducible, deduplicated library. Each step speaks a different library, auth flow, and data format.
## Solution
`scitex-scholar` provides a unified workflow:
- **Search** across CrossRef, Semantic Scholar, PubMed, arXiv, and OpenAlex
- **Resolve** DOIs from titles; enrich BibTeX with abstracts, citation counts, impact factors (JCR 2024), PMIDs, and arXiv IDs
- **Download** PDFs through institutional access (OpenAthens / SSO) with Playwright browser automation
- **Organize** papers in a MASTER-hash library with per-project symlinks at `~/.scitex/scholar/library/`. One-button maintenance via `library refresh` (reconcile → regenerate readable names → optional rsync to remote hosts)
- **Highlight** each sentence of a PDF by rhetorical role — claim, method, limitation, supportive citation, contradicting citation — via Claude
- **Automate** the same operations from the CLI, a Python API, or the SciTeX MCP server
## Installation
```bash
pip install scitex-scholar # core
pip install "scitex-scholar[pdf]" # PDF text extraction
pip install "scitex-scholar[mcp]" # MCP server deps (fastmcp)
pip install "scitex-scholar[browser]" # Playwright automation
pip install "scitex-scholar[all]" # everything
```
## 4 Interfaces
<details open>
<summary><strong>Python API</strong></summary>
<br>
```python
from scitex_scholar import Scholar, Paper, Papers, apply_filters, to_bibtex
scholar = Scholar()
papers = scholar.search("deep learning EEG", year_min=2020) # auto-enriched
papers.save("results.bib")
# Filter + export
top = apply_filters(papers, min_citations=50, min_impact_factor=5.0)
print(to_bibtex(top))
```
</details>
<details>
<summary><strong>CLI</strong></summary>
<br>
Entry point: `scitex-scholar <subcommand>` (Click-based).
```bash
# Discover everything
scitex-scholar --help
scitex-scholar --help-recursive # full overview, every leaf
scitex-scholar --version # or -V
# Paper(s)
scitex-scholar paper fetch --doi 10.1038/nature12373 --project demo
scitex-scholar paper fetch-batch --dois 10.1038/xxx --dois 10.1126/yyy --project demo --num-workers 4
# BibTeX file
scitex-scholar bibtex import --bibtex refs.bib --project demo --output refs.enriched.bib
# PDF post-processing
scitex-scholar pdf highlight paper.pdf
# Library — daily workflow
scitex-scholar library list # all projects
scitex-scholar library list neurovista # one project, per-paper
scitex-scholar library open-urls neurovista --watch # browser + auto-import
scitex-scholar library refresh neurovista # one-button maintenance
scitex-scholar library refresh neurovista --sync spartan # +rsync push (repeatable)
# Library — manual PDF import (when DBs haven't indexed yet, or no auto-download)
scitex-scholar paper fetch --doi 10.1002/epi.70076 \
--pdf-main ~/Downloads/Liu_2026.pdf \
--pdf-supple ~/Downloads/MOESM1_ESM.pdf \
--attachment ~/Downloads/dataset.csv \
--project neurovista
# Library — layout / share / integrity
scitex-scholar library bind neurovista ~/proj/neurovista # one symlink, no data move
scitex-scholar library export neurovista --format bibtex
scitex-scholar library audit-files --project neurovista
scitex-scholar library db build --dry-run
scitex-scholar library db audit --json
# Auth (institutional SSO — OpenAthens / EZProxy / Shibboleth)
scitex-scholar auth status # exit 0 if any session valid, 1 otherwise
scitex-scholar auth login # trigger SSO flow now (debug-friendly)
scitex-scholar auth logout -y # clear cached cookies (--yes required)
scitex-scholar auth refresh # logout + login
# MCP server
scitex-scholar mcp start
scitex-scholar mcp list-tools --json
# Shell completion
scitex-scholar install-shell-completion --shell bash
scitex-scholar print-shell-completion --shell bash
# Skills + Python API introspection
scitex-scholar skills list
scitex-scholar list-python-apis -v
```
### Debugging the SSO automator
Every browser-automation step writes a screenshot + HTML pair to
`~/.scitex/scholar/cache/engine/screenshots/` and
`~/.scitex/browser/cache/debug/`. When a selector breaks (e.g. an
Okta UI refresh), `ls -lt` the artifact dirs to get a frame-by-frame
storyboard — the screenshot shows what was rendered, the HTML
shows what the locator was reasoning over. See
`_skills/scitex-browser/11_debugging-visuals.md` for the full pattern.
Mutating verbs accept `--dry-run` and `-y/--yes`. Read verbs support `--json`.
Common paper/bibtex flags: `--browser-mode {stealth,interactive}`, `--chrome-profile NAME`, `--force`.
> **Migration (1.3.0):** the CLI moved to noun-verb groups. Old top-level commands
> (`single`, `parallel`, top-level `bibtex --bibtex`, `highlight`, `link-project-tree`,
> `materialize`, `dematerialize`, `db`) still work but emit a `DeprecationWarning`
> and will be removed in 1.4.0. See [CHANGELOG.md](CHANGELOG.md) for the full
> migration table.
</details>
<details>
<summary><strong>MCP Server</strong></summary>
<br>
The package ships MCP tool handlers consumed by the unified `scitex serve`
server (tools prefixed `scholar_*`). A standalone server at
`scitex_scholar.mcp_server` is still shipped but deprecated. See the
[Skills documentation](https://scitex-scholar.readthedocs.io/en/latest/skills.html)
for the full tool list.
</details>
<details>
<summary><strong>Skills</strong></summary>
<br>
Agent skill pages are published at
[scitex-scholar.readthedocs.io/en/latest/skills.html](https://scitex-scholar.readthedocs.io/en/latest/skills.html).
The `semantic-highlight` skill documents the PDF-highlighting workflow.
</details>
## Core API
| Symbol | Purpose |
|-----------------|---------|
| `Scholar` | Main search / enrich / download / save interface |
| `Paper`, `Papers` | Single paper / collection with export methods |
| `ScholarConfig` | Paths, API keys, auto-enrich toggle, browser settings |
| `apply_filters` | Filter a `Papers` collection |
| `to_bibtex`, `to_ris`, `to_endnote`, `to_text_citation` | Export formats |
| `generate_cite_key`, `make_citation_key` | Deterministic BibTeX keys |
| `CitationGraphBuilder`, `plot_citation_graph` | Optional citation graph |
| `pdf_highlight.highlight_pdf` | Overlay semantic highlights on a PDF |
Sources: `core/`, `search_engines/`, `metadata_engines/`, `pdf_download/`, `pipelines/`, `browser/`, `auth/`, `storage/`, `pdf_highlight/`, `_mcp/`.
## Semantic PDF Highlighting
Overlay colour-coded highlights on a PDF that separate what the paper **claims** from its
**methods**, **self-admitted limitations**, and stance toward related work. Highlights are
standard PDF annotation objects placed on a copy of the source — the original bytes are unchanged
and any viewer can show or strip them.
| colour | category | meaning |
|---|---|---|
| green | `focal_claim` | what the paper clarifies, suggests, demonstrates |
| purple | `focal_method` | novel method, model, cohort, or analysis |
| red | `focal_limitation` | self-admitted caveat or threat to validity |
| blue | `related_supportive` | prior work whose finding supports the paper |
| orange | `related_contradictive` | prior work whose finding contradicts the paper |
A compact colour legend + signature (model name, timestamp) is stamped in the lower-right corner
of the last page. See [docs](https://scitex-scholar.readthedocs.io/en/latest/semantic_highlight.html)
for full details.
```bash
export ANTHROPIC_API_KEY=sk-ant-...
scitex-scholar pdf highlight paper.pdf # sentence-level, Haiku, writes paper.highlighted.pdf
scitex-scholar pdf highlight paper.pdf --stub # offline keyword heuristic (no API calls)
```
```python
from scitex_scholar.pdf_highlight import highlight_pdf
result = highlight_pdf("paper.pdf", output_path="paper.highlighted.pdf")
print(result.counts(), result.annotations_added)
```
Also exposed as the `scholar_highlight_pdf` MCP tool (unified `scitex serve` server) and as the
`semantic-highlight` agent skill (see
[skills documentation](https://scitex-scholar.readthedocs.io/en/latest/skills.html)).
## Storage layout
```
~/.scitex/scholar/library/
├── MASTER/<HASH>/ # Canonical per-paper storage (metadata.json + PDF)
└── <project>/<human-label> -> ../MASTER/<HASH>
```
Cache and auth state live under `~/.scitex/scholar/cache/` (URL resolver, Chrome profiles, OpenAthens cookies). Override with `SCITEX_DIR`.
## Architecture
```
scitex_scholar/
├── __init__.py ← public API (Scholar, Paper, Papers, apply_filters, to_bibtex)
├── _cli_main.py ← `scitex-scholar` Click entry point (noun-verb groups)
├── core/ ← Scholar / Paper / Papers / ScholarConfig
├── search_engines/ ← CrossRef, OpenAlex, Semantic Scholar, arXiv, PubMed federation
├── metadata_engines/ ← DOI resolution, abstract / IF / citation enrichment
├── auth/ ← OpenAthens / EZProxy / Shibboleth / SSO automators
├── browser/ ← persistent-profile Playwright manager (stealth + interactive)
├── url_finder/ ← PDF URL discovery (translators + heuristic strategies)
├── pdf_download/ ← `paper fetch` strategies (chrome viewer, direct, fallback)
├── pdf_highlight/ ← claim/method/limitation overlay via Claude
├── pipelines/ ← end-to-end @session-decorated workflows
├── storage/ ← MASTER-hash library + per-project symlinks
├── citation_graph/ ← optional citation network builder + plot
├── integration/ ← Zotero / Mendeley / RefWorks / EndNote / Paperpile importers
├── _mcp/ ← MCP tool handlers (`scholar_*` tools for `scitex serve`)
└── _skills/ ← agent-facing skill files (semantic-highlight, etc.)
```
The CLI is a thin layer over the Python API: every `scitex-scholar <noun> <verb>` command dispatches into one of `core/`, `pipelines/`, `storage/`, or `auth/`. The MCP server (`_mcp/`) exposes the same handlers as `scholar_*` tools consumed by the unified `scitex serve` server.
## Demo
```mermaid
flowchart LR
Q["search query<br/>or .bib / DOI"] --> S["Scholar.search() /<br/>bibtex import"]
S --> E["metadata_engines<br/>(CrossRef · OpenAlex · arXiv ·<br/>PubMed · Semantic Scholar)"]
E --> P["Papers<br/>(deduped, enriched)"]
P --> U["url_finder<br/>(translators + heuristics)"]
U --> A["auth<br/>(OpenAthens / EZProxy /<br/>Shibboleth)"]
A --> D["pdf_download<br/>(Playwright strategies)"]
D --> L["~/.scitex/scholar/library/<br/>MASTER/<HASH>/"]
P -.-> H["pdf_highlight<br/>(claim / method / limitation)"]
L -.-> M["MCP: scholar_* tools<br/>(scitex serve)"]
```
`scitex-scholar paper fetch --doi 10.1038/...` exercises the full chain in one call: enrich → resolve URL → authenticate → download → store under `MASTER/<HASH>/` with metadata + PDF + per-project symlinks.
## License
AGPL-3.0-only.
## Part of SciTeX
`scitex-scholar` is part of [**SciTeX**](https://scitex.ai). Install via
the umbrella with `pip install scitex[scholar]` to use as
`scitex.scholar` (Python) or `scitex scholar ...` (CLI).
> Four Freedoms for Research
>
> 0. The freedom to **run** your research anywhere — your machine, your terms.
> 1. The freedom to **study** how every step works — from raw data to final manuscript.
> 2. The freedom to **redistribute** your workflows, not just your papers.
> 3. The freedom to **modify** any module and share improvements with the community.
>
> AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.
---
<p align="center">
<a href="https://scitex.ai" target="_blank"><img src="docs/scitex-icon-navy-inverted.png" alt="SciTeX" width="40"/></a>
</p>
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available