moodle-downloader

<div align="center"> # Tool List **Sync your UNSW Moodle course materials — then let AI revise them with you.** [](https://www.python.org/) [](LICENSE) [](tests/) [](#ai-revision) [](https://github.com/KunyQi/moodle-downloader/releases) [English Version](#english-version) </div> <div align="center"> **Downloader + AI Revision Assistant** · Okta SSO Login · 24-thread Parallel · MCP · Obsidian · Jupyter · EN </div> > [!IMPORTANT] > **Disclaimer** > > This tool is intended **only** for students downloading materials from courses they are **enrolled in**, for **offline study**. Always comply with your institution's IT Acceptable Use Policy. --- ## English Version ### About `moodle-downloader` is a course-material downloader for UNSW Moodle (`moodle.telt.unsw.edu.au`). It signs you in through a real browser window (Okta SSO), lists the courses you are enrolled in, deep-scans course pages for PDFs, PPTs and other files, and downloads them in parallel. Re-running it only fetches what's new — think of it as "sync" for your lecture materials. **But downloading is only the start.** Since v2.2.0 it ships an [MCP](https://modelcontextprotocol.io) server that turns your materials into **a knowledge base AI agents can query directly** — connect Claude Code or Claude Desktop and it can search your files, summarise a course, build a spaced-repetition plan, and quiz you from your own lecture notes. One command also exports an Obsidian vault and a Jupyter revision notebook. 👉 **[Jump to "AI agent revision"](#ai-revision-en)** <p align="center"> <img src="assets/demo-en.png" alt="moodle-downloader terminal demo (rendered from the real UI)" width="760"> </p> ### How it works 1. **Log in once** — a real browser window opens for Okta verification; your session is saved and reused 2. **Pick a course** — all your enrolled courses are listed automatically; just type a number 3. **Deep scan** — the course page and its sub-pages are crawled to surface every nested file link 4. **Parallel download** — multi-threaded downloads that skip anything already on disk 5. **Hand it to AI** — index the files, connect an agent, start revising ### Highlights - 🤖 **AI Agent Integration (MCP)** — built-in MCP server; connect Claude Code / Claude Desktop; 5 tools cover search, overview, and revision scheduling; zero extra dependencies - 🧠 **Spaced Repetition Plan** — schedule materials across N days; auto-arrange d-1 / d-3 / d-7 reviews - 📓 **Obsidian Vault + Jupyter Notebook** — export with one command; notes come with frontmatter, links, and 'blank recall' quiz sections - **Real-browser login, Okta SSO support** — use Selenium to open a browser window; cookies are saved and reused locally - **Auto-list enrolled courses** — read your course list from Moodle homepage; select by number or enter course ID directly - **Deep scan** — crawl sub-pages beyond the course homepage; find nested / embedded files others might miss - **Parallel download** — 24 threads by default (configurable via `config.toml` or `--workers`) - **Refined terminal UI** — progress bars with ETA; scanned results in a table with file type icons; download dashboard and result banners (powered by Rich) - **Incremental sync** — existing files on disk are skipped; re-run only downloads new files - **Atomic writes** — write to `.part` temp files then rename; no corrupted files are left behind - **Filter by keywords and extensions** — configure in `config.toml`; download only files you care about - **Auto-identify course names** — save files to folders named after the courses - **Bilingual interface (v2.1.0+)** — switch with `--lang en` or a config line - **Auto-detect browser** — try Chrome → Edge → Firefox; use `--browser` to force a specific one - **Crash protection** — console doesn't close on errors; full error messages are visible - **One-click start on Windows** — double-click `启动下载器.bat` to use - **Privacy-first** — runs locally; doesn't handle your password; collects and uploads no data ### Installation Requires **Python 3.11+**. ```bash git clone https://github.com/KunyQi/moodle-downloader.git cd moodle-downloader pip install -r requirements.txt ``` Dependencies: `requests`, `beautifulsoup4`, `selenium`, `rich`. Also need a browser driver (matching your browser version): | Browser | Driver | | --- | --- | | Chrome | [chromedriver](https://chromedriver.chromium.org/) | | Edge | [msedgedriver](https://developer.microsoft.com/microsoft-edge/tools/webdriver/) | | Firefox | [geckodriver](https://github.com/mozilla/geckodriver/releases) | ### Usage ```bash python main.py # auto-login → list courses → select download python main.py 98120 # download course 98120 directly python main.py --browser chrome # specify browser (chrome / edge / firefox) python main.py --lang en # interface language (zh / en) python main.py --workers 8 # parallel download threads ``` Windows users can also double-click `启动下载器.bat`. The first run will open a browser window for Okta login; subsequent runs usually don't require re-login. **This file contains your session credentials; don't commit it or share it.** ### Configuration Edit `config.toml` in the repo root and re-run: ```toml [download] max_workers = 24 # parallel download threads [filters] file_keywords = ["lecture"] # file name keywords (empty = accept all) file_extensions = [".pdf", ".ppt", ".pptx"] # extension whitelist (empty = accept all) [ui] language = "en" # interface language: "zh" or "en" [browser] # type = "chrome" # empty = auto-detect (chrome → edge → firefox) ``` <a id="ai-revision-en"></a> ### 🤖 AI Agent Revision (added in v2.2.0) Downloading is just the start. These commands turn your local files into a **queryable knowledge base**, letting AI agents like Claude Code or Claude Desktop revise with you — all operations are local; **no login required**. ```bash python main.py --index # index local files (default: current dir) python main.py --obsidian vault # export Obsidian vault python main.py --obsidian vault --copy-files # also copy original files python main.py --notebook revision.ipynb --days 7 # generate revision notebook ``` **MCP Server — the agent's entry point** ```bash python -m moodle_scraper.mcp_server --root . ``` A standard [MCP](https://modelcontextprotocol.io) server (stdio / JSON-RPC; zero extra dependencies). Any MCP client can connect. In Claude Code: ```bash claude mcp add moodle -- python -m moodle_scraper.mcp_server --root /path/to/materials ``` Exposes 5 tools: | Tool | Function | | --- | --- | | `list_courses` | list local courses and material stats | | `search_materials` | full-text search across materials | | `get_material_text` | retrieve a material's text | | `course_overview` | summarise a course by week / type | | `revision_plan` | schedule materials across N days with d-1/d-3/d-7 spacing | Once connected, simply talk to the agent: ```text I've got a PHYS1231 exam next week; help me plan a 5-day revision schedule What did Week 3 cover? Use lecture notes to create 5 questions to quiz me Find all materials mentioning 'Second Law of Thermodynamics' ``` The agent will use these tools to answer from your **local files** — not fabricated. **Obsidian Vault**: each course has a MOC index page; materials are organised by week; each note has YAML frontmatter (course / week / type / tags), embedded files, `## Revision Notes` blank sections, and text summaries; all interlinked with wikilinks. **Revision Skill**: the repo includes [`skills/moodle-revision/`](skills/moodle-revision/); this lets agents that support Skills learn a full workflow: select course → overview → schedule spaced repetition → quiz you with real materials. > 💡 **Full-text search is an optional enhancement**: PDF text extraction requires `pip install pypdf` (lazy-loaded; not installing means search degrades to filename matching). `.txt` / `.md` files don't need extra dependencies. ### Tests ```bash python -m pytest tests/ -q ``` All 280+ tests run offline — a fake HTTP client injects preset responses; no real network access. ### Engineering Quality - Full type annotations: all functions have type hints for parameters and return types - HTTP access is hidden behind a `Protocol` seam; the scanner and downloader only depend on the interface, not on `requests`; offline tests are thus possible - A single file or page failure won't interrupt the entire download batch; failed items are listed in the final report ### Contributing Feel free to open issues and PRs. Before submitting: 1. Run `python -m pytest tests/ -q` to ensure tests pass 2. Keep full type annotations; when adding external dependencies, follow the `HttpClient` Protocol seam pattern 3. Test through a fake HTTP client; don't access the real network 4. User-facing text goes through `moodle_scraper/i18n.py`'s `t()`; sync both English and Chinese directories ### FAQ <details> <summary><b>What if my cookie expires?</b></summary> Nothing to do. The tool checks your local session on each run; if expired, it re-opens the browser for login and reuses the session. </details> <details> <summary><b>Where do downloaded files go?</b></summary> In a course-named folder under the repo directory; e.g. `PHYS1231_Higher_Physics_1B/`. Re-running only downloads new files. </details> <details> <summary><b>Why didn't some files get scanned?</b></summary> Check `config.toml`'s `[filters]`. By default, only files with 'lecture' in their name or with `.pdf/.ppt/.pptx` extensions are kept — set both lists to `[]` to download all files. </details> <details> <summary><b>Is my account info safe?</b></summary> Login happens in a real browser window on your machine; the tool doesn't handle your password; it only saves session cookies locally in `moodle_cookies.json` (excluded by `.gitignore`); no data is collected or uploaded. Code is open-source; feel free to audit. </details> <details> <summary><b>Does AI functionality upload my files to the cloud?</b></summary> The tool itself doesn't. The MCP server only runs locally; it exchanges data with the agent client of your choice through stdio; indexing, Obsidian vault, and notebook are all written locally. Note: **when you let a cloud AI (e.g. Claude) read these contents, that service will naturally receive them** — this depends on which agent you use; unrelated to this tool. For complete offline use, connect a local model. </details> <details> <summary><b>Can I use it without AI?</b></summary> Yes. Downloading, scanning, and filtering are independent; AI is optional. Even just using `--obsidian` to export a knowledge base is useful without an agent. </details> ### License [MIT](LICENSE) --- **⭐ If it saves you time navigating Moodle, star this repo — it's also how more students find it.** ### Highlights - 🤖 **AI agent access (MCP)** — a built-in MCP server any client can connect to (Claude Code, Claude Desktop); 5 tools covering search, overview and revision planning, hand-rolled JSON-RPC with zero extra dependencies - 🧠 **Spaced-repetition planning** — materials ordered by week and kind, spread across N days, with automatic d-1 / d-3 / d-7 review slots - 📓 **Obsidian vault + Jupyter notebook** — one command each; notes ship with frontmatter, wikilinks and blank-recall self-testing - **Real-browser login with Okta SSO** — Selenium opens a browser window for you to sign in; cookies are persisted locally and reused, so login is usually one-time - **Lists your enrolled courses** — read straight from the Moodle dashboard; pick by number or enter a course ID directly - **Deep scan** — drills into course sub-pages to find nested / embedded files that other tools miss - **Parallel downloads** — 24 threads by default (configurable via `config.toml` or `--workers`) - **Polished terminal UI** — progress bars with ETA, scan-result tables with file-type icons, a download report dashboard and result banners (built on Rich) - **Incremental sync** — files already on disk are skipped; re-run anytime to fetch only what's new - **Atomic writes** — downloads go to a `.part` temp file, then rename; no corrupt half-downloads, ever - **Keyword and extension filters** — configure in `config.toml` to download only what you care about - **Auto-detects the course name** — files are saved into a matching folder - **Fully bilingual interface** (new in v2.1.0) — switch between English and English with `--lang en` or one line of config - **Browser auto-detect** — tries Chrome → Edge → Firefox, or force one with `--browser` - **Crash guard** — the console never closes silently on error; you always see what went wrong - **Windows one-click launcher** — double-click `启动下载器.bat` and go - **Privacy-first** — everything runs locally; the tool never handles your password, collects nothing and uploads nothing ### Install Requires **Python 3.11+**. ```bash git clone https://github.com/KunyQi/moodle-downloader.git cd moodle-downloader pip install -r requirements.txt ``` Dependencies: `requests`, `beautifulsoup4`, `selenium`, `rich`. You also need at least one WebDriver matching your browser: | Browser | Driver | | --- | --- | | Chrome | [chromedriver](https://chromedriver.chromium.org/) | | Edge | [msedgedriver](https://developer.microsoft.com/microsoft-edge/tools/webdriver/) | | Firefox | [geckodriver](https://github.com/mozilla/geckodriver/releases) | ### Usage ```bash python main.py # login → list courses → pick & download python main.py 98120 # download course 98120 directly python main.py --browser chrome # force a browser (chrome / edge / firefox) python main.py --lang en # UI language (en) python main.py --workers 8 # number of parallel download threads ``` On Windows you can also just double-click `启动下载器.bat`. The first run opens a browser window — complete the Okta login there. Your session is saved locally to `moodle_cookies.json`, so subsequent runs usually skip the login entirely. **That file holds your session credentials — never commit it or share it.** ### Configuration Edit `config.toml` in the repo root; changes take effect on the next run: ```toml [download] max_workers = 24 # parallel download threads [filters] file_keywords = ["lecture"] # filename keywords (empty = accept all) file_extensions = [".pdf", ".ppt", ".pptx"] # extension whitelist (empty = accept all) [ui] language = "en" # UI language: "en" [browser] # type = "chrome" # empty = auto-detect (chrome → edge → firefox) ``` <a id="ai-revision-en"></a> ### 🤖 AI agent revision (new in v2.2.0) Downloading is only step one. These commands turn the files on your disk into a **searchable knowledge base** that Claude Code, Claude Desktop and other AI agents can revise with you — all local, **no login required**. ```bash python main.py --index # index local materials (defaults to cwd) python main.py --obsidian vault # export an Obsidian vault python main.py --obsidian vault --copy-files # also copy the source files in python main.py --notebook revision.ipynb --days 7 # build a revision notebook ``` **MCP server — the agent entry point** ```bash python -m moodle_scraper.mcp_server --root . ``` A standard [MCP](https://modelcontextprotocol.io) server (stdio / JSON-RPC, zero extra dependencies) that any MCP client can talk to. Register it with Claude Code: ```bash claude mcp add moodle -- python -m moodle_scraper.mcp_server --root /path/to/materials ``` It exposes 5 tools: | Tool | What it does | | --- | --- | | `list_courses` | List local courses with material counts | | `search_materials` | Full-text search across your course files | | `get_material_text` | Pull the text of one material | | `course_overview` | Break a course down by week and type | | `revision_plan` | Split the material across an N-day plan with d-1/d-3/d-7 review slots | Once connected, just talk to your agent in plain language: ```text I have a PHYS1231 exam next week — build me a 5-day revision plan What was covered in week 3? Quiz me on it using the lecture notes Find every material that mentions the second law of thermodynamics ``` The agent answers by calling those tools against **your real local files** — not from memory. **Obsidian vault**: one MOC per course, materials bucketed by week, and every note carrying YAML frontmatter (course / week / kind / tags), an embed of the source file, a blank "key points" section and a text summary — all wikilinked together. **Revision Skill**: the repo ships [`skills/moodle-revision/`](skills/moodle-revision/), which teaches a Skills-capable agent the whole loop: pick a course → review the overview → build a spaced-repetition plan → quiz you from the real material. > 💡 **Full-text search is an optional upgrade**: extracting text from PDFs needs `pip install pypdf` (lazily imported — without it everything still runs, search just falls back to filename matching). `.txt` / `.md` need nothing extra. ### Tests ```bash python -m pytest tests/ -q ``` All 280+ tests run fully offline — they inject pre-registered responses through a fake HTTP client and never touch the network. ### Engineering notes - Fully type-annotated: every function parameter and return value carries a type hint - HTTP access sits behind a `Protocol` seam: the scanner and downloader depend only on the interface, never on `requests`, which is what makes the test suite completely offline - Per-item failure isolation: one bad page or file never aborts the batch; failures are listed in the final report ### Contributing Issues and PRs are welcome. Before submitting: 1. Run `python -m pytest tests/ -q` and make sure everything passes 2. Keep full type annotations, and follow the `HttpClient` Protocol seam pattern when adding external dependencies 3. Write tests against the fake HTTP client — no real network access in tests 4. Route every user-visible string through `t()` in `moodle_scraper/i18n.py`, keeping the en catalogs in sync ### FAQ <details> <summary><b>My cookies expired — what do I do?</b></summary> Nothing. On every run the tool first probes whether your saved session is still valid; if it isn't, a browser window opens automatically for a one-time re-login, and the new session is reused afterwards. </details> <details> <summary><b>Where do my files go?</b></summary> Into a folder named after the course, right in the repo directory — e.g. `PHYS1231_Higher_Physics_1B/`. Re-running only fetches new files; nothing is downloaded twice. </details> <details> <summary><b>Why are some files missing from the scan?</b></summary> Check `[filters]` in `config.toml`. By default only filenames containing `lecture` or ending in `.pdf/.ppt/.pptx` are kept — set both lists to `[]` to download everything. </details> <details> <summary><b>Is my account safe?</b></summary> You log in inside a real browser window on your own machine — the tool never sees your password. It only stores the session cookie locally in `moodle_cookies.json` (already in `.gitignore`), collects nothing, and uploads nothing. The code is fully open source — audit away. </details> <details> <summary><b>Do the AI features upload my materials to the cloud?</b></summary> Not by this tool. The MCP server runs entirely on your machine and hands data to whichever agent client *you* choose over stdio; the index, the Obsidian vault and the notebook are all written to local disk only. Worth being clear about: **when you ask a cloud AI (Claude, for instance) to read that content, the parts it reads do go to that provider** — that follows from the agent you picked, not from this tool. Point it at a local model if you want to stay fully offline. </details> <details> <summary><b>Can I use it without any AI?</b></summary> Absolutely. Downloading, scanning and filtering are completely independent — the AI layer is optional. Even `--obsidian` on its own is useful without ever connecting an agent. </details> ### License [MIT](LICENSE) --- **⭐ If this tool saved you from clicking through Moodle page by page, consider leaving a star — it's the best way to help other students find it.**