world-scanner

# 🛰 World Scanner **Turn drone and phone videos into explorable 3D worlds — entirely on your Mac.** Drop an FPV drone clip (or any walkthrough video) onto a local dashboard and get back an interactive 3D reconstruction: fly through the point cloud, view it as Gaussian splats, and replay the camera's flight path through the world it scanned. No cloud, no GPU rig, no accounts — everything runs on an Apple Silicon laptop. Built on [LingBot-Map](https://github.com/Robbyant/lingbot-map) ([paper](https://arxiv.org/abs/2604.14141)), a feed-forward streaming 3D reconstruction model, ported here to run on macOS (details below).  ## What it does | | | |---|---| |  |  | | **Trajectory mode** — the camera's path as a ribbon through the reconstructed world, with heading markers, a flying tracer, and a top-down minimap | **Splat mode** — the scene as 3D Gaussian splats (instant fuzzy conversion built in; optional HQ training via OpenSplat) | 1. **Drop a video** → pick **Drone** (fast motion, 6 fps sampling) or **Walkthrough** (4 fps). 2. The model watches the frames and predicts, for each one, the **camera's pose** and a **dense depth map**, fusing them into a colored point cloud — one feed-forward pass, no COLMAP, no offline optimization. 3. **Explore** the scene three ways — Points, Splats, Trajectory — straight from the scene card. Every scene also exports standard formats for other tools: `points.ply`, nerfstudio `transforms.json`, and a COLMAP text model. There's also an **MCP server**, so Claude Code can drive the same pipeline conversationally (*"reconstruct this video", "render the courthouse from above"* — images come back into the chat). ## Quickstart (Apple Silicon Mac) ```bash git clone https://github.com/hem-318/world-scanner.git && cd world-scanner ./setup.sh # Homebrew tools + Python env + 4.6 GB model (sha256-verified) + patch demo-kit/run_dashboard.sh # → http://localhost:8100 — drop a video on it ``` Requirements: Apple Silicon Mac (tested: M1 Pro, 16 GB, macOS 27), [Homebrew](https://brew.sh), ~15 GB free disk. First reconstruction downloads nothing further. <details> <summary><b>CLI usage & other entry points</b></summary> ```bash # Bundled example scene with the upstream interactive viewer (viser): demo-kit/run_demo.sh --image_folder "$PWD/lingbot-map/example/courthouse" \ --first_k 80 --num_scale_frames 2 --kv_cache_sliding_window 16 \ --offload_to_cpu --port 8081 --export_dir "$PWD/scenes/courthouse" # Headless: video → exports only demo-kit/run_demo.sh --video_path ~/Movies/flight.mp4 --fps 6 --first_k 100 \ --num_scale_frames 2 --kv_cache_sliding_window 16 --offload_to_cpu \ --no_viewer --export_dir "$PWD/scenes/flight" # Instant splats from any exported scene: lingbot-map/.venv/bin/python demo-kit/points_to_splats.py scenes/flight ``` **MCP:** `.mcp.json` registers the `lingbot-map` server for Claude Code sessions opened in this folder — tools: `reconstruct`, `make_splats`, `render_view`, `open_viewer`, `list_scenes`. Full runbook with all measured numbers: [demo-kit/README-mac.md](demo-kit/README-mac.md). </details> ## How the macOS port works The upstream project targets NVIDIA CUDA + FlashInfer. This repo carries a small patch ([demo-kit/patches/](demo-kit/patches/), applied automatically by `setup.sh`) that makes it run on Macs: - device selection `cuda → mps → cpu` with a `--device` override; SDPA attention replaces FlashInfer off-NVIDIA; CUDA-graph compilation and CUDA-only calls are guarded - the RoPE frequency tables are complex128, which Apple's MPS backend can't store — they're built on CPU in full precision and transferred as complex64 (lossless for inference, which only ever reads `.real`/`.imag` in fp32) - `--export_dir` / `--no_viewer` hooks feed the exporter and headless pipelines **Performance, measured on an M1 Pro (16 GB):** CPU fp32 at ~7–11 s/frame (an 80-frame scene: ~9 min, 8.7 M points, peak 3.9 GB RAM). MPS inference runs after the patches but exhausts the 20 GB allocator watermark on a 16 GB machine — likely viable on 32 GB+ Macs. OpenSplat HQ training is CPU-bound (~13 s/step) without full Xcode; install Xcode and rebuild with `-DGPU_RUNTIME=MPS` for Metal speed. ## Repo layout ``` demo-kit/ everything this repo adds ├── dashboard/ FastAPI app + single-page UI (upload, jobs, scene cards) ├── splat_viewer/ browser Gaussian-splat viewer (three.js, vendored by setup.sh) ├── export_predictions.py model output → PLY / nerfstudio / COLMAP ├── points_to_splats.py instant point→gaussian converter ├── render_scene.py headless numpy rasterizer (for MCP image replies) ├── mcp_server.py Claude Code integration ├── patches/ macOS port of upstream (git am format) └── tests/ pytest suite for all of the above docs/ design spec, implementation plan, screenshots setup.sh one-command bootstrap ``` Reconstructions land in `scenes/<name>/` (gitignored), models in `models/` (gitignored). ## Credits & license - [LingBot-Map](https://github.com/Robbyant/lingbot-map) — model, weights and demo code (Apache-2.0); paper: *Geometric Context Transformer for Streaming 3D Reconstruction* ([arXiv:2604.14141](https://arxiv.org/abs/2604.14141)) - [OpenSplat](https://github.com/pierotofy/OpenSplat) (optional HQ splat training), [GaussianSplats3D](https://github.com/mkkellogg/GaussianSplats3D), [three.js](https://threejs.org), [viser](https://viser.studio) This repo's code and the macOS patch are licensed **Apache-2.0** (see [LICENSE](LICENSE)).