topo_scholar

# Tool List [English](README_EN.md)     **TopoScholar** is a Chinese toponym knowledge base for Agents, RAG, and toponym research: it not only preserves "what place names exist" but also continuously answers "why a place name is called that way". The project has currently established a five-level basic database of provinces, cities, counties, townships/streets, and villages/communities, and has started to systematically fill in the origins, meanings, historical evolutions, old names, and evidence sources for each place name. The goal is to form a queryable, disambiguatable, traceable, and extensible Chinese place name knowledge system. ## Why this project Chinese place names contain a lot of historical, migration, mountain and river, water system, orientation, surname,屯垦, ethnic language, and administrative evolution information. Ordinary administrative division tables can only tell us "where it is", but it's hard to tell us: - Why `Wuhan` is called Wuhan, and what is its relationship with Wuchang, Hankou, and Hanyang? - What does the "Nan" in `Nan Gao Village` refer to? - How many `Peace Villages` and `Unity Villages` are there in the country, and where are they located? - What are the mergers, name changes, cancellations, and resets that a place name has experienced? - How can Agents avoid misjudging and fabricating origins when answering place name questions? TopoScholar attempts to turn these questions into structured data, reproducible pipelines, and Agent-callable tools. ## Current Progress The latest verification results can be seen in `data/metadata/quality_report.json` and `docs/quality_assessment.md`. | Module | Current Scale | |---|---:| | Basic Place Names `places` | 665,276 | | Administrative Level Edges `admin_edges` | 665,245 | | Alias Index `place_aliases` | 2,981,225 | | Place Name Origin Knowledge `place_knowledge` | 1,837 | | Origin Collection Queue `collection_queue` | 3,227 | | Basic Level | Quantity | |---|---:| | Provincial | 31 | | Prefecture-level | 342 | | County-level | 2,978 | | Township/Street-level | 41,352 | | Village/Community-level | 620,573 | Place name origin collection progress: - Provincial administrative regions: 31/31 completed. - Collectible prefecture-level administrative regions: 333/333 completed. - County-level administrative regions: 1,471 completed, continuously batch-processed. - Village-level examples: retain `Nan Gao Village` for verification of village-level place names and origin associations. ## Core Capabilities ### 1. Five-level Place Name Base Based on publicly available administrative division data, generate unified `places`, `admin_edges`, and `place_aliases`: - Five-level structure of provinces, cities, counties, townships/streets, and villages/communities. - Can be drilled down or traced back to the superior level by administrative code. - Can handle a large number of homonymous villages, communities, and district counties. ### 2. Aliases and Disambiguation Place name queries do not require users to input the complete standard name: - `Nan Gao Village` can hit `Nan Gao Village Committee`. - Common suffixes such as `community`, `residents' committee`, and `village committee` will be included in the alias index. - Support narrowing down homonymous results by province, city, county, and level constraints. ### 3. Place Name Origin Knowledge Table `place_knowledge` saves structured place name knowledge: - Place name origin `origin` - Name meaning `meaning` - Historical evolution `history` - Old names `old_names` - Source title, source URL, evidence summary, and confidence level Currently, priority is given to pulling from China's National Place Name Information Database as needed, and candidate audit records are retained to avoid incorrect matching. ### 4. Agent / MCP Friendly The project provides JSON CLI and reserves MCP Server tools: - Place name resolution - Alias resolution - Administrative subordinate query - Homonymous place name comparison - Local place name origin query - Place name origin supplementation on demand - Full-text retrieval of cached knowledge ## Quick Start ### Clone the Project ```powershell git clone https://github.com/snailfrying/topo_scholar.git cd topo_scholar ``` ### Direct Query When running for the first time, if `data/processed/topo_scholar.sqlite` does not exist, the project will automatically build a local database from the original CSV and small knowledge table in the repository. ```powershell python topo_scholar_cli.py resolve 武汉市 --province 湖北省 --city 武汉市 --limit 1 ``` ### Common Commands ```powershell # Precise disambiguation python topo_scholar_cli.py resolve 武汉市 --province 湖北省 --city 武汉市 --limit 1 # Alias disambiguation: Nan Gao Village -> Nan Gao Village Committee python topo_scholar_cli.py alias 南高村 --province 山东省 --city 东营市 --county 广饶县 --level village # Query subordinate administrative units python topo_scholar_cli.py children 4201 --limit 10 # Query homonymous place name distribution python topo_scholar_cli.py same-name 和平村委会 --limit 20 # Query locally cached place name origin python topo_scholar_cli.py origin 南高村 --province 山东省 --city 东营市 --county 广饶县 --limit 1 # Supplement place name origin on demand python topo_scholar_cli.py fetch-origin 武汉市 --province 湖北省 --city 武汉市 --place-type 地级行政区 --limit 1 # Search cached origin knowledge python topo_scholar_cli.py search-knowledge 洪洞 --limit 10 ``` ## Place Name Origin Collection The project has a built-in conservative collection pipeline: first generate a queue, then fill in the queue at a limited speed, and write it into the knowledge table and SQLite. This can continuously improve the data and avoid putting pressure on the source site. ```powershell # Construct province/city/county three-level collection queue python scripts\build_collection_queue.py --levels province,city,county # Preview the first 3 items to be collected, without accessing the network python scripts\batch_fetch_origins.py --max-items 3 --dry-run # Small batch completion of county-level place name origins python scripts\batch_fetch_origins.py --max-items 100 --levels county --sleep 0.8 --max-pages 3 --max-attempts 3 ``` Collection results will be written to: - `data/processed/place_knowledge.csv` - `data/processed/collection_queue.csv` - `data/metadata/mca_candidate_audit.csv` - `data/processed/topo_scholar.sqlite` in `place_knowledge` and `place_knowledge_fts` ## Local Construction and Verification If you want to explicitly rebuild the database: ```powershell python scripts\build_places.py python scripts\build_collection_queue.py --levels province,city,county python scripts\build_sqlite.py python scripts\validate_data.py ``` After construction, the following will be generated: - `data/processed/places.csv` - `data/processed/admin_edges.csv` - `data/processed/topo_scholar.sqlite` These files are large and are not submitted as ordinary Git files; the repository submits necessary raw data, small knowledge tables, collection queues, audit tables, scripts, and documents to ensure that they can be reproduced after cloning. See `docs/release_data_policy.md` for details. ## MCP Server MCP SDK is an optional dependency. After installation, MCP Server can be started: ```powershell pip install -r requirements.txt python -m topo_scholar.mcp_server ``` Currently planned/implemented MCP tools include: | Tool | Purpose | |---|---| | `resolve_place_name_tool` | Standard place name resolution | | `resolve_place_alias_tool` | Alias resolution | | `search_toponyms_tool` | Place name search | | `search_aliases_tool` | Alias search | | `get_place_by_code_tool` | Query by administrative code | | `get_admin_children_tool` | Query subordinate administrative units | | `compare_same_name_tool` | Compare homonymous place names | | `get_place_origin_tool` | Query local place name origin | | `search_knowledge_tool` | Search origin knowledge | | `fetch_place_origin_tool` | Supplement origin knowledge on demand | ## Data Directory ```text data/ raw/ # Necessary raw CSV data submitted to the repository processed/ # Locally generated data; submit small place_knowledge.csv, collection_queue.csv, and .gitkeep cache/ # API response cache, not submitted metadata/ # Data source, hash, quality report, candidate audit scripts/ # Data construction, collection, and verification scripts topo_scholar/ # Python query library and MCP Server docs/ # Requirements, data plan, quality assessment, and release strategy ``` ## Data Sources Currently accessed or planned data sources: - `modood/Administrative-divisions-of-China`: Five-level administrative division basic CSV. - `xiangyuecn/AreaCity-JsSpider-StatsGov`: Four-level administrative division supplementary data. - China's National Place Name Information Database: Source of place name origin, meaning, and historical evolution. - Local government websites, local records, and place name records: Subsequent supplementary cultural and historical materials. - OpenStreetMap/Geofabrik: Subsequent road and spatial data candidate sources. Specific sources, licenses, and usage see `docs/dataset_sources.md`. ## Roadmap - [x] Establish a five-level basic place name database. - [x] Establish an alias index and basic disambiguation capability. - [x] Establish a place name origin collector, queue, and candidate audit. - [x] Complete provincial and collectible prefecture-level place name origins. - [ ] Continue to complete county-level place name origins. - [ ] Establish place name naming type labels: surname, orientation, hydrology, topography, migration,屯垦, historical figures, etc. - [ ] Expand BM25/vector retrieval to support topic research such as "named after water" and "Hongdong immigration". - [ ] Distinguish administrative streets and road streets, and access road place name data. - [ ] Add administrative division version difference table, record cancellation, merger, name change, code change. - [ ] Improve MCP Server and Agent Skill usage instructions. ## Current Limitations - The five-level basic database is mainly based on the 2023 statistical caliber, not equal to the latest civil affairs/place name database caliber. - Current "streets" mainly refer to administrative street offices, not road streets. - Hong Kong, Macau, and Taiwan are not included in the current five-level main table. - The place name origin knowledge database is still being collected, and batch scripts are executed with default quantity and speed limits. - The National Place Name Information Database is suitable for on-demand query and caching, and high-frequency crawling or redistribution without authorization is not recommended. ## Documents - `docs/requirements.md`: Product requirements and capability planning. - `docs/data_plan.md`: Data construction plan. - `docs/origin_enrichment.md`: Place name origin completion plan. - `docs/tool_contracts.md`: CLI/MCP tool interface contract. - `docs/quality_assessment.md`: Data quality assessment and optimization route. - `docs/release_data_policy.md`: Repository data submission strategy. ## License and Compliance The license and documentation of the code and this project will be stated separately. Third-party data retains its original license and source description; please confirm the license terms of the corresponding data source before use, distribution, or commercial use. For place name origin materials, this project emphasizes source recording, evidence retention, and confidence level labeling; without reliable sources, Agents should not fabricate place name origins.