nanzi-api-data-platform

> **Project Statement** > This project is a **personal open-source** project for free learning and communication, following the [MIT](LICENSE) open-source protocol, allowing free distribution. > The original project name "云枢" was renamed to "NanZi" to avoid confusion with other enterprise projects. > "NanZi" comes from my long-used nickname (南孜), meaning "diligent and unremitting," symbolizing AI's continuous learning and evolution. # Tool List **Simplified Chinese** | [English](README_EN.md) > **Enterprise-level Data API and Metadata Governance Platform** > *Connect Data. Serve Intelligence.* [](https://www.python.org/) [](https://fastapi.tiangolo.com/) [](https://vuejs.org/) [](https://tailwindcss.com/) [](https://clickhouse.com/) [](https://www.mysql.com/) [](https://redis.io/) [](LICENSE)  **NanZi · Data Service Platform** is a one-stop **Data-as-a-Service (DaaS)** hub for enterprise data consumption scenarios. It uniformly encapsulates physical tables, custom SQL, and semantic metadata into governable, auditable, and observable RESTful APIs, providing standardized data access capabilities for AI Agents, operation consoles, and business systems. The platform focuses on the following capability matrix: * 🚀 **Dynamic Resource Service (Resource-as-an-API)**: `TABLE` zero-code mapping and `SQL` complex logic encapsulation dual modes; Jinja2 dynamic template + unified DSL query entry. * 🧪 **SQL Lab**: Online writing, debugging, AI-assisted generation and repair of SQL, one-click publishing as production API resources. * 🗂️ **Metadata and Data Source Governance**: Multi-source connection management (MySQL / ClickHouse / Oracle), semantic metadata, health score, and RBAC fine-grained authorization. * 📦 **Data Product Catalog**: API resource productization release, business domain browsing, permission application/approval, asset panoramic KPI, and call volume insight. * 🛡️ **Enterprise-level Security Audit**: Daily partitioned audit logs, AST static SQL guardrails, API Key + Session dual authentication, and data desensitization strategy. * 📊 **Full-link Observability**: 24/7 call trend, Top ranking, minute-level statistical aggregation, and connection pool health monitoring. * 🔌 **Open Integration**: Standardized `/api/v1` external interface and management background Portal API, which can serve as the data base of the upstream **NanZi · Intelligent Platform**. ## 🏛️ Architecture  ## 🖼️ Interface Snapshots | 🔐 Unified Login Entry (Login) | 📊 Observability Dashboard | | :---: | :---: | |  |  | | **🚀 API Resource Management** | **🧪 SQL Lab** | |  |  | | **🛡️ Permission and Role (RBAC)** | **📋 Full-link Audit (Audit Logs)** | |  |  | | **📦 Data Product Catalog** | **🌐 Asset Panorama** | |  |  | ## 🌟 Core Capabilities  ### 1. 🚀 Dynamic Resource Service (Resource-as-an-API) * **Dual-mode Engine**: `TABLE` mode directly maps physical tables; `SQL` mode encapsulates complex query logic. * **Jinja2 Dynamic Template**: Inject condition branches into custom SQL to achieve high-performance parameterized filtering. * **Unified Query DSL**: `/api/v1/query` and `/api/v1/resources/{key}` support `EQ` / `IN` / `LIKE` / range comparison filters. ### 2. 🧪 AI-driven SQL Lab * **Intelligent SQL Assistance**: LLM generation, syntax error correction, and field Chinese label completion. * **Conversational Analysis**: Streaming returns analysis reports, automatically generating ECharts visualization configuration. * **One-click Publishing**: Debug-passed SQL is published as RBAC-controlled API resources. ### 3. 🗂️ Metadata and Data Source Management * **Multi-source Connection Pool**: Unified management of MySQL, ClickHouse, and Oracle connections and role isolation. * **Semantic Metadata**: Structured modeling of datasets, tables, fields, indicators, and entity relationships. * **Health Governance**: Metadata health scoring, creator tracking, and permission simulator. ### 4. 🛡️ Security, Audit, and Permission (Security & RBAC) * **Fine-grained RBAC**: Permissions are precise to data sources, physical tables, API resource points, and UI elements. * **Full-link Audit**: `api_access_logs_YYYYMMDD` is partitioned by day, supporting massive log reading and writing. * **SQL Security Guardrails**: `sqlparse` static analysis intercepts high-risk operations like `DELETE`/`DROP`, and forces `LIMIT`. * **Data Desensitization**: Global / role / user-level desensitization strategy, field rules configurable. ### 5. 📊 Observability and O&M * **Real-time Dashboard**: Call trend, Top interface/user, and online user statistics. * **Minute-level Aggregation**: `APScheduler` asynchronously summarizes `api_access_stats_1m`, and the dashboard opens in seconds. * **Connection Pool Monitoring**: Visualized active and healthy status of each data source connection pool. ### 6. 📦 Data Product Catalog and Asset Panorama * **Product Release**: One-click release or batch listing of API resources to the catalog, supporting draft → published → offline full lifecycle. * **Catalog Browsing**: Filtering by business domain, call volume, and featured recommendations; published product metadata is visible to all, and permission status is clear at a glance. * **Permission Application**: Users without permission can submit applications on the product details page, and responsible persons/administrators approve and automatically synchronize API resource permissions. * **Asset Panorama**: Domain distribution, zero-call alerts, KPI statistics, and call volume trend (`/api/portal/catalog/panorama`). * **Governance Assistance**: Redundant product detection, CSV export, responsible person batch designation, and Playground quick debugging entry. ## 🔄 Typical Data Consumption Process 1. **Register Data Source**: Configure MySQL / ClickHouse / Oracle connections in the management background. 2. **Define Resources**: Create `sys_resource_meta` resource entries through table mapping or SQL Lab. 3. **Publish to Catalog**: Publish API resources as data products, supplementing business domain, introduction, and responsible person metadata. 4. **Discovery and Application**: Users browse and filter products in the data product catalog; submit applications without permission, and obtain access rights after approval. 5. **External Call**: Client carries `X-API-Key` to call `/api/v1/query` or resource direct connection interface. 6. **Audit Trace**: All calls write logs partitioned by day, supporting Trace ID troubleshooting. See [architech/design/API_INTEGRATION_GUIDE.md](architech/design/API_INTEGRATION_GUIDE.md) · [docs/guides/getting-started.md](docs/guides/getting-started.md) ## 📚 Documentation | Document | Description | |------|------| | [HOW_TO_INSTALL.md](HOW_TO_INSTALL.md) | Installation and deployment and FAQ | | [architech/README.md](architech/README.md) | Architecture document index | | [architech/design/API_INTEGRATION_GUIDE.md](architech/design/API_INTEGRATION_GUIDE.md) | External API integration guide | | [architech/api-schema/sql_execution_api_usage.md](architech/api-schema/sql_execution_api_usage.md) | SQL execution API description | | [docs/guides/getting-started.md](docs/guides/getting-started.md) | Developer quick start | | [architech/design/ORACLE_INTEGRATION_GUIDE.md](architech/design/ORACLE_INTEGRATION_GUIDE.md) | Oracle data source access | | [db-prod/README.md](db-prod/README.md) | Database migration and idempotent apply tool | | [docker/README.md](docker/README.md) | Docker construction and deployment | | [architech/design/redis_key_design.md](architech/design/redis_key_design.md) | Redis Key design description | | [tests/CHECKLIST.md](tests/CHECKLIST.md) | Automated test acceptance checklist | ## 📂 Project Structure ```text . ├── app/ # Backend core code (FastAPI) │ ├── api/ # API interface layer (v1 external API / portal management background) │ ├── core/ # Core configuration (middleware, database, Redis) │ ├── services/ # Business logic (metadata, permission, AI, query engine, product catalog) │ │ └── data_adapter/ # Multi-source adapter (MySQL / ClickHouse / Oracle) │ ├── utils/ # Tool class (partition routing, encryption and decryption) │ └── jobs/ # Asynchronous scheduling (statistical aggregation, log cleaning) ├── frontend/ # Vue 3 management background (Vite + TailwindCSS) ├── architech/ # Architecture design documents and API Schema description ├── db-prod/ # Database version migration script (V0-VNN) and apply tool ├── docker/ # Containerized packaging and Docker Compose deployment ├── docs/ # Integration guide and O&M document ├── scripts/ # O&M auxiliary script ├── tests/ # Pytest automated test and acceptance checklist └── openspec/ # Interface specification change tracking (OpenSpec) ``` ## 🚀 Quick Start ### Environment Requirements | Component | Version Requirements | |------|----------| | Python | 3.10+ (recommended 3.13) | | Node.js | 18+ | | MySQL | 8.0+ | | Redis | 6.0+ (optional, recommended to enable) | ### 🐳 Docker Deployment (Recommended) **1. Configure Environment** ```bash cd docker cp ../env.example .env # Configure database, Redis, ENCRYPTION_KEY, etc. ``` **2. Build Image and Export Tar** | Script | Target Environment | | :--- | :--- | | `./build_linux_x86.sh <version>` | x86_64 Linux server (most common) | | `./build_linux_arm.sh <version>` | ARM64 Linux (Kunpeng / Ampere, etc.) | | `./build_native.sh <version>` | Local native architecture, only for local trial operation | ```bash # Production environment (x86 server) — Mac also uses x86 package with this script ./build_linux_x86.sh 1.0.0 ``` The product is output to **`docker/release/`**, for example, `nanzi-api_1.0.0_linux-amd64_20260628.tar`. Offline deployment: ```bash docker load -i docker/release/nanzi-api_1.0.0_linux-amd64_*.tar docker tag nanzi-api:1.0.0 nanzi-api:latest ``` > If you are on a Mac (Apple Silicon) deploying to an x86 server, be sure to use `build_linux_x86.sh` instead of `build_native.sh`. **If `docker buildx` is not available** (common with Homebrew docker + Colima): ```bash ./install-buildx.sh ./build_linux_x86.sh 1.0.0 ``` See [docker/README.md](docker/README.md) **3. Start Service** ```bash ./start-nanzi-api-server.sh ``` The service listens on **http://localhost:8000** by default (management background, `/docs` API documentation). ### 🛠️ Local Development #### 1. One-click Development and Debugging (Recommended) ```bash ./dev.sh ``` Automatically clean up old builds, compile the front-end, release port 8000, and start the back-end in `--reload` mode, with logs written to `app.log`. #### 2. Traditional Step-by-Step Startup ```bash # 1. Prepare environment python3 -m venv venv && source venv/bin/activate pip install -r requirements.txt cp env.example .env # 2. Database initialization (interactive, supports idempotent repeated execution) ./db-prod/apply-sql.sh # Without Python environment: ./db-prod/apply-sql-native.sh # 3. Create administrator (if INIT-USER-ADMIN.sql is not imported) python3 scripts/create_admin_user.py # 4. Compile front-end cd frontend && npm install && npm run build && cd .. # 5. Start back-end uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload ``` #### 3. Front-end and Back-end Separate Development ```bash # Terminal 1: Back-end uvicorn app.main:app --reload --port 8000 # Terminal 2: Front-end hot update cd frontend && npm run dev ``` ## 🔗 Relationship with NanZi Ecosystem This platform is the **data service layer** in the **NanZi · Intelligent · AI** system, complementary to [NanZi · Intelligent Platform](https://github.com/RandyChen1985/nanzi-ai-agent-platform): | Platform | Positioning | |------|------| | **Data Service Platform** (this repository) | Data API, metadata governance, SQL Lab, data product catalog, asset panorama, audit, and RBAC | | **Intelligent Platform** | AI dialogue, Agent orchestration, ChatBI, knowledge base, MCP plugin | ## 🤝 Contribution Guide Welcome to submit Issues and Pull Requests to improve this platform together. 1. **Branch Specification**: Develop based on `main`, and name functional branches `feature/your-feature-name`. 2. **Commit Information**: Use **Chinese** Commit Message, following [Conventional Commits](https://www.conventionalcommits.org/) specification. 3. **Pull Request**: When creating a PR, please fill in the description and test list according to [PULL_REQUEST_TEMPLATE.md](PULL_REQUEST_TEMPLATE.md). 4. **Test Acceptance**: Please update [tests/CHECKLIST.md](tests/CHECKLIST.md) for new functions. 5. **Database Change**: Add `V{N}-description.sql` in `db-prod/`, ensuring that the migration script can be idempotent and repeatable. ## 💬 Contact and Communication - **Issue**: Welcome to feedback problems and functional suggestions in [GitHub Issues](https://github.com/RandyChen1985/nanzi-api-data-platform/issues). - **Email**: You can contact us through the Issue message. If you have any questions, functional suggestions, or want to get more technical information during use, welcome to scan our WeChat official account: <img src="docs/images/weixin.png" alt="WeChat Official Account" width="200" /> ## 📄 License This project uses the [MIT License](LICENSE), allowing free use, copying, modification, merging, publishing, distributing, re-licensing, and selling copies of the software. --- Copyright © 2025-2026 NanZi API Data Platform Contributors. All Rights Reserved.