yunshu-api-data-platform

# Tool List **English** | [English](README_EN.md) > **Enterprise-level Data API and Metadata Governance Platform** > *Connect Data. Serve Intelligence.* [](https://www.python.org/) [](https://fastapi.tiangolo.com/) [](https://vuejs.org/) [](https://tailwindcss.com/) [](https://clickhouse.com/) [](https://www.mysql.com/) [](https://redis.io/) [](LICENSE)  **Yunshu API Data Platform** is a one-stop **Data-as-a-Service (DaaS)** hub for enterprise data consumption scenarios. It uniformly encapsulates physical tables, custom SQL, and semantic metadata into governable, auditable, and observable RESTful APIs, providing standardized data access capabilities for AI Agents, operation control consoles, and business systems. The platform focuses on the following capability matrix: * 🚀 **Dynamic Resource Service (Resource-as-an-API)**: `TABLE` zero-code mapping and `SQL` complex logic encapsulation dual modes; Jinja2 dynamic template + unified DSL query entry. * 🧪 **SQL Lab**: Online writing, debugging, AI-assisted generation and repair of SQL, one-click publishing as production API resources. * 🗂️ **Metadata and Data Source Governance**: Multi-source connection management (MySQL / ClickHouse / Oracle), semantic metadata, health score, and RBAC fine-grained authorization. * 📦 **Data Product Catalog**: API resource productization release, business domain browsing, permission application/approval, asset panoramic KPI, and call volume insight. * 🛡️ **Enterprise-level Security Audit**: Day-partitioned audit logs, AST static SQL guardrails, API Key + Session dual authentication, and data desensitization strategies. * 📊 **Full-link Observability**: 24/7 call trends, Top ranking, minute-level statistical aggregation, and connection pool health monitoring. * 🔌 **Open Integration**: Standardized `/api/v1` external interface and management background Portal API, can be used as the data base of **Yunshu AI Agent Platform**. --- ## 🏛️ Architecture  --- ## 🖼️ Interface Snapshots | 🔐 Unified Login Entry (Login) | 📊 Observability Dashboard (Dashboard) | | :---: | :---: | |  |  | | **🚀 API Resource Management (Resource Management)** | **🧪 SQL Lab (SQL Lab)** | |  |  | | **🛡️ Permission and Role (RBAC)** | **📋 Full-link Audit (Audit Logs)** | |  |  | | **📦 Data Product Catalog (Catalog)** | **🌐 Asset Panorama (Asset Panorama)** | |  |  | --- ## 🌟 Core Capabilities  ### 1. 🚀 Dynamic Resource Service (Resource-as-an-API) * **Dual-mode Engine**: `TABLE` mode directly maps physical tables; `SQL` mode encapsulates complex query logic. * **Jinja2 Dynamic Template**: Inject condition branches into custom SQL to achieve high-performance parameterized filtering. * **Unified Query DSL**: `/api/v1/query` and `/api/v1/resources/{key}` support `EQ` / `IN` / `LIKE` / range comparison and other filters. ### 2. 🧪 AI-driven SQL Lab * **Intelligent SQL Assistance**: LLM generates, syntax error correction, field Chinese label completion. * **Conversational Analysis**: Streaming returns analysis reports, automatically generates ECharts visualization configuration. * **One-click Publishing**: Debug-passed SQL is immediately published as RBAC-controlled API resources. ### 3. 🗂️ Metadata and Data Source Management * **Multi-source Connection Pool**: Unified management of MySQL, ClickHouse, and Oracle connections and role isolation. * **Semantic Metadata**: Structured modeling of datasets, tables, fields, indicators, and entity relationships. * **Health Governance**: Metadata health score, creator tracking, and permission simulator. ### 4. 🛡️ Security, Audit, and Permission (Security & RBAC) * **Fine-grained RBAC**: Permissions are precise to data sources, physical tables, API resource points, and UI elements. * **Full-link Audit**: `api_access_logs_YYYYMMDD` day-partitioned table, supports massive log reading and writing. * **SQL Security Guardrails**: `sqlparse` static analysis intercepts high-risk operations such as `DELETE`/`DROP`, and forces `LIMIT`. * **Data Desensitization**: Global / role / user-level desensitization strategy, field rules configurable. ### 5. 📊 Observability and O&M * **Real-time Dashboard**: Call trends, Top interface/user, online user statistics. * **Minute-level Aggregation**: `APScheduler` asynchronously summarizes `api_access_stats_1m`, dashboard opens in seconds. * **Connection Pool Monitoring**: Visualized active degree and health status of each data source connection pool. ### 6. 📦 Data Product Catalog and Asset Panorama * **Product Release**: One-click release or batch listing of API resources to the catalog, supporting draft → published → offline full lifecycle. * **Catalog Browsing**: Screening by business domain, call volume, and featured recommendation; published product metadata is visible to all, and permission status is clear at a glance. * **Permission Application**: Users without permission can submit applications on the product details page, and approvers/administrators approve and automatically synchronize API resource permissions. * **Asset Panorama**: Domain distribution, zero-call alarms, KPI statistics, and call volume trends (`/api/portal/catalog/panorama`). * **Governance Assistance**: Redundant product detection, CSV export, responsible person batch designation, and Playground quick debugging entrance. --- ## 🔄 Typical Data Consumption Process 1. **Register Data Source**: Configure MySQL / ClickHouse / Oracle connections in the management background. 2. **Define Resources**: Create `sys_resource_meta` resource entries through table mapping or SQL Lab. 3. **Publish to Catalog**: Publish API resources as data products, supplement business domain, introduction, and responsible person and other metadata. 4. **Discovery and Application**: Users browse and filter products in the data product catalog; submit applications when permissionless, and obtain access rights after approval. 5. **External Call**: Client carries `X-API-Key` to call `/api/v1/query` or resource direct connection port. 6. **Audit Tracking**: All calls are written to day-partitioned logs, supporting Trace ID troubleshooting. See [architech/design/API_INTEGRATION_GUIDE.md](architech/design/API_INTEGRATION_GUIDE.md) · [docs/guides/getting-started.md](docs/guides/getting-started.md) for details. --- ## 📚 Documentation and Architecture | Document | Description | |------|------| | [HOW_TO_INSTALL.md](HOW_TO_INSTALL.md) | Installation and deployment and FAQ | | [architech/README.md](architech/README.md) | Architecture document index | | [architech/design/API_INTEGRATION_GUIDE.md](architech/design/API_INTEGRATION_GUIDE.md) | External API integration guide | | [architech/api-schema/sql_execution_api_usage.md](architech/api-schema/sql_execution_api_usage.md) | SQL execution API description | | [docs/guides/getting-started.md](docs/guides/getting-started.md) | Developer quick start | | [architech/design/ORACLE_INTEGRATION_GUIDE.md](architech/design/ORACLE_INTEGRATION_GUIDE.md) | Oracle data source access | | [db-prod/README.md](db-prod/README.md) | Database migration and idempotent apply tools | | [docker/README.md](docker/README.md) | Docker construction and deployment | | [architech/design/redis_key_design.md](architech/design/redis_key_design.md) | Redis Key design description | | [tests/CHECKLIST.md](tests/CHECKLIST.md) | Automated test acceptance checklist | --- ## 📂 Project Structure (Structure) ```text . ├── app/ # Backend core code (FastAPI) │ ├── api/ # API interface layer (v1 external API / portal management background) │ ├── core/ # Core configuration (middleware, database, Redis) │ ├── services/ # Business logic (metadata, permission, AI, query engine, product catalog) │ │ └── data_adapter/ # Multi-source adapter (MySQL / ClickHouse / Oracle) │ ├── utils/ # Tool class (sub-table routing, encryption and decryption) │ └── jobs/ # Asynchronous scheduling (statistical aggregation, log cleaning) ├── frontend/ # Vue 3 management background (Vite + TailwindCSS) ├── architech/ # Architecture design documents and API Schema description ├── db-prod/ # Database version migration script (V0-VNN) and apply tools ├── docker/ # Containerized packaging and Docker Compose deployment ├── docs/ # Integration guide and O&M documents ├── scripts/ # O&M auxiliary scripts ├── tests/ # Pytest automated testing and acceptance checklist └── openspec/ # Interface specification change tracking (OpenSpec) ``` --- ## 🚀 Quick Start ### Environment Requirements | Component | Version Requirements | |------|----------| | Python | 3.10+ (recommended 3.13) | | Node.js | 18+ | | MySQL | 8.0+ | | Redis | 6.0+ (optional, recommended to enable) | ### 🐳 Docker Deployment (Recommended) **1. Configure Environment** ```bash cd docker cp ../env.example .env # Configure database, Redis, ENCRYPTION_KEY, etc. ``` **2. Build Image and Export Tar** | Script | Target Environment | | :--- | :--- | | `./build_linux_x86.sh <version>` | x86_64 Linux server (most common) | | `./build_linux_arm.sh <version>` | ARM64 Linux (Kunpeng / Ampere, etc.) | | `./build_native.sh <version>` | Native architecture of the host machine, only for local trial run | ```bash # Production environment (x86 server) — Mac builds x86 package also use this script ./build_linux_x86.sh 1.0.0 ``` The output is stored in **`docker/release/`**, for example `yunshu-api_1.0.0_linux-amd64_20260628.tar`. Offline deployment: ```bash docker load -i docker/release/yunshu-api_1.0.0_linux-amd64_*.tar docker tag yunshu-api:1.0.0 yunshu-api:latest ``` > If you prompt `docker buildx` is not available (Homebrew docker + Colima common): > ```bash ./install-buildx.sh ./build_linux_x86.sh 1.0.0 ``` See [docker/README.md](docker/README.md) for details. **3. Start Service** ```bash ./start-yunshu-api-server.sh ``` The service listens to **http://localhost:8000** by default (management background, `/docs` API documentation). --- ### 🛠️ Local Development #### 1. One-click Development and Debugging (Recommended) ```bash ./dev.sh ``` Automatically clean up old builds, compile the front end, release port 8000, and start the back end in `--reload` mode, with logs written to `app.log`. #### 2. Traditional Step-by-Step Startup ```bash # 1. Prepare environment python3 -m venv venv && source venv/bin/activate pip install -r requirements.txt cp env.example .env # 2. Database initialization (interactive, support idempotent repeated execution) ./db-prod/apply-sql.sh # Without Python environment: ./db-prod/apply-sql-native.sh # 3. Create administrator (if INIT-USER-ADMIN.sql is not imported) python3 scripts/create_admin_user.py # 4. Compile front-end cd frontend && npm install && npm run build && cd .. # 5. Start back-end uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload ``` #### 3. Front-end and Back-end Separate Development ```bash # Terminal 1: Back-end uvicorn app.main:app --reload --port 8000 # Terminal 2: Front-end hot update cd frontend && npm run dev ``` --- ## 🔗 Relationship with Yunshu Ecosystem This platform is the data service layer in the **Yunshu · Intelligent · AI** system, complementary to [Yunshu · AI Agent Platform](https://github.com/RandyChen1985/yunshu-ai-agent-platform): | Platform | Positioning | |------|------| | **Data Service Platform** (this repository) | Data API, metadata governance, SQL lab, data product catalog, asset panorama, audit and RBAC | | **AI Agent Platform** | AI dialogue, Agent orchestration, ChatBI, knowledge base, MCP plugin | --- ## 🤝 Contribution Guide Welcome to submit issues and pull requests to improve this platform. 1. **Branch Specification**: Develop based on `main`, feature branch naming `feature/your-feature-name`. 2. **Commit Information**: Use **Chinese** Commit Message, follow [Conventional Commits](https://www.conventionalcommits.org/) specification. 3. **Pull Request**: Create PR, please fill in the description and test list according to [PULL_REQUEST_TEMPLATE.md](PULL_REQUEST_TEMPLATE.md). 4. **Test Acceptance**: Update [tests/CHECKLIST.md](tests/CHECKLIST.md) for new functions. 5. **Database Change**: Add `V{N}-description.sql` in `db-prod/`, ensure migration script can be idempotent repeated execution. --- ## 💬 Contact and Communication - **Issue**: Welcome to feedback problems and functional suggestions in [GitHub Issues](https://github.com/RandyChen1985/yunshu-api-data-platform/issues). - **Email**: Can contact us through Issue to get in touch. If you have any questions, functional suggestions, or want to get more technical information, welcome to scan our WeChat official account: <img src="docs/images/weixin.png" alt="WeChat Official Account" width="200" /> --- ## 📄 License This project uses [MIT License](LICENSE) open source, allowing free use, copy, modify, merge, publish, distribute, sublicense, and sell copies of this software. --- Copyright © 2025-2026 Yunshu API Data Platform Contributors. All Rights Reserved.