cockpit-agent

# Tool List [](https://github.com/SuperdeMan/cockpit-agent/actions/workflows/ci.yml) [](https://github.com/SuperdeMan/cockpit-agent/actions/workflows/nightly-e2e.yml)     [](LICENSE) > Cloud-edge collaborative smart cockpit AI Agent system. Call out "Xiaozhou Xiaozhou" to complete tasks from millisecond-level vehicle control to multi-day travel planning and minute-level in-depth research with a single voice input. LLM is only responsible for understanding and planning, while the deterministic system is responsible for execution - no vehicle control instructions are directly issued by LLM. **14** domain agents · **30** services can be started with one click · **150** end-side intention patterns / **62** vehicle control objects · **2287** backend use cases can be run through single command + **35** journey-level utterances · Streaming TTS first frame **469ms** / end-to-end voice first audio **609ms** · All data sources are real (Gaode / He Feng / Exa / Tushare / api-football) · M0a→M4 intelligent upgrade fully passed **through cross-stage combined experience acceptance** (2026-07-26) ## What can it do | You say | What the system does | |---|---| | "Adjust the air conditioner to 22 degrees" | End-side fast path, millisecond-level execution, available offline, zero LLM involvement | | "Turn on the air conditioner, play a song by Lin Junjie, and navigate to the company" | Hybrid multi-intention semantic grouping and shunting: local vehicle control/media immediate execution, slow intention parallel cloud processing, same request collaborative completion | | "Navigate to that building that looks like a bamboo shoot" | Visual landmark → LLM parsing official name → Gaode real POI verification and navigation | | "Help me plan a two-day trip to Hangzhou for the weekend" | LLM proposes a framework + deterministic pipeline lands real POI + weaves charging stations according to real power and checks daily travel | | "Create a fishing mode: seat recline, ambient light dimming" | One sentence creates a scene: LLM only compiles and creates (passes VAL word list whitelist), activation and execution are zero LLM, exit returns to the state before activation | | "Remind me to submit my weekly report before arriving at the company" | Calculate reminder time based on navigation ETA, one-click task, proactive notification when the time comes | | "Remind me to watch the game tomorrow" | Event agent and reminder agent cross-domain handover: automatic reminder before the game starts | | "Change the itinerary to indoor activities if it rains" | Deterministic rescheduling based on weather forecasts | | "In-depth research on the progress of solid-state battery production, take your time, and tell me when you're done" | Asynchronous in-depth research: millisecond-level acceptance, multi-perspective iterative retrieval, and active push of sectional reports with references | | "How is the research going?" / "Stop the research" | Long tasks in the background have accounts: can ask for progress, can stop in the middle, and restart interrupted tasks with honest notification instead of pretending to still be running | ## Interface Preview HMI "Aurora Glass · Extreme Light Cabin": horizontal screen two-column layout (left conversation flow + right "context stage" switching scenes with dialogue), liquid glass material, "Xiaozhou" light ball avatar; **bubble ↔ card ↔ right stage** three interactive elements, about 20 types of structured information cards rebuilt according to Figma source.  | Weather (card + right stage live scene) | Nearby POI (card + distance measuring map, "Nth" linkage) | |:---:|:---:| |  |  | | **Trip planning (structured trip card + dangerous operation confirmation bar)** | **Settings (voice engine / tone / LLM provider switching)** | |  |  | > All screenshots are based on **real backend data** (weather = He Feng, POI/trip = Gaode). Access `http://localhost:5173` after starting the stack locally, and speak by holding down the "Xiaozhou" light ball. ## Design Principles Four architecture commitments that remain unchanged, each with testing and verification: 1. **Planning and execution separation** - LLM only produces "intent/plan", all vehicle control is issued by deterministic Executor after VAL (vehicle control abstraction layer) permission verification, and dangerous actions require secondary confirmation. Intelligence can make mistakes, but safety cannot. 2. **Dual fast and slow systems** - high-frequency, deterministic, and safety-sensitive instructions stay on the end-side for millisecond-level response and offline availability; complex, cross-domain, and multi-turn intentions are handed over to the cloud for LLM Planner orchestration. Delay and availability are architectural constraints, not optimization goals. 3. **Agent plug-and-play** - all agents implement unified gRPC contracts + Manifest declaration (capabilities / permissions / deterministic routing fallback / card priority), discovered through the registration center. **Adding a new domain agent does not modify a single line of orchestration core code**, this rule is guarded by contract testing. 4. **Truth prioritization** - navigation/weather/search/news/stocks/sports all connect to real data sources; external data cards carry `_prov` source tracing tags (authenticity/source/data time, visible on HMI badges), strict mode `REQUIRE_REAL_PROVIDERS=on` rejects any mock resolution; if real source fails, **honest downgrade and say cannot get**, no fallback to mock fake data. Demonstrations can be downgraded, but not fabricated. ## System Architecture ```mermaid flowchart TB subgraph EDGE["End-side · Vehicle (Offline Available)"] HMI["HMI Cabin Front-end<br/>Wake-up KWS · VAD · Streaming ASR/TTS"] EGW["Edge Gateway (Go)"] EO["Edge Orchestrator<br/>Fast Intent Intention Shunting"] VAL["VAL Vehicle Control Abstraction Layer<br/>Unique Vehicle Control Outlet"] HMI <--> EGW EGW --> EO EO -->|"T0 Fast Path · Millisecond-level"| VAL end subgraph CLOUD["Cloud-side · LLM Orchestration"] CGW["Cloud Gateway (Go)"] CP["Cloud Planner<br/>T1 Single DAG · T2 Bounded Loop"] REG["Registry Registration Center<br/>Capability Semantic Retrieval"] LLM["LLM Gateway<br/>Multi-model Runtime Hot Switching"] MEM["Memory<br/>pgvector Semantic Memory"] AG["12 × Domain Agents<br/>Unified gRPC Contract + Manifest"] CGW --> CP CP <--> REG CP <--> LLM CP <--> MEM CP <--> AG end EO <-->|"Slow Intention Cloud · Persistent Bidirectional Stream"| CGW CP -.->|"DispatchToEdge: Vehicle Control Plan Back to End, Deterministic Execution + Permission Verification"| VAL ``` Service synchronization calls use gRPC (`proto/` as the only contract source), asynchronous and proactive push use NATS; short-term state uses Redis, long-term/vector uses PostgreSQL + pgvector. Requests are divided into three layers based on complexity: | Layer | What to Handle | Form | |---|---|---| | **T0 End-side Fast Path** | High-frequency deterministic instructions like vehicle control/media | Rules + Knowledge Base, millisecond-level local execution, offline available | | **T1 Cloud Single DAG** | Complex/cross-domain/multi-intention requests | LLM Planner plans once, deterministic engine executes in parallel | | **T2 Bounded Agentic Loop** | Tasks that need to adjust plans based on intermediate results | Iterative times and time budget controlled, adaptive replanning | Agent access is fully declarative: manifest declares capabilities and permissions, `route_hints` for deterministic routing fallback, `verification` declares what to verify after execution, `_escalate` for execution phase reassignment, and `heavy` marks driving thinking and process area - orchestration core has zero hardcoding for specific agents. Long tasks that survive across requests (like asynchronous in-depth research) have a persistent task ledger to record "how far it has gone / let it continue or not", access only needs to call three SDK functions. ### Safety Rules (Architecture-level, violating means bug) 1. Vehicle control can only be issued through VAL, no component (including LLM/Agent) directly operates CAN/SOME-IP. 2. LLM is only responsible for understanding and planning, deterministic Executor/Dispatcher is responsible for execution. 3. Dangerous actions (`require_confirm=true`) must be confirmed by the user a second time. 4. New agents can only be accessed through the registration center, without modifying the orchestration core. 5. Keys only go into `.env`, not into code, logs, or submissions. 6. Sensitive data (precise location/vehicle audio/video/payment) defaults to not leaving the vehicle, and cloud-side issuance is minimized according to manifest `context_scopes`. The only source of architectural truth: [`docs/architecture/cockpit-agent-architecture.md`](docs/architecture/cockpit-agent-architecture.md) - any implementation conflicting with it is considered a bug. ## Capability Panorama ### Voice: Full-duplex Interactive Loop From wake-up to interruption, the entire chain is streaming, and all engines are switchable: - **Wake-up**: browser local KWS (sherpa-onnx WASM, self-built chain), preset wake-up words "Xiaozhou Xiaozhou / Hello Xiaozhou…", audio not leaving the browser before wake-up, voice response after wake-up. - **Listening**: silero VAD endpoint detection + DashScope real-time streaming ASR - speaking and uploading simultaneously, pause and finalize automatically send, failed fallback to batch processing. - **Speaking**: server-side streaming TTS (incremental text input, PCM slice output), cosyvoice first frame 469ms (4.7~7.2× faster than batch processing), also has dialect tone engine; can be interrupted during playback (barge-in). - **Continuous Dialogue without Wake-up**: directly continue conversation within follow-up window; "退下吧" local exit without going to the cloud. - **Rejection and Clarification (confidence level three-stage)**: hands-free scene passenger dialogue etc. non-received statements silently rejected - no disturbance, no storage, no image formation (built-in evaluation mis-rejection 3.4% / interception 88.9%); real ambiguity sentences choose card questions and ask again, clear sentences never ask questions again. - **End-to-end Voice Direct Connection (optional gear)**: casual conversation and common sense by voice model directly listening and answering, first audio **609ms** (P50), multi-turn more coherent; **requests that need execution or real-time information query are automatically handed back to the deterministic chain** - no vehicle control path bypasses permission verification and secondary confirmation (tested 100% handover rate, including "it's hot today, turn down the air conditioner a bit" such sentences in casual conversation); **dangerous actions' "confirmation/cancellation" are always decided by the deterministic main chain** - while waiting for confirmation, voice model has no power to answer or cancel on behalf of you. Automatically reconnect and re-inject context if disconnected, otherwise fall back to conventional chain. Default off: it uploads raw voice within wake-up window, user needs to enable explicitly in settings. - **Distinguish between passengers (optional gear)**: after wake-up, first sentence edge judgment who is speaking (1.5 seconds enough), each passenger's taste, habit, and frequent places are independent, ask "do you know who I am" and call out the name. **If not recognized, treat as main driver** - fallback to previous behavior before enabling this option, not a new bad state; if two voices are too similar, do not draw conclusions. **Voiceprint only used for distinction, not for any permission or payment voucher** (misidentified only loss of personalization, not loss of safety). - **Look and See (optional gear)**: say "what is that / what car is this" grab a frame of current screen image and hand it to multi-modal model recognition (image only lives two minutes in gateway memory, no disk storage, no dialogue chain). If no image, directly say cannot get. ### End-side: Vehicle Control and Hybrid Multi-intention 150 intention patterns cover 62 vehicle control/media objects (air conditioner / seat / window / ambient light / 360 surround view / Bluetooth / broadcast…), knowledge base-driven normalization, verification, safety gate and dialogue; hybrid multi-intention semantic grouping and shunting, local actions and cloud slow intentions are coordinated within the same request. ### Cloud-side: 14 Domain Agents | Agent | One-sentence capability | |---|---| | `navigation` | Gaode navigation: POI search, visual landmark parsing, passing points, navigation preferences | | `nearby` | Nearby discovery (Gaode POI 2.0): catering/hotel/scenery/movie theater/parking/charging, price and business status screening | | `trip-planner` | Structured multi-day trip planning: real POI landing + power perception charging weaving + partial rescheduling no drift + in-transit status | | `charging-planner` | Charging planning: along the way/destination charging station, destination candidate secondary confirmation | | `info` | Weather (He Feng) / Search (Exa landing synthesis: mandatory reference, no evidence rejection) / News / Stocks (Tushare) / Sports (api-football) | | `deep-research` | In-depth research: multi-perspective sub-questions → bounded parallel retrieval → sectional report with reference + progressive voice briefing, support asynchronous minute-level | | `reminder` | Natural language schedule/reminder/todo: rescheduling / snooze / repeat rules, proactive notification when time comes, cross-domain handover | | `scene-orchestrator` | User-defined scene: one-sentence creation, environment adaptive strategy evaluation, exit true recovery, execution honest accounting | | `road-safety` | Road safety and responsive active announcement | | `parking-payment` | Parking payment (via unified payment gateway, Agent does not hold payment voucher) | | `manual-rag` | Car owner manual question answering (RAG) | | `chitchat` | Casual conversation and common sense direct answer (wall clock/date based on system clock deterministic direct answer, never let LLM compile time) | | `vision` | Look and See: say "what is that", grab current screen frame and hand to multi-modal model recognition (image only lives two minutes in gateway memory, no disk storage) | | `mcp-bridge` | Controlled MCP ecosystem bridge: one Agent bears N external MCP server, **artificial admission + version locking + schema fingerprint** triple lock; write operation has idempotent key/ledger accounting / uncertain oral diameter (no fake success/failure, no promised non-existent verification entry) / compensation statement / audit (first batch for demo merchants, card and dialogue constant marked) | Request mode judgment routing: one sentence accurately falls into **direct answer / network query / news / in-depth research** four modes (common sense no network, timeliness must be networked, browse a batch go to news, system understand promotion research), evaluation first (five buckets of data + confusion matrix, baseline accuracy 98.9%); Agent misconnection via `_escalate` mechanism zero broadcast automatic reassignment. ### Memory, Context, and Personalization - **Semantic Memory** (pgvector): automatically extract preferences and personal entities from conversation, semantic recall injection planning and casual conversation; privacy level, query and deletion. - **Context Assembly**: unified token budget assembly capability directory (semantic pre-screening) + conversation history + long-term memory + structured focus state (cross-wheel substitution not relying on chewing on the original text); sensitive context minimized issuance). | **Active Governance Layer**: seven active (routine / scene trigger / road condition announcement / reminder to point / deep research completion / morning news / low battery route suggestion) pass through **Unified Active Engine** - situation assertion at delivery moment, cross-domain de-duplication, high driving load stash, merge into one if arriving at the same window, and send to HMI via NATS. If governance layer absent, automatically fall back to direct issue, not silently swallow explicit user agreement reminder. ### Multi-LLM / Multi-engine Runtime - **LLM**: MiMo / MiniMax / DeepSeek / Tongyi Qianwen four vendor process registry, HMI setting page runtime hot-swapping, persistence; request-level pin, 429 and streaming failure classification downgrade, health probe; embedding and chat vendor decoupling. - **ASR**: DashScope real-time streaming (qwen3 / fun dual protocol) + chunked fallback; **TTS**: cosyvoice / qwen3 (dialect) / MiMo / MiniMax four engines, engine → tone two-stage selection. - Provider report locked (mid-drift means invalid), cross-model comparison credible. ### Observability and Badcase Closed-loop trace_id from HMI bubble corner one-click copy, throughout every LLM invocation (tokens / latency / gate content); collector SQLite persistence; Dashboard four views - conversation three-level drill-down, overview, log, badcase favorites one-click playback, also has LLM consumption attribution view. Prometheus `/metrics` + OTel span export + Grafana dashboard via `--profile observability` optional enable. ## Quick Start Dependency: Docker Desktop, Python 3.11+; local development also requires Go 1.24+, Node 20+, buf. ```bash cp .env.example .env # No key configuration can run: LLM fall back to MockProvider, external data sources use mock make proto # Generate gRPC code (proto changes require this) python test/smoke_edge.py # Optional: edge-side smoke test without Docker make up # Start 30 services ``` Windows PowerShell: ```powershell Copy-Item .env.example .env ./scripts/gen-proto.ps1 python test/smoke_edge.py docker compose -f compose.yaml up --build -d ``` After starting: - **HMI Cabin** <http://localhost:5173> - Click/hold "Xiaozhou" light ball to speak or type. - **Observability Platform** <http://localhost:5174> - Conversation drill-down, trace, LLM consumption attribution. Caution: - Only start from root `compose.yaml` ( `make up` encapsulates); direct use of `deploy/docker-compose.yaml` will lose root `.env`, real providers silently fall back to mock. - Real data sources and LLM credentials see `.env.example`; Dashboard vehicle dynamic debugging interface limited to local demo, non-development environment set `DEBUG_VEHICLE_CONTROL=false`. ## Engineering and Verification Test pyramid with all single commands reproducible: | Layer | What | Current Status | |---|---|---| | Unit Test / Contract | Full-service pytest (orchestration / Agent / security / memory / gateway) single command run-through | 2287 passed (2026-07-26) | | Frontend | HMI `node --test` / Dashboard vitest | 195 / 16 | | Evaluation Baseline | End-to-end intent coverage (8k+ real-life utterances), cloud routing, four-mode routing, rejection/clarification | Report-type, CI non-blocking gate | | Single Link e2e | WS full link / memory / context / resilience self-healing / TTS stream / voice loop / degradation matrix / task ledger / execution reconciliation / proactive governance / voiceprint / vision / S2S | Real-stack script, access to `run_e2e` (Windows/Linux list consistent) | | L3 Journey-level | 35 utterances: cross-Agent autonomous execution (getting things done) × full-scenario continuous dialogue | Regression-level evergreen (red light single re-run qualitative system) | | L4 HMI CDP | Real browser rendering / click → WS frame assertion | Secondary interaction use cases | | CI | Push full (green = local full green); nightly assertion-type e2e (mock lane) + nightly self-evolution pipeline (bad case mining → attribution → gate) | GitHub Actions + Task Scheduler | ```bash make test # = python -m pytest --import-mode=importlib -q cd hmi && npm test && npm run build cd dashboard && npm test && npm run build # After full-stack setup python test/e2e_ws.py # WS full link smoke make e2e # Local full e2e list (Windows: ./scripts/run_e2e.ps1) python test/e2e_journeys.py # L3 journey-level (--provider lock evaluation LLM) node test/hmi_cdp/run_cases.mjs # L4 real browser CDP ``` Three engineering cultures: - **Documentation First**: 60+ design and implementation records ( `docs/design/` ), each topic aligned with design before implementation; "architecture single source of truth" institutionalized. - **Bad Case Driven**: Real device/real mic feedback → Dashboard trace drill-down → fix → original sentence real-stack re-verification, full link documentation. - **Iron Law Testing**: "New Agent no orchestration core change" and "dangerous actions must be confirmed" and other architecture conventions solidified by contract testing, direct red light for violations. ## Directory Structure ```text proto/ gRPC contract - single source of truth for all interfaces gateway/ Go access gateway (edge/ end-side, cloud/ cloud-side) orchestrator/ edge/ end-side orchestration + FastIntent + VAL (PoC simulation); cloud/ cloud-side LLM Planner agents/ 14 domain Agents; _sdk/ public SDK (BaseAgent / retrieval and grounding kernel / task ledger) llm-gateway/ LLM multi-model gateway - single exit for LLM / Embedding / ASR / TTS registry/ Agent registration center (manifest + capability semantic retrieval) memory/ Memory / portrait service (pgvector) security/ Permission engine, scope definition, content review, injection protection payment-gateway/ Unified payment gateway (Agent does not hold payment credentials) proactive/ Unified proactive engine - single decision point for "whether to disturb the driver now" observability/ NATS event export, collector, trace / metrics hmi/ React cockpit frontend (Aurora Glass) dashboard/ React development / demonstration observable platform runtime/ Shared runtime (gRPC keepalive / mTLS / graceful shutdown + proactive message export) deploy/ docker-compose / certificate generation test/ e2e, evaluation baseline, journey utterances, CDP use cases docs/ Architecture (single source of truth), design records, guides ``` ## Documentation Navigation | What you want to know | Look here | |---|---| | Current real progress, first step, self-inspection entry | [`AGENTS.md`](AGENTS.md) | | Engineering conventions, directory specifications, security red lines | [`CLAUDE.md`](CLAUDE.md) | | Why this design (architecture single source of truth) | [`docs/architecture/cockpit-agent-architecture.md`](docs/architecture/cockpit-agent-architecture.md) | | Phase plan and production DoD | [`docs/architecture/phase1-implementation-plan.md`](docs/architecture/phase1-implementation-plan.md) | | Design and implementation records for each topic (60+, by date) | [`docs/design/`](docs/design/) | | How to connect real Provider (Gaode/He Feng sample) | [`docs/guides/provider-integration.md`](docs/guides/provider-integration.md) | | Environment / port / naming / error code quick check | [`docs/dev-guide.md`](docs/dev-guide.md), [`docs/conventions.md`](docs/conventions.md) | | Test pyramid and operation instructions | [`test/README.md`](test/README.md) | Each service subdirectory has its own README. ## Current Status and Boundaries Currently in **Phase 1 Engineering PoC**: T0 / T1 / T2 operation model, cloud hub, voice loop, memory/context, observability, and journey-level verification system have been implemented. Known boundaries to production: - **VAL is Python simulation** ( `orchestrator/edge/val.py` ): real SOME-IP/CAN connection, vehicle resource constraints, and OTA are production-stage. - **Parking/manual still mock Provider** (strict mode default exemption domain), access by environment. - **Single instance status**: Cloud Gateway vehicle long-connection status in single instance memory; Registry has PostgreSQL persistence and periodic re-registration self-healing, multi-instance expansion pending. - **Security capabilities implemented but disabled by default**: two-layer session authentication ( `AUTH_REQUIRED` ) and inter-service mTLS ( `GRPC_TLS` ) controlled by env gate, enabled for full-stack effect; real IdP, certificate rotation are follow-up. - **Acoustic layer metrics** (real mic hit rate / false wake rate) are manual acceptance; browser KWS / VAD link has been verified on real devices. - **MCP ecosystem bridge has been implemented but first batch is demo merchants**: protocol, access (version lock + schema fingerprint), write operation idempotence, and timeout have been verified and interoperated with official reference implementation; **compensation path currently only has access verification (zero call at runtime), user query/cancellation入口未接** — must be implemented before connecting real merchants (verified on 2026-07-26). Universal HTTP tool surface not yet implemented; third-party Agent outbound has domain whitelist forward proxy. Real-time status, test evidence, and to-do list are in [`AGENTS.md`](AGENTS.md). ## License This project is released under the [Apache License 2.0](LICENSE).