chatlog_alpha

# Tool List WeChat 4.x chat log local query tool, supporting `macOS` and `Windows`. > Warning: The Windows platform has not been fully tested and may not run normally. Please verify in a test environment before using it for formal data. ## Recent Update Log ### 2026-04-26 - Experimental semantic capabilities default switch to local Ollama: embedding defaults to `qwen3-embedding:8b`, rerank defaults to `dengcao/Qwen3-Reranker-8B:Q5_K_M`, default address `http://127.0.0.1:11434`. - GLM/DeepSeek adjusted to optional Chat provider: still capable of vector index, semantic search, and Ollama fine ranking without configured Chat API Key; chat capabilities like session Q&A, LLM summary, and time knowledge graph extraction require configuration. - Frontend "Configuration and Index Management" adds Ollama Base URL, DeepSeek API Key/Base URL, Embedding/Rerank/Chat Provider configuration; switching providers automatically synchronizes corresponding default model names. - Ollama call adds serial scheduling: embedding, rerank, Ollama chat share the same execution lock to avoid multiple local models loading simultaneously; the same stage reuses loaded models, and the stage switches, task ends, or connectivity tests release models. - Vector index and time knowledge graph status add processing speed and estimated completion time display for judging whether local Ollama or remote LLM tasks need suspension, concurrency reduction, or model replacement. - Correction of Ollama `qwen3-embedding:4b` dimension identification: automatically uses `2560` dimensions to avoid index writing but covering conversation, recent index messages, and retrieval by error dimensions. ### 2026-04-25 - Added "Time Knowledge Graph" experimental platform: supports chat messages, business data, and external events entering local graph extraction queues, automatically maintaining entities, relationships, events, facts, evidence, and time versions, and provides frontend graph visualization, timeline, fact relationship list, and graph Q&A. - Vector index adds pause/resume: pause cancels current construction and retains breakpoints; resume rebuilds tasks continuing from breakpoints; real-time incremental indexing does not trigger during pause. - Chunk index incremental optimization: session chunks no longer delete and rebuild entire conversations; only recalculate changed segments based on `content_hash` and clean up expired chunks, reducing repeated vectorization costs for large group chats and LLM chunks. - Chunk tail window optimization: new messages typically rebuild tail segments only from the start of the last indexed session chunk, avoiding scanning entire long conversations. - Entity candidate optimization: contact/group member candidates sort by precise matching, group hits, fuzzy matching, and vector recall, and merge the same entity sources to reduce short nickname mis-matching and unnecessary ambiguity prompts. - Frontend adds vectorized content visualization: preview message vectors, entity vectors, and chunk vectors'入库 text, type distribution, vector dimensions, norms, and top dimension bar charts in "Configuration and Index Management"; project sample vectors into a draggable 3D semantic space to observe clustering and outliers. - Vectorized content visualization enhancement: supports `message/entity/chunk` mixed views, automatically marking suspected outliers by recent neighbor distance; chunk views draw time trajectory lines for observing semantic segment continuity. - Vectorized content visualization supports quick filtering by session: search recent conversations and select group chats/contacts to view only the session's messages, chunks, and associated entities; chunk trajectories focus on selected sessions. - Experimental function page removes "Retrieve Evidence Only" entry, retaining chat-style Q&A and continuing to display evidence in answers. - Dashboard hotspot summary changed to manual trigger; click "Generate Summary" to call `summary=1` and GLM to avoid automatic consumption of LLM requests when entering the dashboard or switching time windows. ### 2026-04-24 - Removed semantic matching push: keyword hooks remove GLM Reranker-based semantic matching, retaining only literal `strings.Contains` keyword hits. - Dashboard expansion: added global overview card (total messages/active group chats/participating人数/index coverage), group chat comparison card (click to jump to message retrieval), speaker ranking (top 15 across groups), group message type grouping bar chart, 24h activity multi-line chart, daily message trend bar chart, semantic analysis summary (hot topics + privacy reminders). - Dashboard adds time range selector: overview/comparison/ranking/type/24h share one group, defaulting to "Today"; trend/summary have an independent group, defaulting to "Last 1 Month"; all support quick switching between today/last 7 days/last 1 month/last quarter/last 1 year/all. - `parseSemanticWindow` adds `90d` (last quarter) and `1y` (last year) support, aligning with frontend time selector windows. - Optimized configuration log security: TUI/HTTP service startup logs desensitize sensitive fields like `data_key`, `img_key`, `api_key`, `token`, and `secret`. - Optimized semantic incremental index: HTTP service startup actively executes an incremental index to make up for new chat records added during program closure. - Optimized vector index status: construction progress now displays as "success + failure + pending"; `pending` no longer includes failed items. - Optimized frontend database panel: database query detect改为限流并发 to avoid excessive instantaneous requests when there are many database files. - Synchronized TUI help: supplemented dashboard, push page, experimental semantic capabilities, and global search entries. - GLM experimental function access `glm-5.1` Chat Completions: session Q&A upgraded to retrieval evidence-based LLM answers, displaying cited evidence. - Session Q&A defaults to GLM streaming output, with frontend displaying answers in a typewriter effect. - Experimental function page reconstructed into GPT web-side chat interaction; model configuration, index status, delete index, rebuild index, and parameter tuning unified under the "Configuration and Index Management" secondary panel. - Semantic Q&A/search data source changed to "recent conversation -> username -> time window chat record" scope logic, supporting recent conversation quantity, specifying single chat, multiple session selection, and time window filtering. - Session Q&A adds LLM intent routing: GLM outputs limited JSON plans, and the system calls structured queries, LLM summaries, or vector RAG by `sender_messages`, `sender_semantic_search`, `chat_summary`, `stats`, `keyword_search`, and `semantic_search` intents; frontend displays `intent/entity/topic/route` debug information. - Entity candidate supports frontend click confirmation: if there are duplicate names or ambiguities, the next question carries precise filtering of `username` to send. - Direct query supports `answer_mode=list/summary/stats`: list mode returns raw evidence, summary mode calls GLM to summarize based on structured evidence, and empty recall returns clear reasons. - Optimized session Q&A RAG: automatically supplements hit message context, supports frontend multi-turn follow-up questions with context, and strengthens evidence anti-injection prompts. - Adjusted GLM Embedding default dimension to `2048`; embedding batch requests split into at most 64条, single input截断 to approximately 3072 token上限. - Optimized semantic index入库 content: filters pure image/video/voice placeholders, voice calls, retracted messages, and common short confirmations to reduce low-information messages' interference with recall and theme analysis. - Theme trend and contact profile add GLM-5.1 summary to help explain trends, profiles, and points of attention. - Vector index rebuild changed to background task; interface immediately returns task received, and frontend views progress through the status panel. - Incremental index changed to "scan conversation, recalculate only added or changed content messages," covering old message subsequent补解析导致的内容变化. - Failed index items made partially available: existing completed indexes still usable for semantic search/Q&A; failed sessions displayed separately in status. ### 2026-04-23 - Added "Experimental Function" page, carrying `GLM semantic retrieval and reranking` full entry (configuration, connectivity test, index management, semantic search, session Q&A, theme trend, contact profile). - Semantic capability made available after index readiness; frontend action buttons automatically disable/enable by index status. - Security adjustment: `semantic api_key` has no default value; `GET /api/v1/semantic/config` does not echo real key, only returns `has_api_key`; saving leaves existing key unchanged. - Index status enhancement: added `last_incremental_at / last_incremental_added / last_incremental_error` and `last_rerank_at / last_rerank_applied / last_rerank_error`. - Incremental index mechanism upgrade: besides search triggering, server adds background automatic monitoring (detecting conversation `NOrder` changes and automatically triggering incremental construction). - Optimized vector recall utilization: semantic search/Q&A candidate pool changed from "only recent records" to "recent records + time-stratified sampling"; long time windows and `all` queries cover earlier historical messages; pre-search bottom-up incremental add 30 seconds throttling to reduce repeated index checks during continuous Q&A. - Added entity vector index: contacts, group chats, group member nicknames, remark names, nicknames, and WeChat IDs vectorized separately; Q&A routing analysis entities fuse precise matching, fuzzy matching, and entity vector recall to reduce missed matches due to nicknames/aliases not in message content. - Added chunk-level semantic index: same conversation divided into conversation chunks by dynamic boundaries (about 30条/30 minutes, longer silence intervals, low theme overlap); and generates time segment summary chunks, theme word chunks; semantic search/Q&A fuses single message recall and chunk recall to improve long discussion and cross-message theme hit issues. - RAG Q&A enhancement: evidence deduplicated and compressed; low-confidence recall rejects answers and prompts narrowing scope; answering prompt forces critical fact citation of evidence numbers; frontend evidence table displays `source/chunk_type/score/rerank_score` for judging answer credibility. - Theme trend and contact profile upgraded to chart display (time window support: today, last 7 days, last 1 month, all). - History/检索 interface径修正: `history/search` adds unified `total_count + limit + offset`; filtering changed to "filter then paginate" to fix hour filter statistical inconsistencies. ### 2026-04-22 - WeChat keyword push supports Hermes Agent Weixin Channel; frontend can read/save Hermes WeChat configuration and perform availability checks. - Added Hermes Agent QQ push channel, supporting read/save `QQ_APP_ID`, `QQ_CLIENT_SECRET`, `QQBOT_HOME_CHANNEL`, and sending text and media through Hermes `QQAdapter`. - Push page capability integration: supports keyword push, real-time full forwarding, specified contact forwarding, specified group chat forwarding, and displays results of each push method. ### 2026-04-21 - Friend circle media proxy decryption enhancement: aligned reference implementation fixes `keystream reverse` and decryption verification strategy to reduce "successful return but media unplayable" probability. - Added official WASM priority decryption link (fallback to local implementation if failed), improving video号 sample compatibility. ## Platform and Capability - Database Key acquisition: built-in scanning process (compatible with `all_keys.json`) - Picture Key acquisition: built-in scanning and verification process - Data query: HTTP + MCP (wx-cli style interface) - Data source: built-in `wcdb_api` compatible query link (no external DLL) - Global search: supports fast search across all databases / deep search - Friend circle media: supports image, video, and real-time media decryption - Keyword push: frontend/TUI synchronous configuration, supporting MCP active push and POST notification - Also supports pushing to Hermes Agent's WeChat home channel - Also supports pushing to Hermes Agent's QQ home channel - Push event: supports persistent saving, startup recovery, and one-click cleanup - Time knowledge graph: supports business/event push, chat message extraction, relationship evolution, evidence chain, timeline, and graph Q&A ## GitHub Automatic Build Product The current `Release` workflow automatically builds the following platforms and architectures: - `darwin/amd64` - `darwin/arm64` - `windows/amd64` - `windows/arm64` Release will generate corresponding compressed packages and binary files (Windows as `.exe`) in `dist/`. ## Quick Start ### Local Run ```bash go run . ``` or: ```bash go build -o chatlog ./cmd/chatlog ./chatlog ``` ### Common Commands (CLI) ```bash chatlog http list ``` ```bash chatlog http call --endpoint history --query chat=<chat ID> --query limit=100 --query format=json ``` ### HTTP Interface Command Line Call (Full Interface) ```bash # List all callable HTTP interface aliases chatlog http list # Call by alias (example: chat log) chatlog http call --endpoint history --query chat=<chat ID> --query limit=100 --query format=json # Call by raw path (example: execute SQL) chatlog http call --path /api/v1/db/query --query group=message --query file=message_0.db --query sql='select count(*) c from MSG' # Global search (quick / deep) chatlog http call --path /api/v1/db/search --query keyword=朋友圈 --query mode=deep --query limit=100 # Media interface (template path parameter) chatlog http call --endpoint image --path-param key=<image_key> # Friend circle media proxy decryption chatlog http call --path /api/v1/sns/media/proxy --query key=<enc_key> --query url='<sns_media_url>' ``` Skill documentation: `skills/chatlog-http-cli/SKILL.md` ## macOS Permission Description (Must Read) ### 1) Recommended to run with `sudo` macOS memory reading relies on `task_for_pid`, suggesting root startup. ### 2) If using setuid solution (executable file automatically roots) Please execute after recompilation: ```bash BIN_PATH="/your actual path/chatlog" sudo chown root:wheel "$BIN_PATH" sudo chmod 4755 "$BIN_PATH" ls -l "$BIN_PATH" ``` See `-rwsr-xr-x` to indicate effectiveness. ### 3) SIP On most machines, only root may still be insufficient to read WeChat process memory. To stably scan Key, usually also need to disable SIP (System Integrity Protection). ## Windows Permission Description - Please start the program with "administrator privileges"; otherwise, it may fail to read WeChat process memory. ## HTTP Interface (Summary) Basics: - `GET /health` - `GET /api/v1/ping` Media: - `GET /image/*key` - `GET /video/*key` - `GET /file/*key` - `GET /voice/*key` - `GET /data/*path` - `GET /api/v1/sns/media/proxy` Queries (wx-cli style): - `GET /api/v1/sessions` - `GET /api/v1/history` - `GET /api/v1/search` - `GET /api/v1/unread` - `GET /api/v1/members` - `GET /api/v1/new_messages` - `GET /api/v1/stats` - `GET /api/v1/favorites` - `GET /api/v1/sns_notifications` - `GET /api/v1/sns_feed` - `GET /api/v1/sns_search` - `GET /api/v1/contacts` - `GET /api/v1/chatrooms` Database Debugging: - `GET /api/v1/db` - `GET /api/v1/db/search` - `GET /api/v1/db/tables` - `GET /api/v1/db/data` - `GET /api/v1/db/query` - `POST /api/v1/cache/clear` Semantic Search (Ollama Embedding + Rerank, GLM/DeepSeek Chat optional): - `GET /api/v1/semantic/config` - `POST /api/v1/semantic/config` - `POST /api/v1/semantic/test` - `GET /api/v1/semantic/index/status` - `POST /api/v1/semantic/index/rebuild` - `POST /api/v1/semantic/index/clear` - `GET /api/v1/semantic/search` - `POST /api/v1/semantic/qa` - `POST /api/v1/semantic/qa/stream` (SSE streaming QA, frontend default usage) - `GET /api/v1/semantic/topics` - `GET /api/v1/semantic/profiles` Time Knowledge Graph: - `POST /api/v1/graph/ingest/message` - `POST /api/v1/graph/ingest/business` - `POST /api/v1/graph/ingest/event` - `GET /api/v1/graph/status` - `POST /api/v1/graph/rebuild` - `POST /api/v1/graph/pause` - `POST /api/v1/graph/resume` - `GET /api/v1/graph/query` - `GET /api/v1/graph/timeline` - `GET /api/v1/graph/visualize` - `POST /api/v1/graph/qa` Keyword Push (Frontend "Keyword Push" page sync with TUI): - `GET /api/v1/hook/config` - `POST /api/v1/hook/config` - `GET /api/v1/hook/status` - `GET /api/v1/hook/events` - `POST /api/v1/hook/events/clear` - `GET /api/v1/hook/stream` (SSE real-time events) - `GET /api/v1/hook/hermes/weixin` - `POST /api/v1/hook/hermes/weixin` - `GET /api/v1/hook/hermes/qq` - `POST /api/v1/hook/hermes/qq` Output Format: - Default `YAML` - Optional `JSON` (`format=json`) Query Interface (Latest): - `GET /api/v1/history` - Added optional filter parameters: `hour` (0-23), `is_self` (`1/0`), `sub_type`, `has_media` (`1/0`) - `hour` not passed or empty means "all hours" - Filter order is "filter first, then paginate" to avoid `limit` truncation leading to statistical deviation - Returned fields include: - `total_count`: filtered total count - `count`: current page count - `limit` / `offset` - `GET /api/v1/search` - Supports `offset` - Result process is "aggregate and sort first, then paginate" - Returned fields include `total_count` / `count` / `limit` / `offset` - `GET /api/v1/stats` - Now is real-time calculation (removed server-side cache) - Returned fields: `query_since` / `query_until` / `query_range_label` - Group chat `active_senders` is real deduplicated speakers (not TopN length) - `GET /api/v1/contacts` - Default `limit=500` - Supports `is_friend` filter (`1/0/true/false`) - `GET /api/v1/chatrooms` - Default `limit=500` YAML Readability Optimization: - `history/search/stats` changed to structured output (fixed field order, avoid map random order) - Merged forwarding/note media content, when host is missing, no longer generates `http:///...` empty link, but falls back to text placeholder ## Global Search - Frontend page: access root page `http://127.0.0.1:5030/`, switch to "Global Search" tab. - Interface: `GET /api/v1/db/search` - Parameters: - `keyword`: search keyword - `mode`: `quick` or `deep` - `limit`: result total count upper limit, default `100`, max `500` - Returned content includes hit: - Database group - Database file - Table name - Column name - Row identifier - Hit content preview Description: - `quick`: prioritize performance, suitable for frontend real-time search. - `deep`: more comprehensive, will additionally try to parse compressed message body and part of binary fields, slower. ## Dashboard Frontend entry: root page `http://127.0.0.1:5030/` "Dashboard" tab, integrated group chat data visualization. Group statistics: top dropdown selects private/group chat session, displays message type pie chart, active time period bar chart, and send/receive ratio. Group chat data overview (6 items): 1. **Overview card** — group chat message total count, active group count, active members (accumulated by group), semantic index coverage count 2. **Group chat comparison card** — each group message type ratio, active speakers, peak time period, click card to jump to corresponding group chat message retrieval 3. **Speaker ranking** — cross-group top 15 speakers and message count 4. **Group chat comparison chart** — combined display of each group message type structure and 24-hour activity, avoid repetitive module dispersion display 5. **Message trend** — daily message volume bar chart based on raw message statistics, not dependent on semantic index 6. **Hotspot summary** — based on raw message extraction hotspot theme; GLM available, completed by LLM Chinese word segmentation, synonymous merging, and noise filtering, then backend scans full message statistics support; GLM unavailable, automatically falls back to local word segmentation. Also, statistics "most @ed" ranking, and supports Markdown summary display. Time range filter: - Overview/group chat comparison/ranking/type comparison/24h activity share a set of time selectors, default "today", support recent 7 days, recent 1 month, recent quarter, recent 1 year, all. - Message trend/hotspot summary share independent time selector, default "recent 1 month", support today, recent 7 days, recent quarter, recent 1 year; hotspot summary needs to click "generate summary" manual trigger, avoid entering dashboard automatic consumption of LLM requests. Description: - Overview card, group chat comparison, and ranking filtered by `/api/v1/stats` `time` parameter (`last-1d` / `last-7d` / `last-30d` / `last-3m` / `last-1y` / `all`). - Message trend and hotspot summary filtered by `/api/v1/dashboard/trend` `window` parameter (`today` / `7d` / `30d` / `90d` / `1y`), this interface directly reads raw messages; trend chart uses `summary=0` quick return, hotspot summary clicks "generate summary" then uses `summary=1`. GLM input includes raw messages sampled and deduplicated by date and session, daily trend, LLM merged topic, and @ ranking; response returns `topics_source` / `topics_error`, frontend will indicate topic source as "LLM topic merging" or "local word segmentation fallback". - Semantic index coverage card displays `detecting / not enabled / building / not established / indexed count / unavailable`, avoid interface exception or empty index performance. - Group statistics (private/group chat independent analysis) use each panel's `since` parameter filter, default "recent 7 days". ## Experimental Semantic Capability (Ollama Embedding/Rerank + optional GLM/DeepSeek Chat) Frontend entry: - Root page `http://127.0.0.1:5030/` "Experimental Features" tab now is GPT webpage-style chat entry. - Right side can set time window, recent session count, specify single chat, select multiple recent sessions, and retrieval depth as data source/strategy. - Time window is default range; if question contains explicit time expression like "today/yesterday/April 23, 2026/nearly a month/historical", backend will automatically override default time window. - Under dialogue box "configuration and index management" sub-panel provides model parameters, connectivity test, index status, delete index, rebuild index (resume from breakpoint), and topic/profile tools. Configuration and test: - Supports configuration of `ollama_base_url`, `embedding_provider`, `rerank_provider`, `chat_provider`, `api_key`, `base_url`, `deepseek_api_key`, `deepseek_base_url`, `embedding_model`, `rerank_model`, `chat_model`, `chat_max_tokens`, `chat_temperature`, `embedding_dimension`, `recall_k`, `top_n`, `similarity_threshold`. - Supports configuration of `index_workers` (concurrent index threads, default 1, max 32). - Semantic capability is experimental fixed capability (frontend cannot close); `POST /api/v1/semantic/test` only does temporary connectivity test for current form configuration, does not save configuration, start index, or change functional state. Embedding not configured or disconnected, index, retrieval, and QA prohibit startup. - Ollama default address is `http://127.0.0.1:11434`, default uses `qwen3-embedding:8b` and `dengcao/Qwen3-Reranker-8B:Q5_K_M`. - Local Ollama model uses serial scheduling: only one Ollama model runs at a time, avoids embedding, rerank, chat occupying memory/GPU simultaneously; same stage reuses loaded model, stage switching, task ending, or connectivity test ends release actively. Cost is cold start overhead when switching embedding/rerank/chat stage. - Performance hint: `qwen3-embedding:8b`, 8B-level rerank/chat model has high memory and CPU/GPU pressure, 16GB memory machine recommends keeping `index_workers=1`, if necessary, change to `qwen3-embedding:4b` or smaller model; Ollama scenario increasing concurrency usually does not linearly accelerate, but may increase queuing, model switching, and memory pressure. - Fee hint: GLM, DeepSeek, and other remote APIs charge by model, input token, output token, and request count. Rebuilding vector index, enabling LLM chunk summary, time knowledge graph extraction/verification, hotspot summary, topic/profile, and session QA may generate a large number of requests; recommends testing with small time window and few sessions first, then full rebuild or high concurrency extraction. - `api_key` only used for GLM Chat or GLM provider, `deepseek_api_key` only used for DeepSeek Chat; both have no default value, configuration interface does not echo real key, only returns `has_api_key` / `has_deepseek_api_key` marker. - `POST /api/v1/semantic/config` if `api_key` left blank, will retain saved key (will not clear). - Default model: - embedding provider: `ollama` - embedding: `qwen3-embedding:8b` - rerank provider: `ollama` - rerank: `dengcao/Qwen3-Reranker-8B:Q5_K_M` - chat provider: `glm` - chat: `glm-5.1` (optional; if not configured GLM API Key, session QA, LLM summary, time knowledge graph extraction unavailable) - DeepSeek Chat optional configuration: `chat_provider=deepseek`, default `deepseek_base_url=https://api.deepseek.com`, default model `deepseek-chat`; can also manually change to `deepseek-reasoner`. - Default vector dimension is `4096`, consistent with `qwen3-embedding:8b` Ollama output; `qwen3-embedding:4b` automatically identifies as `2560` dimension. Dimension or embedding model change requires rebuilding vector index. - Embedding request limit: single array max 64 items; single input max about 3072 tokens, server-side will approximate truncate and auto-batch. - `chat_model` default calls GLM Chat Completions, request path is `<base_url>/chat/completions`; if choose DeepSeek, requests `<deepseek_base_url>/chat/completions`; if choose Ollama Chat, requests `<ollama_base_url>/api/chat`. ## Time Knowledge Graph - Frontend entry: "Time Knowledge Graph" tab page. - Local graph database path: `<WorkDir>/.chatlog_graph/temporal_graph.db`. - Graph schema includes `graph_entities`, `graph_relations`, `graph_events`, `graph_facts`, `graph_evidence`, `graph_jobs`, `graph_meta`, and `graph_source_records`. - Extraction model reuses "Experimental Features" Chat configuration; GLM API Key is required by default. No local rules are used as a fallback; if Chat is not configured, the model request fails, JSON parsing fails, or the model returns an empty extraction result, no graph result will be produced, and an error will be displayed in the graph status. - Chat message access: The graph module will scan recent conversations and establish a source queue after startup, then independently poll recent conversations and enqueue messages incrementally based on the `seq` of each conversation; no keyword/forward hook is required. The message source will carry the previous 5, subsequent 2 context messages, and conversation participants; historical enqueuing will also generate a conversation chunk source for extracting facts and relationship changes across multiple messages. Clicking the frontend "Incremental Rebuild" or calling `POST /api/v1/graph/rebuild` can manually supplement. - Graph chat data source will filter pure media placeholders, recall notifications, voice/call placeholders, attachment placeholders, extremely short confirmation statements, and pure @ summon content; the same type of low-information source in the existing queue will be directly marked as processed and no longer call GLM. - Pre-ingestion quality enhancement: alias normalization based on conversation participants, contact display names, wxid, and business entity prompts; relationship predicates will be mapped to stable normative predicates; time expressions like "today/tomorrow/next week/end of month/April 25" will be parsed into absolute time combined with message time. - Extraction queue supports concurrent workers: 1 Chat extraction worker and 1 historical enqueue worker by default, adjustable on the frontend; sources will be atomically claimed as `processing`, and automatically restored to `pending` after interruption and restart, to avoid duplicate extraction or getting stuck. - Performance and cost hints: graph extraction usually calls the Chat model for extraction, and some processes also do evidence verification/normalization, and remote GLM/DeepSeek will incur token costs; Ollama local Chat does not incur API costs, but 8B-level models have slower extraction speeds and higher memory usage, so it's recommended to start with 1 concurrent worker and adjust after confirming machine resources and model stability. - Business data and external events can be pushed through the ingest API or frontend form; the interface supports single object or array batch submission. - The `entities` of business data and `actors/targets` of external events will be used as strong entity prompts for ingestion and transmitted to the Chat model as extraction constraints. - When the Chat extraction result contains changes like `ended/updated/conflict`, the backend will close the old `active` version of the same relationship/fact and write `valid_to` for displaying the time evolution of relationships and facts. - After graph extraction, the Chat model will be called again for evidence verification and normalization: facts/relationships not supported by evidence will not be ingested; retained items will be written to `verified`, `support_score`, `canonical_statement/canonical_predicate`, and optional `conflict_group`. - Entity ingestion maintains `canonical_key/canonical_name/canonical_id` for merging nicknames, remarks, group nicknames, and different addresses of the same entity; the frontend fact/relationship list will display verification status, support degree, normalized statement, and conflict group. - Graph question answering prioritizes retrieving entities, relationships, facts, and events in the time graph, then calls the Chat model to generate answers about fact, relationship changes, and evidence sources; if Chat is not configured, it returns local evidence summaries. - `POST /api/v1/graph/rebuild` by default only supplements/runs incomplete content; passing `{"reset":true}` will clear entities/relationships/events/facts/evidence and reset the original source to be extracted. - `pause/resume` only controls the graph extraction queue, not affecting the semantic vector index. - Graph status: `GET /api/v1/graph/status` returns `source_count` / `pending` / `processing` / `processed` / `failed` / `progress_pct`, and displays `started_at` / `processing_rate_per_minute` / `estimated_seconds_left` for estimating the remaining time of the current extraction task. Vector Index: - Real-time status: `GET /api/v1/semantic/index/status` - Status fields include: - Basic construction status: `indexed_count` / `processed` / `failed` / `pending` / `total` / `progress_pct` - Estimated completion status: `started_at` / `processing_rate_per_minute` / `estimated_seconds_left` - Incremental status: `last_incremental_at` / `last_incremental_added` / `last_incremental_error` - Re-ranking status: `last_rerank_at` / `last_rerank_applied` / `last_rerank_error` - `pending` only represents the number of unprocessed conversations; construction progress is calculated based on `processed + failed`, and failed items will be displayed separately. - Index content preview: `GET /api/v1/semantic/index/preview?kind=message|entity|chunk|all&limit=20&talker=...` - Returns ingested text, type distribution, vector dimensions, vector norms, sample of the first few dimensions, 3D projection coordinates, and outlier scores, but not the complete high-dimensional vector. - `talker` is optional; if the session username is passed, messages/chunks are filtered by conversation, and entities return related entities of the conversation. - Rebuild index: `POST /api/v1/semantic/index/rebuild` - `reset=0` (default): resumes from the last interruption progress - `reset=1`: rebuilds from scratch (clears the index first) - The current task runs in the background: after the interface returns `accepted=true`, view the progress through `GET /api/v1/semantic/index/status`. - Pause index: `POST /api/v1/semantic/index/pause` - Cancels the currently running index task and retains the written index and rebuild breakpoint. - The paused status is written to the index library metadata and remains paused after service restart. - Resume index: `POST /api/v1/semantic/index/resume` - If paused before was a rebuild task, it continues in the breakpoint continuation mode with `reset=0`. - If paused before was an incremental task, it is supplemented by the real-time incremental watcher or the next retrieval. - Clear index: `POST /api/v1/semantic/index/clear` - Local index database path: `<WorkDir>/.chatlog_semantic/vector_index.db` - The same index database also maintains an entity index table for contact, group name, and group member alias recall; the `entity_count` in the index status represents the current entity index quantity. - The same index database also maintains a Chunk index table for conversation fragments, time period summaries, and topic word fragments recall; the `chunk_count` in the index status represents the current Chunk index quantity. Integrated Capabilities (6 items): 1. Semantic global search: `GET /api/v1/semantic/search?query=...&chat=...&window=7d&source_limit=50` 2. Retrieval precision ranking: `semantic/search` enables Ollama rerank by default 3. Session-level Q&A (RAG retrieval evidence + context expansion + Chat flow generation): `POST /api/v1/semantic/qa/stream` 5. Topic clustering/trends (statistical charts + LLM summary): `GET /api/v1/semantic/topics` 6. Contact/sender semantic profile (keyword aggregation + LLM summary): `GET /api/v1/semantic/profiles` Description: - Semantic Q&A and search, when no `chat/chats` is specified, will first read the recent conversation list, then retrieve chat records within the specified time window from the vector database based on each conversation's `username`. - Q&A entity resolution prioritizes contact/group chat exact matching, then combines entity vector index for contact nickname, remark, WeChat ID, group name, and group member nickname; matched candidates will be displayed in the frontend debug information. - `chat` represents single conversation forced filtering; `chats` supports comma-separated multiple `username`; `window` supports `today`, `yesterday`, `7d`, `30d`, `90d`, `1y`, `all`, and `YYYY-MM-DD`. - The time priority in the question is higher than the frontend time window and is also used for vector RAG branch to avoid deviation. - The frontend chat main entry no longer exposes manual TopN and no longer provides "only retrieve evidence" mode; retrieval depth is used to control Q&A evidence quantity: `standard` (8 items), `deep` (16 items), `wide` (30 items). Professional parameters `recall_k/top_n` are still used for default configuration and API parameter adjustment. - For precise condition questions like "a person's messages today / a person said something yesterday / a person's messages in the last 7 days", the Q&A interface prioritizes contact/group member entity resolution + original message sender filtering, not relying on vector similarity. - For hybrid questions like "did someone mention something", the Q&A interface first resolves the speaker, then retrieves the speaker's time window messages and uses GLM for evidence judgment and summary. - For media filtering questions like "what are the images/files/voice/video/emojis today", the Q&A interface directly filters original messages by media type, not through vector recall. - The index status adds coverage: displays indexed conversation count, known conversation count, uncovered conversation count, and recent indexed message time. - Entity resolution returns candidate count and ambiguity in debug information; multiple contacts or group members with the same name will be marked as `candidates=N(ambiguity)` on the frontend, and you can click candidates to confirm the precise `username`. - LLM routing now performs schema validation and one retry; returns include `answer_mode` (list/summary/stats), and empty results display clear reasons. - When there is entity ambiguity, the frontend lists candidate entities (display name, type, source, username) in the evidence area; after confirming the candidate, the next Q&A will precisely filter the sender through `entity_override`. - The frontend Q&A result top displays debug information, such as `intent=sender_semantic_search | route=llm/direct/sender+llm | entity=张三 | topic=合同延期`, for troubleshooting entity resolution, time window, and retrieval path. - The current 5/6 items are still based on lightweight statistics, and LLM summaries are used for result interpretation; subsequent replacements can be made with stronger Chinese word segmentation, clustering algorithms, or long-term image models. - Vector recall is not direct full-table brute-force comparison. The system adaptively expands the candidate pool by conversation scope and time window and uses a mix of "recent records + time-stratified sampling" to avoid `all` or long time windows only recalling recent chats. - RAG recall combines three types of vector evidence: single message, entity index, and Chunk index. When Chunk is hit, it retrieves the corresponding time period's original messages as evidence context, and summary/topic chunks are only responsible for recall, not directly replacing original chat evidence. - Optional `enable_llm_chunk` can be enabled, and rebuilding the index will call the Chat Model to generate higher-quality summaries and topics for chunks; this capability increases index time and API cost and is disabled by default. - Before Q&A generation, duplicate evidence is compressed, and weak recall executes low-confidence rejection; model answers require key facts using `[1]`, `[2]`, etc., evidence numbers, and do not allow quoting non-existent numbers. - When `realtime_index` is enabled, the service running period automatically triggers incremental index building according to conversation `NOrder` changes; semantic retrieval also performs 30-second throttling for bottom-line incremental supplement, balancing real-time and continuous Q&A performance. - After the HTTP service starts, it executes an incremental index once to supplement messages generated during the program or WeChat client shutdown but not written to the index database. - Incremental indexing scans conversation messages and skips unchanged content by `content_hash`; new messages and changed content messages are re-vectorized. - Semantic indexing skips low-information content: pure media placeholders (e.g., `[image]`, `[video]`, `[voice]`), voice calls, recall messages, and common short confirmations. Historical interfaces still return these messages. - If the index fails for a conversation, the completed part can still be used for search/Q&A; failed conversations will be displayed in the status field `failed_talkers`. ## WeChat Media Proxy and Decryption - Interface: `GET /api/v1/sns/media/proxy` - Typical parameters: - `url`: WeChat media resource URL - `key`: corresponding XML `<enc key="...">` - Behavior: - Images: decrypts with `reversed` keystream, attempts `raw` if failed - Video: XOR decrypts the first `128 KB` and uses `reversed` keystream - File header verification prioritizes; will not skip decryption just because the response header's `Content-Type` seems like an image/video - Prioritizes using official WeChat `wasm_video_decode.wasm/js` to generate keystream, falls back to local Go implementation if failed - `sns_feed` / `sns_search` returned `media_list` includes proxy address, can be directly accessed. Example: ```bash curl "http://127.0.0.1:5030/api/v1/sns_feed?limit=5" curl "http://127.0.0.1:5030/api/v1/sns/media/proxy?key=2503144471&url=https%3A%2F%2F..." ``` Automatic key supplement: - If `sns_feed` / `sns_search` has parsed the corresponding WeChat media message, the server caches the media `url -> key` mapping. - Subsequent requests to `/api/v1/sns/media/proxy?key=0&url=...` will try to automatically supplement the real key from the cache. - If the message has never been parsed and the request does not provide the correct `key`, it cannot be decrypted. Prerequisites: - To use the official WASM decryption path, the runtime environment needs to install `node`. - Without `node`, the program will fall back to the built-in Go implementation, but compatibility may be slightly worse. ## Keyword Push and Persistence - Frontend page: access the root page `http://127.0.0.1:5030/`, switch to the "Keyword Push" tab. - Configuration items are consistent with TUI: - `keywords` (multiple separated by `｜`) - `notify_mode` (`mcp` / `post` / `both` / `weixin` / `qq` / `all`, also supports combinations like `mcp,weixin`, `mcp,qq`) - `post_url` - `before_count` / `after_count` - MCP active push method name: `notifications/chatlog/keyword_hit` - Frontend "Keyword Push" page displays all triggered events, not affected by `notify_mode`. - WeChat Channel push: - Automatically reads Hermes Agent's `HERMES_HOME` or default `~/.hermes` - Prioritizes reading `.env` for `WEIXIN_HOME_CHANNEL`, `WEIXIN_ACCOUNT_ID`, `WEIXIN_TOKEN`, `WEIXIN_BASE_URL` - Also supports reading `config.yaml` `platforms.weixin.token` - If `.env` does not provide token / base_url, continues reading `weixin/accounts/<account_id>.json` - Also attempts reading `config.yaml` `platforms.weixin.extra` (e.g., `account_id/token/base_url/cdn_base_url`) - Frontend can directly read and modify the above WeChat configurations, and saving will write back to Hermes Home's `.env` - Enabling `weixin` mode checks if Hermes Agent is installed and the WeChat channel is configured complete - iLink interface limit reminder: - The interface has session state limitations; after a long time without interaction, active push may be rejected or invalid. - Experience: after actively sending a message to `clawbot`, usually can continuously push about `10` times (actual number may vary with account status and interface strategy). - QQ Channel push: - Automatically reads Hermes Agent's `HERMES_HOME` or default `~/.hermes` - Prioritizes reading `.env` for `QQ_APP_ID`, `QQ_CLIENT_SECRET`, `QQBOT_HOME_CHANNEL`, `QQBOT_HOME_CHANNEL_NAME` - Also supports reading `config.yaml` `platforms.qqbot.extra.app_id/client_secret` and `platforms.qqbot.home_channel` - Compatible reading `config.yaml` `platforms.qq` section - Frontend can directly read and modify the above QQ configurations, and saving will write back to Hermes Home's `.env` - Enabling `qq` mode checks if Hermes Agent is installed and the QQ channel is configured complete - home channel defaults to private chat treatment; to push to group chat or channel, use `group:group_openid` or `channel:channel_id` prefix - Event persistence file: - Prioritize: `<DataDir>/chatlog_hook_events.json` - Fallback: `<WorkDir>/chatlog_hook_events.json` - Clearance: - Frontend "Clear Events" button - Or call `POST /api/v1/hook/events/clear` ## MCP Endpoints: - `ANY /mcp` - `ANY /mcp/` - `ANY /sse` - `ANY /message` ### Hermes Agent Integration This project can be used as an HTTP MCP Server for Hermes. 1. Ensure chatlog HTTP service is started (default `127.0.0.1:5030`). 2. Add MCP configuration to `~/.hermes/config.yaml`: ```yaml mcp_servers: chatlog: url: "http://127.0.0.1:5030/mcp" enabled: true connect_timeout: 60 timeout: .chatlog_graph/temporal_graph.db`。 tools: resources: false prompts: false ``` 3. Or use Hermes CLI: ```bash hermes mcp add chatlog --url http://127.0.0.1:5030/mcp hermes mcp test chatlog ``` 4. Execute `/reload-mcp` in Hermes session. Loaded, tool names will appear with `mcp_chatlog_` prefix. ## Security and Privacy - All processing is done locally - Please handle decryption data and key files properly ## Disclaimer See [DISCLAIMER.md](./DISCLAIMER.md)