java-obs-agent

# java-obs-agent [](LICENSE) [](README.zh-CN.md) A **framework-agnostic Java observability sidecar agent** — drop a single `-javaagent` jar onto any Java (Servlet) service, then **chat with it** (via a built-in web UI, an MCP client like Claude Code, or curl) to understand the service's live state: request volume, latency, slow calls, errors, logs, thread dumps, method-level traces. It's the thing between **Arthas** (in-JVM diagnostics, no AI/aggregation) and **SkyWalking** (APM with distributed traces, no conversational agent) — plus the **Datadog Bits AI** pattern (chat-with-your-service), but self-hosted and attachable to a single JVM. > Built and validated end-to-end on a real Spring Boot service (the `interview-guide` project): attach → chat → "which route is slowest?" → agent calls its tools → answers with real metrics, then `trace_request` to pin the slow method in the call tree. --- ## What it does Attach the agent to a running Java service (no restart needed — dynamic attach like Arthas). The agent runs **inside the target JVM**: - Captures per-route request metrics (count, p50/p95/p99, error rate), slow calls, errors — into in-JVM ring buffers (zero external storage). - Captures recent logs (Logback appender) + thread snapshots (JMX). - Exposes a **MCP server** (JSON-RPC over HTTP) — any MCP client (Claude Code, Cursor) can chat with the service. - Exposes a **built-in chat web UI** (`/chat`) — an OpenAI-compatible LLM (BYO API key) orchestrates the tools to answer your questions in natural language. - On-demand **method-level diagnostics**: `watch_method` (capture a method's invocations) and `trace_request` (full method call-tree, Arthas-`trace` style). - Leaves a **Source SPI** seam so a future SkyWalking/Tempo backend plugs in without changing tools (Phase 3). ## Quick start ### 1. Build ```bash cd <repo-root> JAVA_HOME=<path-to-jdk-21> ./gradlew :obs-agent-core:shadowJar ``` Produces `obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar` (shaded + relocated ByteBuddy/Jackson; slf4j uses the host's). Use the `-all` jar (has the manifest + deps); not the plain one. > JDK 21 (GraalVM 21.0.11) builds Java 17 bytecode. The system default `java` is JDK 8 — always set `JAVA_HOME` to JDK 21 for `./gradlew`. ### 2. Attach to a running service (dynamic, no restart) ```bash # find the service PID (e.g. on port 8080) PID=$(netstat -ano | grep LISTENING | grep ":8080" | head -1 | awk '{print $NF}') # attach the agent into the running JVM (Arthas-style) JAVA_HOME=<path-to-jdk-21> JAR="<repo-root>/obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar" ARGS='port=9911,token=tok,enableInstrumentation=true,chat.apiKey=<YOUR_LLM_KEY>,chat.base_url=<https://...>,chat.model=<glm-4|gpt-4o-mini|...>' "$JAVA_HOME/bin/java" --add-modules jdk.attach -cp scripts Attach "$PID" "$JAR" "$ARGS" ``` The agent starts an MCP server + (optional) chat UI on port `9911` inside the target JVM. It stays until the JVM restarts. > The `Attach` helper is a one-liner using `com.sun.tools.attach.VirtualMachine.loadAgent` (the `agentmain` entry point). It ships at `scripts/Attach.java`. To use it, compile once: `"$JAVA_HOME/bin/javac" --add-modules jdk.attach -d scripts scripts/Attach.java`. ### 2b. Or attach at JVM startup (`-javaagent`, persistent across restarts) ```bash java -javaagent:<repo-root>/obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar=port=9911,token=tok,enableInstrumentation=true -jar your-app.jar ``` Use this if you want the agent resident from boot (survives restarts). Dynamic attach (above) is for ad-hoc diagnosis without restart. ### 3. Use it **Chat web UI** (browser): open `http://localhost:9911/chat?token=tok` — a chat box; type questions, the LLM calls the tools and answers. **Claude Code** (MCP client): ```bash claude mcp add --transport http obs-agent http://localhost:9911/mcp --header "Authorization: Bearer tok" # then chat: "which route is slowest? any recent errors?" ``` **curl** (manual tool call): ```bash curl -X POST http://localhost:9911/mcp \ -H "Authorization: Bearer tok" -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"recent_metrics","arguments":{"route":"/api/x"}}}' ``` > All endpoints require `Authorization: Bearer <token>`. The chat **UI page** also accepts `?token=<token>` (browsers can't send headers on URL navigation). --- ## Configuration All config is in the agent args (comma-separated `key=value`), optionally overridden by env vars (`OBSAGENT_PORT`, `OBSAGENT_HOST`, `OBSAGENT_TOKEN`). | Key | Default | Description | |---|---|---| | `port` | 9000 | MCP + chat HTTP port | | `host` | `127.0.0.1` | Network interface to bind. Defaults to loopback (**not** exposed). Set `0.0.0.0` only behind a firewall/reverse proxy **and** with `token` set. | | `token` | (none → no auth) | MCP/chat Bearer token. **Set it** if the port is exposed beyond localhost. | | `window` | 300s | Metrics/log retention window | | `bucket` | 1s | Metrics time bucket | | `slowThreshold` | 500ms | Slow-call threshold | | `logBuffer` | 2000 | Log ring buffer size | | `enableInstrumentation` | false | Master switch for `watch_method`/`trace_request` (instrumentation tier) | | `watchLimit` | 10 | Max captures per `watch_method` | | `watchTtl` | 60s | `watch_method` auto-uninstall TTL | | `scrubPatterns` | (none) | Regexes (semicolon-separated) to scrub from tool results (tokens/PII) | | `chat.apiKey` | (none → chat off) | LLM API key. Set to enable the built-in chat. | | `chat.base_url` | — | OpenAI-compatible base URL (e.g. `https://api.openai.com/v1`, `https://open.bigmodel.cn/api/paas/v4`, `https://ark.cn-beijing.volces.com/api/coding/v3`) | | `chat.model` | — | Model name (e.g. `gpt-4o-mini`, `glm-4`, `glm-5.2`) | When `chat.apiKey` is unset, only MCP is exposed (chat UI returns 404). --- ## Tools **Read-only** (always available): | Tool | What it does | |---|---| | `get_routes` | Known routes + health (count, error rate, p95) | | `recent_metrics(route?, window_seconds)` | Per-route count / p50/p95/p99 / error rate / qps | | `slow_calls(top)` | Slowest recent requests (with traceId if host sets MDC) | | `recent_errors(top)` | Recent error requests | | `search_logs(query, level?, traceId?)` | Search the in-JVM log ring (host Logback) | | `thread_dump(top, by=cpu\|blocked)` | Top CPU threads + deadlocks | **Instrumentation tier** (gated by `enableInstrumentation=true`, default off): | Tool | What it does | |---|---| | `watch_method(class, method, limit?, ttl_seconds?)` | **Start** capturing a method's invocations (args/return/duration/exception). Returns immediately (two-phase). | | `watch_results(class, method)` | **Read** the captures from a prior `watch_method`. | | `trace_request(class_pattern, ttl_seconds?, limit?)` | **Start** a method call-tree trace (Arthas-`trace`) for classes matching the regex. Returns immediately. | | `trace_results()` | **Read** the call trees from a prior `trace_request`. | **Two-phase design**: `watch_method`/`trace_request` install + return immediately; the trace/watch stays active for `ttl_seconds`. Generate traffic (hit the endpoint, or rely on frontend traffic), then call `watch_results`/`trace_results` to read. This is critical — a single blocking call would prevent the caller (LLM) from generating traffic during the window (→ empty results). ### trace_request usage - `class_pattern` is a **regex**. Use the FQCN for specific classes. - To see **cross-class call trees** (controller → service), trace **both** classes with `|`: e.g. `class_pattern="interview.guide.modules.x.KnowledgeBaseController|interview.guide.modules.x.service.KnowledgeBaseListService"`. - Repeated `trace_request` calls **reset** the prior trace before installing the new one (no advice stacking). - Example flow (via chat or MCP): 1. `trace_request("...KnowledgeBaseController|...KnowledgeBaseListService", ttl_seconds=15)` 2. hit the endpoint a few times 3. `trace_results()` → tree: `controller.getStatistics (335ms) → service.getStatistics (333ms)` --- ## Architecture ``` ┌──────────────────────────────────────────────────────────┐ │ LLM clients (BYO): chat UI (/chat) / Claude Code / curl │ └──────────────────┬───────────────────────────────────────┘ │ MCP (JSON-RPC over HTTP) + /chat (POST), token auth ┌──────────────────▼───────────────────────────────────────┐ │ obs-agent-core (inside the target JVM, shaded jar) │ │ ┌───────────┐ ┌────────────┐ ┌─────────────────────┐ │ │ │ MCP server│ │ ChatEndpoint│ │ ToolRegistry │ │ │ │ (JdkHttp) │ │ (LlmClient, │ │ (10 tools) │ │ │ │ │ │ agent loop)│ │ │ │ │ └───────────┘ └────────────┘ └──────────┬──────────┘ │ │ ┌──────────────────────────────────────────▼──────────┐ │ │ │ Source SPI (extension seam): │ │ │ │ MetricsSource / LogSource / TraceSource / │ │ │ │ TopologySource (+ capability flags) │ │ │ └──────────────────────────┬───────────────────────────┘ │ │ ┌──────────────────────────▼──────────────────────────┐│ │ │ Phase 1 in-JVM impls: ring buffers / Logback / JMX ││ │ └──────────────────────────────────────────────────────┘│ │ ┌──────────────────────────────────────────────────────┐│ │ │ ByteBuddy instrumentation (shaded + isolated): ││ │ │ - ServletMetricsInstrumentation (javax+jakarta) ││ │ │ - MethodInstrumenter (watch) - MethodTracer (trace) ││ │ └──────────────────────────────────────────────────────┘│ └────────────────────────────────────────────┬──────────────┘ (future) ┌──────────────────────▼──────────────┐ │ Phase 3: SkyWalkingTraceSource etc. │ │ (query OAP, zero-change to tools) │ └──────────────────────────────────────┘ ``` **Key design points:** - **Single shaded jar** — ByteBuddy/Jackson relocated to `io.obsagent.shaded.*`; slf4j is `compileOnly` (uses the host's, so the agent can read host Logback/MDC). No host-classpath pollution. - **No new deps for chat** — uses JDK `java.net.http.HttpClient` (OpenAI-compatible). - **Advice never throws into user code** — all ByteBuddy advice is wrapped in try/catch + JUL logging. - **Advice-accessed members are `public static`** — ByteBuddy inlines advice into target classes in *other packages*; package-private fields/methods would `IllegalAccessError` (a real gotcha — see Troubleshooting). - **`agentmain`** — enables dynamic attach to a running JVM (not just `-javaagent` at startup). --- ## Roadmap - ✅ **Phase 1** — core agent: servlet metrics + ring buffers + Source SPI + in-JVM sources + MCP server + read-only tools + scrubber/guard + shaded jar + e2e smoke test. - ✅ **Phase 2** — `watch_method` (method capture) + `trace_request` (call-tree) + validation demo. - ✅ **Built-in chat** — LlmClient (OpenAI-compatible) + ChatEndpoint (agent loop) + web UI at `/chat`. - ✅ **Dynamic attach** (`agentmain`) + two-phase trace/watch + trace reset (no stacking) + cross-class call trees. - ⏳ **Phase 3** (deferred) — `SkyWalkingTraceSource`/`SkyWalkingTopologySource` (query OAP) → `distributed_trace`/`service_topology` tools → cross-service business-chain root cause (order→pay). The Source SPI is already in place. - ⏳ **Optional tail** — Spring Boot adapter (route templates/Actuator/Micrometer traceId); streaming chat (token-level SSE); `obs-agent-cli`; reactive/WebFlux; Java 8 bytecode broadening. --- ## Known limitations - **`trace_request` is pattern-based, not traceId-based** — the host must set MDC `traceId` (e.g. Micrometer Tracing) for `slow_calls`/`search_logs` to correlate by traceId. Without it, those fields are `null` (route+latency diagnosis still works). - **Broad `.*` patterns** capture the entry method but may not show cross-class callees as children — trace the **specific classes** (with `|`) for cross-class trees. - **Chat is non-streaming** (POST `/chat/completions`, `stream=false`) — returns the full answer after the tool-calling loop. Live token streaming is a follow-up. - **Servlet-only** (blocking) — reactive/WebFlux not yet instrumented. - **Java 17 bytecode** — broadening to Java 8 (attach to older JVMs) is a follow-up. - **Agent can't be hot-updated in a running JVM** — re-attaching a new jar doesn't redefine already-loaded classes (system classloader caches them). To pick up code changes, restart the target JVM + re-attach (or use `-javaagent` at startup). - **`watch_method`/`trace_request` default OFF** — set `enableInstrumentation=true` to use them. --- ## Troubleshooting **MCP client (Claude Code) fails to connect with "expected object, received undefined" for `capabilities`/`serverInfo`** — the `initialize` result must include both (MCP spec). Fixed (returns `{"protocolVersion","capabilities":{"tools":{}},"serverInfo":{"name":"obs-agent","version":"0.1.0"}}`). **MCP tools return "content expected array, received object" / content blocks fail validation** — `tools/call` result must be `{"content":[{"type":"text","text":"<json>"}],"isError":false}` (array of content blocks, each with a `type`). Fixed. **`search_logs` returns "no log source" / `traceId` always null** — slf4j must NOT be shaded (so the agent uses the host's slf4j to reach Logback + MDC). slf4j is `compileOnly` + not relocated. If you see this, check the shadow config. **Browser opens `/chat` → `{"code":-32001,"message":"unauthorized"}`** — browsers can't send `Authorization` headers on URL navigation. Open `http://localhost:9911/chat?token=tok` (query param accepted). **`trace_request` returns empty `{"trees":[]}`** — (1) it's **two-phase**: call `trace_request` (install), generate traffic, then `trace_results` (read). The install call returns immediately, it does NOT capture. (2) Make sure `enableInstrumentation=true`. (3) Verify the `class_pattern` regex matches (use the FQCN; `.` matches any char, `*` is a quantifier). **`trace_request` shows duplicate/nested trees** — fixed: repeated `trace_request` now `reset()`s the prior transformer before installing. If you still see it, you're on an old jar (rebuild + re-attach). **ByteBuddy advice silently does nothing (empty captures, no error)** — the advice accesses package-private members from a cross-package target → `IllegalAccessError` swallowed by the try/catch → returns null. Advice logic must be in `public static` methods (e.g. `onEnter`/`onExit`) so the inlined advice only calls public methods. The unit test passes (same-package test class) but live cross-package targets fail — always verify on a fresh JVM with a real cross-package target. **Build fails with JDK issues** — system `java` is JDK 8; the project needs JDK 21. Always run `./gradlew` with `JAVA_HOME=<path-to-jdk-21>`. **Re-attaching a second agent doesn't apply new code** — the first attach's classes are already loaded by the system classloader; `loadAgent` won't redefine them. Restart the target JVM + attach once. --- ## Development ```bash # build JAVA_HOME="..." ./gradlew :obs-agent-core:build # tests JAVA_HOME="..." ./gradlew :obs-agent-core:test # e2e smoke (real -javaagent attach + MCP) JAVA_HOME="..." ./gradlew :obs-agent-core:test --tests 'io.obsagent.EndToEndSmokeTest' -Pe2e=true # shaded agent jar JAVA_HOME="..." ./gradlew :obs-agent-core:shadowJar ``` Layout: - `obs-agent-core/src/main/java/io/obsagent/` — `AgentPremain`, `config/`, `data/` (ring buffers + `model` records), `source/` (SPI + `injvm/` impls), `instr/` (ByteBuddy: servlet/watch/trace), `mcp/` (JdkHttpMcpServer), `tools/` (ReadOnlyTools + InstrumentingTools), `safety/` (Scrubber + Guard), `chat/` (LlmClient + ChatEndpoint + ChatPage). - `docs/superpowers/specs/` — design spec. - `docs/superpowers/plans/` — Phase 1 / Phase 2 / chat+trace implementation plans. - `.superpowers/sdd/progress.md` — SDD execution ledger (gitignored). --- ## How it was built Designed via brainstorming → spec → implementation plans → subagent-driven development (12 Phase-1 tasks + 4 Phase-2/chat-trace tasks), each task = implementer subagent + review + fixes. ~40 commits, 29 main + 20 test Java files. Validated live on the real `interview-guide` Spring Boot backend (dynamic attach → MCP chat → real metrics → `trace_request` call-tree pinning the slow method). See `docs/superpowers/specs/` and `docs/superpowers/plans/` for the full design + task breakdown.