Content
# Ingero Fleet
[](LICENSE)
<!-- ingero-version:install-header product=ingero-fleet channel=stable -->
**Version: 0.17.1**
Cluster-side OpenTelemetry Collector distribution for GPU clusters.
Fleet receives OTLP from every [Ingero](https://github.com/ingero-io/ingero)
agent in your fleet, runs three GPU-specific computations on the stream
(MAD-based straggler threshold, NCCL collective skew, provider cost
attribution), and forwards everything to your existing observability
stack. One binary covers what would otherwise be three separate hops.
| Role | Component | Headline output |
|---|---|---|
| Straggler detection | ingero processor + extension | per-cluster threshold; pushed back in OTLP response headers so agents self-classify in real time |
| NCCL collective health | nccl processor | per-rank skew + slow-rank attribution from `libnccl` uprobes (v0.12.0+) |
| Cost attribution | provider-lookup processor | enriches every metric with `ingero.provider` so cost dashboards just work (v0.11+) |
| EE / health surface | healthee extension | `/healthz/ee` open probe + bearer-authed `/internal/ee/state` rich body (v0.12.8) |
| OTEL backbone | standard collector | OTLP receivers, TLS/mTLS, auth, batching, retry, backends -- nothing custom |
Companion service: [Ingero Echo](cmd/ingero-echo/) -- the cluster-wide event
store and MCP query layer that Fleet forwards to (v0.12.4+).
No agent needs inbound network access; everything is outbound push and
pull. Threshold is delivered to agents in the OTLP push response, so the
straggler classification path needs no extra polling.
## Quick Start
> **Looking for a worked end-to-end example?** Three multi-node
> quickstart guides take you from zero to a detected straggler on
> three GPU hosts in about 20 minutes. Pick the deployment style that
> matches your environment:
>
> - [Kubernetes (Helm)](docs/quickstart-k8s_fleet.md)
> - [Bare-metal binary](docs/quickstart-binary_fleet.md)
> - [Docker](docs/quickstart-docker_fleet.md)
>
> See [`docs/quickstart_fleet.md`](docs/quickstart_fleet.md) for a one-page
> comparison if you are not sure which to pick.
### Option A: Use the pre-built Fleet distribution
```bash
# Docker
docker run -p 4317:4317 -p 8080:8080 ghcr.io/ingero-io/ingero-fleet:latest
# Binary
# ingero-version:install-curl-version product=ingero-fleet channel=stable
VERSION=0.17.1
curl -fsSL "https://github.com/ingero-io/ingero-fleet/releases/download/v${VERSION}/ingero-fleet_${VERSION}_linux_amd64.tar.gz" | tar xz
./ingero-fleet --config fleet-config.yaml
```
### Option B: Kubernetes (Helm)
```bash
helm install ingero-fleet ./helm/ingero-fleet
```
(Chart default is `replicaCount: 1`; see High Availability section below for multi-replica guidance.)
### Option C: Add to your existing OTEL Collector
Advanced path: drop the Ingero Go modules into your existing
[OCB](https://opentelemetry.io/docs/collector/custom-collector/) manifest
and rebuild your collector instead of running ours.
```yaml
# builder-config.yaml
processors:
# ingero-version:builder-gomod-processor product=ingero-fleet channel=stable
- gomod: github.com/ingero-io/ingero-fleet/processor v0.17.1
extensions:
# ingero-version:builder-gomod-extension product=ingero-fleet channel=stable
- gomod: github.com/ingero-io/ingero-fleet/extension v0.17.1
```
```bash
ocb --config builder-config.yaml
```
### Configure the agent
Point your Ingero agent at Fleet:
```yaml
# ingero.yaml (on each GPU node)
fleet:
endpoint: https://fleet.example.com:4317
```
## Architecture
<img src="docs/assets/architecture.svg" width="800" alt="Ingero architecture: per-node agent emits OTLP to Fleet collector; Fleet aggregates via ingeroprocessor, ncclprocessor, and providerlookupprocessor; healtheeextension serves the K8s probe + EE rich body; backends are Prometheus, Grafana, MCP clients, and UDS sinks">
The detailed Fleet-Agent component view is below.
### Fleet-Agent Overview
```mermaid
graph TB
subgraph GPU Cluster
A1[Ingero Agent<br/>gpu-node-01<br/>score: 0.92]
A2[Ingero Agent<br/>gpu-node-02<br/>score: 0.91]
A3[Ingero Agent<br/>gpu-node-03<br/>score: 0.58]
A4[Ingero Agent<br/>gpu-node-04<br/>score: 0.93]
end
subgraph Fleet Service
R[OTLP Receiver<br/>gRPC :4317 / HTTP :4318]
P[Ingero Processor<br/>Score Map + MAD + EMA]
E[Ingero Extension<br/>Threshold API :8080<br/>Middleware Piggyback]
EX[Prometheus Exporter]
end
subgraph Observability
PR[Prometheus]
GR[Grafana]
end
A1 -->|OTLP push| R
A2 -->|OTLP push| R
A3 -->|OTLP push| R
A4 -->|OTLP push| R
R --> P
P -->|Set threshold| E
P --> EX
EX --> PR
PR --> GR
E -.->|threshold in<br/>push response| A1
E -.->|threshold in<br/>push response| A2
E -.->|threshold in<br/>push response| A3
E -.->|threshold in<br/>push response| A4
style A3 fill:#f66,stroke:#333,color:#fff
```
*Node 03 (score 0.58) is below the fleet threshold (0.87) - detected as straggler.*
### Detailed Communication Flow
```mermaid
sequenceDiagram
participant A as Ingero Agent<br/>(GPU Node)
participant R as OTLP Receiver<br/>:4317 gRPC / :4318 HTTP
participant M as Ingero Extension<br/>(Middleware)
participant P as Ingero Processor
participant S as ThresholdStore<br/>(in-memory)
participant T as Timer<br/>(every 10s)
participant API as Threshold API<br/>:8080
Note over A: Computes health score<br/>from 4 GPU signals
A->>R: OTLP push (HTTP :4318)<br/>metric: ingero.node.health_score = 0.92<br/>attrs: node.id, cluster.id, state<br/>header: ingero.cluster.id = cluster-prod
R->>M: HTTP request passes through middleware
M->>R: Injects response headers:<br/>X-Ingero-Threshold: 0.87<br/>X-Ingero-Quorum-Met: true
R->>P: ConsumeMetrics(OTLP payload)
P->>P: Extract health_score from payload<br/>Write to score map[cluster:node]
R-->>A: HTTP 200 + threshold headers
Note over A: Reads X-Ingero-Threshold<br/>0.92 > 0.87 = healthy
T->>P: Timer tick (every push_interval)
P->>P: Read score map (RLock)<br/>Compute MAD per cluster<br/>Apply EMA smoothing<br/>Check quorum, panic mode
P->>S: Set(cluster_id, ThresholdResult)
Note over M: Next push reads<br/>updated threshold from store
A->>API: GET /api/v1/threshold?cluster_id=cluster-prod<br/>(fallback if no piggyback)
API->>S: Get(cluster_id)
S-->>API: ThresholdResult
API-->>A: {"threshold": 0.87, "quorum_met": true}
```
### Ports and Protocols
| Port | Protocol | Component | Direction | Purpose |
|------|----------|-----------|-----------|---------|
| 4317 | gRPC | OTLP Receiver | Agent -> Fleet | Health score push (binary protobuf) |
| 4318 | HTTP | OTLP Receiver | Agent -> Fleet | Health score push (JSON). Threshold returned in response headers. |
| 8080 | HTTP | Ingero Extension | Agent -> Fleet | `GET /api/v1/threshold` fallback endpoint |
| 8081 | HTTP | Ingero Extension | Admin -> Fleet | Diagnostics endpoint (loopback only) |
| 8088 | HTTP | Healthee Extension | K8s / EE -> Fleet | `/healthz/ee` open probe + bearer-authed `/internal/ee/state` rich body (v0.12.8+) |
| 55679 | HTTP | zPages | Internal | Health/readiness probes |
| 8888 | HTTP | OTEL Telemetry | Prometheus -> Fleet | Fleet self-monitoring metrics |
### Data Sent per Push
**Agent -> Fleet (OTLP metric payload):**
```
Resource attributes:
ingero.node.id: "gpu-node-01"
ingero.cluster.id: "cluster-prod"
Gauge: ingero.node.health_score = 0.92
ingero.node.state: "active"
ingero.workload_type: "training"
HTTP header:
ingero.cluster.id: cluster-prod (for middleware routing)
```
**Fleet -> Agent (push response headers):**
```
X-Ingero-Threshold: 0.870348
X-Ingero-Quorum-Met: true
```
**Fleet -> Agent (GET fallback response):**
```json
{"threshold":0.870348,"quorum_met":true}
```
Fleet is built as a custom [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) distribution. Two custom components, everything else is standard OTEL:
| Component | Type | What it does |
|-----------|------|-------------|
| Ingero Processor | OTEL processor | Accumulates health scores, computes MAD threshold with EMA smoothing |
| NCCL Processor | OTEL processor | Per-rank skew + slow-rank attribution from `libnccl` uprobe events (v0.12.0+) |
| Provider-Lookup Processor | OTEL processor | Enriches metrics with `ingero.provider` for cost-attribution dashboards (v0.11+) |
| Ingero Extension | OTEL extension | Threshold API for agent polling and diagnostics |
| Healthee Extension | OTEL extension | `/healthz/ee` open probe + bearer-authed `/internal/ee/state` rich body (v0.12.8+) |
| Everything else | Standard OTEL | OTLP receiver, exporters, TLS, auth, batching - zero custom code |
## Key Properties
- **Stateless.** No database, no disk. Health scores and threshold live in memory. Restart rebuilds state from incoming pushes in ~10 seconds.
- **Fail-open.** If Fleet goes down, agents use their cached threshold, then fall back to local baselines. Straggler detection degrades gracefully, never blocks workloads.
- **Outbound-only.** Agents push to Fleet and poll from Fleet - all outbound connections from GPU nodes. Zero firewall changes for enterprise GPU clusters with restricted inbound access.
- **Composable.** Run Fleet as a drop-in distribution, OR add the Ingero processors and extensions to your existing OTEL Collector via [OCB](https://opentelemetry.io/docs/collector/custom-collector/) and skip the new operational footprint entirely.
- **Tiny.** ~50MB RAM, negligible CPU for typical clusters.
## How It Works
### Health Score
Each agent computes a health score (0.0 - 1.0) from four signals:
| Signal | Weight | What it measures |
|--------|--------|-----------------|
| CUDA throughput | 0.40 | CUDA operations/sec relative to baseline |
| Compute efficiency | 0.25 | Kernel launch rate relative to baseline |
| Memory headroom | 0.20 | Available VRAM fraction |
| CPU availability | 0.15 | Inverse of scheduler contention |
The throughput signal is workload-agnostic - it works for both training (step throughput) and inference (request processing rate). Baselines adapt via exponential moving average.
All four signals are normalized to [0.0, 1.0] against the agent's rolling fast-window baseline, then combined as a weighted sum. A hard floor per signal catches "close to zero" conditions (deep stalls, OOM pressure) that a weighted average could otherwise hide. The agent classifies itself against its local baseline during warmup and switches to the Fleet-computed peer threshold once quorum is met.
### Threshold
Fleet computes the straggler threshold using [Median Absolute Deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) (MAD):
```
threshold = median(scores) - k * MAD * 1.4826
```
MAD resists outliers (50% breakdown point vs 0% for mean/stddev). A single straggler - or even several - cannot shift the threshold. The `k` parameter (default 2.0) controls sensitivity.
The threshold is delivered to agents via the OTLP push response headers, eliminating a separate polling round-trip.
### Straggler Classification
```
if my_score < threshold:
I am a straggler
```
That's it. The agent emits a straggler event via OTLP and the existing remediation protocol (`--remediate` flag).
## High Availability
### Single replica (recommended for most clusters)
`replicaCount: 1` is the chart default. Vertical scale is the path to larger clusters: a single g4dn.xlarge-class node carries 100+ pushing agents at 5s intervals with p99 handler latency under 20 ms.
If a single Fleet pod dies, agents use their cached threshold (~5 min grace), then fall back to local baseline. Restarts repopulate within 1-2 push intervals.
### Multi-replica HA (when you need it)
Each Fleet replica maintains its own in-memory score map. An agent push reaches ONE replica (selected by DNS or the service mesh); that replica's map is the only one that sees the score. Each replica computes its own threshold from its subset of agents.
For multi-replica deployments, put an L7 load balancer with **consistent-hash on the `cluster_id` query parameter** (Envoy / nginx / service mesh) in front of Fleet. Every agent from one cluster lands on the same replica, eliminating cross-replica drift.
Size `statistical_min` for the per-replica visible node count, not the cluster-wide count. Alert on `sum_over_replicas(ingero_fleet_active_nodes) < expected_total_nodes` for replica starvation.
Larger-cluster topologies (gateway-based shared state) are out of scope for this release. Talk to us if you're approaching the per-replica vertical-scale ceiling.
See `docs/architecture_fleet.md` for the full behavior model and rationale.
## Fleet Configuration
```yaml
# fleet-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
ingero:
threshold:
k: 2.0 # MAD sensitivity (default: 2.0)
ema_alpha: 0.2 # Threshold smoothing (default: 0.2)
quorum:
statistical_min: 5 # Min active nodes for valid threshold
coverage_fraction: 0.80 # Coverage alert threshold
push_interval: 10s # Expected agent push interval
ttl_multiplier: 5 # Node expiry = push_interval * ttl_multiplier
extensions:
ingero_threshold:
agent_endpoint: 0.0.0.0:8080 # Agent threshold poll (fallback)
admin_endpoint: 127.0.0.1:8081 # Diagnostics (management plane only)
exporters:
prometheus:
endpoint: 0.0.0.0:9090
service:
extensions: [ingero_threshold]
pipelines:
metrics:
receivers: [otlp]
processors: [ingero]
exporters: [prometheus]
```
## Observability
Fleet emits its own metrics:
| Metric | Type | Description |
|--------|------|-------------|
| `ingero_fleet_threshold` | Gauge | Current straggler threshold per cluster |
| `ingero_fleet_active_nodes` | Gauge | Nodes actively reporting |
| `ingero_fleet_idle_nodes` | Gauge | Nodes in idle state |
| `ingero_fleet_coverage_low` | Gauge | 1 if coverage quorum not met |
| `ingero_fleet_panic_mode` | Gauge | 1 if panic mode active |
| `ingero_fleet_median` | Gauge | Fleet health score median |
| `ingero_fleet_mad` | Gauge | Fleet MAD value |
Agent-side metrics:
| Metric | Type | Description |
|--------|------|-------------|
| `ingero_agent_health_score` | Gauge | This node's health score |
| `ingero_agent_detection_mode` | Gauge | Current detection tier (fleet/cached/local/none) |
| `ingero_agent_fleet_reachable` | Gauge | 1 if Fleet is reachable |
## Documentation
- [Architecture](docs/architecture_fleet.md) - components, data flow, threshold computation
- [Deployment Guide](docs/deployment_fleet.md) - K8s, Slurm, bare metal
- [Configuration Reference](docs/configuration_fleet.md) - all config parameters
- [API Reference](docs/api_fleet.md) - threshold endpoints
- [Integrations](docs/integrations_fleet.md) - add Fleet to vanilla OTel Collector, Grafana Alloy, Datadog, New Relic, Groundcover, Splunk, AWS ADOT
- [End-to-end walkthrough on Lambda Cloud](examples/lambda-e2e/) - A100 + GH200 (arm64) reference deploy
- [Ingero Echo](cmd/ingero-echo/) - companion cluster-wide event store + MCP query layer that Fleet forwards to (v0.12.4+)
## Requirements
- Ingero agent v0.10+ on each GPU node
- Go 1.22+ (for building from source)
- Kubernetes 1.24+ (for Helm deployment)
## License
Apache License 2.0. See [LICENSE](LICENSE).
