Content
# ProductArena (ultrametric.ai/productarena)
*(formerly Product Arena)*
[](https://github.com/ultrametricai/productarena/actions/workflows/ci.yml)
[](https://ultrametric.ai/productarena)
<!-- stat-badges:start -->
[](https://ultrametric.ai/productarena)
[](https://ultrametric.ai/productarena/everything)
[](https://ultrametric.ai/productarena/methodology)
<!-- stat-badges:end -->
[](./LICENSE)
[](./DATA-LICENSE)
**The open evidence layer for the agent economy.** ProductArena measures how ready the
world's software is for AI agents — and how Built-in AI it is for humans — with evidence, not
opinion: crawled docs, community sources, hands-on probes, recorded proofs, and an LLM judge
whose every verdict carries citations, a confidence grade, and a contest button.
It has grown past rankings into an operating map of the AI era:
- **Arenas** — head-to-head, story-judged rankings across 60 software categories
- **Processes** — how startups actually run, step by step, with the honest agent ceiling:
what an agent can do today and where the market still forces a human
- **Simulator** — dry-run a founder process through real market options and watch the API calls
- **Stacks & lenses** — evidence-backed toolchains, buyer-type (ICP) views, comparisons
- **Proofs** — recorded terminal/MCP sessions and a prove-it protocol vendors can submit to
- **Agent-native access** — MCP server, JSON API, llms.txt: agents are first-class readers
> ⭐ **If evidence-based software rankings are useful to you, [star the repo](https://github.com/ultrametricai/productarena)** — stars are how other builders find this, and every ranking, verdict, and proof here is open for you to audit or contest.
ProductArena is an evidence-based comparison site: evidence in, rankings out. For each of 80+
product arenas (500+ products and growing) we crawl vendor docs, GitHub, and community sources,
extract per-product evidence, and
have an LLM judge every product against a shared set of user stories. The result is a
leaderboard, a head-to-head battle log, a per-product story matrix, and — across every arena —
global rankings of how agent-ready and Built-in AI every product is. Every score traces back to
cited evidence, not opinion.
Live site: https://ultrametric.ai/productarena
<!-- stats:start -->
As of the last full pipeline run: **93 arenas, 578 products, 32,949 judged verdicts.**
<!-- stats:end -->
Arena, product, and verdict counts above (and in the badge row) are regenerated from data/ by
`pnpm stats` — never a hand-maintained claim; see "Status & roadmap" for what's next.
## Try it in 10 seconds
Nothing to install — the rankings are a public JSON API and agent-readable markdown:
```bash
curl https://ultrametric.ai/productarena/data/ai-coding/rankings.json # any arena, machine-readable
curl https://ultrametric.ai/productarena/arena/ai-coding/llms.md # same arena, markdown for agents
```
Agents: start from [`/llms.txt`](https://ultrametric.ai/productarena/llms.txt) — every arena and
product has agent-readable markdown, and the data API needs no auth. (A first-party Ultrametric
MCP server + API is coming; ProductArena itself is not offered over MCP or a CLI.)
## What's new
The news is generated from the committed score history, never hand-written: the live
[changelog](https://ultrametric.ai/productarena/changelog) (rank flips, PA Score moves, new
arenas), [weekly arena reports](https://ultrametric.ai/productarena/reports), and point-in-time
[dataset releases](https://github.com/ultrametricai/productarena/releases) if you want a stable
snapshot to build against.
## The rankings, at a glance
*Dated snapshot — rankings move whenever new evidence lands, so treat every
number in this image as historical. The live table is at
[ultrametric.ai/productarena](https://ultrametric.ai/productarena); machine-readable rankings
are at `/data/<arena>/rankings.json` per arena.*
## Quickstart
**I want to contribute (fix a verdict, add evidence, add a product):**
```bash
git clone https://github.com/ultrametricai/productarena.git && cd productarena
pnpm install
pnpm dev # http://localhost:3000
pnpm test
```
Then see [CONTRIBUTING.md](./CONTRIBUTING.md) — contesting a verdict or adding evidence is a
first-class workflow, not a favor. Founders: the ["Add your product"](./CONTRIBUTING.md#3-add-your-product)
section covers both the quick issue path (via [/submit](https://ultrametric.ai/productarena/submit))
and the PR path, plus how to make your product probe well (llms.txt, OpenAPI, remote MCP,
`.md` docs mirrors) so the pipeline sees everything you've shipped.
**I want to consume the data (curl, or an agent reading /llms.txt):**
```bash
curl https://ultrametric.ai/productarena/data/categories.json
curl https://ultrametric.ai/productarena/data/ai-coding/rankings.json
curl https://ultrametric.ai/productarena/llms.txt # index for agents
```
(A first-party Ultrametric MCP server + API is coming; ProductArena itself is not offered over
MCP or a CLI — the JSON data API and llms.txt/llms.md surfaces above are the supported ways in.)
**I want to contest a verdict (I think a score is wrong):**
Every verdict on the site has a "⚑ contest" link that opens a prefilled issue. Or file one
directly: [new contest-verdict issue](https://github.com/ultrametricai/productarena/issues/new?template=contest-verdict.yml) —
cite the evidence URL and quote the relevant excerpt so a maintainer can verify it.
## How it fits together
```
crawl → extract → normalize → collect-community → probe → judge → derive
│ │ │ │ │ │ │
└── vendor/GitHub/community pages ──┴── evidence ───┴────────┴── rankings.json
│
▼
data/{category}/*.json (checked in)
│
┌───────────────┼────────────────┐
▼ ▼ ▼
static site /data JSON API MCP server
ultrametric.ai/productarena + llms.md/txt (mcp/)
+ /openapi.json
```
## The arenas
<!-- arenas:start -->
| Arena | Products |
|---|---|
| Desktop OS (`desktop-os`) | macos, omarchy, ubuntu, fedora, windows |
| Startup Banking (`startup-banking`) | mercury, brex, ramp, wise, relay, jeeves, airwallex |
| Project Management (`project-management`) | linear, asana, clickup, notion, monday, jira |
| Web Scraping APIs (`web-scraping`) | firecrawl, crawl4ai, jina-reader, apify, scrapingbee, browserbase, riveter, context-dev |
| Mobile AI Dev Tools (`mobile-dev`) | termius, tailscale, blink-shell, a-shell, working-copy, github-mobile |
| Code Hosting (`code-hosting`) | github, gitlab, bitbucket, gitea |
| AI Coding Agents (`ai-coding`) | codex, claude-code, cursor, github-copilot, gemini-cli, opencode, devin, aider, cline, cubic, antigravity, random-labs, conductor, byteask |
| Edge & App Platforms (`edge-platforms`) | cloudflare, vercel, netlify, fly-io, railway, render |
| Frontend Frameworks (`frontend-frameworks`) | react, vue, svelte, angular, solid |
| Local LLM Runtimes (`local-llm-runtimes`) | ollama, llama-cpp, vllm, lm-studio, jan, localai, llamafile, runanywhere |
| Payroll & HR Ops (`payroll`) | gusto, rippling, deel, justworks |
| Product Feedback & Intent (`product-feedback`) | canny, featurebase, productboard, foreloop |
| Software Factory (`software-factory`) | foreloop, factory, devin, openhands, codegen, jules, omnara, yylo, humanlayer, superset |
| Mobile & In-Person Payments (`mobile-payments`) | stripe-terminal, square, sumup, adyen-pos |
| API platforms (`api-platforms`) | postman, kong, bruno, hoppscotch, insomnia |
| Team Chat (`team-chat`) | slack, discord, ms-teams, zulip, buzz |
| Backend as a Service (`backend-as-a-service`) | supabase, firebase, convex, appwrite |
| Online Payments (`payments`) | stripe, adyen, paypal, square, autumn, checkout-com, mollie, airwallex, paddle, polar |
| Accounting & Bookkeeping (`accounting`) | quickbooks, xero, puzzle, pilot, freshbooks, zoho-books, wave, digits, kick, mercury-books, bench |
| Security Scanners (`security-scanners`) | trufflehog, semgrep, snyk, gitleaks, trivy, gecko-security |
| Infrastructure as Code (`infra-as-code`) | terraform, pulumi, opentofu, crossplane |
| Vibe-Coding App Builders (`vibe-coding`) | lovable, bolt, v0, replit, base44, floot |
| Model Gateways & Routers (`model-gateways`) | openrouter, litellm, portkey, vercel-ai-gateway, cloudflare-ai-gateway, requesty, kong-ai-gateway |
| LLM Evals & Observability (`llm-evals-observability`) | langfuse, langsmith, braintrust, arize-phoenix, wandb-weave, helicone, galileo, cekura |
| AI Search APIs (`ai-search-apis`) | exa, tavily, perplexity-sonar, brave-search-api, serpapi |
| Agent Frameworks & SDKs (`agent-frameworks`) | langgraph, openai-agents, claude-agent-sdk, crewai, pydantic-ai, mastra, google-adk, autogen, smolagents |
| Agent Sandboxes & Code Execution (`agent-sandboxes`) | e2b, daytona, modal, cloudflare-sandbox, vercel-sandbox, runloop, blaxel, maritime |
| Product Analytics (`product-analytics`) | posthog, amplitude, mixpanel, plausible |
| CRM (`crm`) | hubspot, attio, salesforce, twenty |
| Startup Legal & Incorporation (`legal-ops`) | stripe-atlas, clerky, docusign, firstbase, ironclad, beglaubigt, legalzoom |
| Robotics Software Platforms (`robotics-platforms`) | formant, gazebo, nvidia-isaac, ros2, viam |
| Terminals (`terminals`) | warp, ghostty, iterm2, alacritty, wezterm, kitty |
| AI Assistants (`ai-assistants`) | chatgpt, claude, gemini, perplexity, copilot, grok, muse, poke, martin, jo |
| AI Research Agents (`ai-research-agents`) | elicit, consensus, futurehouse, undermind, sakana-marlin, notebooklm |
| Package & Toolchain Managers (`package-managers`) | homebrew, nix, pnpm, uv, bun, mise |
| Vector Databases & Memory Stores (`vector-databases`) | pinecone, weaviate, qdrant, chroma, milvus, helixdb, lancedb |
| Auth & Identity (`auth-platforms`) | auth0, clerk, workos, keycloak, better-auth |
| AI Inference Providers (`inference-providers`) | groq, together-ai, fireworks-ai, cerebras, deepinfra, baseten, morph |
| Workflow Automation (`workflow-automation`) | n8n, zapier, make, temporal, pipedream, windmill, afk, gumloop, lindy, activepieces, trigger-dev |
| Observability & Monitoring (`observability`) | datadog, grafana, sentry, new-relic, honeycomb, signoz |
| MCP Infrastructure & Registries (`mcp-infrastructure`) | composio, smithery, glama, pipedream-mcp, gram, manufact, metorial |
| Browser Automation for Agents (`browser-agents`) | browser-use, stagehand, skyvern, hyperbrowser, steel, notte, smooth |
| AI Memory Layers (`ai-memory`) | mem0, zep, letta, supermemory, cognee, airweave |
| Voice Agent Platforms (`voice-agents`) | vapi, retell, elevenlabs-agents, bland, livekit-agents, pipecat, bolna, telli, deepgram |
| Notes & Knowledge Bases (`notes-knowledge`) | obsidian, logseq, anytype, capacities, reflect, poly |
| Meeting AI & Notetakers (`meeting-ai`) | granola, fireflies, otter, fathom, fellow |
| GPU Clouds (`gpu-clouds`) | runpod, lambda-labs, coreweave, vast-ai, paperspace |
| Feature Flags & Experimentation (`feature-flags`) | launchdarkly, statsig, growthbook, flagsmith, unleash |
| Serverless & Developer Databases (`serverless-databases`) | neon, turso, planetscale, clickhouse, cockroachdb, supabase |
| Agent Skills & Extensions (`agent-skills`) | superpowers, anthropic-skills, mattpocock-skills, skills-cli, codex-plugins, gstack |
| Data Warehouses & Lakehouses (`data-warehouses`) | snowflake, databricks, bigquery, motherduck |
| Search Infrastructure (`search-infra`) | algolia, meilisearch, typesense, elastic, orama |
| Scheduling & Calendar (`scheduling`) | cal-com, calendly, motion, reclaim, savvycal |
| Design & Prototyping (`design-tools`) | figma, penpot, framer, sketch, canva, claude-design, spline, rive |
| Email Clients & Email AI (`email`) | superhuman, shortwave, missive, zero, fastmail, agentmail, gmail |
| Incident Management & On-call (`incident-management`) | pagerduty, incident-io, firehydrant, rootly, betterstack |
| Developer Docs Platforms (`docs-platforms`) | mintlify, gitbook, readme, docusaurus, fern |
| Data Pipelines & ELT (`data-pipelines`) | airbyte, fivetran, dagster, dlt, meltano |
| E-commerce Platforms (`ecommerce-platforms`) | shopify, medusa, woocommerce, bigcommerce, swell |
| Customer Data Platforms (`customer-data-platforms`) | segment, rudderstack, mparticle, jitsu, hightouch |
| Durable Execution Engines (`durable-workflows`) | temporal, inngest, trigger-dev, restate, hatchet, dbos |
| AI Customer Support Agents (`ai-support-agents`) | intercom-fin, decagon, sierra, pylon, lorikeet, parahelp |
| AI Code Review (`ai-code-review`) | coderabbit, greptile, graphite, qodo, cursor-bugbot, cubic |
| Document Extraction APIs (`document-extraction`) | reducto, llamaparse, extend, datalab, unstructured, mistral-document-ai |
| Cap Table & Equity (`equity-management`) | carta, pulley, cake-equity, fidelity-private-shares, ledgy, vestd |
| Error Tracking (`error-tracking`) | sentry, bugsnag, rollbar, honeybadger, glitchtip, raygun |
| Expense Management (`expense-management`) | ramp, brex, expensify, navan, bill-spend-expense |
| Billing & Subscriptions (`billing-subscriptions`) | stripe-billing, chargebee, recurly, lago, orb, metronome, revenuecat |
| Payment Fraud Prevention (`fraud-prevention`) | stripe-radar, sift, signifyd, forter, riskified |
| Agentic Commerce (`agentic-commerce`) | stripe-agentic-commerce, shopify-ucp, paypal-agent-commerce, coinbase-x402, crossmint, visa-intelligent-commerce, skyfire |
| Card Issuing Platforms (`card-issuing`) | stripe-issuing, lithic, marqeta, highnote, adyen-issuing |
| Sales Tax Automation (`tax-automation`) | stripe-tax, avalara, anrok, taxjar, numeral, kintsugi |
| Marketplace & Platform Payments (`marketplace-payments`) | stripe-connect, adyen-for-platforms, finix, mangopay, rainforest, tilled |
| Banking as a Service (`banking-as-a-service`) | stripe-treasury, unit, increase, column, synctera, treasury-prime |
| Processors (`processors`) | apple-m5, apple-m4-max, apple-m4-pro, qualcomm-snapdragon-x2-elite-extreme, amd-ryzen-ai-max-plus-395, amd-ryzen-9-9950x3d, intel-core-ultra-9-285k, intel-core-ultra-7-258v |
| GPUs & AI Accelerators (`gpus`) | nvidia-rtx-5090, nvidia-rtx-5080, nvidia-rtx-5070-ti, amd-rx-9070-xt, nvidia-rtx-pro-6000, nvidia-h200-sxm, nvidia-b200, amd-mi355x |
| Hardware Security Keys (`security-keys`) | yubikey, google-titan, nitrokey, solokeys, feitian, token2 |
| Authenticator Apps (`authenticator-apps`) | google-authenticator, microsoft-authenticator, authy, 1password, bitwarden, proton-pass, ente-auth, 2fas |
| Game Engines (`game-engines`) | unity, unreal, godot, threejs, babylonjs, bevy, playcanvas, phaser |
| Identity Verification & KYC (`identity-verification`) | stripe-identity, persona, entrust-onfido, sumsub, veriff, plaid-idv |
| Banking Data APIs (`banking-data-apis`) | stripe-financial-connections, plaid, mx, mastercard-open-finance, teller, truelayer, yapily |
| Stablecoin Payments (`stablecoin-payments`) | stripe-crypto, circle, bvnk, coinbase-payments, moonpay, paxos |
| Email Marketing (`email-marketing`) | loops, customer-io, klaviyo, mailchimp, kit, bento |
| Self-Hosted AI Assistants (`self-hosted-assistants`) | openclaw, open-webui, librechat, anythingllm, khoj, lobe-chat |
| Frontier Models (`frontier-models`) | jev, claude, gpt, gemini, llama, deepseek, mistral, grok |
| Compliance Automation (`compliance-automation`) | vanta, drata, secureframe, oneleet, sprinto, thoropass |
| Applicant Tracking (`applicant-tracking`) | greenhouse, lever, ashby, workable, recruitee |
| Domain Registrars (`domain-registrars`) | porkbun, name-com, cloudflare-registrar, godaddy, dynadot, namecheap |
| SSO & Workforce Identity (`sso-identity`) | okta, microsoft-entra, google-workspace, jumpcloud, rippling-it |
| Cloud Storage (`cloud-storage`) | dropbox, google-drive, box, onedrive |
| Cloud Platforms (`cloud-platforms`) | aws, google-cloud, azure, oracle-cloud |
| Transactional Email APIs (`email-apis`) | sendgrid, resend, postmark, mailgun, amazon-ses |
| Virtual Mailboxes (`virtual-mailboxes`) | stable, virtualpostmail, earth-class-mail, anytime-mailbox |
<!-- arenas:end -->
Regenerated by `pnpm stats` — never hand-edited. …and growing — see the live site for the current set.
See `data/categories.json` for each arena's full description, personas, and themes.
## Methodology
New to the site and just want plain-language answers ("what does `na` mean," "why does this
score look low," "how do I disagree")? See **[docs/SCORING.md](./docs/SCORING.md)** — a
one-page, jargon-free companion to the technical writeup below. For a critical self-assessment
of the scoring formula's limits (coverage sensitivity, judge noise, blend-weight choices) and
the improvement roadmap, see [docs/SCORING-REVIEW.md](./docs/SCORING-REVIEW.md).
### 1. Evidence tiers
Every claim about a product is backed by an **evidence** item with one of four tiers:
- `claimed-docs` — vendor site/docs/changelog copy (what the vendor says about itself)
- `github` — README/repo content
- `community` — independent community sources (forums, reviews, social posts)
- `probe` — direct, hands-on observation of the product
Evidence is stored per product at `data/{category}/evidence/{product}.json`, each item with
a stable id, tier, source URL, verbatim excerpt, and fetch timestamp.
#### Evidence-strength ladder
Tiers aren't just labels — they're ranked, strongest first:
**`probe` (tested) > `github` (code) > `community` (independent) > `claimed-docs` (vendor claim)**
A direct, hands-on probe of the product (see "Probe harness" below) outranks a GitHub
README, which outranks independent community commentary, which outranks the vendor simply
describing its own product. `lib/verification.ts`'s `strongestEvidence()` walks a verdict's
cited evidence down this ladder and returns the single best-supported item — that's the
source behind every "proof ↗" link on the site (product page verdict rows, story matrix
cells, and battle round cards). It's a different, finer-grained ranking than the coarser
`verificationLevel` badge (`tested`/`corroborated`/`vendor-claim`/`disputed`), which groups
`github` in with `claimed-docs` for display purposes; `strongestEvidence` keeps them distinct
so the proof link always points at the most credible single source, not just the highest
badge tier.
#### Probe harness
`pnpm pipeline probe` runs a small set of keyless, hands-on checks per product and turns each
*definitive* result — positive or negative — into a `probe`-tier evidence item (ambiguous
results, e.g. a 403 from a WAF, produce no item rather than a guess): whether `/llms.txt`
resolves, whether the docs URL serves a `.md` variant, whether a conventional OpenAPI spec
path resolves, and whether the curated `links.mcp`/`links.cli` URLs are live. See
`pipeline/stages/probe.ts` (`runProbeChecks`) for the exact rules and
`pipeline/__tests__/probe.test.ts` for coverage with a mocked fetcher (no live network calls
in tests). Like `collect-community`, re-running `probe` for a product replaces only its prior
`probe`-tier items — other tiers are untouched.
### 2. Judged cells
For every (product, story) pair, an LLM judge reads only that product's evidence pack for
that story and returns a **verdict**:
| Verdict | Meaning |
|---|---|
| `full` | Clearly delivers the story |
| `partial` | Delivers, with significant caveats or extra tooling required |
| `disputed` | Vendor claims it, but community/hands-on evidence contradicts — must cite both sides |
| `none` | No evidence it delivers (never used for capabilities that don't apply — see `na`) |
| `na` | The story's axis doesn't apply to this product at all (wrong-axis, e.g. an OS-install story for a SaaS API) |
Each verdict also carries a `quality` score (0–10, how *well* it delivers — 0 for `none`/`na`),
a `confidence` level, a short rationale, and the specific `evidenceIds` the judge relied on.
The judge is instructed to use **only** the evidence pack, never outside/training knowledge —
absence of evidence for a well-known capability still yields `none`, not a guess.
Verdicts are cached and keyed on a hash of `(storyId, story title, evidence ids+excerpts,
prompt version)`, so re-running `judge` is a no-op unless the story or evidence actually
changed.
#### Judge model & prompt version
The judge is an Anthropic model — `claude-sonnet-5` by default, overridable via the
`PA_MODEL` env var (`pipeline/llm.ts`) — driven by a versioned system prompt in
`pipeline/stages/judge.ts` (`PROMPT_VERSION`, currently **`v3`**). Because the prompt
version is part of every cell's cache key, bumping it deliberately invalidates all cached
verdicts, so a prompt change is always followed by a full re-judge rather than mixing
verdicts from different prompt generations.
What each version added:
- `v2` — evidence-only judging (no training knowledge), `na` wrong-axis tier, disputed
requires citations from two distinct evidence tiers.
- `v3` — three calibration fixes, each traceable to an audited defect:
1. **`na` vs `none` decision procedure** with few-shot boundary examples (applicability is
decided *before* looking at evidence; lack of evidence for an applicable axis is always
`none`).
2. **Explicit 0–10 quality rubric**, plus a hard requirement that any quality below 10
names its gap in the rationale (`"missing for 10: …"`) — mechanically enforced by
`validateVerdictRules`, which rejects sub-10 `full`/`partial` verdicts missing the
clause.
3. **Evidence-relevance rule**: a citation may only support or undermine a verdict if its
content is *about the story's subject* — off-topic negative sentiment (e.g. unrelated
security news) must not move a verdict or appear in the rationale.
Calibration is gated by `pnpm calibrate` (`pipeline/scripts/calibration-check.ts`), which
re-runs the live judge against a hand-verified golden-cell set
(`pipeline/golden/golden-cells.json`) and fails on more than 2 tier mismatches.
### 3. Scoring formula
A cell's score is `story.weight × quality × verdictFactor`, where the verdict factor is
`full=1.0, partial=0.6, disputed=0.3, none=0, na=excluded`. A product's overall score (and
each per-theme score) is the weighted percentage across all **applicable** (non-`na`) cells:
```
score = 100 × Σ(cellScore) / Σ(story.weight × 10) — over non-na cells only
```
`na` cells are excluded from both numerator and denominator entirely — they neither help nor
hurt a product's score. Head-to-head battles use the same per-cell scores: each story is a
"round" won by whichever product scores higher on it (ties are draws, `na` rounds are
excluded), and the overall battle winner is whoever wins more story-weight.
**Important:** scores measure **evidenced story coverage**, not absolute product quality. A
product with a thin crawl (fewer/weaker evidence items) will score lower even if it's
objectively excellent — the judge can only score what's in the evidence pack. Conversely, a
wrong-axis story is marked `na` and excluded rather than counted against a product.
#### Claims vs reality — the claims-integrity index
Every product's vendor claims (extracted from its own docs/GitHub materials by
`pipeline/stages/claims.ts`) are reconciled against our judge's independent verdicts
(`lib/claims.ts`). Each claim that maps onto a story lands in one of three **testable**
buckets: *verified* (full/partial verdict backed by corroborated/tested evidence),
*unverified* (full/partial verdict, but only the vendor's own claim backs it), or
*contradicted* (verdict disputed/none/na). Claims mapping onto no story are **untestable**
(a taxonomy gap, not a mark against the product) and are excluded entirely.
The claims-integrity score (`lib/claimsIntegrity.ts`) rewards claims we independently
verified and actively penalizes ones the evidence contradicts:
```
testable = verified + unverified + contradicted — untestable claims excluded
integrity = 100 × max(0, verified − 2 × contradicted) / testable
```
Verified claims count fully, unverified claims count for nothing (they only inflate the
denominator), and each contradicted claim cancels **two** verified ones — overpromising is
worse than staying silent — with the score clamped at 0.
**Null, never zero:** a product with no claims data (or no testable claims) gets `null`,
not a fabricated `0` — same rule as every other index here: "we don't know" is not "the
worst", and nulls sort last. The full cross-arena ranking lives at
[/rankings/claims-integrity](https://ultrametric.ai/productarena/rankings/claims-integrity),
and each product page's "Claims vs evidence" section opens with its integrity summary.
### 4. The Agenticness Index
Every arena includes the same 9 canonical "agenticness" stories, injected verbatim (never
LLM-authored) so agent-readiness is comparable across categories. Defined in
`pipeline/agentic-stories.ts`:
| Story id | Story | Weight |
|---|---|---|
| `agentic-public-api` | I can drive the product through a documented public API | 3 |
| `agentic-official-cli` | I can use an official CLI | 2 |
| `agentic-mcp-server` | I can connect an agent via an official MCP server | 3 |
| `agentic-mcp-client` | I can plug MCP servers into this product so it can use their tools | 3 |
| `agentic-webhooks` | I can subscribe to events via webhooks | 2 |
| `agentic-sdks` | I can build against official SDKs | 2 |
| `agentic-agent-docs` | I can point an agent at llms.txt or agent-oriented docs | 2 |
| `agentic-scoped-keys` | I can issue scoped/least-privilege API credentials for an agent | 2 |
| `agentic-headless` | I can run the product headlessly / in CI for automation | 2 |
`agentic-mcp-server` and `agentic-mcp-client` are two ends of the same protocol, deliberately
split into separate axes: `-server` asks whether the product *ships* an MCP server for other
agents to connect to, `-client` asks whether the product itself *consumes* MCP servers. For
agent products (e.g. a coding-agent CLI), the serving axis is often the wrong question — the
product IS the agent — while the consuming axis is exactly the right one; see the applicability
notes in [METHODOLOGY.md](./METHODOLOGY.md).
A product's "agenticness" score on the leaderboard is its weighted percentage across just
these 9 cells (theme `agenticness`, group `agent-access`).
Two sibling group-scoped indexes live under the same `agenticness` theme: `agentic-features`
("does the product act agentically itself" — `agenticApp` on the leaderboard) and, since v2.4,
`api-quality` (see below).
### 5. The PA Score (formerly the Arena Score / AI-Era Index)
v2.4 adds a sixth canonical group, **API quality** (theme `agenticness`, group `api-quality`),
alongside `agent-access`. Where `agent-access` asks "can an agent reach the product at all,"
`api-quality` asks "how good is that surface once an agent is there":
| Story id | Story | Weight |
|---|---|---|
| `api-interactive-docs` | I can explore an interactive API reference with runnable examples | 2 |
| `api-machine-spec` | I can download a machine-readable API spec (OpenAPI or equivalent) | 2 |
| `api-versioning-policy` | I can rely on versioned APIs with a documented deprecation policy | 2 |
| `api-sandbox` | I can test against a sandbox environment without touching production data | 1 |
On top of that, every leaderboard entry now carries a **PA Score** (`aiEra` internally,
displayed on-site as a bare "{n}/100" badge under a "PA Score" label — this score used to be
called the "Arena Score" and before that the "AI-Era Index," same
formula, new name) — a single
number meant to answer "how ready is this product for a world where agents, not just humans,
are the primary users?" It's a weighted blend of five existing leaderboard components:
| Component | Weight | What it measures |
|---|---|---|
| `agentReady` | 0.30 | can an agent reach the product (API/CLI/MCP/webhooks/SDKs/docs) |
| `apiQuality` | 0.20 | how good is that API surface (docs, spec, versioning, sandbox) |
| `openness` | 0.20 | can you self-host, export your data, and read the source |
| `agenticApp` | 0.15 | does the product act agentically on its own behalf |
| `automation` | 0.15 | how deep are its rules/scheduling/bulk/versioned-automation primitives |
```
aiEra = Σ(component × weight) / Σ(weight) — over non-null components only
```
Weights are renormalized over whichever components are non-null for a given product, so a
product missing one axis (e.g. no `openness` theme applies to its category) isn't penalized
twice — once for the missing axis, once for a shrunken blend. `aiEra` is `null` only when every
component is null. The exact weights live in `AI_ERA_WEIGHTS` in `lib/scoring.ts`.
**Why lead with this instead of the coverage score.** The coverage score measures evidenced
story coverage across a product's whole category — useful, but it treats "has a nice settings
UI" the same as "has an MCP server." As of v2.4, leaderboards sort primarily by `aiEra` (nulls
last, ties broken by coverage score) because we think products in the AI era should be ranked
first by how well agents and automation can actually work with them — the coverage score is
still shown, just demoted to a secondary line.
**These weights are a starting position, not a verdict.** We picked them because agent-access
and API quality are the most direct proxies for "can an agent use this at all," while
openness/agenticApp/automation matter but are one step removed. If you think the weighting is
wrong, [contest it via an issue](./CONTRIBUTING.md) — like every verdict on this site, the
formula is open to challenge.
#### Score confidence (A–D)
Scores never pretend: untested cells can't score, and the confidence grade says how much of a
score is backed by probes. Every PA Score badge carries a small A–D chip
(`lib/confidence.ts`) derived from two fractions over the product's applicable (non-`na`)
cells:
| Signal | Meaning |
|---|---|
| **coverage** | fraction of applicable cells whose verdict cites *any* evidence — the complement is "we found nothing either way," which already scores 0 but is unknown, not failed |
| **testedShare** | fraction whose *strongest* cited evidence is a tested tier (`probe` or `github` — hands-on runs or inspectable source), per `lib/verification.ts`'s evidence ladder |
Grades: **A** = coverage ≥ 0.85 and testedShare ≥ 0.40 · **B** = coverage ≥ 0.70 and
testedShare ≥ 0.25 · **C** = coverage ≥ 0.55 · **D** = below that. The grade never changes any
published score — two products can post the same 60 while one earned it from probes and the
other from vendor docs, and the chip is where that difference shows. Thresholds are calibrated
against the live dataset so the letters actually discriminate (see
`CONFIDENCE_THRESHOLDS`), and, like the PA Score weights, they're open to challenge.
### 6. Story provenance
Every story in `data/{category}/stories.json` optionally carries an `origin` field
(`lib/schemas.ts`'s `StoryOriginSchema`) recording where it came from and when:
| `origin.kind` | Meaning |
|---|---|
| `canonical` | one of the 29 fixed agenticness/openness/automation-depth/privacy-posture stories (`pipeline/agentic-stories.ts`), injected verbatim into every category by `normalize.ts`'s `assembleTaxonomy` — never LLM-authored |
| `normalized` | assembled into the category's taxonomy by the LLM-driven `normalize` stage; carries the judge `promptVersion` in force at the time |
| `contest` | added or adjusted via a contest issue (not yet exercised — `contest-check.ts` only appends evidence today, never stories) |
| `manual` | hand-edited |
`origin` is additive and never participates in `cellHash` (`pipeline/stages/judge.ts`) —
stamping or backfilling it can never invalidate the judge cache or change a verdict. Hover a
story title or matrix cell on any product page to see its origin in the tooltip (e.g.
"canonical" or "normalized · v2").
### 7. Judge model and prompt version
The judge model is `claude-sonnet-5` by default (override with the `PA_MODEL` env var), and
the judge prompt is versioned (`PROMPT_VERSION = 'v3'` in `pipeline/stages/judge.ts`) — the
cache key includes the prompt version, so bumping it forces a full re-judge.
**Re-judge stability policy.** LLM judging has measurable re-roll variance (~9% of cells can
change verdict or quality on a re-judge with no relevant evidence change). To keep rankings
evidence-driven rather than noise-driven, large re-judge waves are reviewed against the prior
state and pure churn is reverted under audited rules: applicability (`na`↔`none`) never flips
without new evidence, verdicts that cite nothing new don't move close races, and negative
mechanical probe results only affect the story axis they actually test. Every revert is
recorded in the commit that applies it. A future prompt version will pass the prior verdict
as an anchor to reduce this variance at the source.
### 8. Score intervals — the ± band on every PA Score
Every PA Score carries a **68% confidence band** ("42 ±3 /100" on product pages; the exact
low–high band in the score badge's tooltip, including on the homepage mega-table). The band is
an honest statement of how much the published number could move under the judge noise we have
actually **measured** — it is *analytic v1*: computed from existing data with **no new judging**.
How it's built (`pipeline/scripts/compute-confidence-intervals.ts`, math in
`lib/scoreIntervals.ts`):
- **Per-cell noise comes from measured re-roll statistics, never invented rates.** The
multi-judge uncertainty pass (`data/*/uncertainty.json`, see §7) re-judged 650+ decisive
cells twice more against unchanged evidence; ~20% showed some disagreement. From those
samples we build a cached-tier → re-rolled-tier transition matrix (e.g. a `full` cell
re-rolls to `partial` ~7% of the time), and every evidenced `full`/`partial`/`disputed`/`none`
cell resamples its verdict from its measured row.
- **Untested cells carry wider, epistemic uncertainty.** A `none` verdict citing zero evidence
means "we found nothing either way", not "it failed" — it scores 0 today but could plausibly
be a `partial` on new evidence. Each such cell flips to `partial` with probability equal to
the measured any-disagreement rate (~0.20), with quality bounded to the plausible [3, 7]
range. Products whose score rests on many untested cells therefore get honestly wider bands.
- **Applicability is pinned.** `na` cells never resample — the re-judge stability policy (§7)
reverts `na`↔`none` churn that cites no new evidence, so published applicability is
policy-stable and modeling it as noise would overstate the band.
- **Propagation is exact.** Each of 500 Monte Carlo draws per product resamples every relevant
cell and recomputes the score through the *same* `weightedPercent` + `computeAiEra` code that
produces the published number (roundings included). The band is the 16th–84th percentile of
the resulting distribution; agent-readiness gets its own band the same way. The PRNG is
seeded (mulberry32, keyed per product) — builds are byte-reproducible, no `Math.random`.
Results land in `data/{category}/score-intervals.json` (committed, re-run post-derive by the
story-runner). Display is tolerant-optional (`lib/scoreIntervals.ts`): no interval data ⇒ no
band is ever rendered — never a fabricated one. Fleet-wide as of the first pass the median band
width is ~4.5 PA Score points.
**What the band is not (yet):** it reflects propagated judge-*sampling* noise plus
untested-cell ignorance, **not** model-family disagreement — the same evidence judged by a
non-Anthropic model could move scores in ways this band doesn't capture. Adding a second judge
model and folding cross-model disagreement into the interval is listed as future work.
### 9. Bias disclosure — the judge is an Anthropic model
**Owner-product disclosure:** the Product Feedback & Intent arena includes Foreloop, built by
Ultrametric Inc — the company that operates ProductArena. Foreloop is judged by the identical evidence
rules as every other product (it placed third of four in its own arena as of this writing), its
product page carries an affiliation banner, and every one of its verdicts is contestable like
any other. An adversarial bias audit of Foreloop's verdicts (2026-09-21) found and corrected 14
overcalls in Foreloop's favor — four quality/tier downgrades on the `agentic-*` cells (including
`agentic-public-api` `full` 8 → `partial` 5, applying the same "documented public REST/HTTP API"
bar used for its competitors, per the claude-code precedent below) and ten `na` → `none`
reclassifications where the wrong-axis call had improperly shrunk Foreloop's score denominator
relative to peers graded `none` on the same stories. Each corrected verdict carries a dated
audit note in its rationale, applied in the committed judge cache so re-judges preserve it; the
corrections moved Foreloop from second to third of four.
**Read this before trusting the `ai-coding` arena's numbers.** The judge model
(`claude-sonnet-5`) is made by Anthropic, and the `ai-coding` arena includes Anthropic's own
product, Claude Code, which leads that arena's **PA Score** (29.5) as of v2.4 — though on
raw coverage score it now sits second (34.6) behind GitHub Copilot (35.0), a lead that flipped
when the v2.4 `api-quality` cells were added (Claude Code's own coverage score was 35.2 as of
the last full audit below, before those cells existed). This is a real conflict of interest and
we want it visible, not buried.
What we did about it:
- **We ran an adversarial bias audit** of every `claude-code` verdict scored `full` in the
`ai-coding` arena (14 cells), checking each cited evidence excerpt against the claim it
was used to support, and separately compared every `agentic-*` cell head-to-head against
`codex`.
- **One cell was downgraded** as a result: `live-app-debugging` went from `full` (quality 6)
to `partial` (quality 5) after adjudication. The sole citation was a bare, title-only doc
fragment ("Debug live web applications | Chrome") with no scope or mechanism detail, while
every competing product's comparable-or-better evidence for the same story capped at
`partial`/`none`. The verdict's rationale in `data/ai-coding/verdicts.json` documents the
downgrade and the shared-vendor conflict explicitly. This changed Claude Code's overall
score from 35.5 to 35.2 (it remained the category leader).
- **A second adversarial review pass, run for v2.4, audited the new api-quality/agent-access
cells** and applied two cross-vendor corrections, stated plainly in both directions:
one **against** Claude Code's favor (`agentic-public-api` downgraded `full`→`partial`,
quality 8→5 — the Agent SDK/CLI don't clear the same "documented public REST/HTTP API" bar
applied to competitors), and one **in** Claude Code's favor, applied to a competitor
(GitHub Copilot's `agentic-mcp-server` downgraded `partial`→`none`, quality 5→0 — its cited
evidence showed MCP *client* administration, not an official MCP server offered by the
product). Both corrections are recorded in `data/ai-coding/verdicts.json` with rationale
suffixes citing the review.
- **The audit's calibration samples also found the judge was *harsher* on claude-code in
several cells**, not just lenient. The clearest example: on
`natural-language-feature-implementation`, Claude Code's own marketing describing its core
workflow was judged `disputed` (because cited community complaints contradicted it),
while Codex was judged `full` on the same story from its evidence pack. A biased judge that
favored its own vendor would not do this.
- **Two additional cells carry documented caveats** (left as computed, not adjusted,
per the audit's own rule of "flag, don't silently override" except where adjudicated
above):
- `persistent-project-instructions` — `full`, quality 9. The cited community evidence
partly complains about the feature's practical downsides (config sprawl, some output
degradation), which the verdict's rationale doesn't fully surface. The verdict *tier*
(`full`) is considered correct — the feature (CLAUDE.md) unambiguously exists and is used
— but the quality score is generous given the mixed community signal.
- `background-cloud-tasks` — both `claude-code` and `codex` scored `full`, quality 8. The
story specifies an "isolated cloud environment"; Codex's cited evidence explicitly says
"isolated cloud environments," while Claude Code's cited evidence confirms background/
cloud execution but never uses the word "isolated." Both were scored `full`, but only one
product's evidence actually supports that specific qualifier.
- **Every verdict cites evidence ids** resolvable in `data/{category}/evidence/`, so anyone
can independently check any verdict against its source. If you disagree with a call, see
[CONTRIBUTING.md](./CONTRIBUTING.md) — contesting a verdict is a first-class, expected
workflow, not a one-off.
We think shipping this disclosure — including the fact that the audit itself was run by the
same vendor's model — is more honest than pretending the conflict doesn't exist. Judge for
yourself using the cited evidence.
## Local development
```bash
pnpm install
pnpm dev # http://localhost:3000
pnpm test # vitest — schema and scoring unit tests
pnpm build # next build (static)
```
## Pipeline refresh workflow
The pipeline is local-only (not run on Vercel). Each stage accepts `--category <id>` to
scope a run, and most accept `--product <id>` to scope further to one product:
```bash
pnpm pipeline crawl --category <id> [--product <id>] # fetch site/docs/github pages
pnpm pipeline extract --category <id> [--product <id>] # LLM: pull candidate stories from crawled pages
pnpm pipeline normalize --category <id> # LLM: assemble the category's story taxonomy
pnpm pipeline collect-community --category <id> [--product <id>] # gather community evidence
pnpm pipeline probe --category <id> [--product <id>] # keyless hands-on checks (llms.txt, openapi, etc.) → probe-tier evidence
pnpm pipeline judge --category <id> [--product <id>] # LLM: judge every (product, story) cell
pnpm pipeline derive --category <id> # compute rankings.json (scores + battles) from verdicts
pnpm pipeline logos --category <id> [--product <id>] # fetch product logos into public/logos/
pnpm pipeline popularity --category <id> [--product <id>] # keyless GitHub/npm/PyPI momentum signal (display-only, no LLM, not scored)
```
To refresh a single product's data after editing its evidence (e.g. after a contributed
correction), you don't need to re-run the whole category — see
[CONTRIBUTING.md](./CONTRIBUTING.md) for the minimal `judge --product` + `derive` flow.
After any re-judge/`derive` that changes rankings, re-run `node scripts/generate-badges.mjs`
and commit the `public/badges/` diff — the embeddable score badges (see `/badges` on the site)
are committed static SVGs, deliberately not generated during `pnpm build`, so consumers who
hotlink them pick up the new scores on the next deploy.
Requires `ANTHROPIC_API_KEY` in a local `.env` (see `.env.example`) for the LLM-driven stages
(`extract`, `normalize`, `collect-community`, `judge`).
**`ANTHROPIC_API_KEY` is local-pipeline-only. It must never be set as an environment variable
on the Vercel project** — the deployed site only serves pre-computed static data from `data/`
and never calls the Anthropic API at build or request time.
### Arena Notes (editorial drafts — never auto-published)
`pnpm tsx pipeline/scripts/generate-arena-notes.ts` drafts a short signed-POV essay (5
paragraphs max: the week's flip that matters, the gap nobody's filling, the vendor move to
watch, one honest self-critique of our own data, a closing line) into
`drafts/arena-notes/YYYY-MM-DD.md`. It is LLM-drafted but honesty-gated in code: the model only
sees a fact sheet derived from the committed changelog/report/uncertainty/gap-closers data, and
every paragraph must cite its site source inline (`(source: /changelog)`, `(source:
/arena/...)`) from that fact sheet — violations fail schema validation and are retried.
Publishing is a human act, by design. Every generated file carries
`status: draft — requires founder sign-off before publishing` in its frontmatter, and the
[/notes](https://ultrametric.ai/productarena/notes) page (see `lib/notes.ts`) renders **only**
files whose frontmatter a human has edited to exactly `status: published`. Review the draft,
edit the frontmatter, commit — that's the whole publish flow. Unreviewed drafts never render.
## Data layout
```
data/
categories.json # arena metadata: id, name, description, personas, themes
{category}/
products.json # product metadata (id, name, vendor, type, urls, logo)
stories.json # the category's story taxonomy (incl. the 29 canonical stories, each with an optional `origin`)
evidence/
{product}.json # evidence items for one product: id, tier, url, excerpt, fetchedAt
verdicts.json # one verdict per (productId, storyId) cell
rankings.json # derived: leaderboard + battles (generated by `pipeline derive`)
popularity.json # optional: keyless GitHub/npm/PyPI momentum signal, keyed by productId (generated by `pipeline popularity`)
popularity-history.jsonl # optional: one snapshot line per product per popularity run, for future velocity tracking
```
`popularity.json`/`popularity-history.jsonl` are display-only — see "Popularity — a signal, not
a score" in [METHODOLOGY.md](./METHODOLOGY.md). Neither file is read by the scoring pipeline
(`pipeline derive`) or `lib/scoring.ts`, and `loadCategory` tolerates their absence entirely.
`rankings.json` is derived data — never hand-edit it; regenerate it with
`pnpm pipeline derive --category <id>` after any verdict change.
`pnpm run build`/`pnpm run dev` mirror all of `data/` verbatim to `public/data/` (a gitignored
build artifact, via `scripts/copy-data.mjs`) so it's served at stable URLs — see "For AI
agents" below.
**Dataset releases.** Point-in-time snapshots of `data/` are periodically tagged and published
as [GitHub releases](https://github.com/ultrametricai/productarena/releases) (e.g.
`data-2026-09-02`), each with a zip of the full `data/` tree attached and release notes listing
arena/product/verdict counts at that snapshot — useful if you want a stable dataset to build
against instead of tracking `main`.
## For AI agents
ProductArena is built to be read by agents, not just browsed by humans:
- **[/llms.txt](https://ultrametric.ai/productarena/llms.txt)** — the top-level index per the
[llms.txt convention](https://llmstxt.org): site purpose, methodology one-liner, and links to
every arena's markdown endpoint, the data API, and `/openapi.json`.
- **Markdown endpoints** — every arena has a full-content markdown rendering at
`/arena/{category}/llms.md` (leaderboard, business models, grouped story matrix with proof
URLs), and every product has a deep-dive at `/arena/{category}/product/{productId}/llms.md`
(every verdict, rationale, and proof URL). These are the pages an agent should actually read.
- **Data API** — the same JSON the site renders from is mirrored verbatim to stable URLs at
build time (`scripts/copy-data.mjs`, a `prebuild` step): `/data/categories.json`,
`/data/{category}/{products,stories,verdicts,rankings}.json`,
`/data/{category}/evidence/{productId}.json`. `public/data/` is a build artifact
(gitignored) — it doesn't exist until `pnpm run build` or `pnpm run dev` regenerates it.
- **[/openapi.json](https://ultrametric.ai/productarena/openapi.json)** — an OpenAPI 3.1 document
describing every data endpoint above, with hand-written JSON Schema summaries of each shape
(mirrors `lib/schemas.ts`).
- **[/methodology](https://ultrametric.ai/productarena/methodology)** — a tight, on-site summary of
the methodology below (evidence tiers, judging, scoring, PA Score weights, story provenance,
re-judge stability, bias disclosure), linked from the header next to Arenas and from
`/llms.txt`.
- **MCP / CLI** — retired as offerings (2026-09-23): ProductArena is not served over its own
MCP server or CLI; a first-party Ultrametric MCP + API is coming instead. The `mcp/` and
`cli/` workspaces remain in-repo for that successor work but are not published entry points.
- **schema.org** — arena pages embed an `ItemList` of `SoftwareApplication` entries and product
pages embed a `SoftwareApplication`, both with `additionalProperty` entries for our own
metrics (`aiEra`, `score`, etc). No `aggregateRating` — we don't have star ratings, and faking
one would be dishonest.
- **sitemap.xml / robots.txt** — `app/sitemap.ts` lists every route including the `llms.md`
endpoints; `public/robots.txt` explicitly allows `GPTBot`, `ClaudeBot`, `Claude-Web`,
`PerplexityBot`, `Googlebot`, and `Bingbot`, with a `Sitemap:` pointer.
## Status & roadmap
ProductArena is live at **[ultrametric.ai/productarena](https://ultrametric.ai/productarena)** and under active expansion. In flight:
- **Finer-grained arenas** — splitting broad categories (e.g. project management, edge
platforms) into narrower slices as products diverge enough to need it.
- **New arenas incoming**: the full build queue lives in
[`data/arena-roadmap.json`](./data/arena-roadmap.json) (170+ planned arenas in priority
tiers — see [`docs/COVERAGE-STRATEGY.md`](./docs/COVERAGE-STRATEGY.md)); founder-ops
next steps are specced in [`docs/FOUNDER-OPS-ROADMAP.md`](./docs/FOUNDER-OPS-ROADMAP.md)
and classic head-to-heads in [`docs/CLASSIC-BATTLES.md`](./docs/CLASSIC-BATTLES.md).
- **Recorded proofs** — CLI/MCP probe sessions are now captured as sanitized, replayable
transcripts (and pilot browser video) published on product pages; the
[`docs/PROVE-IT.md`](./docs/PROVE-IT.md) protocol lets vendors submit reproducible proof
specs our runner executes and publishes — pass or fail.
- **The Agentic Depth Program** — a deeper, runtime-conformance-tested successor to the
current agenticness/API-quality cells (MCP handshake checks, API probe suites, CLI
conformance, agent task trials, and a monthly agentic-velocity leaderboard). See
[`docs/AGENTIC-DEPTH-PROGRAM.md`](./docs/AGENTIC-DEPTH-PROGRAM.md) for the full plan.
- **The contest flow** — every verdict is contestable today (see
[Contributing](#contributing) below); `.github/workflows/contest-check.yml` can automate
the add-evidence → re-judge → PR loop once a maintainer wires up the `ANTHROPIC_API_KEY`
secret.
- **Continuous runners** — `.github/workflows/story-runner.yml` re-runs one arena's full
pipeline every 6 hours on rotation and opens a reviewable PR (idle until the
`ANTHROPIC_API_KEY` secret is set); an arena-builder runner that works through the
roadmap automatically is designed in `docs/COVERAGE-STRATEGY.md`.
Counts on this page (arenas/products/verdicts) are generated by `pnpm stats` — see
`scripts/update-readme-stats.mjs` — and will keep moving as arenas and products are added;
treat them as a snapshot, not a promise.
## Contributing — how the community can help
Every score on the site is only as good as its evidence, so evidence work is the
contribution. Pick your angle:
- **Contest a verdict** — every verdict has a "⚑ contest" link that opens a prefilled
issue. Bring a citation that contradicts (or supports) the ruling.
- **Prove a story** — submit a reproducible proof spec (setup + commands + expected
result) via the *Prove a story* issue form; our runner executes and records it, and the
recording is published as probe-tier evidence — pass or fail. See
[`docs/PROVE-IT.md`](./docs/PROVE-IT.md).
- **Respond as a vendor** — put an official, verified statement on the record next to a
verdict about your product (CVE-style) via the *Vendor response* issue form. Published
verbatim; it never changes a verdict by itself, but feeds the next re-judge. See
[`docs/VENDOR-RESPONSES.md`](./docs/VENDOR-RESPONSES.md).
- **Add evidence by PR** — new doc pages, changelogs, or community sources for any
product; the pipeline re-judges only the cells whose evidence changed.
- **Submit a product or arena** — the *Submit a product* issue form (prefilled from the
[/submit](https://ultrametric.ai/productarena/submit) quick scan), or open a PR adding your
`data/<arena>/products.json` entry — the exact entry shape, which URLs to include and why,
and how to make your product probe well are all in
[CONTRIBUTING.md § Add your product](./CONTRIBUTING.md#3-add-your-product).
- **Make a prediction** — `/predictions` auto-generates yes/no questions from live close
races ("will the #2 overtake the #1 within 30 days?") and settles them mechanically from
the changelog's rank-flip events. The "Predict" link opens a prefilled issue
(`.github/ISSUE_TEMPLATE/prediction.yml`) — v1 records predictions as submitted issues;
an accuracy leaderboard comes once enough questions have settled.
- **Report an inaccuracy** — wrong metadata, broken link, stale evidence, missing logo.
- **Adopt an arena** — become the standing reviewer for one arena's story taxonomy,
evidence freshness, and contest triage.
- **Run probes locally** — `pnpm pipeline probe --category <id>` needs no API key;
publishing discrepancies you find is exactly the point.
- **Cite the data** — verdicts, scores, and evidence excerpts may be quoted with
attribution (see DATA-LICENSE); bulk reuse needs written permission from Ultrametric.
- **Spread the Weekly Arena Report** — `pnpm tsx pipeline/scripts/generate-weekly-report.ts`
renders the last 7 days (biggest movers, rank flips, new arenas, close races) into
committed markdown at `reports/YYYY-MM-DD.md`, listed on the site at `/reports`. The
markdown is copy-paste ready for a newsletter or community digest — links, attribution,
and honest empty-states included. The rolling equivalent lives at `/changelog`.
Found a verdict you think is wrong, or evidence we missed? See
[CONTRIBUTING.md](./CONTRIBUTING.md) — contesting a verdict and adding evidence are both
first-class, expected contribution paths. Every verdict on the site has a "⚑ contest" link
that opens a prefilled GitHub issue with the category/product/story and current verdict
already filled in; a maintainer (or, in the future, a GitHub Action) does the deeper check —
adding evidence, then `pnpm pipeline judge --category <category> --product <product>` followed
by `pnpm pipeline derive --category <category>` — before any verdict actually changes.
## Citing ProductArena
Quoting a ranking, verdict, or evidence excerpt (with attribution, per DATA-LICENSE) in a
paper, post, or dataset card:
> ProductArena — evidence-graded software rankings for the agent economy. Ultrametric Inc,
> 2026. https://ultrametric.ai/productarena
```bibtex
@misc{productarena,
title = {ProductArena: evidence-graded software rankings for the agent economy},
author = {{Ultrametric Inc}},
year = {2026},
howpublished = {\url{https://ultrametric.ai/productarena}},
note = {Methodology: \url{https://ultrametric.ai/productarena/methodology}}
}
```
Rankings move as evidence lands — cite the dataset release tag (e.g. `data-2026-09-02`) when
you need a pinned snapshot.
## License
Code: © 2026 Ultrametric Inc, all rights reserved (source-available — see LICENSE).
Data (`data/`): © 2026 Ultrametric Inc, all rights reserved (see DATA-LICENSE) — viewable
and quotable with attribution; bulk reuse requires written permission.
Rankings are research outputs provided "as is" — no responsibility for decisions made in
reliance on them (see [/terms](https://ultrametric.ai/productarena/terms)); published
`rankings.json` files carry a `_provenance` watermark, verifiable with
`pnpm tsx pipeline/scripts/verify-provenance.ts <file>`.
Connection Info
You Might Also Like
No recommendations available