Crocodile

<p align="center"><img src="assets/logo.svg" width="84" alt="Crocodile"></p> <h1 align="center">Crocodile</h1> <p align="center"> <b>A deterministic market-data engine for crypto <i>and</i> US equities.</b><br> Pull order books, trades, funding, option chains, filings and on-chain DEX events from 109 crypto venues and nine equity sources into one Parquet lake you own —<br>then replay any window and get the same bytes back. </p> <p align="center"> <a href="LICENSE"><img alt="Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-2ea043"></a> <img alt="Python 3.12+" src="https://img.shields.io/badge/python-3.12%2B-3572A5"> <img alt="3,941 tests" src="https://img.shields.io/badge/tests-3%2C941-2ea043"> <img alt="62 CLI commands" src="https://img.shields.io/badge/CLI-62%20commands-a3620a"> <img alt="API keys mostly optional" src="https://img.shields.io/badge/API%20keys-mostly%20optional-f0a92e"> </p>  Every market-data tool can fetch a price. Three things make this one worth the disk space. **One schema for everything.** A hand-written Deribit connector, a ccxt Kraken venue, a Uniswap V3 pool on Base and an SEC EDGAR Form 4 all emit the *same* record types — 30 of them, defined once. "Show me every trade across all my sources" is one SQL statement, not four codebases. **One capability, both markets.** Crocodile is the merge of two engines — Crypcodile for crypto, Stockodile for equities — and both histories are in this repository's `git log`. What came out is a single registry of **49 capabilities**, 42 of which answer for both asset classes under one name and one parameter schema. The seven that do not each carry a written argument for why no equity analogue can exist. The CLI, the REST API and the MCP server are *projections* of that registry, so none of them can drift from it. **The lake is yours, and it says where every number came from.** What lands on disk is normalized, validated and replayable. Every record also carries how it came to exist: a provenance level, the method it rests on, and a confidence computed by a registered formula rather than chosen by hand. A modelled answer says so before you read a row. ## How it compares The market-data business is mostly rented. You pay monthly, you query an API, and when you stop paying you are contractually required to delete what you pulled. Massive's Market Data Terms §8 put it plainly: > you agree to cease all use of the Market Data and delete all Market Data in your > possession. Crocodile writes Parquet into a directory you chose. There is nothing to stop paying for.  Three honest notes, because a comparison nobody can check is worth nothing: - **Crocodile is not a licensed archive.** Kaiko and Databento sell SLA-backed, audit-grade history under venue agreements. Crocodile ingests what public endpoints return. If you have a compliance requirement, these are not interchangeable. - **Deterministic replay is not unique.** Tardis.dev ships four documented replay endpoints, and NautilusTrader has nanosecond L3-to-bar event replay. Both are good. What is unusual here is the *combination* — both asset classes, an arbitrary window, over a store you own, for nothing. - **Neither is MCP.** CoinAPI, Massive and CryptoQuant all ship MCP servers. What is unusual is that Crocodile's MCP tools, REST routes and CLI commands are generated from one declaration, so an agent, a `curl` and a shell cannot disagree about what a capability does. _{Verified 2026-07-29 against each vendor's own pricing, docs and terms pages. Kaiko has acquired Amberdata; counted once. Polygon.io was renamed Massive.com on 2025-10-30. NautilusTrader reaches US equities through a paid Databento subscription (from $199/mo) or an Interactive Brokers account.} ## Install **Not on PyPI.** That name belongs to an unrelated project, so install from the repository: ```bash uv pip install "crocodile[full] @ git+https://github.com/nazmiefearmutcu/Crocodile.git" ``` Drop `[full]` and you still get the whole streaming core: every native connector, the Parquet lake, replay, and the 62-command CLI. Heavier surfaces are opt-in, so you never pay for a dependency tree you don't use: | Extra | Adds | |---|---| | `[market]` | 100+ exchanges through the universal ccxt connector | | `[ml]` | funding prediction, Black-Scholes, 3D vol surfaces (xgboost, scipy, matplotlib) | | `[web]` | FastAPI server + Streamlit examples | | `[onchain]` | Base L2 / GMX / Superchain connectors (web3) | | `[full]` | everything above | There is no desktop extra. Nothing in this repository needs a display server. ## Five minutes ```bash # stream Deribit BTC-perp trades + book deltas into a local lake crocodile collect --asset-class crypto --sources deribit --symbols BTC-PERPETUAL \ --channels trade --channels book_delta --data-dir data # the lake is just partitioned Parquet — one DuckDB view per channel on disk crocodile query --asset-class crypto "SELECT count(*) AS n FROM trade" # replay that window later — same bytes, every run crocodile replay --asset-class crypto --channels trade --symbols BTC-PERPETUAL \ --start-ns 0 --end-ns 9223372036854775807 ``` Three things in there are load-bearing and easy to get wrong: - **`--sources`, not `--exchange`/`--provider`.** The merge collapsed the two forks' partitions into one `source=`, and the flag went with it. Lakes written under the old `exchange=` prefix want `crocodile migrate-lake` first. - **`--asset-class` when the symbol does not name its market.** A bare `BTC-PERPETUAL` could belong to either fork. Guessing is how a request lands in the wrong market's implementation and comes back with plausible numbers, so it refuses instead. - **There is no `records` table.** The lake registers one view per `channel=` directory it finds — `trade`, `book_snapshot`, `ohlcv` — so the channel is the view name rather than a column. `crocodile catalog-channels` lists what a given lake actually has. `replay` and `query` read the lake you point them at and nothing else. There is no bundled sample and no offline fallback, so both answer empty on a fresh clone until you have collected something. There is also an interactive shell — `crocodile shell` — with history and tab-completion, and every command runs inside it. ## Reaching the whole market  On the crypto side Crocodile speaks to **109 venues**: ten native connectors, hand-written for fidelity, plus the entire [ccxt](https://github.com/ccxt/ccxt) family behind one universal connector. Six names exist in both — binance, bybit, coinbase, deribit, derive, okx — and the native connector wins, so the total is a union rather than a sum: 10 ∪ 105 = 109. That 105 is not a fixed number. The dependency is `ccxt>=4.5`, and 105 is what the version resolved here ships. `crocodile markets --asset-class crypto` prints the count your install actually has. | | Venues | |---|---| | **Native** | Binance · Bybit · Coinbase · Deribit · OKX · Base on-chain (Uniswap V3, Aerodrome) · GMX/Synthetix · Derive · Superchain · CoinGecko | | **Universal** | every ccxt exchange id — Kraken, KuCoin, MEXC, Gate, HTX, Bitget, … | On the equity side there are **nine sources**, and what each may be asked for is derived from the source itself rather than from a shared menu — so a channel is offered for the providers that serve it and nowhere else: | Source | Writes | Key | |---|---|---| | `alpaca` | `trade` `quote` `ohlcv` | yes | | `yahoo` | `options_chain` `ohlcv` `corp_action` `insider` | no | | `sec_edgar` | `insider` `holding_13f` `filing` `fundamental` | no (contact string) | | `treasury` | `macro_series` | no | | `tiingo` | `ohlcv` `corp_action` | yes (free tier) | | `stooq` | `ohlcv` `index_value` | no | | `msn_money` | `ohlcv` `corp_action` | no | | `google_finance` | `trade` `index_value` `fundamental` | no | | `finnhub` | `trade` | yes | `openfigi` is deliberately not a source: it enriches an instrument universe and writes no channel of its own. You don't have to name symbols. Name a *slice of the market* and Crocodile resolves the concrete list from the live universe: ```bash # the 200 most-liquid spot pairs on Binance, streamed over a single WebSocket crocodile collect-market --asset-class crypto --sources binance --top 200 \ --kinds spot --use-ws --channels trade --channels book_ticker # every USDT perpetual across three venues at once, order books included crocodile collect-market --asset-class crypto --sources bybit,okx,mexc \ --all-symbols --quote USDT --kinds perpetual --channels book_snapshot \ --max-symbols 400 # the whole coin universe — 17k+ coins, including the long tail no CEX lists crocodile collect --asset-class crypto --sources coingecko --symbols _ \ --channels ohlcv ``` Two design choices make that scale honestly. The ccxt path is **REST-poll-first**, which works on every venue, but upgrades to a **single multi-symbol WebSocket** per channel where the exchange supports it (`watchTradesForSymbols` / `watchTickers`) — the difference between one socket for three symbols and one socket for a whole exchange's book. And `universe` ranks any venue's markets by live 24h volume, so `--top N` covers the liquid core instead of ten thousand dead pairs. ### The whole market, on one screen `crocodile census` measures the market live and prints one JSON document — connector reach, venue market counts, the coin universe, market cap, dominance, and total value locked. Every figure comes from a keyless public feed. A run over the twelve major venues, 2026-07-28: > **109** reachable venues · **34,208** markets across the majors · **17,863** active coins · > **$2.28T** market cap · **$75.4B** DeFi TVL ```bash crocodile census --asset-class crypto \ --venues binance,bybit,okx,coinbase,kraken,kucoin,mexc,gate,htx,bitget,bingx,cryptocom ``` Name the venues explicitly. A bare `crocodile census` reports zero of them today — an empty tuple default reaches the parameter as a literal string — and zero venues is indistinguishable from a census that ran and found nothing. The same bug makes `--kinds` mandatory on `collect-market`. ## One declaration, three projections  Three numbers describe the same CLI, and which one you want depends on the question. **62** is what `--help` lists. **54** folds away eight retired spellings kept resolving for callers already wired to them. **49** is the capability registry — every command below except the five launchers is a projection of one declaration, which is also what the REST route and the MCP tool are projections of. | Cluster | Commands | |---|---| | **Lake** | `collect` · `collect-market` · `backfill` · `replay` · `query` · `export` · `resample` | | **Discovery** | `census` · `markets` · `universe` · `search` · `resolve-symbols` · `data-coverage` · `list-exchanges` · `catalog` · `catalog-channels` · `catalog-dates` · `catalog-exchanges` · `catalog-inventory` · `catalog-scan` · `catalog-stats` · `catalog-summary` · `catalog-symbols` | | **Options & funding** | `iv-surface` · `term-structure` · `vol-skew` · `risk-reversal` · `funding-apr` · `funding-predict` · `basis` · `perp-basis` · `spot-future-basis` · `open-interest` | | **Microstructure** | `ofi` · `slippage` · `whale-alerts` · `liquidity-depth` · `depth` · `indicators` | | **On-chain / L2 risk** | `sequencer-latency` · `peg-deviation` · `chaos-score` · `lending-stress` · `gas-vol` · `smart-money` · `label-transfers` · `mev-sandwich` · `onchain-price` · `base-market-data` | | **Servers** | `mcp` · `api` | | **Operator** | `shell` · `update` · `migrate-lake` | | **Retired spellings** | `inventory_snapshot` · `list_data_channels` · `list_dates` · `list_exchanges_on_disk` · `list_symbols` · `query_market_data` · `search_symbols` · `simulate-price-impact` | The last row is a ledger that only shrinks. Each entry was on the wire under one of the two forks and resolves to a single declaration, so there is never a second implementation behind a second name. Ingest survives disconnects with sequence-gap bridging and a dead-letter queue ([`src/crocodile/core/ingest/`](src/crocodile/core/ingest/)); whatever reaches disk is normalized against the [30-record schema](src/crocodile/core/schema/records.py) and replayable. ## Where every number came from  Four fields ride along on every record. `prov` says what class of thing the value is — `native` when the venue reported it, `derived` when it was computed from native records, `synthetic` when it was modelled from another data class, `unavailable` when it is a typed hole rather than a zero. `prov_basis` names the registered method. `prov_inputs` lists what that method consumed. `prov_confidence` is filled by the basis's own pure, documented formula — there is no supported way to type a confidence in by hand. On a join, `worst_provenance()` wins, so a modelled input cannot launder itself into a native-looking answer. ## Analytics The lake feeds an options + microstructure library, reachable three ways — CLI, MCP tool, or plain Python over a `Catalog`. Nine runnable scripts in [`examples/`](examples/): ```python from crocodile.crypto.analytics.funding import funding_apr from crocodile.crypto.analytics.volsurface import iv_surface from crocodile.core.store.catalog import Catalog catalog = Catalog(data_dir="data") apr = funding_apr(catalog, "BTCUSDT", from_ns, to_ns) # Polars DataFrame surface = iv_surface(catalog, "BTC", at_ns, rate=0.0) # strike × expiry × IV ``` Symbols here are the venue's own spelling, as stored. The canonical `source:RAW` form the CLI reads to work out which market you mean is not what the lake's `symbol` column holds, so passing `binance:BTCUSDT` to a reader matches nothing and returns an empty frame rather than raising. The full set spans OFI, slippage, whale alerts, term structure, vol skew, risk reversal, spot–perp / spot–future basis, open interest, and an L2/DeFi-risk family (sequencer latency, peg deviation, lending stress, MEV-sandwich detection). Each reads the same normalized records — native venue or ccxt, it can't tell the difference. ## For agents (MCP) ```bash crocodile mcp --data-dir data # Model Context Protocol server over stdio ``` Every tool is read-only and deterministic — answers come from the lake and the chain, never the model's imagination. Tools cover market-data reads (bounded DuckDB SQL, on-chain prices), catalog discovery, and the analytics library. Because the tool list is generated from the same registry as the CLI, an agent cannot call a capability the shell doesn't have, and each input schema is derived from the capability's parameter type rather than written out a second time. Works with Claude, Cursor, or anything that speaks MCP. ## REST API `crocodile api` serves the same lake over FastAPI at `/api/v1/*` — ops and catalog discovery, a bounded read-only `POST /query`, the derivatives and microstructure analytics, and a payment-gated demo route over the x402 protocol. ## Base L2 `BaseOnchainConnector` reads Uniswap V3 and Aerodrome swap/reserve events from Base RPC logs and emits the same record types as the CEX connectors, so a cross-venue query is one SQL statement instead of two codebases. Start with [docs/base_quickstart.md](docs/base_quickstart.md); there is a Streamlit dashboard and a Farcaster frame server under [`examples/`](examples/). Public data needs no keys; on-chain reads use a default Base RPC you can override. ## Performance  A network-free, synthetic-data benchmark of the offline path (normalize → store → query → resample → replay), fixed seed, every number written by the run itself: [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md). Reproduce with `uv run python benchmarks/bench.py`. There is no GUI in this repository, and therefore no frame rate to quote. ## Tests ```bash uv sync --all-extras pytest tests/ ``` 2,682 test functions across 240 files, **3,941 cases** once parametrisation expands them. Under that headline: a local mock-RPC server for degraded-network E2E runs (`tests/e2e/`), adversarial payload suites, a regression file seeded by real exchange API anomalies, and `tests/conformance/`, which asserts the *arguments* this codebase runs on rather than its behaviour. CI-friendly — Matplotlib runs headless and BLAS thread caps keep imports fast on Apple Silicon. One of them is worth naming. `tests/replay/test_byte_identity.py` replays the same window in two subprocesses under two different `PYTHONHASHSEED` values and compares raw stdout, because the claim at the top of this page is about bytes and a same-process repeat would not notice a silent dependency on `set` iteration order. It is the gate under the hero image. Neither `mypy --strict` nor Ruff gates the whole of `src/`, and saying otherwise was the more expensive kind of wrong — it described a gate nobody would re-derive. What actually holds: - **Zero Ruff findings** in `core`, `contrib`, `capabilities`, `surfaces` and the two deprecation shims — every line the merge wrote. - **A ratchet, currently 202,** over `crypto` and `equity`, the two inherited trees. It only ever falls; `tests/conformance/test_phase1_exit.py` fails if it rises, and a third test fails if any file lands under neither gate. Deleting the desktop layer took it from 222 to 202 on its own — a gate that tightens when you remove code needs nobody to remember to tighten it. - **`mypy --strict` clean** across 229 files, with `crocodile.crypto.*` and `crocodile.equity.*` exempted in `pyproject.toml` until they are migrated. There is also a conformance gate that fails when a sentence in the source states a number the code no longer holds. It is the reason the figures on this page can be trusted: they are not maintained by hand. ## What it isn't - **Not a trading bot.** There is no order-execution path, by design. - **Not a hosted service.** Everything runs on your machine, against your lake. - **Not a licensed data archive.** It ingests what public endpoints return. - **Not a chart.** Crocodile has no GUI. [FlowMap](https://github.com/nazmiefearmutcu/flowmap) is a separate dual-market order-flow visualizer that reads a lake like this one. - **Not magic.** Options analytics need options data — point `iv-surface` at a lake with Deribit snapshots in it, not an empty directory. ## Contributing PRs welcome. Skim [`CHANGELOG.md`](CHANGELOG.md) for direction, keep the `mypy --strict` and Ruff gates described above green — new code goes in the zero-findings half, and the ratchet over the inherited half never rises — and make sure the E2E and adversarial suites pass before opening one. Apache-2.0 — see [LICENSE](LICENSE).