Content
# DataQ
> Data quality monitoring platform built around Great Expectations — Snowflake (DEV/QA/UAT), ADLS Gen2, S3, Unity Catalog (Databricks), Apache Iceberg (native read), with ADF + Airflow + dbt orchestration integrations.
**📖 Documentation site: <https://theurgicduke771.github.io/DataQ/>** (MkDocs Material — quickstart, concepts, architecture, guides).
**Status:** **v1.0.0 released (2026-07-04)** — the 8-week roadmap is complete (187/189; 2 items consciously re-scoped to post-v1). v1 is **deployed to Azure Container Apps** — API + worker + a runtime-configured **frontend Container App** (the sole public surface; the api runs on internal ingress behind it), with Key Vault, App Insights, and orchestration polling live. Auth is a **generic OIDC client** (validated against Azure AD; ADR [0028](docs/adr/0028-cloud-neutral-image-runtime-config-generic-oidc.md)). The completed v1 ledger is archived at [docs/progress-v1.md](docs/progress-v1.md); live post-v1 progress at [docs/progress.md](docs/progress.md).
## What it does
- **Checks on 5 datasources** — Snowflake, Unity Catalog (Databricks), ADLS Gen2 + S3
flat files (CSV/Parquet, batch patterns), and Apache Iceberg (native `pyiceberg` read).
Five check styles: **GX expectations**, **custom SQL** (rows returned = failures),
**freshness / volume monitors**, and **comparison** (reconcile two datasets across
connections — ADR 0015); plus a column profiler and dry-run preview on every
datasource. [Feature matrix →](https://theurgicduke771.github.io/DataQ/feature-matrix/)
- **Assets, lineage & incidents** — the table/file is a first-class entity: health rolled
up across every suite that targets it, **table-level lineage** (a left-to-right graph of
provenance and blast radius, from dbt's manifest or an OpenLineage catalog), and open
incidents. Assets are the primary lens — the dashboard and sidebar lead with them (ADR
0034). [Concepts →](https://theurgicduke771.github.io/DataQ/concepts/)
- **Quality by dimension** — every check is classified (accuracy, completeness,
consistency, integrity, timeliness, uniqueness, validity), filled in automatically from
the check type. The asset **scorecard** turns that into the question people actually
ask: not just "are the checks passing" but **"what isn't being watched at all"** — an
asset with no Timeliness checks says so, and never shows a green tick for it.
[Dimensions →](https://theurgicduke771.github.io/DataQ/datasources-checks/)
- **Three run modes** — run now (live progress + cancel), **cron schedules**
(timezone/DST-aware), and **pipeline triggers**: ADF, Airflow **and dbt** runs are
monitored, and a successful pipeline can trigger the bound suite. [Scheduling →](https://theurgicduke771.github.io/DataQ/scheduling/) · [Orchestration →](https://theurgicduke771.github.io/DataQ/orchestration/)
- **Severity + alerting** — warn/fail/critical tiers band each check's unexpected-%;
alerts to **Teams / Slack / email** with severity-aware routing, first-failure dedup,
and per-check snooze. [Notifications →](https://theurgicduke771.github.io/DataQ/notifications/)
- **Results you can share** — dashboard health score + trends, per-run drill-down with
**PII-redacted** failing-row samples, suite-level sharing (view/edit), admin control
centre. [Best practices →](https://theurgicduke771.github.io/DataQ/best-practices/)
## Stack
| Layer | Tech |
|---|---|
| Backend | FastAPI · Celery · Great Expectations · SQLAlchemy + Alembic · PostgreSQL · Redis |
| Frontend | React · Vite · Ant Design · generic OIDC (`oidc-client-ts`) |
| Auth / secrets | OIDC — Azure AD validated (`AUTH_*` contract, provider-neutral) · Azure Key Vault |
| Hosting | Azure Container Apps (API · worker · frontend) + Application Insights (deployed) |
| AI integration | FastMCP — 8 curated MCP tools at `/mcp` for Claude Desktop / Copilot / Cursor |
## Quick start
### Run DataQ — prebuilt images (recommended)
Evaluate or self-host in ~5 minutes: **no source checkout, no Azure tenant.** Just Docker.
```bash
curl -O https://raw.githubusercontent.com/TheurgicDuke771/DataQ/main/docker-compose.ghcr.yml
docker compose -f docker-compose.ghcr.yml up
```
Open **<http://localhost:3000>** — the stack comes up migrated and seeded with demo data, on **dev-bypass auth** (every request is a fixed demo user; no sign-in). API + Swagger at `http://localhost:8000/docs`. Images are pulled from GHCR and are **multi-arch** (amd64 + arm64), so Apple-Silicon runs native. Ports bind to `127.0.0.1` only. To pin a release instead of the moving tags: `DATAQ_BACKEND_TAG=vX.Y.Z DATAQ_FRONTEND_TAG=vX.Y.Z docker compose -f docker-compose.ghcr.yml up`.
> Self-hosting with **your own** Azure AD? The published frontend is **one generic image** — the compose eval runs it with `DATAQ_AUTH_MODE=bypass` (auth off). For real SSO, **no rebuild**: run that same image with `DATAQ_AUTH_MODE=oidc` + `DATAQ_AUTH_AUTHORITY` / `DATAQ_AUTH_CLIENT_ID` / `DATAQ_AUTH_API_SCOPE` (auth config is injected at runtime, ADR 0028), and run the backend with `AUTH_DEV_BYPASS` off. See [Getting started](https://theurgicduke771.github.io/DataQ/getting-started/).
### Develop DataQ — from source
```bash
git clone https://github.com/TheurgicDuke771/DataQ.git
cd DataQ
./scripts/setup.sh # conda env + pre-commit + docker-compose + migrations
conda activate dataq
docker-compose up
```
Backend at `http://localhost:8000` (Swagger at `/docs`), frontend at `http://localhost:3000`.
## MCP (AI assistant access)
DataQ exposes 8 curated MCP tools at `/mcp` (streamable HTTP) — `list_suites`, `get_suite_results`, `get_health_score`, `get_adf_pipeline_status`, `trigger_suite_run`, `get_run_status`, `create_check`, `profile_column`. The endpoint is **Azure AD–protected**: present the same bearer token the web UI uses (validated against the same tenant / audience / scope). Without Azure auth configured the endpoint is only mounted in local dev-bypass mode — never unauthenticated in a deployed environment (ADR [0008](docs/adr/0008-mcp-server.md)).
Point any MCP client at `https://<your-dataq-host>/mcp/` (keep the **trailing slash** — `/mcp` 307-redirects and some clients drop the `Authorization` header on redirect) with an `Authorization: Bearer <token>` header. Once configured, all 8 tools are available to natural-language queries (e.g. *"what failed in the orders suite today?"*, *"run the orders suite on DEV"*).
Per-client configuration (Claude Desktop / Claude.ai, VS Code / Copilot, Cursor), how to get a token, token hygiene, and troubleshooting: **[AI assistants (MCP setup)](https://theurgicduke771.github.io/DataQ/mcp-setup/)** on the docs site.
## Documentation
| | |
|---|---|
| **Working agreements + commit/PR conventions** | [CONTRIBUTING.md](CONTRIBUTING.md) |
| **Documentation site (user guides)** | <https://theurgicduke771.github.io/DataQ/> · source in [docs/](docs/), built by [.github/workflows/docs.yml](.github/workflows/docs.yml) |
| **Deployment guide + env-var reference** | [deploy/README.md](deploy/README.md) · [.env.app.example](.env.app.example) |
| **Project guide for AI assistants** | [CLAUDE.md](CLAUDE.md) |
| **Architecture diagram + invariants** | [docs/architecture.md](docs/architecture.md) |
| **Architecture Decision Records** | [docs/adr/](docs/adr/) |
| **Live task tracker** | [docs/progress.md](docs/progress.md) |
| **Product roadmap (8 weeks, 100 tasks)** | [context/DataQ_platform_roadmap.md](context/DataQ_platform_roadmap.md) |
| **Security policy + responsible disclosure** | [SECURITY.md](.github/SECURITY.md) |
## License
MIT — see [LICENSE](LICENSE).
