Content
<p align="center">
<img src="assets/banner.svg" alt="AI Engineering from Scratch" width="100%">
</p>
<p align="center">
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="License: MIT"></a>
<a href="CONTRIBUTING.md"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" alt="PRs Welcome"></a>
<img src="https://img.shields.io/badge/Lessons-200+-purple" alt="200+ Lessons">
<img src="https://img.shields.io/badge/Phases-20-orange" alt="20 Phases">
<img src="https://img.shields.io/github/stars/rohitg00/ai-engineering-from-scratch?style=social" alt="GitHub Stars">
</p>
<p align="center">
<a href="#the-journey">Journey</a> •
<a href="#getting-started">Get Started</a> •
<a href="#course-output-the-toolkit">Toolkit</a> •
<a href="ROADMAP.md">Roadmap</a> •
<a href="CONTRIBUTING.md">Contribute</a> •
<a href="glossary/terms.md">Glossary</a>
</p>
---
200+ hands-on lessons across 20 phases. From linear algebra to autonomous agent swarms. Python, TypeScript, Rust, Julia. Every lesson produces something reusable: prompts, skills, agents, MCP servers.
You learn AI. You build real things. You ship tools others can use.
| | Other Courses | This Course |
|---|---|---|
| **Scope** | One slice (NLP or Vision or Agents) | Everything: math, ML, DL, NLP, vision, speech, transformers, LLMs, agents, swarms |
| **Languages** | Python only | Python, TypeScript, Rust, Julia |
| **Output** | "I learned something" | A portfolio of tools, prompts, skills, and agents |
| **Depth** | Surface-level or theory-heavy | Build from scratch first, then use frameworks |
| **Format** | Videos or docs | Runnable code + notebooks + docs + web app |
---
## The Journey
<table>
<tr><td>
### Phase 0: Setup & Tooling `4 lessons`
> Get your environment ready for everything that follows.
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | [Dev Environment](phases/00-setup-and-tooling/01-dev-environment/) | Build | Python, Node, Rust |
| 02 | Git & Collaboration | Learn | -- |
| 03 | GPU Setup & Cloud | Build | Python |
| 04 | APIs & Keys | Build | Python, TS |
</td></tr>
</table>
<details>
<summary><strong>Phase 1: Math Foundations</strong> <code>10 lessons</code> <em>The intuition behind every AI algorithm, through code.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | [Linear Algebra Intuition](phases/01-math-foundations/01-linear-algebra-intuition/) | Learn | Python, Julia |
| 02 | Vectors, Matrices & Operations | Build | Python, Julia |
| 03 | Matrix Transformations & Eigenvalues | Build | Python, Julia |
| 04 | Calculus for ML -Derivatives & Gradients | Learn | Python |
| 05 | Chain Rule & Automatic Differentiation | Build | Python |
| 06 | Probability & Distributions | Learn | Python |
| 07 | Bayes' Theorem & Statistical Thinking | Build | Python |
| 08 | Optimization: Gradient Descent Family | Build | Python |
| 09 | Information Theory: Entropy, KL Divergence | Learn | Python |
| 10 | Dimensionality Reduction: PCA, t-SNE, UMAP | Build | Python |
</details>
<details>
<summary><strong>Phase 2: ML Fundamentals</strong> <code>13 lessons</code> <em>Classical ML - still the backbone of most production AI.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | What Is Machine Learning: Types & Taxonomy | Learn | -- |
| 02 | Linear Regression from Scratch | Build | Python |
| 03 | Logistic Regression & Classification | Build | Python |
| 04 | Decision Trees & Random Forests | Build | Python |
| 05 | Support Vector Machines | Build | Python |
| 06 | K-Nearest Neighbors & Distance Metrics | Build | Python |
| 07 | Unsupervised Learning -K-Means, DBSCAN | Build | Python |
| 08 | Feature Engineering & Selection | Build | Python |
| 09 | Model Evaluation -Metrics, Cross-Validation | Build | Python |
| 10 | Bias, Variance & the Learning Curve | Learn | Python |
| 11 | Ensemble Methods: Boosting, Bagging, Stacking | Build | Python |
| 12 | Hyperparameter Tuning & AutoML | Build | Python |
| 13 | ML Pipelines & Experiment Tracking | Build | Python |
</details>
<details>
<summary><strong>Phase 3: Deep Learning Core</strong> <code>13 lessons</code> <em>Neural networks from first principles. No frameworks until you build one.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | The Perceptron: Where It All Started | Build | Python |
| 02 | Multi-Layer Networks & Forward Pass | Build | Python |
| 03 | Backpropagation from Scratch | Build | Python |
| 04 | Activation Functions: ReLU, Sigmoid, GELU & Why | Learn | Python |
| 05 | Loss Functions: MSE, Cross-Entropy, Contrastive | Build | Python |
| 06 | Optimizers -SGD, Momentum, Adam, AdamW | Build | Python |
| 07 | Regularization -Dropout, Weight Decay, BatchNorm | Build | Python |
| 08 | Weight Initialization & Training Stability | Build | Python |
| 09 | Learning Rate Schedules & Warmup | Build | Python |
| 10 | Build Your Own Mini Framework | Build | Python |
| 11 | Introduction to PyTorch | Build | Python |
| 12 | Introduction to JAX | Build | Python |
| 13 | Debugging Neural Networks | Learn | Python |
</details>
<details>
<summary><strong>Phase 4: Computer Vision</strong> <code>16 lessons</code> <em>From pixels to understanding - image, video, and 3D.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Image Fundamentals: Pixels, Channels, Color Spaces | Learn | Python |
| 02 | Convolutions from Scratch | Build | Python |
| 03 | CNNs: LeNet to ResNet | Build | Python |
| 04 | Image Classification | Build | Python |
| 05 | Transfer Learning & Fine-Tuning | Build | Python |
| 06 | Object Detection -YOLO from Scratch | Build | Python |
| 07 | Semantic Segmentation -U-Net | Build | Python |
| 08 | Instance Segmentation -Mask R-CNN | Build | Python |
| 09 | Image Generation -GANs | Build | Python |
| 10 | Image Generation -Diffusion Models | Build | Python |
| 11 | Stable Diffusion -Architecture & Fine-Tuning | Build | Python |
| 12 | Video Understanding -Temporal Modeling | Build | Python |
| 13 | 3D Vision: Point Clouds, NeRFs | Build | Python |
| 14 | Vision Transformers (ViT) | Build | Python |
| 15 | Real-Time Vision: Edge Deployment | Build | Python, Rust |
| 16 | Build a Complete Vision Pipeline | Build | Python |
</details>
<details>
<summary><strong>Phase 5: NLP: Foundations to Advanced</strong> <code>18 lessons</code> <em>Language is the interface to intelligence.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Text Processing: Tokenization, Stemming, Lemmatization | Build | Python |
| 02 | Bag of Words, TF-IDF & Text Representation | Build | Python |
| 03 | Word Embeddings: Word2Vec from Scratch | Build | Python |
| 04 | GloVe, FastText & Subword Embeddings | Build | Python |
| 05 | Sentiment Analysis | Build | Python |
| 06 | Named Entity Recognition (NER) | Build | Python |
| 07 | POS Tagging & Syntactic Parsing | Build | Python |
| 08 | Text Classification -CNNs & RNNs for Text | Build | Python |
| 09 | Sequence-to-Sequence Models | Build | Python |
| 10 | Attention Mechanism -The Breakthrough | Build | Python |
| 11 | Machine Translation | Build | Python |
| 12 | Text Summarization | Build | Python |
| 13 | Question Answering Systems | Build | Python |
| 14 | Information Retrieval & Search | Build | Python |
| 15 | Topic Modeling: LDA, BERTopic | Build | Python |
| 16 | Text Generation | Build | Python |
| 17 | Chatbots: Rule-Based to Neural | Build | Python |
| 18 | Multilingual NLP | Build | Python |
</details>
<details>
<summary><strong>Phase 6: Speech & Audio</strong> <code>12 lessons</code> <em>Hear, understand, speak.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Audio Fundamentals: Waveforms, Sampling, FFT | Learn | Python |
| 02 | Spectrograms, Mel Scale & Audio Features | Build | Python |
| 03 | Audio Classification | Build | Python |
| 04 | Speech Recognition (ASR) | Build | Python |
| 05 | Whisper: Architecture & Fine-Tuning | Build | Python |
| 06 | Speaker Recognition & Verification | Build | Python |
| 07 | Text-to-Speech (TTS) | Build | Python |
| 08 | Voice Cloning & Voice Conversion | Build | Python |
| 09 | Music Generation | Build | Python |
| 10 | Audio-Language Models | Build | Python |
| 11 | Real-Time Audio Processing | Build | Python, Rust |
| 12 | Build a Voice Assistant Pipeline | Build | Python |
</details>
<details>
<summary><strong>Phase 7: Transformers Deep Dive</strong> <code>14 lessons</code> <em>The architecture that changed everything.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Why Transformers: The Problems with RNNs | Learn | -- |
| 02 | Self-Attention from Scratch | Build | Python |
| 03 | Multi-Head Attention | Build | Python |
| 04 | Positional Encoding: Sinusoidal, RoPE, ALiBi | Build | Python |
| 05 | The Full Transformer: Encoder + Decoder | Build | Python |
| 06 | BERT -Masked Language Modeling | Build | Python |
| 07 | GPT -Causal Language Modeling | Build | Python |
| 08 | T5, BART -Encoder-Decoder Models | Build | Python |
| 09 | Vision Transformers (ViT) | Build | Python |
| 10 | Audio Transformers -Whisper Architecture | Build | Python |
| 11 | Mixture of Experts (MoE) | Build | Python |
| 12 | KV Cache, Flash Attention & Inference Optimization | Build | Python, Rust |
| 13 | Scaling Laws | Learn | Python |
| 14 | Build a Transformer from Scratch | Build | Python |
</details>
<details>
<summary><strong>Phase 8: Generative AI</strong> <code>14 lessons</code> <em>Create images, video, audio, 3D, and more.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Generative Models: Taxonomy & History | Learn | -- |
| 02 | Autoencoders & VAE | Build | Python |
| 03 | GANs: Generator vs Discriminator | Build | Python |
| 04 | Conditional GANs & Pix2Pix | Build | Python |
| 05 | StyleGAN | Build | Python |
| 06 | Diffusion Models -DDPM from Scratch | Build | Python |
| 07 | Latent Diffusion & Stable Diffusion | Build | Python |
| 08 | ControlNet, LoRA & Conditioning | Build | Python |
| 09 | Inpainting, Outpainting & Editing | Build | Python |
| 10 | Video Generation | Build | Python |
| 11 | Audio Generation | Build | Python |
| 12 | 3D Generation | Build | Python |
| 13 | Flow Matching & Rectified Flows | Build | Python |
| 14 | Evaluation: FID, CLIP Score | Build | Python |
</details>
<details>
<summary><strong>Phase 9: Reinforcement Learning</strong> <code>12 lessons</code> <em>The foundation of RLHF and game-playing AI.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | MDPs, States, Actions & Rewards | Learn | Python |
| 02 | Dynamic Programming | Build | Python |
| 03 | Monte Carlo Methods | Build | Python |
| 04 | Q-Learning, SARSA | Build | Python |
| 05 | Deep Q-Networks (DQN) | Build | Python |
| 06 | Policy Gradients -REINFORCE | Build | Python |
| 07 | Actor-Critic -A2C, A3C | Build | Python |
| 08 | PPO | Build | Python |
| 09 | Reward Modeling & RLHF | Build | Python |
| 10 | Multi-Agent RL | Build | Python |
| 11 | Sim-to-Real Transfer | Build | Python |
| 12 | RL for Games | Build | Python |
</details>
<details>
<summary><strong>Phase 10: LLMs from Scratch</strong> <code>14 lessons</code> <em>Build, train, and understand large language models.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Tokenizers: BPE, WordPiece, SentencePiece | Build | Python |
| 02 | Building a Tokenizer from Scratch | Build | Python, Rust |
| 03 | Data Pipelines for Pre-Training | Build | Python |
| 04 | Pre-Training a Mini GPT (124M) | Build | Python |
| 05 | Distributed Training, FSDP, DeepSpeed | Build | Python |
| 06 | Instruction Tuning -SFT | Build | Python |
| 07 | RLHF -Reward Model + PPO | Build | Python |
| 08 | DPO -Direct Preference Optimization | Build | Python |
| 09 | Constitutional AI | Build | Python |
| 10 | Evaluation -Benchmarks, Evals | Build | Python |
| 11 | Quantization: INT8, GPTQ, AWQ, GGUF | Build | Python, Rust |
| 12 | Inference Optimization | Build | Python |
| 13 | Building a Complete LLM Pipeline | Build | Python |
| 14 | Open Models: Architecture Walkthroughs | Learn | Python |
</details>
<details>
<summary><strong>Phase 11: LLM Engineering</strong> <code>13 lessons</code> <em>Put LLMs to work in production.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Prompt Engineering: Techniques & Patterns | Build | Python |
| 02 | Few-Shot, CoT, Tree-of-Thought | Build | Python |
| 03 | Structured Outputs | Build | Python, TS |
| 04 | Embeddings & Vector Representations | Build | Python |
| 05 | Vector Databases | Build | Python, TS |
| 06 | RAG -Retrieval-Augmented Generation | Build | Python, TS |
| 07 | Advanced RAG -Chunking, Reranking | Build | Python |
| 08 | Fine-Tuning with LoRA & QLoRA | Build | Python |
| 09 | Function Calling & Tool Use | Build | Python, TS |
| 10 | Evaluation & Testing | Build | Python |
| 11 | Caching, Rate Limiting & Cost | Build | Python, TS |
| 12 | Guardrails & Safety | Build | Python |
| 13 | Building a Production LLM App | Build | Python, TS |
</details>
<details>
<summary><strong>Phase 12: Multimodal AI</strong> <code>11 lessons</code> <em>See, hear, read, and reason across modalities.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Multimodal Representations | Learn | -- |
| 02 | CLIP: Vision + Language | Build | Python |
| 03 | Vision-Language Models | Build | Python |
| 04 | Audio-Language Models | Build | Python |
| 05 | Document Understanding | Build | Python |
| 06 | Video-Language Models | Build | Python |
| 07 | Multimodal RAG | Build | Python, TS |
| 08 | Multimodal Agents | Build | Python, TS |
| 09 | Text-to-Image Pipelines | Build | Python |
| 10 | Text-to-Video Pipelines | Build | Python |
| 11 | Any-to-Any Models | Learn | Python |
</details>
<details>
<summary><strong>Phase 13: Tools & Protocols</strong> <code>10 lessons</code> <em>The interfaces between AI and the real world.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Function Calling Deep Dive | Build | Python, TS |
| 02 | Tool Use Patterns | Build | TS |
| 03 | MCP: Model Context Protocol | Learn | -- |
| 04 | Building MCP Servers | Build | TS, Python |
| 05 | Building MCP Clients | Build | TS, Python |
| 06 | MCP Resources, Prompts & Sampling | Build | TS |
| 07 | Structured Output Schemas | Build | TS, Python |
| 08 | API Design for AI | Build | TS |
| 09 | Browser Automation & Web Agents | Build | TS |
| 10 | Build a Complete Tool Ecosystem | Build | TS, Python |
</details>
<details>
<summary><strong>Phase 14: Agent Engineering</strong> <code>15 lessons</code> <em>Build agents from first principles.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | [The Agent Loop](phases/14-agent-engineering/01-the-agent-loop/) | Build | Python, TS |
| 02 | Tool Dispatch & Registration | Build | TS |
| 03 | Planning: TodoWrite, DAGs | Build | TS |
| 04 | Memory: Short-Term, Long-Term, Episodic | Build | TS, Python |
| 05 | Context Window Management | Build | TS |
| 06 | Context Compression & Summarization | Build | TS |
| 07 | Subagents: Delegation | Build | TS |
| 08 | Skills & Knowledge Loading | Build | TS |
| 09 | Permissions, Sandboxing & Safety | Build | TS, Rust |
| 10 | File-Based Task Systems | Build | TS |
| 11 | Background Task Execution | Build | TS |
| 12 | Error Recovery & Self-Healing | Build | TS |
| 13 | Hooks: PreToolUse, PostToolUse | Build | TS |
| 14 | Eval-Driven Agent Development | Build | Python, TS |
| 15 | Build a Complete AI Agent | Build | TS |
</details>
<details>
<summary><strong>Phase 15: Autonomous Systems</strong> <code>11 lessons</code> <em>Agents that run without human intervention safely.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | What Makes a System Autonomous | Learn | -- |
| 02 | Autonomous Loops | Build | TS, Python |
| 03 | Self-Healing Agents | Build | TS |
| 04 | AutoResearch: Autonomous Research | Build | TS, Python |
| 05 | Eval-Driven Loops | Build | TS |
| 06 | Human-in-the-Loop | Build | TS |
| 07 | Continuous Agents | Build | TS |
| 08 | Cost-Aware Autonomous Systems | Build | TS |
| 09 | Monitoring & Observability | Build | TS, Rust |
| 10 | Safety Boundaries | Build | TS |
| 11 | Build an Autonomous Coding Agent | Build | TS |
</details>
<details>
<summary><strong>Phase 16: Multi-Agent & Swarms</strong> <code>14 lessons</code> <em>Coordination, emergence, and collective intelligence.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Why Multi-Agent | Learn | -- |
| 02 | Agent Teams: Roles & Delegation | Build | TS |
| 03 | Communication Protocols | Build | TS, Rust |
| 04 | Shared State & Coordination | Build | TS, Rust |
| 05 | Message Passing & Mailboxes | Build | TS |
| 06 | Task Markets | Build | TS |
| 07 | Consensus Algorithms | Build | TS, Rust |
| 08 | Swarm Intelligence | Build | Python, TS |
| 09 | Agent Economies | Build | TS |
| 10 | Worktree Isolation | Build | TS |
| 11 | Hierarchical Swarms | Build | TS |
| 12 | Self-Organizing Systems | Build | TS, Rust |
| 13 | DAG-Based Orchestration | Build | TS, Rust |
| 14 | Build an Autonomous Swarm | Build | TS, Rust |
</details>
<details>
<summary><strong>Phase 17: Infrastructure & Production</strong> <code>11 lessons</code> <em>Ship AI to the real world.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | Model Serving | Build | Python |
| 02 | Docker for AI Workloads | Build | Python, Rust |
| 03 | Kubernetes for AI | Build | Python |
| 04 | Edge Deployment: ONNX, WASM | Build | Python, Rust |
| 05 | Observability | Build | TS, Rust |
| 06 | Cost Optimization | Build | TS |
| 07 | CI/CD for ML | Build | Python |
| 08 | A/B Testing & Feature Flags | Build | Python, TS |
| 09 | Data Pipelines | Build | Python, Rust |
| 10 | Security: Red Teaming, Defense | Build | Python, TS |
| 11 | Build a Production AI Platform | Build | Python, TS, Rust |
</details>
<details>
<summary><strong>Phase 18: Ethics, Safety & Alignment</strong> <code>6 lessons</code> <em>Build AI that helps humanity. Not optional.</em></summary>
| # | Lesson | Type | Lang |
|:---:|--------|:----:|------|
| 01 | AI Ethics: Bias, Fairness | Learn | -- |
| 02 | Alignment: What & Why | Learn | -- |
| 03 | Red Teaming & Adversarial Testing | Build | Python |
| 04 | Responsible AI Frameworks | Learn | -- |
| 05 | Privacy: Differential Privacy, FL | Build | Python |
| 06 | Interpretability: SHAP, Attention | Build | Python |
</details>
<details>
<summary><strong>Phase 19: Capstone Projects</strong> <code>5 projects</code> <em>Prove everything you learned.</em></summary>
| # | Project | Combines | Lang |
|:---:|---------|----------|------|
| 01 | Build a Mini GPT & Chat Interface | Phases 1, 3, 7, 10 | Python, TS |
| 02 | Build a Multimodal RAG System | Phases 5, 11, 12, 13 | Python, TS |
| 03 | Build an Autonomous Research Agent | Phases 14, 15, 6 | TS, Python |
| 04 | Build a Multi-Agent Dev Team | Phases 14, 15, 16, 17 | TS, Rust |
| 05 | Build a Production AI Platform | All phases | Python, TS, Rust |
</details>
---
## Course Output: The Toolkit
Every lesson produces something reusable. By the end you have:
```
outputs/
├── prompts/ Prompt templates for every AI task
├── skills/ SKILL.md files for AI coding agents
├── agents/ Agent definitions ready to deploy
└── mcp-servers/ MCP servers you built during the course
```
Real tools. Install them with [SkillKit](https://github.com/rohitg00/skillkit), plug them into Claude Code, Cursor, or any AI agent.
---
## How Each Lesson Works
```
phases/XX-phase-name/NN-lesson-name/
├── code/ Runnable implementations (Python, TS, Rust, Julia)
├── notebook/ Jupyter notebooks for experimentation
├── docs/
│ └── en.md Lesson documentation
└── outputs/ Prompts, skills, agents produced by this lesson
```
Every lesson follows 6 steps:
1. **Motto** - one-line core idea
2. **Problem** - why this matters
3. **Concept** - visual diagrams and intuition
4. **Build It** - implement from scratch
5. **Use It** - same thing with real frameworks
6. **Ship It** - the prompt, skill, or agent this lesson produces
---
## Getting Started
```bash
git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch
python phases/00-setup-and-tooling/01-dev-environment/code/verify.py
python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py
```
### Prerequisites
- You can write code (Python or any language)
- You want to understand how AI actually works
---
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for how to add lessons, translations, and outputs.
Want to fork this for your team or school? See [FORKING.md](FORKING.md).
See [ROADMAP.md](ROADMAP.md) for progress tracking.
---
<p align="center">
MIT License. Use it however you want.
</p>
