ai-basics

# AI Basics: From Neurons to Language Models > 🌐 **This guide is available in two languages.** You're reading **English**. Click **[हिंदी / Hindi](./hindi/README.md)** for the full Hindi version (all chapters translated). A complete, beginner-to-pro guide to building modern language models from scratch. Written in plain English with PyTorch code you can actually run. ## Who this is for - **Beginners** who know a little Python and want to understand how ChatGPT-style models work, end to end. - **Intermediate folks** who can use `nn.Linear` but want to know why a transformer block looks the way it does. - **Pros** who want refreshers on RoPE math, GQA cache savings, MoE routing, or quantization formats. You should not feel stuck. If a section feels too dense, skip to the code, run it, and come back. Reading and running is faster than reading alone. ## The learning path The chapters build on each other. If you only have time for a fast tour, read the **TL;DR** at the top of each file. | # | Topic | What you walk away with | |---|-------|--------------------------| | 00 | [Essential Math](./00-essential-math.md) | Matmul, gradients, chain rule, softmax, cross-entropy — the math you'll see every chapter. | | 00a | [Math for Beginners](./00a-math-for-beginners.md) | The same math as 00, but rebuilt from scratch for high schoolers — verbose, friendly, with a Google Colab walkthrough at the end. Read this first if `00` felt too dense. | | 01 | [Neural Networks](./01-neural-networks.md) | A neural net from scratch in NumPy + PyTorch. Backprop intuition. | | 02 | [PyTorch](./02-pytorch.md) | Tensors, autograd, modules, training loops, mixed precision. | | 03 | [GPU Computing](./03-gpu-computing.md) | Why GPUs are fast, memory hierarchy, kernel fusion, profiling. | | 04 | [Data](./04-data.md) | FineWeb-Edu, DCLM, MinHash dedup, classifier filtering, sharding. | | 05 | [Model Scale](./05-model-scale.md) | Scaling laws, Chinchilla, over-training, test-time compute. | | 06 | [Tokenization & Embeddings](./06-tokenization-embeddings.md) | BPE, vocab choices, embedding tables, tied heads. | | 07 | [Positional Encodings](./07-positional-encodings.md) | RoPE (full math + code), YaRN, NoPE layers. | | 08 | [Attention Mechanisms](./08-attention-mechanisms.md) | Q/K/V, multi-head, causal masking, Flash Attention 3, FlexAttention. | | 09 | [KV Cache, MQA, GQA](./09-kv-cache-mqa-gqa.md) | The single biggest inference win, plus MLA, PagedAttention, speculative decoding. | | 10 | [Building Blocks](./10-building-blocks.md) | RMSNorm, SwiGLU, residuals, pre-norm, QK-norm, looped / recurrent-depth blocks. | | 11 | [Building Qwen from Scratch](./11-building-qwen-from-scratch.md) | Glue everything into a real, working LLM with weights you can load. | | 12 | [Quantization](./12-quantization.md) | INT8/INT4/FP8/NVFP4, GPTQ, AWQ, GGUF, QLoRA. | | 13 | [Mixture of Experts](./13-mixture-of-experts.md) | Sparse models, routing, DeepSeek-V3-style fine-grained MoE. | | 14 | [Training Small Language Models](./14-training-small-language-models.md) | Full pretraining + SFT + DPO/GRPO pipeline with Muon and FSDP2. | | 15 | [Reading Training Logs](./15-reading-training-logs.md) | W&B charts, what each metric means, debugging loss curves. | | 16 | [Frontier Models in 2026](./16-frontier-models-2026.md) | Gemma 4 (E2B/E4B/26B-A4B/31B Dense), Qwen 3.6 (27B Dense / 35B-A3B MoE), and the GPT-6 Astra / Sol picture. | | 17 | [Production Inference](./17-production-inference.md) | Serving 10,000 requests: vLLM/SGLang, FP8, prefix cache, HBM bandwidth, token caching, pricing per archetype. | | 18 | [AI Apps & Agents](./18-ai-apps-and-agents.md) | The harness above the model: 12 usage patterns, MCP, real-world apps (Cursor, Claude Code, Cline), provider pricing, 300-line agent. | | 19 | [Evaluation](./19-evaluation.md) | Capability benchmarks, LLM-as-judge, golden sets, agent evals (SWE-bench, OSWorld, GAIA), eval-driven development. | | 20 | [Fine-tuning Recipes](./20-fine-tuning-recipes.md) | Decision tree, LoRA/QLoRA/DoRA, SFT/DPO/GRPO recipes, end-to-end Qwen 3 fine-tune in 150 lines. | | 21 | [Embeddings & RAG](./21-embeddings-and-rag.md) | Embedding models, vector DBs, hybrid search, reranking, chunking, agentic RAG, complete RAG in 80 lines. | | 22 | [Multimodal Models](./22-multimodal.md) | VLM architectures, SigLIP-2, native multi-resolution, audio (Whisper, Realtime), QLoRA for VLMs, hallucination evals. | | 23 | [Safety & Alignment](./23-safety-and-alignment.md) | Four kinds of safety, prompt injection, content moderation, red-teaming with PyRIT, compliance (EU AI Act, GDPR). | | 24 | [Gemma 4 Deep Dive](./24-gemma-4-deep-dive.md) | Full Gemma 4 reference: vocabulary glossary, architecture (PLE, hybrid attention, p-RoPE), dtype guide, real tok/s benchmarks, multimodal, thinking mode, end-to-end script. | | 25 | [Model Releases & Debugging](./25-model-releases-and-debugging.md) | The release ladder (major/minor/point/patch), continual vs fresh pretrain, tool stack per role, looking inside the model with hooks, TransformerLens, logit lens, ablation, activation patching, SAEs. | | 26 | [Mechanistic Debugging in Practice](./26-mechanistic-debugging.md) | Hands-on: safetensors surgery, circuit tracing step-by-step, sycophancy fix with steering vectors, SAE feature suppression, breakpoints inside neurons, chart visualizations, real before/after measurements. | | 27 | [Useful, Not Benchmaxxed](./27-useful-not-benchmaxxed.md) | **The most important chapter.** Goodhart's Law, what useful actually means, how to evaluate and train for users not leaderboards, the anti-pattern catalog, and case studies of products that win by ignoring benchmarks. | ## How to use this guide 1. **Start with [00-essential-math.md](./00-essential-math.md)** if any of "matrix multiplication / chain rule / softmax / cross-entropy" sound fuzzy. It takes 30 minutes and the rest of the book reads twice as fast afterward. 2. **Read 01-23 in order** the first time. 00-03 are foundations. 04-13 build the model. 14-15 train it. 16-17 deploy. 18 wraps it in a product. 19-23 are the operational maturity layer. **24** is a Gemma 4 reference. **25-26** are release engineering and mechanistic debugging. **Read 27 before you publish a single benchmark number.** It's the chapter on building *useful* models instead of leaderboard champions — the philosophy that decides whether the rest of the work mattered. 3. **Type the code yourself.** Reading code is not the same as writing it. Open a Jupyter notebook and recreate the snippets. 4. **Run on real hardware when possible.** A free Colab T4 GPU is enough for most exercises. For training, even a small CPU run teaches you the loop. 5. **Re-read the math once you've coded it.** The equations make more sense after you've seen the tensor shapes flow through them. ## Prerequisites - Python 3.10+ comfort (functions, classes, list comprehensions). - High-school algebra. We explain matrix multiplication when it first appears. - A working PyTorch install: `pip install torch numpy`. - Optional but useful: `pip install transformers datasets matplotlib`. ## Conventions - **Simple English first**, equations and code second. - All code is **PyTorch 2.x**. We mark CUDA-only sections clearly. - **Tensor shapes** are spelled out in comments: `# x: (B, T, D)` means batch, time/sequence, dimension. - **B** = batch, **T** = sequence length, **D** = model dimension, **H** = heads, **V** = vocab size. ## A note on style The goal is understanding, not impressive jargon. If you find a sentence here that sounds smart but doesn't help, that's a bug — open the file and rewrite it for the next reader. Now go to **[00-essential-math.md](./00-essential-math.md)** if you need the math refresher, or jump straight to **[01-neural-networks.md](./01-neural-networks.md)**.