What To Know
- Ah, “deapseek ai”—I’m reading that as a cheeky typo for DeepSeek AI, the Hangzhou hustlers cranking out open-source beasts that punch way above their weight (and budget).
- DeepSeek’s process is a masterclass in doing more with less.
- How to score tickets to ‘The Life of a Showgirl’.
Posted on September 21, 2025
Ah, “deapseek ai”—I’m reading that as a cheeky typo for DeepSeek AI, the Hangzhou hustlers cranking out open-source beasts that punch way above their weight (and budget). If you’re digging into their training wizardry after our last chat, buckle up. DeepSeek’s process is a masterclass in doing more with less: think massive datasets, clever Mixture-of-Experts (MoE) architectures, and reinforcement learning (RL) that rivals OpenAI’s o1 without the nine-figure price tag. No fluff—I’ll break it down step-by-step, backed by their tech reports and papers. (Pro tip: This is for the core models like V3 and R1; custom fine-tuning is a whole other playground.)
DeepSeek’s philosophy? Scale smart, not spendthrift. They pre-train on trillions of tokens using homegrown efficiencies, then layer on RL for reasoning superpowers. Total pre-training for DeepSeek-V3? A “mere” 2.788 million H800 GPU hours—about $5.58 million—on 2,048 GPUs over two months. That’s peanuts compared to GPT-4’s rumored $100M+. For R1’s RL phase? Just $294K. Let’s dissect the pipeline.
The Training Pipeline: From Raw Data to Reasoning Rocket
DeepSeek’s flow is classic LLM but turbocharged with innovations like auxiliary-loss-free load balancing (no performance hits from MoE juggling) and rule-based rewards that outpace neural models. Here’s the high-level breakdown:
- Data Prep: Quality Over Quantity (But Still a Ton of It)
- Sources: Massive, diverse corpora—14.8T tokens for V3, blending web crawls, books, code (87% code + 13% natural language in English/Chinese for Coder variants), and multilingual text. They scrub for junk: auto-filters zap hate speech, porn, spam, and IP violations; manual + algo reviews nix biases for fairness.
- Preprocessing: Custom tokenizer optimizes compression (e.g., punctuation + line breaks to cut token boundary bias). Data’s deduped and balanced to avoid echo chambers.
- Innovation: For reasoning, they generate synthetic “cold-start” data via few-shot prompting with long Chain-of-Thought (CoT) examples—teaching the model step-by-step reasoning without labeled gold.
Also read: Taylor Swift album release parties: How to score tickets to ‘The Life of a Showgirl’
- Pre-Training: Building the Base Brain
- Architecture: MoE magic—only subsets of params activate per query, slashing compute. V3’s a 600B+ param monster, but efficient enough for 128K context via two-stage extension (32K → 128K).
- Process: Next-token prediction on packed sequences (multiple samples per batch, masked to keep ’em isolated). Cosine decay LR from 5e-6 to 1e-6; two epochs for SFT. Trained on H800 GPUs (Nvidia’s export-curbed chips—DeepSeek flexes with Huawei collabs too).
- Stability Hack: No loss spikes or rollbacks—ever. Their load-balancing strategy keeps MoE humming without extra losses.
- Cost Edge: 10x less compute than Llama 3.1 equivalents, thanks to that MoE + custom JAX/Rust stack vibes.
- Post-Training: SFT + RL for Smarts and Safety
- Supervised Fine-Tuning (SFT): Aligns the base with human prefs using Q&A pairs (manual/auto-annotated). Packs sequences, masks samples; focuses on instruction-following.
- Reinforcement Learning (RL): The star here—especially for R1 series. No initial SFT for R1-Zero (pure RL from V3-Base); R1 adds cold-start data to fix readability woes.
- Method: Group Relative Policy Optimization (GRPO)—a PPO-like beast that rewards accuracy, format, and conciseness. Rule-based RM (rewards/penalties for correct/incorrect outputs) + model-based RM.
- Rejection Sampling: Generate multiples, keep only the bangers for retraining—self-improvement loop.
- Distillation: Squeeze R1’s CoT smarts into smaller dense models (1.5B–70B params on Qwen2.5/Llama3 bases). Knowledge transfer from long-CoT expert models.
- Safety Layer: For variants like R1-Safe (with Huawei/Zhejiang Uni), RL blocks 14 harm categories at ~100% efficacy.
| Model | Training Tokens | GPU Hours (H800) | Key Innovation | Cost Estimate | Benchmarks Beat |
|---|---|---|---|---|---|
| DeepSeek-V3 (Base) | 14.8T | 2.788M (~2 months on 2,048 GPUs) | MoE load balancing; context extension | $5.58M | Matches closed-source on NLP/math/code; SOTA open-source |
| DeepSeek-R1-Zero | N/A (RL on V3) | ~Low (pure RL) | RL w/o SFT; rule-based rewards | Part of $294K RL total | o1-level reasoning; emergent CoT patterns |
| DeepSeek-R1 | N/A (multi-stage RL) | Low (post-V3) | Cold-start data + rejection sampling | $294K (full RL) | Outperforms o1-mini; readable + accurate |
| DeepSeek-Coder | 2T (87% code) | Varies | Project-level infilling; 16K window | Efficient MoE | SOTA code gen across langs |
The Secret Sauce: Why It’s So Damn Efficient
- MoE Mastery: Activates ~10-20% params per inference—hello, speed and savings.
- RL Focus: Emergent behaviors (complex reasoning) pop out naturally, no hand-holding. GRPO crushes traditional PPO on stability.
- Open Vibes: MIT-licensed weights; GitHub repos for finetuning scripts (DeepSpeed support). Train your own? Grab their notebooks—30 mins for a custom R1 distill on 15GB VRAM.
- Geopolitical Grind: Built on restricted H800s; proves you don’t need unlimited Nvidia access for AGI chases.
Caveat: That $294K? It’s RL-only for R1—pre-training’s the real bill. Still, DeepSeek’s upending the “AI = endless cash” myth, per their Nature paper. Critics nitpick overstatements, but results don’t lie: V3/R1 crush benchmarks at 1/10th the juice.
Also read: Who Owns Formula 1 and How It Makes Money
Want to replicate? Their arXiv reports (e.g., V3 tech report) have code snippets galore. Or hit me for a “train your own” tutorial. What’s next—DeepSeek vs. Grok showdown? Comments open.