Welcome to Lab 1 of the Finetuning Sessions!
After a comprehensive, introductory article about Transformers, attention mechanisms, and language model training pipeline, it's time to get our hands dirty and run experiments!
📕 If you haven't read Lesson 1's article, make sure to review it before going forward!
We have prepared a walkthrough video for you so that all ideas are clearly explained and you make steps towards becoming a finetuning ninja 🥷
Here's a detailed breakdown of the main ideas covered.
Continued PreTraining (CPT)
The first stage of a language model training pipeline is usually general pretraining, but what happens when your model needs to speak "medical", "legal", or "highly specific codebase"?
👉 This is where Continued PreTraining (CPT) enters the chat.
While the initial pretraining phase (the "Foundational" stage) exposes a model to a massive crawl of the internet, that data is often a mile wide and an inch deep.
If you are building a model for a specialized industry, generic internet data won't cut it … You need a model that understands the latent relationships between niche technical terms—not just how to autocomplete a sentence.
🙋 Here's the nuance: CPT is fundamentally different from Supervised Finetuning (SFT)!
In SFT, we provide "prompt-response" pairs to teach the model how to act. In CPT, we go back to self-supervised learning. We feed the model raw, unlabeled text from your specific domain (like 50GB of internal engineering documents or medical journals) using the same "predict the next token" objective.
✅ Why go through this extra step?
Domain Vocabulary: It allows the model to learn specialized tokens and their contexts that were underrepresented in the original training set.
World Knowledge: It updates the model’s internal facts. If you're working in a rapidly evolving field like AI research, a model pretrained in 2023 won't know about 2025's breakthroughs unless you perform CPT.
Better Foundation for SFT: An SFT model is only as good as the "brain" underneath it. If the base model doesn't understand the concepts you're asking it to follow instructions on, the finetuning will likely result in "hallucination-heavy" outputs.
Curriculum Learning
While large language models often steal the spotlight, the real magic frequently happens at a smaller, more specialized scale.
In this lab, we are going to get hands-on with continued pre-training, taking a Small Language Model (SLM) and immersing it in the specialized domain of mathematics.
🙋 Rather than starting from scratch, we leverage a pre-trained foundation and refine its "brain" to handle the rigor and logic required for mathematical reasoning.
However, fine-tuning a model on a new domain isn't as simple as dumping a textbook into its memory. To get the best results—especially when dealing with the high-density logic of math—we need a strategic approach to how that information is ingested.
Plus, one of the main challenges regarding causal language modelling is saturation. If you feed a model a mountain of complex data right away, it might struggle to converge, or worse, it might "collapse" into learning low-quality patterns.
🎓 This is where we borrow a concept from human education: Curriculum Learning.
Imagine trying to teach a child calculus before they can add numbers. They might eventually memorize some formulas, but they won't understand the logic. Models are surprisingly similar. If we bombard a fresh model with highly dense, complex mathematical proofs or messy, noisy web data at the very start, the gradients can become unstable.
🙋 The "plot twist" here? The order and quality in which you show data to a model can drastically change the final performance of the weights.
In Curriculum Learning, we organize training data by signal-to-noise ratio or complexity. We start with the most "educational" data—highly curated, clean, and foundational—and gradually increase the difficulty. This is often implemented in two ways:
Data Quality Sorting: Start training on "Gold Standard" datasets (like Wikipedia or textbooks) and slowly introduce the "Long Tail" of the internet (like Reddit or raw web crawls).
Sequence Length Scaling: Start with shorter context windows (e.g., 512 tokens) so the model learns local syntax, then gradually increase to 4k, 8k, or 128k tokens to learn long-range dependencies.
✅ The result? By preventing the model from getting "overwhelmed" by noise early on, we ensure the weights settle into a more robust configuration. It leads to faster convergence, lower final loss, and a model that is significantly more resilient to "garbage in, garbage out" scenarios.
Distillation
A common practice in the last stages of language model training pipelines is knowledge distillation.
If you've ever wondered how a tiny 7B model can sometimes punch way above its weight class—sometimes even outperforming 70B models on specific tasks—this is often the secret sauce!
The idea is simple yet powerful: we take a massive, highly capable "teacher" model (like GPT-4 or Llama 3 405B) and use its "intelligence" to train a much smaller "student" model. But we aren't just asking the student to copy the teacher's homework.
How it works: In standard training, a model sees a "hard label" (e.g., the next word is "Paris"). In Distillation, the student looks at the Teacher's Soft Targets. The Teacher doesn't just say the answer is "Paris"; it provides a probability distribution. It says, "I am 90% sure it's Paris, 8% sure it's Lyon, and 2% sure it's London.”
This "Dark Knowledge"—the relative probabilities of the incorrect answers—contains crucial information about how the Teacher perceives the relationship between words.
By transfering internal semantics from an already trained model into a smaller one, we achieve:
Efficiency: You get a model that is 10x smaller and 10x faster to run in production, but retains 90% of the capability of the giant.
Reasoning Transfer: Through techniques like Chain-of-Thought Distillation, we can even teach a small model the step-by-step reasoning process of a large model by using the Teacher's rationales as the training data for the Student.
Maximizing GPUs capabilities
Training language models is costly and computationally intensive; therefore, having a good understanding of hardware requirements and usage is essential to master finetuning.
You don't want to be the person who hits "Run" only to see an Out of Memory (OOM) error five seconds later because you forgot to account for the optimizer states.
🙋 When we talk about maximizing GPUs, we are fighting a war on two fronts: Memory (VRAM) and Compute (FLOPS).
First, let's look at the Memory Wall. Many beginners think that if a model is 14GB (7B parameters in FP16), they can fit it on a 16GB GPU. Wrong!
During training, you don't just store the weights; you store the gradients, the optimizer states (which can be 3-4x the size of the weights!), and the activations for every layer.
To bypass this, we use several key optimization "tricks":
Mixed Precision (BF16): Most modern hardware (like H100s or A100s) supports Brain Float 16. It offers the range of a 32-bit float but the memory footprint of a 16-bit float, significantly speeding up training without the instability of standard FP16.
Gradient Accumulation: If your GPU is too small for a large batch size, you can calculate gradients over several smaller “micro-batches” and only update the weights after a certain number of steps.
Activation Checkpointing: Instead of storing all the intermediate "thoughts"(activations) of the model during the forward pass, we throw them away and re-calculate them during the backward pass. It's a trade-off: you save massive amounts of VRAM at the cost of about 25% more compute time.
🙋 One final tip: Always monitor your Compute Utilization (SM Utilization). If your GPU memory is full but your utilization is low, your bottleneck is likely your CPU or your data loader—not your GPU!
