Welcome to Lab 4 of the Finetuning Sessions!
In today's lab, we're moving from theory to the terminal. We are stepping onto the 4-bit frontier with a hands-on QLoRA finetuning experiment.
If you haven't read last Wednesday's deep dive, make sure to review it before going forward! Understanding the geometry of the "VRAM Wall" is essential to mastering the code we are about to run.
The shift from standard LoRA to Quantized LoRA (QLoRA) isn't just a minor optimization—it's a fundamental rethinking of the GPU memory map.
While LoRA taught us how to reduce the active parameter count, QLoRA attacks the "static" weight footprint that usually keeps frontier-class models out of reach for individual researchers.
Today, we are going deep into the implementation.
We won't just "run a script"; we will dissect how the Unsloth library leverages 4-bit NormalFloat (NF4) and Paged Optimizers to squeeze every drop of performance out of a single GPU.
We will walk through a live training job on Hugging Face infrastructure, analyzing how to reallocate the VRAM we "save" through quantization into massive context windows and stable gradients.
By the end of this lab, you won't just have a trained adapter—you'll have a first-principles understanding of how to orchestrate high-fidelity training on low-precision silicon.
From LoRA to QLoRA
⚠️ Sorry about the issue with the video. We had some technical problems during the recording. The voice and screen sharing work well, but there is some lag in the window where the person speaking appears. Thanks for your understanding!
In this first clip, we look at the script that drives our experiment.
The beauty of the Unsloth library is that transitioning from a standard LoRA setup to a memory-efficient QLoRA setup is a single-parameter change, but its impact on your VRAM "geometry" is profound.
The 4-bit switch: Notice the
load_in_4bit=Trueargument in theFastLanguageModel.from_pretrainedcall. By flipping this fromFalsetoTrue, you aren't just loading a smaller file; you are triggering the NF4 (NormalFloat 4-bit) quantization we discussed in the theory section. This immediately shrinks the "fixed cost" of your base model weights by 75%.The PEFT Injection: In the
get_peft_modelblock, we target the specific projection layers (q_proj,k_proj, etc.). In a QLoRA context, these 16-bit adapters are being mathematically stitched onto a 4-bit base. We keep the learning high-fidelity while the "memory" remains compressed.The memory safety valve: Look at
optim = "adamw_8bit". This ensures that even our optimizer states—the “silent killers” of VRAM—are compressed, allowing us to fit larger models into smaller hardware slots without losing the "momentum" of the Adam algorithm.
