QLoRA puts LLM fine-tuning on a gaming GPU — here’s the code

June 2021: Microsoft publishes LoRA — a technique that trains less than 1% of a model’s parameters. May 2023: Tim Dettmers and his team at the University of Washington drop QLoRA — the same idea, but on a 4-bit quantized model that fits on consumer hardware. July 2026: the ecosystem is mature enough that anyone with a used RTX 3090 can fine-tune Llama-3-8B overnight.

LLM fine-tuning used to be a lab sport: clusters of eight A100s, five-figure cloud bills, weeks of compute. QLoRA dropped the entry ticket from $10,000 to $500 in hardware — or $0 if the card is already in your case. Here’s how it works, and what you give up.

Full fine-tuning — 8× A100 or go home

Full fine-tuning updates every weight in the network. For Llama-3-8B, that’s 8 billion parameters in bfloat16 — 16 GB just for the model weights. Add AdamW optimizer states (32 GB), gradients (16 GB), and activations that balloon with context length: you’re easily past 60–80 GB of VRAM.

The practical outcome: you need eight A100 80 GB GPUs to fine-tune a 7–8 billion parameter model at full precision. On AWS or GCP, a single A100 runs about $2–3/hour, which puts the cluster at $16–24/hour — roughly $400–600 per day. A proper fine-tuning run — 24 to 72 hours depending on dataset size — lands between $1,500 and $15,000. Add the cost of data engineering, failed experiments, and hyperparameter sweeps, and the real cost of a single successful fine-tune often exceeds $20,000.

For a startup or an individual, that’s a non-starter. For a lab, it’s a line item. LoRA changed the math in June 2021 by asking a simple question: what if you didn’t need to touch all the weights?

LoRA — 1% of the parameters, 90% of the result

The LoRA (Low-Rank Adaptation) paper, published by Edward Hu and his co-authors at Microsoft in June 2021, rests on an elegant mathematical observation. During fine-tuning, the weight update ΔW for each matrix W lives in a low-dimensional subspace. Rather than learning ΔW directly — a d × k matrix with millions of entries — you decompose it into a product of two smaller matrices:

ΔW = B × A, where B ∈ ℝᵈˣʳ and A ∈ ℝʳˣᵏ, with a very small rank r, typically 8 or 16.

In practice, LoRA freezes every pretrained weight and injects these B and A matrix pairs only into the attention layers (Q, K, V, O). The original weights W stay frozen; only the LoRA matrices are trained. The result: for Llama-3-8B, only 20–40 million parameters are updated — roughly 0.5% of the total.

The memory savings are massive. The optimizer no longer stores states and gradients for 8 billion parameters, only for the LoRA adapters. VRAM drops from 60–80 GB to 16–20 GB — the territory of a single RTX 3090 or RTX 4090. Training time drops by a factor of three to five because backpropagation no longer traverses the full computation graph.

The final trick: after training, you merge the LoRA matrices back into the original weights (W’ = W + BA). At inference time, the merged model has exactly the same memory footprint as the original. Zero overhead.

QLoRA — 4-bit quantization meets your gaming card

QLoRA, published by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer in May 2023, pushes the envelope further. The idea: apply LoRA not to bfloat16 weights, but to a 4-bit quantized model.

Quantization reduces the bit-width of each weight from 16 bits to 4 bits — a 4× compression. For Llama-3-8B, the 16 GB of weights becomes 4 GB. But naive quantization loses too much information. QLoRA introduces three innovations that fix this:

• NormalFloat 4-bit (NF4). Instead of slicing the value range into equal-width buckets like standard linear quantization, NF4 assumes weights follow a normal distribution and allocates more quantization levels where weights are dense. It’s an information-theoretically optimal quantizer for Gaussian data — and transformer weights are, roughly, Gaussian.
• Double quantization. The quantization constants themselves consume memory (roughly 0.5 bits per parameter for NF4). Double quantization quantizes those constants as well, reducing the overhead to 0.127 bits per parameter. On a 65-billion-parameter model, this detail saves 3 GB of VRAM.
• Paged optimizers. When a memory spike occurs during backpropagation (typically from gradient checkpointing), QLoRA offloads optimizer states to unified CPU RAM instead of crashing with an OOM. The mechanism is inspired by OS memory paging.

The result is dramatic. Llama-2-65B — sixty-five billion parameters — fine-tunes on a single RTX A6000 48 GB. Llama-3-8B fits in 10–14 GB of VRAM, meaning a RTX 3090 24 GB has room to spare for a batch size of 4. The original paper shows QLoRA matches full 16-bit fine-tuning performance on most benchmarks, including MMLU, TruthfulQA, and GSM8K.

Hands-on: fine-tune Llama-3-8B on a single RTX 3090

Here’s the minimal setup to specialize Llama-3-8B on a domain-specific task — internal knowledge base, technical docs, or an instruction dataset — using a single RTX 3090 24 GB.

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
from trl import SFTTrainer

# 4-bit NF4 quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokenizer.pad_token = tokenizer.eos_token

# Prepare model for k-bit training (freeze quantized weights)
model = prepare_model_for_kbit_training(model)

# LoRA config — r=16, alpha=32, all projections
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~42M | all params: ~8B | trainable%: 0.53

# Dataset — example with a JSONL instruction file
dataset = load_dataset("json", data_files="data/instructions.jsonl", split="train")

training_args = TrainingArguments(
    output_dir="./llama3-8b-qlora-finetuned",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,       # effective batch = 16
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=False,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    optim="paged_adamw_8bit",
    gradient_checkpointing=True,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=dataset,
    max_seq_length=2048,
    dataset_text_field="text",
)

trainer.train()
trainer.save_model()

A few details that prevent sleepless nights.

gradient_checkpointing=True trades compute for memory: intermediate activations are recomputed during backpropagation instead of stored. On an RTX 3090, this is non-negotiable for a 2,048-token context.

paged_adamw_8bit is QLoRA’s paged optimizer: it offloads to CPU RAM automatically when VRAM saturates, without crashing. The alternative adamw_8bit from bitsandbytes uses less CPU but doesn’t handle spikes.

target_modules: targeting all seven linear projections (Q, K, V, O, gate, up, down) yields the best results for instruction fine-tuning. Targeting only Q and V saves memory but leaves performance on the table.

With this setup, fine-tuning on 10,000 examples takes 4–8 hours on an RTX 3090 and consumes 14–18 GB of VRAM. The same operation as full fine-tuning would need a cluster of eight A100s and cost $2,500–$8,000 on the cloud — for a quality gain often under 2% on specialization benchmarks.

Verdict: which approach fits your budget

If you have an RTX 3090 or 4090 in your case, start with QLoRA today. Quality is within 2% of full fine-tuning on most specialization tasks, at marginal hardware cost. Your only investment is experimentation time.

If your cloud budget is under $500, rent an RTX 3090 or RTX 6000 Ada on vast.ai or RunPod ($0.35–$0.80/hour) and use QLoRA. An overnight run costs $3–$10. At that price, you can iterate ten times before hitting the cost of a single full fine-tuning run.

If you have access to a cluster of eight H100s or more and you’re working on a domain where every hundredth of a benchmark point matters — continual pre-training on a massive corpus, adapting a frontier model to a regulated industry — full fine-tuning remains the gold standard. But for 95% of enterprise use cases (internal chatbot, document extraction, specialized classification), QLoRA is more than enough.

If you’re just getting started, use Unsloth. This open-source library optimizes LoRA/QLoRA with custom Triton kernels and cuts training time by a factor of two to three compared to the standard Hugging Face implementation, with no quality degradation. The free Google Colab notebook is enough to fine-tune Llama-3-8B on a small dataset — zero hardware setup required.

LLM fine-tuning is no longer a lab sport. It’s a developer tool. The graphics card that ran Cyberpunk 2077 last night can train your customer support model tonight. QLoRA closed the gap between the data center and the PC case under your desk.

References

• Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models (June 2021)
• Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs (May 2023)
• Hugging Face PEFT — Documentation (accessed July 29, 2026)
• Hugging Face — 4-bit Transformers with bitsandbytes (May 2023)
• Unsloth — Faster LLM Fine-tuning (accessed July 29, 2026)
• Bitsandbytes — GitHub (accessed July 29, 2026)
• TRL — Transformer Reinforcement Learning (accessed July 29, 2026)