Unsloth: fine-tuning an LLM on a large public

#The question to settle before installing anything

Yes, you can fine-tune a 7- or 8-billion-parameter model on a consumer GPU: Unsloth’s documentation gives a minimum of 5 to 6 GB of VRAM in 4-bit QLoRA, so a 12 GB card is suitable, with room for context and the batch. Unsloth rewrites the expensive LoRA and QLoRA fine-tuning steps to achieve, according to the vendor, up to twice the speed with 70% less VRAM, a figure that applies only to certain notebooks. The result can be exported to GGUF to run in Ollama. But the real question comes before installation: fine-tuning changes a model’s behavior (tone, format, register), not what it knows up to date. If you want it to know your documents, build document retrieval first. Also expect dataset preparation to take longer than training.

Fine-tuning changes how a model behaves. Document retrieval changes what it knows when answering. Confusing the two can cost you weeks.

• Fine-tuning or RAG: the detailed decision tree
• LoRA and QLoRA: how they really work
• A complete LoRA fine-tuning example
• Assess whether RAG would do better, with numbers

#What Unsloth changes in practice

The method is not new: the model weights are frozen, and only thin matrices added to each weight are trained—about 1% of the parameters according to Unsloth's guide. LoRA keeps the original model in 16-bit, while QLoRA quantizes it to 4-bit, saving 75% of memory. This is the standard approach; it does not belong to Unsloth.

What the library provides is the implementation, with rewritten compute kernels for the heavy stages. The project exists in three forms: Unsloth Desktop, a native application; Unsloth Studio, a web interface; and Unsloth Core, the code-driven version discussed in this article. It is released under a dual license: Apache 2.0 for the core, and AGPL-3.0 for certain optional components such as the Studio interface. It claims to train language, diffusion, speech-synthesis, and embedding models twice as fast with 70% less VRAM, without loss of accuracy. On mixture-of-experts (MoE) architectures, the published gain is even higher: up to 12 times faster with more than 35% VRAM savings on some recent models. As with all vendor figures, these describe the best case: the README itself shows that the “twice as fast, 70%” claim applies only to certain notebooks.

#Which card for which model

The documentation specifies that these figures are absolute minimums: depending on the model, more may be required. It identifies an excessively large batch size as a frequent cause of exhausted memory, recommending reducing it to 1, 2, or 3, and recommends a context length of 2048 for initial tests.

#The dataset is the real work

A fine-tuning run never fails because of the library. It fails because of the data.

Format

A two-column dataset—usually question and answer—in the conversation structure expected by the model. The guide recommends starting with an Instruct model: it directly accepts conversation templates (ChatML, ShareGPT) and requires less data than a base model. Consistency matters more than volume.

Volume

The Unsloth guide recommends at least 100 lines, and more than 1,000 lines for better results. Tone and format transfer with few examples; a true business behavior requires more, and they must be consistent with one another.

Quality

The model imitates what it is shown, including errors. A systematic flaw in the data becomes a systematic flaw in the model.

Control set

Examples the model never sees during training. Without them, it is impossible to distinguish learning from simple memorization.

No code

Unsloth Studio offers Data Recipes that transform PDFs, CSVs, or DOCX files into datasets through a visual workflow, with a preview before launching the full build.

#From adapter to usable model

The process takes three steps: load a model in 4-bit, train it with one of the official notebooks, then export it. The first code block loads the model and adds the LoRA adapters; the actual training is handled by one of Unsloth’s notebooks, which the guide recommends copying into your local environment.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-8B-unsloth-bnb-4bit",  # quantification dynamique 4 bits d'Unsloth
    max_seq_length=2048,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=16)

The name suffix matters. According to the guide, a model ending in unsloth-bnb-4bit is Unsloth’s dynamic 4-bit quantization: it uses slightly more VRAM than standard BitsAndBytes quantization, but offers significantly higher precision. A name ending only in bnb-4bit refers to the standard version. The guide also notes that training and serving benefit from using the same precision: serving in 4-bit means training in 4-bit.

1. 01
   Train

   The result is a LoRA adapter, approximately 100 MB in Unsloth's example, not a complete model.
2. 02
   Evaluate

   Against the control set and against the original model. “Better” must be demonstrated, not assumed. The guide notes that automatic evaluation tools may poorly reflect your criteria.
3. 03
   Merge or keep separate

   The adapter can remain separate and be swapped, or it can be merged into the weights with: model.save_pretrained_merged using save_method="merged_16bit".
4. 04
   Export to GGUF and quantize

   model.save_pretrained_gguf produces the file, with q4_k_m, q8_0, or f16 as options. That's what makes the result runnable by Ollama or llama.cpp on ordinary hardware.

model.save_pretrained_gguf(
    "mon-modele-q4", tokenizer, quantization_method="q4_k_m"
)
# puis, avec le Modelfile fourni : ollama create mon-modele -f Modelfile

• Import a GGUF into Ollama
• Choosing your model’s quantization level

#Common pitfalls

Starting from a base model without realizing it

The guide recommends Instruct models: they support conversation templates and require less data. A base model requires a different format (Alpaca, Vicuna) and more examples.

Forget the conversation template

Examples formatted differently from what the model expects produce training that is technically successful but practically useless.

Measure on training examples

This is the mistake that leads to spectacular results being announced and disappointing models being delivered.

Believing it will replace document research

What we teach the model becomes outdated when your information changes, and the model does not cite its sources. For changing facts, document search remains safer.

Underestimating example cleanup

Near-identical duplicates in the dataset bias training toward those specific cases without anything indicating it in the tracked metrics.

Change several settings at once

Changing the learning rate, adapter rank, and sequence length in the same run makes it impossible to know which one produced the observed change.

#Concrete use cases

Consistent-tone customer support

A model fine-tuned on real interactions responds in the brand's voice, without repeated style instructions in every prompt.

Extraction in a strict format

A fine-tuner trained on input-output examples fixes the output format more reliably than a prompt alone, which is useful before sending the result to a downstream system.

These cases have one thing in common: each can be measured. Before starting a training run, state the success criterion in one verifiable sentence—“the JSON format is respected in at least 95% of the outputs in the control set,” for example—rather than relying on a general impression of improvement. That criterion, not intuition, will tell you whether the result justifies the time invested.

• Source: official Unsloth repository (features, hardware, dual licensing)
• Source: requirements and minimum VRAM by model size
• Source: Unsloth fine-tuning guide
• Source: GGUF export and template issues
• Source: performance figures for MoE models

#The real cost beyond the price of the card

The graphics card is only one part of the equation. Collecting and cleaning consistent examples, writing a holdout set the model will never see, and then comparing each version with the original model often matters more than the training itself, as the Unsloth guide illustrates with a 60-step run.

That is why Unsloth, as fast as it is for training, does not shorten a project as much as hoped: it removes the technical bottleneck, not the data work.

#FAQ

Recommended hardware: GMKtec EVO-X2 64GB / 1TB (Ryzen AI Max+ 395) — 16 GB NVIDIA card, sufficient for fine-tuning a 7–8B model with QLoRA. All AI hardware →