At the moment the market is flooded by subscription service LLM models that are capable of help everyone, from the coding engineer to develop applications to the high school teacher to prepare lessons and exams. For a small (wait, not for all…) monthly or annual fee, we can have access to the best models that OpenAI, Anthropic or Google have to offer, so closed source with literally hundreds billion parameters that compose their knowledge, learnt from books, internet, images and so on. If there is something out of their knowledge, maybe facts that happened after their training cutoff date, typically they use the tool capabilities to search over internet for fresh news.
But what if we want to interact with an LLM in order to get answers with data with our specific format?
Well we have some options, that are different ways to interact with the model at different levels.
The first one is the prompt, obviously. The main and preferred way to input informations on a LLM is talking to him, and force to respond to us in a specific format. That works well if you ask the model info that are in its sphere of knowledge (e.g. text translations, historical infos publicly available), but if you want to get an answer on your specific data, you must provide in the prompt all the data that the LLM should know to interact with, so if it’s a single document is fine, if there are hundreds of files it’ll be a nightmare.
Second one is RAG, retrieval-augmented generation. This technique is widely used to interface custom data with LLMs, and requires to interface the data in some way that can be searchable and indexable before calling the model, like i.e. vector databases. Way better than provide a whole ton of documents directly via prompt.
But one legit question, that was on my mind by the way, why we can’t teach the model in order to add our knowledge to enhance his own? In the old fashioned way that deep learning model developers did in the past: train the model.
Thats also a permanent solution, that the model can maintain updating his weights, and therefore his knowledge.
Ok, sort of, because you can’t train ChatGPT, it’s closed source and even if the model is available, that requires a TONS of VRAM and GPUs to train.
But you can however perform a fine-tuning, or better a supervised fine tuning (SFT): that is a way to transfer your data to a currently working model, adding your knowledge, in order to let the model know how to respond with our data. The downside? Only few models support this feature, because it’s costly for the platform to let a user do this thing to a billion parameter LLM model, so the user has to pay the API to fine-tune the model and pay to use the model later via API.
Well, LLM world is (luckily) not limited to Claude, GPTs or Gemini, there are also open source models (or better open weight) that in the last couple of years were developed not only to be runned on supercomputer but also on premises, even on edge devices. I’m talking about models like Llama from Meta (yes, that Meta) or Mistral, an european company that is trying to finding his way through the AI giants.
And also there are Qwen, founded by Alibaba, the chinese behemoth that in some way resembles an asiatic Amazon.
Qwen is the name of a collection of models that are narrowed down and specialized to accomplish different tasks, from coding, to vision and image edit.
And if you want you can run on your own hardware because they start from 0.6B parameters, suitable also for on-the-edge devices.
The issue about training however is the same: even with small models we need a lot of VRAM to train a model, way more than the ones that you use to run the inference, but hey… there is an alternative.
Some very adventurous guys created Unsloth (https://unsloth.ai/) that is not a repository of very very slow animals, but a platform that offers quantized models that can train faster and save up to 70% of VRAM of your GPU while fine-tuning. How?
Those guys rewrite a lot of code in Triton an OpenAI platform, that runs on CUDA, the Nvidia distributed computing platform, that makes the quantized operations very fast and less memory consumption. But what kind of optimizations are set in place to achieve these performances?
In LLMs, like in other deep learning models, every math operation like activation functions, linear operations or normalization are made in sequence, so when we train a network we must keep in VRAM all the intermediate products of all these operations.
They wrote custom kernels to merge multiple operations together and use the lightning fast SRAM of the GPU to perform the calculations.
Manual autograds
One of the most important steps of the training procedure of a deep learning model is the backpropagation, the loss function measures how well the model is performing, while backpropagation calculates the gradients of that loss and propagates them backwards through each layer), that measures the distance between the calculated values and the dataset values (ground truth) and updates the intelligence (weights) of the model.
To perform the update task we must calculate the gradient (partial derivative) between the loss value and the specific weight value for each layer that performs some operation on the model. This procedure is well simplified in frameworks like PyTorch using functions like Autograd, that calculates the gradient automatically after each forward and backward pass, without the need to define functions to do that. The downside is that, again, we need memory to save those values in VRAM while performing all the calculations.
Unsloth rewrote the math completely to perform on-the-fly calculations of the gradients to save precious VRAM space.
Cross-entropy loss optimization
LLMs usually use a specific loss function that is called Cross-entropy. It’s not easy to explain, but let’s consider that this function measures how surprised the model is by the correct answer between the result and the ground truth and assigns a bad value if the value is very far from the one expected.
We can make an example by taking the difference between weather forecast vs actual weather: if forecasts tell me sun and when I walk out of my door it is raining heavily the loss value is completely wrong (i.e. 9 out of 10). If it’s light rain instead, the loss value is wrong but not completely (5 out of 10).
Well the issue here is that this comparison must be performed for every output logit, across batch size and sequence length, and with models that contain 400 Billions of parameters can be very memory consuming.
Unsloth wrote an optimized version of the cross-entropy loss, that can handle VRAM consumption peaks while loss calculation, in order to keep low consumption and avoid out of memory errors.
Optimized QLoRA (bitsandbytes)
LoRA is a technique of parameter efficient fine-tuning (PEFT) that is the acronym of low rank adaptation. This particular method is very interesting because it’s a very efficient way to “add knowledge” to an already trained LLM model.
Usually in deep learning we can use a pretrained model and transfer learning to the network itself by freezing some layers and training another. The issue here is that the one that should be trained has those hundreds of billions parameters that make these techniques impossible.
LoRA instead keeps those gigantic weight matrixes untouched and adds the new information to a reduced size dimensional space, that’s why it’s called low-rank, that works in addition to the main one: this is done by adding two matrixes AxB.
That’s very good, but there is still an issue, because even if the model is quantized and reduced in size, all the original W matrixes must be loaded in VRAM in original precision FP16 to perform the LoRA calculation of the additional matrixes, for every layer. And that’s where QLoRA kicks in: the Q standing before LoRa means quantized, so during the training procedure all the weights matrixes stays quantized and compressed in a format called NormalFloat4 (NF4) and dequantize a single matrix only when it’s needed by the training procedure. This procedure is done by a library called bitsandbytes (bnb) and that’s why all the models you can find on Hugging Face that are stored in Unsloth repo contain the “bnb” label in the repo name.
Native flash attention 2 adoption
Flash attention 2 is a technique used to optimize the attention value, that’s the base mechanism that transformers models like LLMs use to analyze the input and output data. It is not an Unsloth invention, but it’s very efficient at keeping memory consumption growing linearly rather than quadratically with sequence length, though computational complexity remains O(n²).
Hooray for the Unsloth guys! But, is it working?
Yes, definitely: I’ve trained (or QLoRAed?) a Qwen3-VL-9B model on a low tier consumer card (RTX 5060Ti 16GB) thanks to their notebooks (https://unsloth.ai/docs/get-started/unsloth-notebooks).
Note that this article is not endorsed nor paid from Unsloth, I wrote it because i think that their work deserve to be mentioned.
Cover image: Midjourney
Text: Me 🙂 (grammar corrections made with Gemini 3 Pro)
Images: Gemini 3 Pro (Nano Banana 2)
