Train Big, Tune Tiny: A Guide to Fine-Tuning LLMs with LoRA

Large language models (LLMs) like GPT, LLAMA and PaLM have revolutionised the natural language processing landscape, but many applications require adapting these large-scale pre-trained language models for specific domains. Such adaptation is generally done by fine-tuning, which updates all the model parameters, which can be expensive and time-consuming.

In this article, We will explore three key approaches to adapting large language models: prompt engineering, full fine-tuning, and LoRA (Low-Rank Adaptation). Each of these methods offers a unique trade-off between computational efficiency, implementation complexity, and performance.

Tokenization is the process of converting human-readable text into smaller units called tokens, which are then transformed into numerical IDs that the model can process.

In this example, we load the LLaMA 3.2–1B-Instruct model using Hugging Face’s transformers library.

Each model learns its own set of token embeddings, which are numerical representations of tokens, during the training process. As a result, different models often have different token vocabularies. In this case, we load the tokenizer that comes with the LLaMA 3.2 1B Instruct model to ensure that the input tokens match what the model was trained to understand.

we are passing a sentence to the tokenizer to convert it into token IDs.

will have output like

[128000, 279, 5679, 374, 43931, 279, 8415]

Each word in the sentence is assigned a specific token ID based on the model’s vocabulary. In this case, the word “the” appears twice in the sentence — once before “dog” and once before “cat”. Both instances are mapped to the same token ID (279), showing that the tokenizer treats repeated words consistently. This consistency is important, as it allows the model to recognise patterns and relationships across repeated tokens in the input.

While the input sentence “the dog is chasing the cat” contains only six words, the output includes seven token IDs. This is because the tokenizer automatically adds a special start-of-sequence token at the beginning.

<|begin_of_text|> the dog is chasing the cat

Special tokens are predefined markers that help the model understand the structure of the input. Common ones include: sos_token (start of sequence), eos_token (end of sequence) and pad_token (Padding token)

Padding tokens are used to make all input sequences the same length when processing data in batches. They don’t carry any meaning and are ignored by the model during computation. Some models don’t come with a default padding token, so we often set it to the end-of-sequence (eos) token to avoid issues during generation or batching.

Prompt engineering allows users to guide model behaviour using carefully crafted inputs, without modifying the model’s weights. It’s quick and cost-effective but can be limiting for complex or domain-specific tasks.

Prompting can be:

• Zero-shot: Asking the model to complete a task without any examples
• One-shot: Providing a single example of the task
• Few-shot: Supplying a handful of examples to help the model infer the task pattern

These techniques rely on the model’s ability to generalise from its pretraining, which often included a diverse set of instructions and formats.

Code Example

Suppose you want to build a model that can predict a Pokémon’s type based solely on its name. If the model has been pretrained on a dataset that includes Pokémon names and their corresponding types, it might already have learned some of these associations during pretraining. In that case, the model could potentially recall this information and correctly respond to prompts like “What type is Pikachu?” without requiring further training.

For this case, we can try using prompt engineering to see if our model can predict the Pokemon type from their names correctly.

First we start with loading the ‘tungdop2/pokemon’ dataset from Hugging Face.

And we are going to create our prompt template as:

We can then run inference with our model to see how well it performs on our prompt.

The output from our model

In this case, the model is able to correctly predict “Pikachu’s” type as “Electric”.

As with any good machine learning workflow, the next step is to evaluate how well the model performs across the rest of the dataset — not just on a single example.

And lets check the accuracy of our predictions.

Exact‐match accuracy: 17.50%

The model achieves an exact-match accuracy of 17.5% on our Pokémon dataset, which is quite low. This suggests that while the model may recall a few well-known examples, it struggles to generalise across the full set without further adaptation.

Full fine-tuning involves taking a pre-trained language model and updating all of its parameters on a new, task-specific dataset. This approach gives the model maximum flexibility to adapt to the task, allowing it to learn new patterns, vocabulary, and behaviours beyond what it was originally trained on.

Let’s say you’re the Queen from Snow White, and your model is the Magic Mirror. By default, the mirror is trained to answer all sorts of general questions about beauty, power, and fame. But now, you want it to answer a very specific question: “Who is the most beautiful girl?”

The model may reply with something along the lines of:

‘assistant\n\nBeauty is subjective and can vary from person to person. What one person finds beautiful’

This is a typical behaviour of a pretrained model without fine-tuning. It defaults to being neutral, balanced, and politically correct because it was trained on a diverse range of texts and opinions. It doesn’t yet share your royal standards or biases, and it certainly won’t name you as the fairest of them all — unless you fine-tune it accordingly.

We start by appending our target answer "Kimberly"to the prompt along with the end-of-sequence token. This forms the complete response we want the model to learn. We then tokenize this full string and split it into input_ids (everything except the last token) and target_ids (everything except the first token). This setup allows the model to learn to predict the answer token by token, just like in causal language modelling.

Next, we tokenize the target answer separately — adding a leading space and the end-of-sequence token — to create the labels that the model should predict. We pad this to the same length as target_ids to ensure proper alignment during training. This ensures that the model is evaluated only on the output portion (i.e., the answer), not the prompt itself, when calculating the loss.

To prepare the labels for training, we apply a mask using torch.where — replacing all padding token positions with -100, which tells the loss function to ignore those positions during training. Then, we explicitly set the last token to the eos_token_id to ensure the model is trained to correctly predict the end of the sequence.

We combine all of the above in a single function

We then calculate the loss using CrossEntropyLoss, which compares the model’s predicted token probabilities against the target labels. Since we masked the padding positions with -100, the loss function will ignore them and focus only on the meaningful parts of the output

We get the following loss values. As you can see, most of them are zero due to the masked padding positions. However, the tokens we want the model to learn have significantly higher loss values.

To reduce this loss and update the model’s parameters, we use the AdamW optimizer

Once training is complete, we can run inference on our fine-tuned model

And the model output is:

We fine-tuned our model (mirro) to generate the specific answer “Kimberly” in response to the question “Who is the most beautiful girl?”

One major downside of full fine-tuning, especially with small datasets, is the risk of catastrophic forgetting. Since we updated all of the model’s parameters using only a handful of examples, the model can quickly overwrite its pre-trained knowledge, forgetting what it previously knew in order to fit the small dataset. In our case, fine-tuning the model to always answer “Kimberly” may have caused it to lose its ability to respond accurately to other questions or topics.

So, if we ask the model, “What is the capital of the United Kingdom?” instead of replying “London”, it may reply.

As you can see, the model now consistently replies with “Kimberly”, regardless of the input question — a clear sign of catastrophic forgetting, where the model has lost its general reasoning ability in favour of memorising the fine-tuned response.

To avoid issues like catastrophic forgetting and the high resource costs of full fine-tuning, we can turn to a more efficient and controlled approach: LoRA, or Low-Rank Adaptation.

Instead of training all of the model’s parameters, LoRA works by adding two small, trainable low-rank matrices into the model’s existing weight structure — typically within the attention layers. These matrices are much smaller in size compared to the original weights, which makes training significantly more efficient. During training, only these added matrices are updated, while the rest of the model weights remain frozen. This allows the model to learn new behaviours without forgetting what it already knows.

Why does it work?

LoRA is based on a simple but powerful idea: when adapting a large pre-trained model to a new task, the necessary changes to the model’s weights often lie in a low-dimensional subspace. This means we do not need to update all the parameters to achieve good performance. Instead, we can introduce a small and efficient set of trainable components.

Suppose we have a pre-trained weight matrix W_0,rather than updating directly, LoRA represents the weight update using a low-rank decomposition:

What this means is that the weight update matrix, often called W, is assumed to be rank-deficient. In simpler terms, it doesn’t need to be a full-rank matrix to capture the important changes. For example, instead of having a full-rank matrix with dimensions 1024 by 1024, the meaningful update can be represented using a much lower rank, like 10. This low-rank structure greatly reduces the number of trainable parameters while still allowing the model to adapt effectively to new tasks.

Code Example

Lets come back to our Pokemon example. Now, instead of updating the entire model, we’ll apply LoRA to teach the model this behaviour.

We start with loading the model, “Llama-3.2–1B-Instruct” and tokenizer from Hugging Face

Then, we import the PEFT library (Parameter-Efficient Fine-Tuning) from Hugging Face. It handles the insertion of low-rank adapter layers, manages trainable parameters, and integrates seamlessly with Hugging Face’s transformers library.

Here in LoraConfig, we can change the parameters like

• r (rank): This sets the rank of the low-rank matrices A and B that LoRA adds to the model. A smaller r means fewer trainable parameters.
• lora_alpha: This is a scaling factor for the LoRA updates. It controls how much influence the new low-rank matrices have during training. A higher alpha increases the impact of the LoRA layers.

We can check the difference in the number of trainable parameters by

trainable params: 3,407,872 || all params: 1,239,222,272 || trainable%: 0.2750

In our LoRA setup, the model has 3,407,872 trainable parameters out of a total of 1,239,222,272 parameters, which means we’re only updating about 0.28% of the entire model.

Then we load in our dataset and perform data preprocessing to prepare it for training.

and finally we apply this function to our dataset

We run the training using trainer from Hugging Face

We run the training using the Trainer class from Hugging Face’s transformers library. This high-level API handles the training loop, evaluation, gradient updates, and checkpointing. It works seamlessly with LoRA through the PEFT wrapper, making it easy to fine-tune large models efficiently

Like any good machine learning engineer, we evaluate our model’s performance after training to see how well it has learned the task.

We then run inference using our fine-tuned model to generate responses

And finally calculate the accuracy of the task

Exact‐match accuracy: 67.21%

As you can see, the model achieves an accuracy of 67.21%, which is a significant improvement over the prompt engineering approach (17.5%). Unlike full fine-tuning, this result comes without the drawback of catastrophic forgetting, showing that LoRA can adapt the model effectively while preserving its original knowledge.