Fine tuning of Large Language Models (LLM)
Harmonizing the Large Language Model to Address Specific Challenges and Needs in Various Fields
Large Language Models, often abbreviated as LLMs, are a significant breakthrough in the field of natural language processing. However, despite their powerful ability to handle a wide range of text-based tasks, these models may not always be compatible to perform certain tasks or domains that are very specific in nature.
To overcome such challenges, we can apply what is known as Fine tuning. Fine tuning of Large Language Models (LLM) is a technique used to customise the model by further training it using specific datasets. The dataset used for Fine tuning is generally much smaller than the dataset used in the initial training of the model.
Fine tuning techniques can significantly improve the consistency and efficiency of model performance. By applying this technique, we can direct the model to adjust to the nuances of vocabulary, patterns, and structures that are specific to a particular domain, according to the needs of the intended application. For example, in the fields of medicine, law, finance, and so on. Fine tuning allows models to become more skilful in understanding and responding to specialised contexts, thereby improving the quality of output produced in various tasks and applications.
Types of Fine Tuning
The following are some of the types of fine tuning that can be done to fine-tune a model
Supervised Fine-Tuning: This fine-tuning technique involves using a pre-trained model and further training it using a customised dataset that has labels. The dataset used usually contains input-output pairs, so that the model can learn to map inputs to their corresponding outputs. This type of fine-tuning is used when we have labelled datasets for specific tasks, such as sentiment analysis, text classification, or entity recognition.
Instruction Fine-Tuning: This type of fine-tuning involves training the model using a dataset of input-output examples that take the form of instructions in the form of template prompts. With this approach, the model is taught to interpret the instructions and produce outputs that match the input context. Instruction Fine-tuning is useful for improving the generalisation ability of the model, especially in the context of tasks that involve interaction or response to specific instructions. Typically, this technique is used when we want to create a chatbot-like model.
PEFT method: Parameter Efficient Fine Tuning (PEFT) is a fine-tuning approach that fine-tunes the model parameters, but does not update all model weights, thus reducing memory requirements significantly. PEFT is usually used when we want to train a large model with limited computational resources.
Soft Prompting: Soft prompting or prompt tuning is a method that involves adding new tokens to be trained into the prompt model. Only the newly added tokens are trained, while all other tokens and model weights are left unchanged. This technique is useful for improving the performance of a model by adjusting its prompts without affecting the structure or other parameters of the model.
Reinforcement Learning Human Feedback (RLHF): This method adopts the concept of reinforcement learning to adjust the model. This approach is usually performed after the fine-tuning stage, with the aim of producing outputs that are more in line with human preferences. RLHF utilises human manual judgement of the model output, and then uses a reinforcement learning algorithm to update the weights of the fine-tuned model based on the best output score given by humans. Thus, this method helps to iteratively improve the quality of the model output through interaction with human feedback.
Steps to perform Fine Tuning
There is actually no standard way to perform fine tuning, as the right approach depends on the specific objectives of the task at hand. However, in general, the following steps are often applied:
Select a base model: Select a pre-trained base model according to the needs of the task and the availability of computational budget.
Determine fine-tuning methods: Consider different fine-tuning methods that match the model’s use cases, the available computational budget, the dataset used, and the desired model size.
Prepare the dataset: Prepare a dataset that matches the task to be performed, the selected fine-tuning method, and the expected input-output format of the selected base model. A good dataset is the key to successful fine-tuning.
Train the model: Use the available libraries such as Transformer, Hugging Face or PyTorch/TensorFlow to train the model with the prepared dataset. This training process transforms the base model into one that is better suited to the task at hand.
Evaluation and iteration: Evaluate the performance of the model against the expected criteria and iterate if necessary to improve the quality of the model. This process often involves iterative processing using different configurations until optimal results are achieved.
By following these steps, the fine-tuning process can be carried out systematically and efficiently to achieve the desired results.
Conclusion
Fine-tuning is an important step in building a model from Large Language Models (LLM) to solve specific tasks. Through the fine-tuning process, we can guide the model to produce outputs that meet the expected applications and criteria with respect to the relevant specialized formats or domains. Thus, fine-tuning allows the model to meet different needs in solving useful tasks, whether on a small, medium or large scale.
