{"id":5545,"date":"2024-07-03T06:24:33","date_gmt":"2024-07-03T06:24:33","guid":{"rendered":"https:\/\/www.moxoff.com\/?p=5545"},"modified":"2025-11-10T15:12:05","modified_gmt":"2025-11-10T15:12:05","slug":"come-specializzare-un-large-language-model","status":"publish","type":"post","link":"https:\/\/www.moxoff.com\/en\/come-specializzare-un-large-language-model\/","title":{"rendered":"How to specialise a Large Language Model"},"content":{"rendered":"

A technical comparison of the various fine-tuning techniques for Large Language Models<\/h2>\n

Introduction<\/h2>\n

In recent years, Large Language Models (LLMs)<\/strong> have captured the world\u2019s attention, revolutionising artificial intelligence thanks to their remarkable capability and versatility in addressing real-world problems. Despite such promising prospects, LLMs are not without significant issues. Among these are bias, low accuracy in specific contexts, and a lack of understanding of the more subtle nuances of human language.
Companies like Moxoff<\/strong>, which work daily with the implementation of solutions utilising LLMs and generative AI, must therefore be ready to take on the challenge of developing technologies that are both cutting-edge and free from such concerns.<\/p>\n

To achieve this, a training<\/strong> method has been developed that enables LLMs to be specialised by controlling their behaviour. This procedure, known as fine-tuning,<\/strong> generally consists of two main steps: Supervised Fine-Tuning (SFT)<\/strong> and Reinforcement Learning from Human Feedback (RLHF)<\/strong>. This reinforcement learning-based approach, which we will examine in detail later in its most common version, Proximal Policy Optimisation (PPO),<\/strong> plays a crucial role in allowing LLMs to specialise and reach appropriate levels of accuracy.
However, this procedure (SFT + RLHF with PPO) has limitations: it is characterised by significant practical implementation complexity, limited versatility, and considerable computational resource consumption. To address these issues, alternative methods for refining language models have recently been proposed. In this article, we intend to analyse in detail the characteristics of the following methods:<\/p>\n

    \n
  1. Direct Preference Optimisation (DPO)<\/strong><\/li>\n
  2. Kahneman-Tversky Optimization (KTO);<\/strong><\/li>\n
  3. Odds Ratio Preference Optimization (ORPO).<\/strong><\/li>\n<\/ol>\n

    We therefore aim to provide a comprehensive and informed overview of the progress in the field of LLM alignment, highlighting the improvements and innovations introduced by the various methods considered.
    To do so, we started from a pre-trained model, Microsoft\u2019s Phi-3-mini-instruct-128k<\/a><\/span>, and carried out four distinct alignment procedures (PPO, DPO, KTO and ORPO). For greater completeness and reproducibility of the results, we have also published all four aligned model versions on the Moxoff Hugging Face page<\/a><\/span>, making them available to the open source community.<\/p>\n

    Methods<\/h2>\n

    This paragraph is dedicated to an in-depth look at the alignment methods mentioned above, analysing their key features.<\/p>\n

    Supervised Fine-Tuning (SFT)<\/h3>\n

    SFT<\/strong> (Supervised Fine-Tuning) is a process in which a pre-trained LLM is optimised using labelled data, consisting of input examples (prompts) and the corresponding expected outputs. Through SFT, it is possible to obtain models whose behaviour aligns with the desired specifications: for example, it allows for the enforcement of certain constraints and the satisfaction of specific requirements, such as adherence to a particular style. This phase of model training is the most important, as it is where the majority of improvements are observed. However, it also demands the most computational resources and time, resulting in the need for a dataset<\/strong> that is significantly larger than those used in subsequent phases\u2014sometimes 10 to 20 times larger.<\/p>\n

    Reinforcement Learning from Human Feedback (RLHF)<\/h3>\n

    SFT alone does not result in sufficient improvements to be used independently with satisfactory outcomes. Therefore, an additional step is introduced, known as RLHF<\/strong>.<\/p>\n

    RLHF is a methodology that combines reinforcement learning<\/strong> with human feedback to enhance model performance. After SFT has been completed, the model generates responses to various inputs. These responses are evaluated by human reviewers, who assign scores based on the quality and relevance of the answers. The scores provided by the reviewers are then used to train a reward model. This reward model learns to predict the quality of the LLM\u2019s responses based on human evaluations. Using the reward model, reinforcement learning techniques are applied to update the LLM, training it to produce responses that maximise the score predicted by the reward model.<\/p>\n

    Proximal Policy Optimisation (PPO)<\/strong> is one of the most commonly used algorithms in the context of RLHF, thanks to its efficiency and stability. PPO improves the model learning process by reducing the risks of undesirable behaviours and performance instability. It achieves this by imposing a limit on the size of the policy updates at each iteration, meaning that changes to the model\u2019s policy are gradual and controlled. Another key feature of PPO is the use of a clipping function, which prevents updates from diverging too far. This helps to maintain stable learning, avoiding oscillations or extreme behaviours. Thanks to these mechanisms, PPO makes reinforcement learning more reliable and smooth, increasing the likelihood of obtaining optimal and stable solutions.
    Despite the strong performance, this type of fine-tuning technique suffers from high implementation complexity, due to the challenges involved in training the reinforcement model.<\/p>\n

    Direct Preference Optimization (DPO)<\/h3>\n

    DPO<\/strong> (Direct Preference Optimisation) represents a significant advancement in language model training, distinguishing itself clearly from more traditional approaches such as RLHF. Also designed as a step following SFT, this method focuses not on optimising a reward, but rather on solving preference classification problems. This means that DPO aims to identify and prioritise actions deemed preferable by a supervisor<\/strong>, rather than to maximise rewards.
    Although DPO offers several advantages compared to methods like RLHF, which utilise human feedback to guide policy learning, it also has its own set of challenges. One of the main issues is the risk of overfitting, as the model may become overly tailored to the example data provided during training, losing the ability to generalise to new, unseen situations and potentially producing hallucinations.<\/p>\n

    Kahneman-Tversky Optimization (KTO)<\/h3>\n

    The KTO methodology<\/strong>, also designed as an alternative to RLHF to be conducted after SFT, is inspired by studies showing that humans evaluate reality asymmetrically, being more adept at quantifying flaws and losses than strengths and gains. Specifically, it is a variant of DPO, intended to address its issues by using a single-preference methodology.<\/p>\n

    KTO outperforms DPO in cases involving noisy and unrefined data, a situation very common in the field of natural language. For this reason, it is an excellent candidate for use on datasets of limited size, a context in which it demonstrates its potential most effectively.<\/p>\n

    Odds Ratio Preference Optimization (ORPO)<\/h3>\n

    The ORPO methodology<\/strong> is, however, fundamentally different from the previous ones. Instead of using the traditional SFT followed by an additional learning phase, ORPO combines these steps. This makes the process faster and more efficient, as well as resulting in lighter models.
    The underlying idea is simple: when increasing the likelihood of obtaining correct answers, one also risks increasing the likelihood of obtaining incorrect answers. ORPO tackles this problem by directly penalising incorrect responses during training. In this way, the model is adapted to a specific scenario from the outset, improving overall performance.<\/p>\n

    Comparison<\/h2>\n

    The comparison was carried out between four distinct fine-tunings of a pre-trained model, specifically Phi-3-mini-instruct. Fine-tuning was performed for 3 epochs using the following datasets, according to the selected method:<\/p>\n

      \n
    1. For KTO<\/strong>, we used the argilla\/distilabel-intel-orca-kto<\/a><\/span> dataset.<\/li>\n
    2. risponda, garantendo la massima qualit\u00e0.\n\nFor PPO, DPO and ORPO<\/strong>, we used the argilla\/ultrafeedback-binarized-preferences-cleaned<\/a><\/span> dataset, selecting only the data with a chosen rating of 5. This means that the responses in the dataset are exactly how we want the model to reply, ensuring the highest quality.<\/li>\n<\/ol>\n

      The alignments were performed on a machine equipped with a 4090 GPU, i7-8700 CPU, and 32GB of RAM. For benchmarking metrics, we chose Hellaswag, Arc-challenge and M_MMLU_EN 5 shot. Here are the benchmarking values of Phi3-mini-instruct before fine-tuning:<\/p>\n\n\n

      Hellaswag<\/td>Arc-challenge<\/td>M_MMLU 5 shot<\/td><\/tr>
      0.7820<\/td>0.5463<\/td>0.6808<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n

      Proximal Policy Optimization (PPO)<\/h3>\n

      \u00a0<\/p>\n\n\n

      \"\"<\/figure>\n\n\n\n
      \"\"<\/figure>\n\n\n

      Training using PPO did not yield optimal results, as can be seen from the metric values.<\/p>\n\n\n

      Hellaswag<\/td>Arc-challenge<\/td>M_MMLU 5 shot<\/td><\/tr>
      0.7044<\/td>0.4701<\/td>0.5814<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n

      Despite some initial improvements, we observed an increase in loss and stagnation in rewards, indicating that the model was not learning effectively. Accuracies and margins showed no significant progress, and logp and logits did not move in the desired direction. This further confirms that, without access to substantial datasets and significant hardware resources, it is generally difficult to achieve good results with this technique. However, by increasing the amount of available data and upgrading the GPU, PPO can become an extremely robust training method<\/strong>.<\/p>\n

      Direct Preference Optimization (DPO)<\/h3>\n\n\n
      \"\"<\/figure>\n\n\n\n
      \"\"<\/figure>\n\n\n

      Training with DPO produced excellent results, with a significant reduction in loss over just 3 epochs. We observed a clear improvement in rewards, accuracies, and margins, as well as in logp and logits, which moved in the desired direction. However, the benchmark results did not show a notable improvement compared to the initial values:<\/p>\n\n\n

      Hellaswag<\/td>Arc-challenge<\/td>M_MMLU 5 shot<\/td><\/tr>
      0.7833<\/td>0.5486<\/td>0.6847<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n

      This result might be due to the fact that we started from an instruct model instead of an SFT, or because we used a relatively small dataset and carried out only a few training epochs.<\/p>\n

      Kahneman-Tversky Optimization (KTO)<\/h3>\n

      \u00a0<\/p>\n\n\n

      \"\"<\/figure>\n\n\n\n
      \"\"<\/figure>\n\n\n

      Alignment using KTO also went very well, with all metrics showing improvements and a reduction in loss. Compared to DPO, we observed an even greater improvement:<\/p>\n

        \n
      1. \n
          \n
        1. \n
            \n
          1. The accuracy and reward metrics improved significantly;<\/li>\n
          2. The logp and logits aligned with expectations.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

            This suggests that the KTO method may be more effective under the conditions in which we conducted the training. Below are the benchmarking metrics:<\/p>\n\n\n

            Hellaswag<\/td>Arc-challenge<\/td>M_MMLU 5 shot<\/td><\/tr>
            0.7915<\/td>0.5606<\/td>0.6939<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n

            Odds Ratio Preference Optimization (ORPO)<\/h3>\n\n\n
            \"\"<\/figure>\n\n\n\n
            \"\"<\/figure>\n\n\n

            Training with ORPO<\/strong> showed positive progress throughout the various training phases, with a steady reduction in loss and improvements in accuracy and margin metrics similar to DPO. However, as with DPO, the positive results of this training were not reflected in the final model\u2019s benchmarks.<\/p>\n\n\n

            Hellaswag<\/td>Arc-challenge<\/td>M_MMLU 5 shot<\/td><\/tr>
            0.7621<\/td>0.5375<\/td>0.6824<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n

            Conclusion<\/h2>\n

            The comparison between the various alignment methods for Large Language Models considered in this analysis (PPO, KTO, DPO and ORPO<\/strong>) highlighted significant differences in terms of time, resource consumption, and performance.
            It is not possible to identify a universally best method, as each one can prove to be the most appropriate choice in certain scenarios, depending on the hardware resources available<\/strong>, the quantity<\/strong> and quality of data<\/strong>, and time constraints.
            Among the methods analysed, PPO proved to be particularly demanding in terms of computation and time, without delivering the expected results. However, this technique shows significant potential: with adequate data and resources<\/strong>, it remains one of the leading candidates to emerge as the most robust and reliable approach.<\/p>\n

            On the other hand, the remaining methods proved to be more accessible and equally valid for use, albeit with differences among them.
            In particular, KTO<\/strong> demonstrated notable improvements over the initial model. This is because, for datasets of limited size, such as the one considered here, this alignment technique tends to outperform the others.
            DPO<\/strong> provided good results in terms of accuracy and margins, but did not show significant improvements on benchmarks compared to the initial values. This may be due to the use of a relatively small dataset and a limited number of training epochs.<\/p>\n

            Finally, ORPO<\/strong> showed positive progress with a steady reduction in loss and improvements in accuracy and margin metrics, although it did not reach the levels of the other methods on the benchmarks considered. However, ORPO<\/strong> offers a combination of efficiency and speed, making it an interesting option in scenarios with time and resource constraints.
            In summary, KTO emerges as the most effective solution for LLM alignment, offering an optimal balance between performance and efficiency. Methods such as DPO and ORPO stand out as valid and convenient alternatives, making model training more accessible without significantly compromising quality. PPO<\/strong>, while requiring substantial time and computational resources, remains a good option when such resources are available, promising high-quality results with appropriate investment.<\/p>\n

            Companies like Moxoff can use this information to optimise their large-scale language model development processes. By selecting the most suitable alignment method based on available resources and specific objectives, Moxoff<\/strong> can improve both the efficiency and quality of its models<\/strong>. For example, in scenarios with abundant computational resources, adopting PPO could ensure high-quality models. Conversely, in situations with time and resource constraints, ORPO may represent a more balanced choice, still allowing good levels of accuracy and performance to be achieved. Flexibility in the choice of alignment method enables Moxoff to better adapt to diverse client needs and operational contexts.
            All aligned versions of the models<\/strong> are available on the Moxoff Hugging Face page<\/a><\/span> for further analysis and experimentation.<\/p>\n

            References<\/h2>\n
              \n
            1. Phi3-mini-instruct, modello: https:\/\/huggingface.co\/microsoft\/Phi-3-mini-128k-instruct<\/span><\/li>\n
            2. Pagina HF di Moxoff: https:\/\/huggingface.co\/MoxoffSpA<\/span><\/li>\n
            3. Allineamento con PPO, modello: https:\/\/huggingface.co\/MoxoffSpA\/Moxoff-Phi3Mini-PPO<\/span><\/li>\n
            4. Allineamento con DPO, modello: https:\/\/huggingface.co\/MoxoffSpA\/Moxoff-Phi3Mini-DPO<\/span><\/li>\n
            5. Allineamento con KTO, modello: https:\/\/huggingface.co\/MoxoffSpA\/Moxoff-Phi3Mini-KTO<\/span><\/li>\n
            6. Allineamento con ORPO, modello: https:\/\/huggingface.co\/MoxoffSpA\/Moxoff-Phi3Mini-ORPO<\/span><\/li>\n
            7. Allineamento con PPO, articolo: <\/span>https:\/\/arxiv.org\/abs\/2307.04964<\/span><\/a><\/span><\/li>\n
            8. Allineamento con DPO, articolo: <\/span>https:\/\/arxiv.org\/abs\/2305.18290<\/span><\/a><\/span><\/li>\n
            9. Allineamento con KTO, articolo: https:\/\/arxiv.org\/abs\/2402.01306<\/span><\/li>\n
            10. Allineamento con ORPO, articolo: https:\/\/arxiv.org\/abs\/2403.07691<\/span><\/li>\n<\/ol>\n

              \u00a0<\/h3>","protected":false},"excerpt":{"rendered":"

              Un confronto tecnico delle varie tecniche di fine tuning di Large Language Model Introduzione Negli ultimi anni, i Large Language Models (LLM) hanno catalizzato l’attenzione di tutto il mondo, rivoluzionando l’intelligenza artificiale grazie alla loro straordinaria capacit\u00e0 e versatilit\u00e0 nell\u2019affrontare problemi reali. Nonostante queste premesse promettenti, i LLM non sono esenti da problematiche significative. Tra […]<\/p>\n","protected":false},"author":3,"featured_media":1850,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[36],"tags":[],"class_list":["post-5545","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t