Fine-tuning
References and useful ressources
- https://www.philschmid.de/fine-tune-llms-in-2024-with-trl
- https://cameronrwolfe.substack.com/p/easily-train-a-specialized-llm-peft
What is Fune-tuning
⚠️ TODO: complete it
Re-train LLM on use case or data company.
Fine-tuning vs Adapter-tuning
In standard fine-tuning, the new top-layer and the original weights are co-trained. In contrast, in adapter-tuning, the parameters of the original network are frozen and therefore may be shared by many tasks.
Optimization Techniques
PEFT (parameter efficient training)
⚠️ TODO: complete it
LLaMA-Adapter
⚠️ TODO: complete it
LoRA
⚠️ TODO: complete it
QLoRA
⚠️ TODO: complete it
DoRA
DoRA (Weight-Decomposed Low-Rank Adaptation), is a new, PEFT technique that claims to enhance the learning capacity and training stability of LoRA, while avoiding any additional overhead.
- https://arxiv.org/abs/2402.09353
- https://huggingface.co/docs/peft/v0.10.0/en/developer_guides/lora#weight-decomposed-low-rank-adaptation-dora
- https://www.linkedin.com/posts/philipp-schmid-a6a2bb196_dora-a-new-better-and-faster-lora-dora-activity-7179756964359360512-_Cez?utm_source=share&utm_medium=member_desktop
Insights:
🏅 DoRA consistently outperforms LoRA
🤗 Supported in Hugging Face PEFT
♻️ Trained Adapters can be merged back into the model
📈 +3.4% on Llama 7B and +1.0% on Llama 13B compared to LoRA on common reasoning
🔐 Improved training stability compared to LoRA
❌ In PEFT, DoRA only supports linear layers at the moment
Figure: Performance of fine-tuned LLaMA2-7B on MT-Bench using different numbers of Alpaca training samples.
Figure: Average accuracy of LoRA and DoRA for varying ranks for LLaMA-7B on the commonsense reasoning tasks.
GaLore
⚠️ TODO: complete it
https://www.linkedin.com/posts/philipp-schmid-a6a2bb196_galore-is-a-new-memory-efficient-fine-tuning-activity-7177599313294827521-kye2
LoReFT
LoReFT (Low-rank Linear Subspace Representation Fine Tuning) from Stanford University doesn't require any update of model weights at all, while being more efficient and often outperforming other PEFT methods like LoRA (which require only a small subset of weights to be updated).
This is achieved instead by intervening with the intermediate representations output from hidden layers of the model.
- https://arxiv.org/abs/2404.03592
- https://github.com/stanfordnlp/pyreft
- https://www.linkedin.com/posts/andrew-iain-jardine_llms-activity-7182723145651433472-i8Fx
Insights:
🔹 LoReFT is 10-50x more efficient that other PEFT methods
🔹 Improves performance on instruction following vs LoRa or full-tuning
🔹 Comparable or better performance for reasoning vs LoRa
🔹 Potentially could be used in conjunction with weight altering PEFT methods
Training process:
1️⃣ Weights of the base LLM are frozen
2️⃣ Specific layers of the model are selected for intervention
3️⃣ Projection matrix for each layer is trained to edit layer output representations
4️⃣ At inference the projection matrix intervenes to change the output
5️⃣ Altered hidden states passed through model layers changing generation
6️⃣ Evaluated on reasoning, instruction and NLU benchmarks vs PEFT methods
Handily the team has introduced a new library Pyreft to handle this process
Preference Alignment algorithms
Model Quantization
⚠️ TODO: complete it