Fine Tunning methods
Self-supervised training: Foundational models are itself trained on self-supervision.
Feature based transfer learning: Embeddings from foundational models are used for other taks or train a separate models.
Infilling finetuning: Making a pre-trained autoregressive model to predict next token or middle tokens based on before or after tokens.
Supervised finetuning (SFT): Pretrained model is further trained on the labeled dataset of input-output pairs.
Reinforcement Learning: Finetune model to generate responses that maximize human preference. This requires the data format (instruction, winning response, losing response).
Long-context finetuning: This modifies the model’s architecture such adjusting the positional embeddings.
Distillation: Finetune a small model to imitate the behavior of a larger model using data generated by the larger model.
When to use:
- SFT: When model needs to adopt specific behavior, format or style consistently.
- RAG: When model lack knowledge
- Prompt: Cheaper first step
Choosing RAG or prompt is whether the model’s failures are information-based or behavior-based.
Memory calculation for training
Model inference: N x M where N=no. of parameters, M is the memory needed for each parameter