Rotary Position Embeddings (RoPE) State Space Model Mixutre of experts Supervised Finetuning (SFT) Preference finetuning Finetuning model to output responses that align with human preference. Reinforcement learning from human feedback (RLHF) Direct Preference Optimization (DPO) Reinforcement Learning from AI feedback (RLAIF) Proximal policy optimization (PPO)
Stopping condition For LLM applications, we can ask model to stop generating when it encounters the end-of-sequence token. Test time compute Instead of generating one answer to query, generate multiple using beam search and choose the best candidate. Best candidate can be chosen by letting humans choose or pick the output with highest probability. The probability of output is the product of the probabilities of all tokens in the output. ...
Functional Correctness
Pretraining evaluation metrics Cross entropy How different the predicted distribution is from the original distribution weighted by true probability. $$H(p, q) = -\sum_{i} p(i) \log q(i)$$ Where: $p$ = the true distribution (what actually happened) $q$ = the model’s predicted distribution Sum is over all possible classes/tokens $i$ Perplexity This measures how certain the model is in predicting next token. Lower value means it is certain while higher value means there are more equally likely options. ...