Rotary Position Embeddings (RoPE) State Space Model Mixutre of experts Supervised Finetuning (SFT) Preference finetuning Finetuning model to output responses that align with human preference. Reinforcement learning from human feedback (RLHF) Direct Preference Optimization (DPO) Reinforcement Learning from AI feedback (RLAIF) Proximal policy optimization (PPO)
Stopping condition For LLM applications, we can ask model to stop generating when it encounters the end-of-sequence token. Test time compute Instead of generating one answer to query, generate multiple using beam search and choose the best candidate. Best candidate can be chosen by letting humans choose or pick the output with highest probability. The probability of output is the product of the probabilities of all tokens in the output. ...
Functional Correctness
Pretraining evaluation metrics Cross entropy How different the predicted distribution is from the original distribution weighted by true probability. $$H(p, q) = -\sum_{i} p(i) \log q(i)$$ Where: $p$ = the true distribution (what actually happened) $q$ = the model’s predicted distribution Sum is over all possible classes/tokens $i$ Perplexity This measures how certain the model is in predicting next token. Lower value means it is certain while higher value means there are more equally likely options. ...
Decoding stragties are the methods that is used to choose the next token prediction in LLMs. Deterministic strategies Greedy decoding In this method, at every step token with the highest probability is chosen. The problem with this method is that this method choose locally optimal token but not globally optimal. This is used when the latency matters most and tasks has fairly unambiguous correct output like code completion for syntax, simple classification style generation. ...
Undersampling One way of handling an imbalanced dataset is to reduce the number of observations from all classes except the minority class. The most well-known algorithm in this group is random undersampling, where samples from the targeted classes are removed at random. These methods can be grouped based on their undersampling strategy into: Prototype generation methods Prototype selection methods Prototype Generation Given an original dataset $S$, prototype generation algorithms will generate a new set $S’$ where $|S’| < |S|$ and $S’ \notin S$. These techniques reduce the number of samples in the targeted classes, but the remaining samples are generated — not selected — from the original set. ...