Stopping condition
For LLM applications, we can ask model to stop generating when it encounters the end-of-sequence token.
Test time compute
Instead of generating one answer to query, generate multiple using beam search and choose the best candidate. Best candidate can be chosen by letting humans choose or pick the output with highest probability. The probability of output is the product of the probabilities of all tokens in the output.
One can use the reward model to score each output.
Structured output
To generate structured output like JSON, one can keep on generating outputs until one fits the expected format.
We can instruct the model in prompting as well to generate the structured output but it depends on instruction following capability of the model. One can use AI itself to validate the structured output of the LLM.
Post-processing: This means if the error in the structured output is same over different trials, then we can fix through the script.
Constrained sampling: This technique guides the generation of text toward certain constraints followed by structured output tools. This sampling filters logit vectors to keep only the tokens that meet the constraints and then sample from these valid tokens.
Finetuning: Finetuning a model with examples of desirable format is the most effective approach. For example classification.
Why LLMs hallucinate? Language model can’t differentiate between the data it’s given and the data it generates (Ortega et al.).
Snowballing hallucination, after making an incorrect assumption, a model can continue to hallucinate to justify the initial wrong assumption (Zhang et al.).
The difference in knowledge of LLM and the labeler (Leo Gao).
Ways to mitigate hallucination:
- Reinforcement learning, make the model differentiate between user-provided prompts and tokens generated by the model.
- Supervised learning in which factual and counterfactual signals are included in the training data.