AI

Two approaches to multi-agent orchestration: Multi-agent workflow (defined orchestration): These systems follow pre-defined collaboration patterns where each agent has clear specified logic and roles. This is deterministic. Autonomous multi-agent orchestration (ai-driven orchestration): The system use AI models to drive orchestration. This is dynamic system and agent collaboration is based on task requirement and itermediate results. Multi-agent systems are most beneficial for tasks exhibiting four key characteristics: requiring planning, needing diverse expertise, involving extensive context and requiring adaptive solutions. ...

How can we make our service faster? We can improve Model Hardware Service Model: Efficient models without computation bottlenecks in the attention mechanism. Hardware: Optimized models for specific hardware Service: Usage, traffic patterns to allocate resources, redunancy, cost, latency Interence workloads Compute-bound: How much computation is needed to complete a task. Memory bandwidth-bound: This is the time it takes to transfer data between memory and processors. Prefill is compute bound and decode is memory bound. ...

When building AI systems, data quality means different things at different stages. During pre-training, quality is measured by the number of tokens processed. During post-training, it shifts to the number of well-crafted examples. Three dimensions matter most when thinking about data: Quality — is the data accurate and clean? Coverage — does it span the problem space? Quantity — is there enough of it? What makes a dataset high quality? ...

Fine Tunning methods Self-supervised training: Foundational models are itself trained on self-supervision. Feature based transfer learning: Embeddings from foundational models are used for other taks or train a separate models. Infilling finetuning: Making a pre-trained autoregressive model to predict next token or middle tokens based on before or after tokens. Supervised finetuning (SFT): Pretrained model is further trained on the labeled dataset of input-output pairs. Reinforcement Learning: Finetune model to generate responses that maximize human preference. This requires the data format (instruction, winning response, losing response). ...

Categories of tools Knowledge augmentation Access to latest information either through web search tools, access to social media accounts etc. Capability extension AI agents having access to small tools instead of AI solving the problems itself. Tools to act based on enviroment AI agents plans the steps itself for task completion. Drawback of autoregressive model is that they can’t backtrack and they need to restart from first step if a path does not lead to goal. This path could be defined by us or AI agent decides. ...

Retrieval Algorithm Term based retrieval (Term based, Lexical retrieval) This vector representation based on TF-IDF. Most values are zero. Embedding-based retrieval (Dense vectores): This is semantic embedding learned. KNN LSH (Local Sensitive Hashing) HNSW (Hierarchical Navigable Small World) Product Quantization IVG (Inverted File Index) Annoy (Approximate Nearest Neighbors Oh Yeah) Metrics to evaluate RAG retrieval Context precision: Out of all the documents retrieved, what percentage is relevant to the query? Context recall: Out of all documents that are relevant to the query, what percentage is retrieved? Indexing: More the fine grained the indexing is more accurate the retrieval process will be. But there is trade off of memory consumption and query time. Indexing and query time of search algorithm is evaluated based on following metrics: ...

Domain-specific capability Functionality of the AI system determines the domain specific capability. Like for coding system, functional correctness, readbility of code etc are important. Usually for open ended systems, MCQ or classification kind of evluation methods are used. Generation capability Fluency: Coherence: Faithfulness: Relevance: Factual consistency: The model’s output is verified against the know facts (local factual consistency) or against open knowledge (Global factual consistency). Self-verification: If a model generates multiple outputs that disagree with one another, the original output islikely hallucinated. ...

Functional Correctness This metric measures whether the performance of any application performs what it is intended to do. This is easy to measure for deterministic systems. Similarity Measurement aginst reference data This checks how similar the LLM generated data is to the gold standard. The gold standard could be generated by AI or human. Exact match This can be one to one mapping of the answer or one to many mapping. ...

Rotary Position Embeddings (RoPE) State Space Model Mixutre of experts Supervised Finetuning (SFT) Preference finetuning Finetuning model to output responses that align with human preference. Reinforcement learning from human feedback (RLHF) Direct Preference Optimization (DPO) Reinforcement Learning from AI feedback (RLAIF) Proximal policy optimization (PPO)

Stopping condition For LLM applications, we can ask model to stop generating when it encounters the end-of-sequence token. Test time compute Instead of generating one answer to query, generate multiple using beam search and choose the best candidate. Best candidate can be chosen by letting humans choose or pick the output with highest probability. The probability of output is the product of the probabilities of all tokens in the output. ...

Pretraining evaluation metrics Cross entropy How different the predicted distribution is from the original distribution weighted by true probability. $$H(p, q) = -\sum_{i} p(i) \log q(i)$$ Where: $p$ = the true distribution (what actually happened) $q$ = the model’s predicted distribution Sum is over all possible classes/tokens $i$ Perplexity This measures how certain the model is in predicting next token. Lower value means it is certain while higher value means there are more equally likely options. ...