Inference Optimzation

How can we make our service faster? We can improve Model Hardware Service Model: Efficient models without computation bottlenecks in the attention mechanism. Hardware: Optimized models for specific hardware Service: Usage, traffic patterns to allocate resources, redunancy, cost, latency Interence workloads Compute-bound: How much computation is needed to complete a task. Memory bandwidth-bound: This is the time it takes to transfer data between memory and processors. Prefill is compute bound and decode is memory bound. ...