Latency is the delay between a request, input, or event and the system’s response. In AI applications, latency affects usability, real-time performance, service design, and infrastructure choices for inference or retrieval.
The time it takes for a model to process input and generate a response. A high latency response takes takes longer to generate than a low latency response. Factors that influence latency of large language models include: - Input and output token lengths - Model complexity - The infrastructure the model runs on Optimizing for latency is crucial for creating responsive and user-friendly applications.
The time that it takes for a model to process an input prompt and generate a response. Key metrics include Time to First Token (TTFT) — the time to produce the first token after receiving the prompt — and Time to Last Token (TTLT) — the total time to process the prompt and generate the complete response.