Задержка — время между запросом, входным сигналом или событием и ответом системы. В AI-приложениях задержка влияет на удобство использования, работу в реальном времени, проектирование сервиса и выбор инфраструктуры для инференса или retrieval.
The time it takes for a model to process input and generate a response. A high latency response takes takes longer to generate than a low latency response. Factors that influence latency of large language models include: - Input and output token lengths - Model complexity - The infrastructure the model runs on Optimizing for latency is crucial for creating responsive and user-friendly applications.
The time that it takes for a model to process an input prompt and generate a response. Key metrics include Time to First Token (TTFT) — the time to produce the first token after receiving the prompt — and Time to Last Token (TTLT) — the total time to process the prompt and generate the complete response.