noun
Inference latency is the delay between when an AI system processes an input (like a text prompt or image request) and when it produces an output. It’s a critical factor in user experience, especially for real-time applications like chatbots or image generators.
This delay depends on factors like model complexity, hardware (GPUs/TPUs), and network speed. For example, a large language model might take 50–500 milliseconds to generate a response, while simpler models can be near-instantaneous.
For developers, high latency can frustrate users, while low latency improves engagement. Optimizing latency often involves trade-offs, like using smaller models or cloud-based acceleration, which may increase costs or reduce accuracy.
For instance, a real-time AI assistant in a customer service app must keep latency below 200ms to feel responsive, or users may abandon the interaction.
A user types a question into an AI chatbot and waits 300ms for the answer to appear.