Inference latency

noun

Inference latency is the delay between when an AI system processes an input (like a text prompt or image request) and when it produces an output. It’s a critical factor in user experience, especially for real-time applications like chatbots or image generators.

This delay depends on factors like model complexity, hardware (GPUs/TPUs), and network speed. For example, a large language model might take 50–500 milliseconds to generate a response, while simpler models can be near-instantaneous.

For developers, high latency can frustrate users, while low latency improves engagement. Optimizing latency often involves trade-offs, like using smaller models or cloud-based acceleration, which may increase costs or reduce accuracy.

For instance, a real-time AI assistant in a customer service app must keep latency below 200ms to feel responsive, or users may abandon the interaction.

A user types a question into an AI chatbot and waits 300ms for the answer to appear.

Découvrez nos autres services

ElpisIA, c’est tout un univers — explorez nos autres services.