Model quantization

noun

Model quantization is a process that reduces the precision of the numerical values in an AI model, typically from 32-bit floating-point numbers to 8-bit integers. This shrinks the model’s size and speeds up its processing, often with minimal loss in accuracy.

Quantization works by approximating the model’s weights and activations to lower-precision formats. For example, a model trained with 32-bit floats can be quantized to 8-bit integers, cutting its memory usage by 75% and improving inference speed by 2-5x on compatible hardware.

For developers, quantization lowers costs by reducing cloud compute expenses and enabling AI models to run on edge devices like smartphones. However, it may require retraining or fine-tuning to maintain performance. A real-world example is Meta’s Llama 2, which offers quantized versions that fit on consumer GPUs while keeping most of their original capabilities.

A developer uses an 8-bit quantized version of a language model to run it on a Raspberry Pi, reducing memory usage from 10GB to 2.5GB.

Découvrez nos autres services

ElpisIA, c’est tout un univers — explorez nos autres services.