Can Quantization Reduce Separability?

Originally published: November 29, 2025

Yes.

I was recently experimenting with the linear separability of quantized embedding vectors, and produced a graph which indicated the embeddings were substantially more linearly separable than the quantized embeddings. A commenter indicated that this didn't make sense, as they thought the quantized vectors should be as separable as the non-quantized vectors. In this brief note, we present two.

The first, and most trivial case, is when your number of classes (since each embedding represents a class data point) than embedding vectors. If you are quantizing your embeddings to a set of \( N \) classes, and you have \( N + 1 \) embeddings vectors, then naturally two classes have to have overlap in this quantized space. However, such a reduced class space isn't always common.

For the second example, consider the XOR problem. This is famously known to be not linearly separable. So in the case where we have two classes and two quantization vectors, then if class 1 maps to \( (0, 1) \) and \( (1, 0) \) and class 2 maps to \( (1, 1) \) and \( (0, 0) \) in quantized space, but the non-quantized vectors are separable then we have shown our goal. Given these four quantization vectors, consider the following embedding points, class 1: \( (0.75, -2) \), \( (-2, 0.8) \), class 2: \( (-0.75, 0.25) \), \( (0.8, 0.8) \), these embed to \( (1, 0) \), \( (0, 1) \), \( (0, 0) \), \( (1, 1) \). Clearly these embeddings are not linearly separable, but with \( f(x, y) = -0.225x - 0.37y - 0.1 \), the non-quantized vectors are.