PR-272: Accelerating Large-Scale Inference with Anisotropic Vector Quantization [Guo et al., ICML 2020] Paper link: https://arxiv.org/abs/1908.10396 Video presentation link: https://youtu.be/cU46yR-A0cs reviewed by Sunghoon Joo

1. LoGANv2: Conditional Style-Based Logo Generation with Generative Adversarial
Networks
PR-272
Our implementation is open-source and available at
https://github.com/google-research/google-research/tree/master/scann

2. 1. Research Background

3. 1. Research Background
Background & Motivation
• Maximum inner product search (MIPS)
3/21
MIPS Query
• , [Cremonesi et al., 2010, ACM],
• label Classification task (millions, billions…) [Dean et al., 2013]
• Training tasks (speed up softmax computation, scalable gradient computation [Yen et al., 2018])

4. 1. Research Background
Background & Motivation
• Maximum inner product search (MIPS)
4/21
q , n dataset X inner product ,
q와 xi의 inner product

5. 1. Research Background
Previous Researches
• Reducing the number of evaluations
5/21
tree search methods, locality sensitive hashing, graph search
• Quantization based techniques
Random projection (Charikar, 2002; Vempala, 2005; Li & Li, 2019), Binary quantization,
Product quantization, Additive quantization, Ternary quantization, Learning quantization
(Ge et al., 2014; Babenko & Lempitsky, 2014; Johnson et al., 2017)
<tree search methods>
https://m.blog.naver.com/laonple/221207919855

6. 1. Research Background
Previous Researches
6/21
• Quantization based techniques
• Reducing the number of evaluations
Random projection, Binary quantization, Product quantization,
Additive quantization, Ternary quantization, Learning quantization
http://kaiminghe.com/cvpr13/index.html
•
•
•

7. 1. Research Background
Previous Researches
7/21
• Quantization based techniques
http://kaiminghe.com/cvpr13/index.html
Optimized Product QuantizationIterative Quantization [CVPR 2011]Product Quantization
[CVPR 2013]
•

8. 1. Research Background
Objective & Approach : a new quantization loss function
• We propose the score-aware quantization loss function. The proposed loss can
work under any weighting function of the inner product and regardless of whether
the datapoints vary in norm.
1) ? (MIPS task)
• SP-TAG (Chen et al., 2018), FAISS (Johnson et al., 2017), hnswlib
(Malkov & Yashunin, 2016)
2) ?
• Product quantization codebook score-aware quantization loss .
• Binary quantization Stochastic Generative Hashing (Dai et al., 2017) loss .
8/21

9. 2. Methods

10. 2. Methods
The goal of quantization
10/21
Traditional Quantization Optimal Quantization
X1, x2 c1, c2 quantize ,
(x1-> c2, x2->c1) quantize inner product
.
Quantization inner product .
https://ai.googleblog.com/

11. 2. Methods
Anisotropic loss
11/21

12. 2. Methods
Quantization Technique
12/21
• X quantization point C .
• k quantized point Quary vector q dot product .
• Dictionary C . Lloyd’s algorithm
(Initialization) Codeword c
(Partition Assignment Step) k .
(codebook update)

13. 2. Methods
Quantization Technique
13/21
Partition Assignment Step
codebook
datapoint
Codebook update and partition assignment

14. 3. Experimental Results

15. 3. Experimental Results
Experiment 1: general l2-reconstruction loss score-aware loss
• T weigth function dot product level, parallel error – orthogonal error
• Threshold T = 0.2 . T = 0.2
• reconstruction loss quantization
15/21
η = 4.125
Traditional :minimizing reconstruction loss
Proposed : minimizing score-aware loss.

16. 3. Experimental Results
Experiment 1: general l2-reconstruction loss score-aware loss
• softmax approximation , <q, x>
• Score-aware loss function -quantized data pair loss
, top-ranking pair
• Amazon 670k
16/21

17. 3. Experimental Results
Experiment 2: bitrate method
17/21
Recall 1@N : 쿼리를 여러 번 수행 했을때, 검색된 상위 N 개 결과에 실제 상위 1개 datapoint가 포함 된 쿼리의 비율
• bitrate Recall 1@N N .
QUIPS : Guo et al., Quantization based fast inner product search. 2016
LSQ: Martinez et al., Lsq++: Lower running time and higher recall in multi-codebook quantization. 2018

18. 3. Experimental Results
Experiment 3: benchmark method – /
18/21
• MIPS end-to-end benchmark
• (http://ann-benchmarks.com/glove-100-angular_10_angular.html)
Benchmarks are all
conducted on an Intel Xeon
W-2135 with a single CPU
thread.

19. 3. Experimental Results
Experiment 4: score-aware loss
19/21
• Score-aware quantization loss quantization technique

20. 4. Conclusion

21. 4. Conclusions 21/21
Thank you.
• quantization method (e.g. product quantization, binary
quantization)
• inner product search quantization loss
function .
• loss function (1) datapoint inner
product , (2) codebook c original x
parallel component .
• state-of-the-art baseline