Slides from Eva Mohedano and Andrea Calafell for the the UPC Computer Vision Reading Group about the paper: Zhong, Yujie, Relja Arandjelović, and Andrew Zisserman. "Faces In Places: compound query retrieval." In BMVC 2016. The goal of this work is to retrieve images containing both a target person and a target scene type from a large dataset of images. At run time this compound query is handled using a face classifier trained for the person, and an image classifier trained for the scene type. We make three contributions: first, we propose a hybrid convolutional neural network architecture that produces place-descriptors that are aware of faces and their corresponding descriptors. The network is trained to correctly classify a combination of face and scene classifier scores. Second, we propose an image synthesis system to render high quality fully-labelled face-and-place images, and train the network only from these synthetic images. Last, but not least, we collect and annotate a dataset of real images containing celebrities in different places, and use this dataset to evaluate the retrieval system. We demonstrate significantly improved retrieval performance for compound queries using the new face-aware place-descriptors.

1. Faces in Places: compound
query retrieval
Y. Zhong, R. Arandjelovic and A. Zisserman: Paper Link
BMVC 2016
1
Slides by Eva Mohedano and Andrea Calafell [GDoc]
UPC Computer Vision Reading Group (14/10/2016)

2. Outline
2
1. Introduction
2. Hybrid Network
3. The “Celebrity in Places” Dataset
4. Synthetic Training Images
5. Experiments and Results
6. Summary

3. Introduction
Large Image Dataset
System
3
Compound query

4. Introduction
Three contributions:
1. Hybrid CNN to produce place descriptors that are aware of faces and their
descriptors.
2. Collect and annotate a dataset of real images containing celebrities in different
places.
3. Image synthesis system to render high quality fully-labelled face-and-place
images to train the network.
4

5. Outline
5
1. Introduction
2. Hybrid Network
3. The “Celebrity in Places” Dataset
4. Synthetic Training Images
5. Experiments and Results
6. Summary

6. Basic Approach
6

7. Hybrid Network
7

8. Hybrid Network
8
AlexNet pre-trained on
Places205
VGG-16 trained on VGG
Face Dataset
FC7
FC7

9. Outline
9
1. Introduction
2. Hybrid Network
3. The “Celebrity in Places” Dataset
4. Synthetic Training Images
5. Experiments and Results
6. Summary

10. The “Celebrity in Places” Dataset
10
Example images from the CIP dataset
Includes:
● 4611 celebrities
● 16 places
Query texts in
Google Image
Search
2,5M
images Duplicate
removal
170K
images Mechanical
Turk
annotation
38K
images

11. The “Celebrity in Places” Dataset
11
Includes:
● 4611 celebrities
● 16 places
Query text in
Google Image
Search
2,5M
images Duplicate
removal
170k
images Mechanical
Turk
annotation
38k
images
Problems with this approach
● Difficult to obtain high quality images with
Image Search engines
● Obtained images highly unbalanced across
classes

12. Outline
12
1. Introduction
2. Hybrid Network
3. The “Celebrity in Places” Dataset
4. Synthetic Training Images
5. Experiments and Results
6. Summary

13. Synthetic Training Images
13

14. Synthetic Training Images
14
178k training images
8.7k validation images
Includes:
● 500 faces
● 16 places

15. Outline
15
1. Introduction
2. Hybrid Network
3. The “Celebrity in Places” Dataset
4. Synthetic Training Images
5. Experiments and Results
6. Summary

16. Experiments and Results
16
Comparison with 3 baselines of late fusion
● FC7 VGG faces + FC7 Places205 + L2norm
● FC7 VGG faces + FC7 Places205 finetuned on 16 places+ L2norm
● FC7 VGG faces + FC7 Places205 finetuned on 16 places+ Platt
Test sets statistics

17. Experiments and Results
17

18. Outline
18
1. Introduction
2. Hybrid Network
3. The “Celebrity in Places” Dataset
4. Synthetic Training Images
5. Experiments and Results
6. Summary

19. Summary
19
● They have presented a hybrid network for compound queries, where place
descriptors are aware of faces and face descriptors. This network outperforms
the baselines.
● They have designed an automatic pipeline to synthesize training images.
● They have collected a new dataset of real images to evaluate their methods.

20. Questions?
20