Columbia Talk: Landmark Search and Community-Contributed Multimedia

The Landmark Revolut ion:
I mproving I mage Search and
Explorat ion
f or Locat ion- Driven Queries

M or N aam an

Y ahoo! R esearch B erkeley
Y ahoo! A dvanced D evelopm ent D i si
vi on

How Flickr Helps us Make Sense of t he
World:
Cont ext and Cont ent in Communit y-
Cont ribut ed
Media Collect ions

M or N aam an

Y ahoo! R esearch B erkeley
Y ahoo! A dvanced D evelopm ent D i si
vi on

Dat a Descript ion

Lyndon Kennedy, Mor Naaman
3 | Y!ADD, 2007

Tag Pat t erns

4 | Y!ADD, 2007

Tag Pat t erns

5 | Y!ADD, 2007

Tag Pat t erns

6 | Y!ADD, 2007

Tag Pat t erns

7 | Y!ADD, 2007

Tag Pat t erns

8 | Y!ADD, 2007

Tag Pat t erns

9 | Y!ADD, 2007

Communit y- cont ribut ed: Bet t er Dat a?

• M edi a
• D escri ve text (ti e , capti , tag)
pti tl on
• Di scussions and com m ents
• V i s and vi patterns
ew ew
• Item use and feedback
• R euse and rem ix
• M i - and expl ci recom m endati
cro it ons
• “ontext M etadata”
C
•…

10 | Y!ADD, 2007

Pat t erns That Make Sense

• S em anti space
c
• A cti ty and vi i data
vi ew ng
• U ser/ personaldata
• S ocialnetw ork
• Locat ion/ t ime

11 | Y!ADD, 2007

Tag Pat t erns: Beyond Geo

12 | Y!ADD, 2007

Flickr Tigers

13 | Y!ADD, 2007

Older Tigers?

• N o tigers, beaches
and sunsets.
ease .
Pl

14 | Y!ADD, 2007

Research Challenges

• C ontent i sti lhard …
s l
• U nstructured data (no sem anti )
cs
• T ags, not ground truth labels
– F al negati and posi ves
se ve ti
– If that even m eans anything
• N oise
• S cale
– Com putation
– Long tai m pl es no supervi
li i sed learning
• B i / feedback / S pam
as

15 | Y!ADD, 2007

That Noise….

• N oi data
sy
• Photographer biases
• W rong data

5 k ms
6 km s

16 | Y!ADD, 2007

Foremost Challenge:

• W hat’s the user probl ?
em
– N avigati / expl
on oration
– R ecom m endation
– N ew appl cati
i on
– O ther?
• G rounded i realneeds
n
• W hat i pact on the
m
com m uni ?
ty

“Social Media Cycle”

17 | Y!ADD, 2007

Talk Out line

• Visual ze
i
– Creati a W orl E xpl
ng d orer
• G enerate know ledge
– E xtracti T ag S em anti
ng cs
• S earch
– Landm ark search

18 | Y!ADD, 2007

Surely, we can do bet t er t han t his

Flickr
“geot agged” in
San Francisco

19 | Y!ADD, 2007

Simple Model

(phot o_ id, user_ id, t ime,
lat it ude, longit ude)
(phot o_ id, t ag)

20 | Y!ADD, 2007

I nt uit ion

More “act ivit y” in a cert ain locat ion
indicat es import ance of t hat locat ion

Tag t hat are unique t o a cert ain locat ion
can represent t he locat ion bet t er

21 | Y!ADD, 2007

Translat ion int o simple algorit hm

• Clusteri of photos
ng
• S cori of tags
ng
– T F / ID F / U F

22 | Y!ADD, 2007

Tag Maps - SF

23 | Y!ADD, 2007

At t ract ion Maps of Paris

S tanley
M i gram ,
l
1976.
”Psychological
Maps of Paris”

24 | Y!ADD, 2007

At t ract ion Maps of Paris

Y !R B , 2006.
”Tag Maps:
World Explorer”

25 | Y!ADD, 2007

Make a World Explorer

ht t p: / / t agmaps. research. yahoo. com
A l see [A hern et al J CD L 2007]
.,
so
26 | Y!ADD, 2007

Summary of San Francisco

Golden Gat e Bridge TransAmerica

AT&T
Baseball Park

Golden Gat e
Twin Peaks
Golden Gat e

Ocean Beach Bay Bridge Chinat own

27 | Y!ADD, 2007

Tag Maps - Paris - Les Blogs?

28 | Y!ADD, 2007

Talk Out line

• Visual ze
i
ng d orer
ng cs
• S earch

29 | Y!ADD, 2007

Tag- based Modeling

• D eri m eani
ve ngfuldata about i vi
ndi dualtags
• B ased on the tag ’s m etadata patterns
• E .g., Yahoo! Mission College, SIGIR 2007.

30 | Y!ADD, 2007

Ext ended Model

(phot o_ id, user_ id, t ime,
lat it ude, longit ude)
(phot o_ id, t ag)

(t ag, locat ion)
(t ag, t ime)

31 | Y!ADD, 2007

Tag Pat t erns

32 | Y!ADD, 2007

Tag Semant ics

• Im proved i age search through query sem anti
m cs
• A utom ati pl - and event-gazetteers
c ace
• A ssoci on of m i ng ti e / pl
ati ssi m ace data based on tags
•…

33 | Y!ADD, 2007

San Francisco Experiment s

~43 k photos
~800 tags

San Francisco Dat aset :
42, 000 Phot os
800+ popular t ags
34 | Y!ADD, 2007

Experiment s

Result s: BYOBW!
We can derive t ag semant ics using locat ion and t ime
met adat a.
[Rat t enbury et al, SI GI R 2007]
byobw
35 | Y!ADD, 2007

Talk Out line

• Visual ze
i
ng d orer
ng cs
• S earch

36 | Y!ADD, 2007

Rolling in Cont ent

• S o far, w e leveraged m etadata patterns to find
– W hat are the geo-driven features
– W here peopl take photos of these features
e
• C an w e uti i
l zed content anal s?
ysi

• Hmmm….

37 | Y!ADD, 2007

Handling scale

• R educe com putati requi
on rem ents
– F i ter usi m etadata
l ng
• U nsupervised m ethods
– E ffecti for l
ve ong tai i
lw thout trai ng
ni

38 | Y!ADD, 2007

Building Visual Summaries

Raw Data Locations and Names
Visual Summary?

39 | Y!ADD, 2007

The Problem, in Short
Find less of and more of t his…
t his…

… hout explicit ly
wit
knowing t he dif f erence.
40 | Y!ADD, 2007

Locat ion can help

E nough visual
si i ari for
m l ty
earni ?
l ng
41 | Y!ADD, 2007

Finding Represent at ive Phot os

42 | Y!ADD, 2007

Visual Feat ures

• Color: m om ents over a 5 x 5 grid
• Text ure: G abor over globali age
m
• I nt erest point s: S IF T

43 | Y!ADD, 2007

Learning f rom noisy labels

44 | Y!ADD, 2007

Clust ering

• K -m eans over l -l
ow evelfeatures
(texture and col )
or
• V ary val of K w i totalnum ber of photographs
ue th
(avg. cluster si ~ 20)
ze

45 | Y!ADD, 2007

Ranking clust ers

• N um ber of users
– M ore users -> m ore shared interest
• T em poralspread
– Persistent over ti e -> m ore l kel to be locati , not event
m iy on
– Alternatel use m ethod descri
y bed earl er
i
• Visualcoherence
– M easure of diversi of vi
ty sualcluster
• Visualconnecti ty
vi
– M ore on thi l
s ater…

46 | Y!ADD, 2007

Finding Represent at ive Phot os

47 | Y!ADD, 2007

Ranking images: low- level similarit y

E ucl dean di
i stance from
cluster centroi i col
dn or
and texture space .

48 | Y!ADD, 2007

Ranking images: discriminat ive model

S am pl pseudo-
e
negati ves from outside
uster.
of cl

Learn S V M m odelover
col / texture space .
or

R ank by distance from
S V M m argi .
n

49 | Y!ADD, 2007

Point - wise Linking

50 | Y!ADD, 2007

Ranking images: point - wise links

F orm l nks betw een
i
i ages vi m atchi
m a ng
S IF T poi .
nts

R ank by degree of
connecti ty.
vi

51 | Y!ADD, 2007

Landmark Graph St ruct ure

Less
connected

More
connected

52 | Y!ADD, 2007

Coit Tower: Two Main Views

Shots from

Coit Tower

Far or
occluded
shots

Shots of
Coit Tower

53 | Y!ADD, 2007

Ranking images: f usion

• S el -si i ari : E ucl dean di
f m l ty i stance from centroi i
dn
l -l
ow evelfeature space .
• Di m nati : di
scri i ve stance from S V M deci on
si
boundary.
• Poi -w i : degree of the photo
nt se
• Fusion: sum of scores, norm al zed vi si oi
i a gm d
function

54 | Y!ADD, 2007

Result s: Palace of Fine Art s

X X
X
XX X
X
Tags-only Tags+Location Tags+Location+Visual

55 | Y!ADD, 2007

Evaluat ion

• D ataset: geo-tagged B ay A rea photos from F l ckr
i
• S elect 10 landm arks to evaluate
• A ppl al thm (and basel ne ) to di
y gori i scover
representati i ages
ve m

56 | Y!ADD, 2007

Perf ormance: Precision

+45%
w/visual
+30%
w/location

57 | Y!ADD, 2007

More Result s: Golden Gat e Bridge

X
X
X
X XX
XX X
T ags-onl T ags+Locati T ags+Locati +V i
y on on sual

58 | Y!ADD, 2007

Evaluat ion I ssues

• Preci <> R epresentati
se ve

59 | Y!ADD, 2007

Evaluat ion I ssues

• Preci <> D i
se verse

60 | Y!ADD, 2007

Perf ormance: Represent at ive

61 | Y!ADD, 2007

Image Search:
Proposed
interface

62 | Y!ADD, 2007

Conclusions

• Locati i strong predi
on s ctor of content
• Landm arks and geo-rel ated queri can be i
es denti ed
fi
• C om puter vi on can w ork . S om eti es.
si m

63 | Y!ADD, 2007

API s f or all!

• E verythi w e can do, you can do (better). A PIs
ng
i ude :
ncl
– Cel ow er ID database
lT
– S uggested T ags based on context
– T agM aps data
– T agM aps W idget

http://developer.yahoo.com/yrb/

64 | Y!ADD, 2007

Thanks

With: L yndo K ennedy, S haneA hern, R ahul N air, T yeR attenbury, J eannieYang, N athan Good, S imon K ing.
n

In the papers: M IR 06, J CD L 07, S IG IR 07, M M 07
A l ask m e about: Z oneT ag , Z urfer, F i E agl
so re e

R ead more, follow: http://www.whyrb.com
P ast talks slides: http://slideshare.net/mor

M or N aaman

65 | Y!ADD, 2007

Columbia Talk: Landmark Search and Community-Contributed Multimedia

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (6)

Plus de mor

Plus de mor (20)

Dernier

Dernier (20)

Columbia Talk: Landmark Search and Community-Contributed Multimedia