Faculty Guest Lecture at University of Michigan

1. Xiao Hu, Ph.D.
Assistant Professor
Library and Information Science
University of Denver
1/11/2012 1

2. An Exercise
What term(s) do you describe the mood of the song?
2
1/11/2012

3. Agenda
 Background
 Music Mood: the Question
 Taxonomies as Organization System
 Developing Music Mood Taxonomies
 Taxonomy from Editorial Labels
 User Evaluation
 Automatic Classification
 Prototype System
 Taxonomy from Social Tags
 Comparisons to Psychological Models
 Automatic Classification
 User Evaluation
1/11/2012 3

4. Music Mood
“main reason behind most people’s engagement with music”
--Juslin and Sloboda
(Lee & Downie,
2004)
(Lamere, 2008)
Lee, J. H. & Downie, J. S. (2004). Survey of music information needs, uses, and seeking behaviours: Preliminary findings. In
ISMIR. Lamere, P. (2008). Social tagging and music information retrieval. Journal of New Music Research 37, 2, 101–114. 4

5. Moods in Previous Research
 Directly borrowed from psychology
Thayer’s stress-energy model gives 4 clusters Farnsworth’s 10 adjective groups
Grounded in music perception
research, but lack social
context of music listening
(Juslin & Laukka, 2004)
Tellegen-Watson-Clark model
Juslin, P. N. and Laukka, P. (2004). Expression, perception, and induction of musical emotions:
a review and a questionnaire study of everyday listening. Journal of New Music Research. 5

6. Taxonomy as An Organization System
From Linnaean taxonomy in Biology
Domain oriented controlled vocabulary
 Contain labels (metadata)
Commonly used on websites
 Pick list; browsable directory, etc.
 Develop taxonomies to organize music
information
 Methods can be applied to other information
types
Stewart, D. (2008) Building Enterprise Taxonomies, Mokita Press
6

7. Agenda
 Background
 Music Mood: the Question
 Taxonomies as Organization System
 Developing Music Mood Taxonomies
 Taxonomy from Editorial Labels
 User Evaluation
 Automatic Classification
 Prototype System
 Taxonomy from Social Tags
 Comparisons to Psychological Models
 Automatic Classification
 User Evaluation
1/11/2012 7

8. Taxonomy from Editorial Labels
Editorial labels:
 Given by professional
editors of online
repositories
 Rooted in realistic social
contexts
 Have a certain level of
control
 allmusic.com: “the most comprehensive music reference
source on the planet”
 179 mood labels created and assigned to music works
1/11/2012 8

9. Mood Label Clustering
 179 labels are too many
 Need a more concise, higher level view of the mood space
 Solution: clustering
 Automatically group similar items
 “Similarity” defined:
 Mood labels assigned to the same pieces of music are similar
 Data
 allmusic.com applies mood labels to albums and songs
 7134 album-mood pairs and 8288 song-mood pairs
 Two independent data sources provide more robust and
meaningful clustering results
Hu, X., & Downie, J. S. (2007). Exploring Mood Metadata: Relationships with Genre, Artist and 9
Usage Metadata. In Proceedings of ISMIR

10. Clustering Results
Mood labels for albums Mood labels for songs
C1 C2 C3 C4 C5 C4 C1 C3 C2 C5
1/11/2012 10

11. A Taxonomy of 5 Mood Clusters
Cluster_1:
passionate, rousing, confident, boisterous, rowdy
Cluster_2:
rollicking, cheerful, fun, sweet, amiable/good natured
Cluster_3:
literate, poignant, wistful, bittersweet, autumnal, brooding
Cluster_4:
humorous, silly, campy, quirky, whimsical, witty, wry
Cluster_5:
aggressive, fiery,tense/anxious, intense, volatile,visceral
11
1/11/2012

12. Verifications of the 5 Clusters
 Survey users of different groups in
labeling a set of songs using the
taxonomy
 Developed an online music
recommendation system based on
the taxonomy
1/11/2012 12

13. User Evaluation: Experts
 1,250 music clips
 21 MIR
researchers
 Each clips had
three judges
% of clips with agreements
Most disagreements are
C1 40.2%
C2 60.2%
between:
C3 70.5% C2 (cheerful) and C4 (humorous)
C4 39.6% C1 (passionate) and C2 (cheerful)
C5 70.8%
Other 16.9%
Hu, X., Downie, J. S., Laurier, C., Bay, M., & Ehmann, A. (2008). The 2007 MIREX Audio Mood
13
Classification Task: Lessons Learned. In ISMIR.

14. User Evaluation: Amazon Mechanic Turk
 AMT:
crowdsourcing
 1,250 music clips
 Each clips had two
judges
% of clips with agreements
C1 39.6%
Most disagreement are
C2 48.9% between:
C3 69.5% C1 (passionate) and C2 (cheerful)
C4 46.3% C2 (cheerful) and C4 (humorous)
C5 60.0% C1 (passionate) and C5 (angry)
Other 21.3%
Lee, J. H. & Hu, X. (Under Review) Generating Ground Truth for Music Mood Classification 14
Using Mechanical Turk

15. Verifications of the 5 Clusters
 Survey users of different groups in
labeling a set of songs using the
taxonomy
 Developed an online music
recommendation system based on
the taxonomy
1/11/2012 15

16. Prototype System
Moodydb.com
Hu, X., et. al (2008). MOODY: A Web-Based Music Mood Classification and
16
Recommendation System, (Demonstration). ISMIR

17. Summary of the 5 Clusters Taxonomy
 Grounded in a real-world music repository
 The first music mood taxonomy undergone
verifications by multiple approaches
Limitations
 “Other”: not sufficiently comprehensive
 Confusions across clusters: multi-label
 Editorial labels vs. end user perspectives
1/11/2012 17

18. Agenda
 Background
 Music Mood: the Question
 Taxonomies as Organization System
 Developing Music Mood Taxonomies
 Taxonomy from Editorial Labels
 User Evaluation
 Automatic Classification
 Prototype System
 Taxonomy from Social Tags
 Comparisons to Psychological Models
 Automatic Classification
 User Evaluation
1/11/2012 18

19. Taxonomy from Social Tags
Social tags
“The largest music tagging
Pros: site for Western music”
 Users’ perspectives
 Large quantity
Cons:
 Non-standardized Linguistic Resources
Human Expertise
 Ambiguous
Hu, X. (2010). Music and Mood: Where Theory and Reality Meet. In Proceedings of the 5th
iConference, (Best Student Paper). 19

20. The Method
1,586 terms in WordNet-Affect
– 202 evaluation terms in General Inquirer
(“good”, “great”, “poor”, etc.)
– 135 non-affect/ ambiguous terms by experts
( “cold”, “chill”, “beat”, etc.)
= 1,249 terms
476 terms are last.fm tags
group the tags by WordNet-Affect and experts
=> 36 categories
20
1/11/2012

21. Verifications
 Compared to influential
psychological models
 Developed multimodal
classification systems
1/11/2012 21

22. Russell’s 2-D Model (1980)
1/11/2012 22

23. Comparison to Russell’s 2-D Model
1/11/2012 23

24. Verifications
 Compared to influential
psychological models
 Developed multimodal
classification systems
1/11/2012 24

25. Automatic Classification
 Multimodal classification using audio and lyrics
 Most comprehensive study
MUSIC
on lyric mood classification
 Significantly outperformed
top-ranked single-source systems Audio Lyrics
 Binary classification
 Each mood has its own
classification model
 Allow one song to have
multiple mood labels
 A data set of 18 categories
Hu, X., & Downie, J. S. (2010). Improving Mood Classification in Music Digital Libraries by
25
Combining Lyrics and Audio. In JCDL (Best Student Paper).

26. 0.75
0.6
0.7
0.8
0.55
0.5
0.85
0.45
0.65
calm
sad
glad
romantic
gleeful
gloomy
angry
Angry
mournful
dreamy
1/11/2012
cheerful
brooding
aggressive
Aggressive
anxious
confident
Accuracy across Categories
hopeful
earnest
cynical
exciting
26

27. Summary of the 2-D Taxonomy
 Grounded in social tags (end users’ perspectives)
 Leverage linguistic resources and experts
 Complement psychological models
 Some categories are easier to predict than others
 Next step: Cultural dependency
 Focus groups and survey of people from different cultures
 Proposal: “Developing A Music Mood Taxonomy: Towards
Understanding Emotion and Culture in the Fast Changing
Information Environment”
1/11/2012 27

28. Question time!
1/11/2012 28