The rapid evolution of deep learning technologies and the explosion of diverse user interaction traces have brought significant challenges and opportunities to recommendation and personalized systems. In this workshop, we discussed recent trends and techniques in user modeling and presented our work on immersive recommendation systems. These systems learn users’ preferences from diverse digital trace modalities (text, image and unstructured data streams) in a wide range of recommendation domains (creative art, food, news, and events). The workshop included a light tutorial on OpenRec, an open source framework that enables quick prototyping of complex recommender systems via modularization.
This workshop is based on research and development done at Cornell Tech as part of the Connected Experiences Lab, supported by Oath and NSF.
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Immersive Recommendation Workshop, NYC Media Lab'17
1. Immersive Recommendation
Deep User and Content Modeling for Personalization
Longqi Yang, Ph.D. student
Connected Experiences Lab, Small Data Lab
Cornell Tech
12. Netflix Challenge (Prize)
We’re quite curious, really. To the tune of one million dollars…
… To help customers find movies, we’ve developed our world-class movie
recommendation system: Cinematch. Its job is to predict whether someone will enjoy a
movie based on how much they liked or disliked other movies …
… We provide you with a lot of anonymous rating data, and a prediction accuracy bar that
is 10% better than what Cinematch can do on the same training data set… If you develop a
system that we judge most beats that bar on the qualifying test set we provide, you get
serious money and the bragging rights …
23. One Example – Bayesian Personalized Ranking (BPR)
𝒖𝒊, 𝒗 𝒑, 𝒗 𝒏For all
m𝑎𝑥 ln 𝜎 𝒖𝒊 ∙ 𝒗 𝒑 − 𝒖𝒊 ∙ 𝒗 𝒏
Rendle, Steffen, et al. "BPR: Bayesian personalized ranking from
implicit feedback." Proceedings of the twenty-fifth conference on
uncertainty in artificial intelligence. AUAI Press, 2009.
user i
An item that the user “click”
An item that the user does not “click”
24. Algorithms (Incomprehensive List)
• Weighted Regularized Matrix Factorization (WRMF)
• Probabilistic Matrix Factorization (PMF)
“Shallow” Models:
• Weighted Approximately Ranked Pairwise Loss (WARP)
“Deep” Models:
Hsieh, Cheng-Kang, et al. "Collaborative metric learning." Proceedings of the 26th International
Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.• Collaborative Metric Learning (CML)
He, Xiangnan, et al. "Neural collaborative filtering." Proceedings of the 26th International Conference
on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.• Neural Collaborative Filtering
Mnih, Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in
neural information processing systems. 2008.
• Wide and Deep Learning for Recommender Systems
Hu, Yifan, Yehuda Koren, and Chris Volinsky. "Collaborative filtering for implicit feedback
datasets." Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on. Ieee, 2008.
Cheng, Heng-Tze, et al. "Wide & deep learning for recommender systems." Proceedings of the 1st
Workshop on Deep Learning for Recommender Systems. ACM, 2016.
Weston, Jason, Samy Bengio, and Nicolas Usunier. "Wsabie: Scaling up to large vocabulary image
annotation." IJCAI. Vol. 11. 2011.
25. Beyond Matrix
Implicit feedback
Deep Content Modeling
Beyond “black-box items”
Beyond Accuracy
Diversity and Fairness
Rich Context and Modality
Learning preference from auxiliary channels
26. Pure Collaborative Filtering is cool (and maybe accurate), but real world
recommendations are far more complex than “likes”
Understanding the contents really matters! - Articles
Is it appropriate to recommend
these two articles together?
27. Pure Collaborative Filtering is cool (and maybe accurate), but real world
recommendations are far more complex than “likes”
Understanding the contents really matters! - Food
Random
(Most healthy)
Trattner, Christoph, and David Elsweiler.
"Investigating the healthiness of internet-
sourced recipes: implications for meal
planning and recommender systems."
Proceedings of the 26th International
Conference on World Wide Web. International
World Wide Web Conferences Steering
Committee, 2017
28. Pure Collaborative Filtering is cool (and maybe accurate), but real world
recommendations are far more complex than “likes”
Understanding the contents really matters! - Food
“Users in general tend to interact most often with the least
healthy recipes. Recommender algorithms tend to score
popular items highly and thus on average promote unhealthy
items.”
Trattner, Christoph, and David Elsweiler.
"Investigating the healthiness of internet-
sourced recipes: implications for meal
planning and recommender systems."
Proceedings of the 26th International
Conference on World Wide Web. International
World Wide Web Conferences Steering
Committee, 2017
29. Pure Collaborative Filtering is cool (and maybe accurate), but real world
recommendations are far more complex than “likes”
Understanding the contents really matters! – Cold Start
A new fashion cloth
A new online course
A new job post
Recommendations
without user feedback
30. Deep Content Modeling for Recommendations
(C/R/Res/Adversarial/Rei
nforcement) NN
User Item Interaction
Optimization
31. Beyond Matrix
Implicit feedback
Deep Content Modeling
Beyond “black-box items”
Beyond Accuracy
Diversity and Fairness
Rich Context and Modality
Learning preference from auxiliary channels
32. Are ratings/clicks/views enough for recommendations?
Context matters! – Music Recommendation
Schedl, Markus, et al. "Music recommender systems."
Recommender Systems Handbook. Springer US, 2015. 453-492.
Schedl, Markus, Peter Knees, and Fabien Gouyon. "New Paths in
Music Recommender Systems Research." Proceedings of the
Eleventh ACM Conference on Recommender Systems. ACM, 2017.
time location weather
Environmental Context
Individual Context
emotion activity social context
Schedl, Markus, Georg Breitschopf, and Bogdan Ionescu.
"Mobile Music Genius: Reggae at the Beach, Metal on a
Friday Night?." Proceedings of International Conference on
Multimedia Retrieval. ACM, 2014.
33. Recommendations are not always “a list”: Rich modality
Sun, Yu, et al. "Contextual intent tracking for personal assistants."
Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining. ACM, 2016.
Kang, Jie, et al. "Understanding How People Use Natural
Language to Ask for Recommendations." Proceedings of the
Eleventh ACM Conference on Recommender Systems. ACM, 2017.
34. Rich Context and Modality
(C/R/Res/Adversarial/Rei
nforcement) NN
User Item Interaction
Optimization
(C/R/Res/Adversarial/Rei
nforcement) NN
(C/R/Res/Adversarial/Rei
nforcement) NN
35. Beyond Matrix
Implicit feedback
Deep Content Modeling
Beyond “black-box items”
Beyond Accuracy
Diversity and Fairness
Rich Context and Modality
Learning preference from auxiliary channels
38. Fairness – Long tail and Minority
# views
(attention)
popular unpopular
39. Fairness – Long tail and Minority
Recommender system
better
worse
Yao, Sirui, and Bert Huang. "Beyond Parity: Fairness
Objectives for Collaborative Filtering." arXiv preprint
arXiv:1705.08804 (2017).
40. Incorporating diversity and fairness into recommendations
(C/R/Res/Adversarial/Rei
nforcement) NN
User Item Interaction
Optimization
(C/R/Res/Adversarial/Rei
nforcement) NN
(C/R/Res/Adversarial/Rei
nforcement) NN
Penalize homogeneous and
unfair recommendations
44. Yum-me
Bringing healthiness into the recommendation of food
Yang, Longqi, et al. "Yum-Me: A Personalized Nutrient-Based Meal Recommender
System." ACM Transactions on Information Systems (TOIS) 36.1 (2017): 7.
45. *Number of Americans Living with Diet-and Inactivity-Related Diseases
Obesity
HBP
Diabetes
113M
50M
15M
Critical Issue of Food
46. The problem is not awareness, but adherence
How can we (efficiently) find meals that are healthy but also
cater to people’s tastes? - Bringing the notion of healthiness into
recommendations!
47. Yum-me: An interactive healthy meal recommendation system
Take a look at the food below and tap all
that look delicious to you.
http:// http://
Compare the food pair below and tap on
whichever looks delicious to you.
Press on Yuck if neither of
them fits to your taste
2iters + 13iters
2iters + 13iters
2iters + 13iters
Browser
Mobile
Wearable
Personal Dietary Profile
(Food Preferences)
… …
Healthy meal recommendations based
on dietary restrictions
Re-ranking
Personalized healthy meal recommendations
…...
…...
Phase I Phase II
Take a look at the food
below and tap all that
look delicious to you.
Compare the food pair below
and tap on whichever looks
delicious to you.
Press on Yuck if neither of
them fits to your taste
Take a look at the food
below and tap all that
look delicious to you.
Compare the food pair below
and tap on whichever looks
delicious to you.
Press on Yuck if neither of
them fits to your taste
http://
Choose the closest diet type to you.
⌾No restrictions ⌾ Vegetarian⌾ Vegan
⌾ Kosher ⌾ Halal
Identify your health goals.
⌾Reduce ⌾ Maintain ⌾ Increase
⌾Reduce ⌾ Maintain ⌾ Increase
⌾Reduce ⌾ Maintain ⌾ Increase
Calories
Protein
Fat
+
Survey
Choose the closest diet
type to you.
Identify your health goals.
⌾Reduce
⌾ Maintain
Calories
⌾No restrictions
⌾ Vegetarian
⌾ Vegan
⌾ Kosher
⌾ Halal
+
+
Choose the closest diet
type to you.
⌾No restrictions
⌾ Vegetarian
⌾ Vegan
⌾ Kosher
⌾ Halal
48. Yum-me: An interactive healthy meal recommendation system
Take a look at the food below and tap all
that look delicious to you.
http:// http://
Compare the food pair below and tap on
whichever looks delicious to you.
Press on Yuck if neither of
them fits to your taste
2iters + 13iters
2iters + 13iters
2iters + 13iters
Browser
Mobile
Wearable
Personal Dietary Profile
(Food Preferences)
… …
Healthy meal recommendations based
on dietary restrictions
Re-ranking
Personalized healthy meal recommendations
…...
…...
Phase I Phase II
Take a look at the food
below and tap all that
look delicious to you.
Compare the food pair below
and tap on whichever looks
delicious to you.
Press on Yuck if neither of
them fits to your taste
Take a look at the food
below and tap all that
look delicious to you.
Compare the food pair below
and tap on whichever looks
delicious to you.
Press on Yuck if neither of
them fits to your taste
http://
Choose the closest diet type to you.
⌾No restrictions ⌾ Vegetarian⌾ Vegan
⌾ Kosher ⌾ Halal
Identify your health goals.
⌾Reduce ⌾ Maintain ⌾ Increase
⌾Reduce ⌾ Maintain ⌾ Increase
⌾Reduce ⌾ Maintain ⌾ Increase
Calories
Protein
Fat
+
Survey
Choose the closest diet
type to you.
Identify your health goals.
⌾Reduce
⌾ Maintain
Calories
⌾No restrictions
⌾ Vegetarian
⌾ Vegan
⌾ Kosher
⌾ Halal
+
+
Choose the closest diet
type to you.
⌾No restrictions
⌾ Vegetarian
⌾ Vegan
⌾ Kosher
⌾ Halal
53. Yum-me: An interactive healthy meal recommendation system
Take a look at the food below and tap all
that look delicious to you.
http:// http://
Compare the food pair below and tap on
whichever looks delicious to you.
Press on Yuck if neither of
them fits to your taste
2iters + 13iters
2iters + 13iters
2iters + 13iters
Browser
Mobile
Wearable
Personal Dietary Profile
(Food Preferences)
… …
Healthy meal recommendations based
on dietary restrictions
Re-ranking
Personalized healthy meal recommendations
…...
…...
Phase I Phase II
Take a look at the food
below and tap all that
look delicious to you.
Compare the food pair below
and tap on whichever looks
delicious to you.
Press on Yuck if neither of
them fits to your taste
Take a look at the food
below and tap all that
look delicious to you.
Compare the food pair below
and tap on whichever looks
delicious to you.
Press on Yuck if neither of
them fits to your taste
http://
Choose the closest diet type to you.
⌾No restrictions ⌾ Vegetarian⌾ Vegan
⌾ Kosher ⌾ Halal
Identify your health goals.
⌾Reduce ⌾ Maintain ⌾ Increase
⌾Reduce ⌾ Maintain ⌾ Increase
⌾Reduce ⌾ Maintain ⌾ Increase
Calories
Protein
Fat
+
Survey
Choose the closest diet
type to you.
Identify your health goals.
⌾Reduce
⌾ Maintain
Calories
⌾No restrictions
⌾ Vegetarian
⌾ Vegan
⌾ Kosher
⌾ Halal
+
+
Choose the closest diet
type to you.
⌾No restrictions
⌾ Vegetarian
⌾ Vegan
⌾ Kosher
⌾ Halal
55. User study
Step 1. Users identify their
diet types and health goals.
Step 2. Users use visual
interace to express their
fine-grained food
preferences.
Step 3. Users identify each of
recommended meals as either
Yummy or No way. (The order
of the items is randomized)
Top 500 healthy items that
meet users’ diet types and
health goals.
Select top 10 items
ranked by user’s
fine-grained dietary
preference.
Randomly select 10
food items from 500
healthy meal pool.
…... …...
…...…...
57. User study
Goal: reduce calories (25 users) Goal: maintain calories (21 users)
Goal: maintain protein (36 users) Goal: increase protein (12 users) Goal: reduce fat (17 users)
Goal: increase calories (2 users)
Goal: maintain fat (30 users)
users’ 20 favorite meals
meals recommended by Yum-me
and accepted by users.
Averageamountofnutrients
perserving(normalized)
Averageamountofnutrients
perserving(normalized)
Averageamountofnutrients
perserving(normalized)
58. Creative Content Recommendation
Bringing unstructured command traces into the recommendation of art
Yang, Longqi, et al. "Personalizing Software and Web Services by
Integrating Unstructured Application Usage Traces." Proceedings of the
26th International Conference on World Wide Web Companion. International
World Wide Web Conferences Steering Committee, 2017.
59. Cold-start creative content recommendation
Day to day work activities
(Commands performed in
professional design software)
72. OpenRec: Experimentation and innovation through Extension
rating
item text item textitem image
rating
user text
item image
item text
rating
rating
user
text
user
demogr
item
text
item
image
integrator
module
extractor
module
interaction
module
R1 R2
R4R3
73. OpenRec: Architecture
Module
Extractor IntegratorInteraction
BPR WARP
PMF CML
NeuMF …
LF
ResNet MLP
LSTM
FoodDist
Concatenation
Average
Weighted sum
……
Recommender
News recommender
system with users’ click
history, Twitter posts
and news topic modeling.
Music recommender system
with users’ listen history,
lyrics and audio analysis
…
Utility
Sampler
Pairwise
sampler
Triplet
sampler
…
Evaluator
…
AUC
Recall@K