Idomaar is an open-source benchmarking framework for recommender systems created by the CrowdRec EU-project. It enables impartial evaluation of recommenders solutions from different aspect: (1) recommendation quality (2) technical aspects (3) business aspects.
Main contributors of Idomaar: Moviri, Gravity RD, Technical University of Delft, Technical University Berlin
2. What is Idomaar?
• A framework enabling impartial evaluation of
recommendation solutions
– A tamer of recommender solutions
– source: https://github.com/crowdrec/idomaar
• Name is coming from the Hungarian world idomár
(animal trainer)
2
3. Goal of Idomaar
• Impartial 3D evaluation of recommendation solutions
– Recommendation’s quality
– Technical aspects
– Business aspects
• Scope of the comparison
– Open-source toolkits (Mahout, Lenskit, MyMediaLite, etc.)
– Recommender vendor’s solution
• Ability to solve a wide range of recommendation tasks
– Training scenarios: held-out set, incremental learning
– Offline and online experiments
3 – Implicit and explicit feedback
4. Use cases
• Tendering – external evaluation
• Algorithm benchmarking – internal evaluation
• Implementation benchmarking
– same algorithm but different implementation
– testing implementation efficiency
• End-to-end measurements of entire solutions
– quality
– robustness
– scalability
4
6. Orchestrator – the central component
• Orchestrates evaluation
• Connects different components
• Access the data from DC
• Cooperates with EVL
– data preparation
– evaluation
• Controls CPEs
– messaging
6
7. Data container – data representation
• Flexibility
• entity – relation model
• multi-graph
• API provided
7
8. Data container – entities
• Entity: the object that can be recommended
• params: type, identifier, properties, links
8
9. Data container – relations
• Relations: is an explicit link b/w entities
• params: type, identifier, timestamp, properties, links
Explicit rating
type: rating
identifier: xyz
timestamp: 2014-
07-06 11:30:00
value: 3
subject: U-1
object: M-1
9
property
links
13. Idomaar users
• Industry
– running tenders
– evaluate open source projects
– qualify in-house solution
• Academia
– comparison for research purpose
– test bed for development
13