Build Your Own Search Engine
•
5 j'aime•2,462 vues
Slides for a talk I gave at The Fifth Elephant conference in 2012. Description copied from the conference's website: No, this is not another tutorial on using Solr/ElasticSearch/Sphinx/Lucene. Imagine that none of these existed and you need a search engine for your shiny new eCommerce startup. What would you do? Build your own search engine, of course. In this session at The Fifth Elephant 2012, Siddhartha Reddy describes what it takes to do that. Here is a video of the talk: https://hasgeek.tv/fifthelephant/2012-1/49-siddhartha-reddy-build-your-own-search-engine
Recommandé
Recommandé
Contenu connexe
Dernier
Dernier (20)
En vedette
En vedette (20)
Build Your Own Search Engine
- 1. Build Your Own Search Engine Siddhartha Reddy @the5el 1
- 2. SOME Ideas Underlying a Search Engine Siddhartha Reddy @the5el 2
- 3. Siddhartha Reddy o Loves Lives Search o Leads Product Discovery (search) team at Flipkart o @sids 3
- 4. “Should I build my own search engine?” No. 4
- 5. “Should I build my own search engine?” No. Use Lucene/Solr/ElasticSearch/Sphinx/etc. “What are we doing here then?” 5
- 9. Text Analysis “I was killed i` the Capitol; Brutus killed me.” Tokenization I, was, killed, i`, the, Capitol, Brutus, killed, me Stop-word Removal I, was, killed, i`, the, Capitol, Brutus, killed, me Normalization Case-folding i, killed, i`, capitol, brutus, killed, me Synonyms (colour = color; pavement = footpath) Others (Accents and diacritics, Abbreviations etc.) Stemming Lemmatization i, killed, i`, capitol, brutus, killed, me (saw = see; saw = hacksaw) 9 - English Porter (Rule-based) - KStemmer (Dictionary-based)
- 11. The Index 11
- 12. Term-Document Matrix Brutus AND Caesar AND NOT Calpurnia Brutus 110100 Ceasar 110111 Calpurnia 10000 NOT Calpurnia 101111 110100 AND 110111 AND 101111 = 100100 12
- 13. Inverted Index Brutus AND Caesar AND NOT Calpurnia 13
- 14. “User” Text Analysis Query/Response Searcher (Ranking/Scoring) Index Text Analysis Documents Indexer
- 16. Ranking/Scoring “mysql performance” Top 25 Best Linux Performance Monitoring and Debugging Tools 8 great MySQL Performance Tips Linux performance: is Linux becoming just too slow and bloated? MySQL Performance Blog 16
- 17. Ranking/Scoring “mysql performance” Term Frequency (Tf) mysql Top 25 Best Linux Performance Monitoring and Debugging Tools 8 great MySQL Performance Tips Linux performance: is Linux becoming just too slow and bloated? MySQL Performance Blog performance Total 1 23 24 5 7 12 3 12 15 6 8 14 * random numbers 17
- 18. Ranking/Scoring “mysql performance” Term Frequency (Tf) mysql performance Total Top 25 Best Linux Performance Monitoring and Debugging Tools Linux performance: is Linux becoming just too slow and bloated? MySQL Performance Blog 1 23 24 3 12 15 6 8 14 8 great MySQL Performance Tips 5 7 12 18
- 19. Ranking/Scoring “mysql performance” • Inverse Document Frequency (Idf) • How common (or rare) is a term? • 1 / (no. of documents the term occurs in) 19
- 20. Ranking/Scoring “mysql performance” score = Tf * Idf • Tf normalized by document length • Idf dampened by applying a function (log) 20
- 21. Ranking/Scoring “mysql performance” Term Idf mysql 10 performance 2 Tf * Idf mysql performance Total 1 * 10 23 * 2 56 Linux performance: is Linux 3 * 10 becoming just too slow and bloated? 12 * 2 54 MySQL Performance Blog 6 * 10 8*2 76 8 great MySQL Performance Tips 5 * 10 7*2 64 Top 25 Best Linux Performance Monitoring and Debugging Tools 21
- 22. Ranking/Scoring “mysql performance” Tf * Idf mysql performance Total MySQL Performance Blog 6 * 10 8*2 76 8 great MySQL Performance Tips 5 * 10 7*2 64 Top 25 Best Linux Performance Monitoring and Debugging Tools 1 * 10 23 * 2 56 Linux performance: is Linux becoming just too slow and bloated? 3 * 10 12 * 2 54 22
- 23. Boolean Search vs. Ranked Search • • Ranked Search Boolean Search o Rich query syntax o No relevance scoring o o o Simple query syntax o Relevance ranking/scoring is key Ex: Patent search, Enterprise search o Ex: Web Search, Flipkart Search Precision Recall controlled by user o Search Engine needs to balance Precision Recall 23
- 25. Indexing 25
- 26. Building an Inverted Documents Text Analysis term,documentId pairs S o r t (Disk) termId = term termId = postingId (dictionary) Persist postingId = postingsList (postings file) 26 term,documentId pairs, sorted
- 31. Distributed Search brutus d1, d3, d6, d7 ceasar d1, d2, d4, d8, d9 noble d5, d10 with d1, d2, d3, d5 killed d8 brutus ceasar noble with killed d1, d3, d6, d7 d1, d2, d4, d8, d9 31 d5, d10 d1, d2, d3, d5 d8
- 32. Distributed Search brutus ceasar noble with killed d1, d3, d6, d7 d1, d2, d4, d8, d9 d5, d10 d1, d2, d3, d5 d8 Partitioning by terms brutu s ceasa d1, d3 d6, d7 d1, d2, d4 brutu s ceasa r noble d5 r noble d10 with d1, d2, d3, d5 killed d8 Partitioning by documents 32 d8, d9
- 33. Images Attribution • Introduction to Information Retrieval o By Christopher D. Manning, Prabhakar Raghavan Hinrich Schütze 33