Presented by David Giffin | Etsy. See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Search at Etsy poses significant challenges. Our marketplace is filled with millions of unique, short-lived items and people trying to find them over 13 million times a day. In this session we'll discuss many of the solutions we've engineered to meet these challenges including, the evolution of indexing at Etsy, how HBase and Hadoop have taken indexing from hours to minutes, how and why we use bittorrent for Solr replication, how we track search performance, our approach to shave crucial milliseconds off every search, and an overview of our continuous deployment strategy, web / search config integration and A/B testing and analytics.

1. Solr, Lucene
& Hadoop
@
Monday, May 14, 12

2. david@etsy
4 Years Lucene and
Solr @ Etsy
Monday, May 14, 12

3. History of Search
@ Etsy
Hadoop + HBase
Indexing
(in development)
Replication
Monday, May 14, 12

4. About
Us
Monday, May 14, 12

5. Monday, May 14, 12

6. Monday, May 14, 12

7. Monday, May 14, 12

8. 13MM Listings
39MM Unique Visitors
880K Shops / 150
Countries
100+ Engineers
Monday, May 14, 12

9. Architectur
e Overview
Monday, May 14, 12

10. Overview
Search Web Database
+n slaves +n webs +n db shards
Memcached
+n caches
Monday, May 14, 12

11. Thrift
Search Web
slave web
query = hats for cats
slave web
result = 402, 283, 837
+n slaves +n webs
Monday, May 14, 12

12. Hydration
Database
shard
shard
Web
web
+n shards
web
Memcached
+n webs
cache
cache
+n caches
Monday, May 14, 12

13. The Results
Monday, May 14, 12

14. History of
Search
Monday, May 14, 12

15. History of Search
2007
•1 Million Listings
•A Single “Master”
Postgres Database
•PHP > Twisted >
Stored Proc > TSearch
Monday, May 14, 12

16. History of Search
2008
•2 Million Listings
•A Single “Master”
Postgres Database
•PHP > Solr
•4 Solr Slaves + 2
Monday, May 14, 12

17. History of Search
2009
•4 Million Listings
•A Single “Master”
Postgres Database
•PHP > Solr
•6 Solr Slaves + 2
Monday, May 14, 12

18. History of Search
2010
•7 Million Listings
•A Single “Master”
Postgres Database
•PHP > Thrift > Solr
•10 Solr Slaves + 1
Monday, May 14, 12

19. History of Search
2011
•10 Million Listings
•“Master” Postgres
Database + DB SHARDS!
•PHP > Thrift > Solr
•24 Solr Slaves + 1
Monday, May 14, 12

20. Future of Search
2012
•?? Million Listings
•MORE DB SHARDS!
•PHP > Thrift > Solr
•?? Solr Slaves + 1
Master
Monday, May 14, 12

21. What Did
We Learn?
Monday, May 14, 12

22. Lucene + Solr
> TSearch
http://www.depesz.com/2010/10/17/why-im-
not-fan-of-tsearch-2/
Monday, May 14, 12

23. Love Lucene +
Solr Trunk!
Monday, May 14, 12

24. Run, Don’t
Walk...
Monday, May 14, 12

25. Deployinator
Fork it: https://github.com/etsy/deployinator
Monday, May 14, 12

26. Smoker
Monday, May 14, 12

27. StatsD, Graph
Everything!
Fork it: https://github.com/etsy/statsd
Monday, May 14, 12

28. Monday, May 14, 12

29. 95th Percentile
Monday, May 14, 12

30. start · build_query · perform_search ·
receive_search_ads · search_side_response ·
create_event_logger · set_tpl_vars · tpl_render ·
receive_search_ads_post_render
Monday, May 14, 12

31. Solr Top Level
Cache >
Memcached
Monday, May 14, 12

32. etsy-
index.properties
$ cat /search/data/person/index/etsy-index.properties
#Tue Mar 27 13:05:51 EDT 2012
max_update_time=2012-03-27T17:05:51.955Z
Monday, May 14, 12

33. Check Index Size
Don’t Install if < 50%
Current Size
Monday, May 14, 12

34. Check if Index is
Too Old
Don’t Update if >
10 Days Old
Monday, May 14, 12

35. What Did We Learn?
Store Nothing
Monday, May 14, 12

36. Keep
Denormalized
Data
Monday, May 14, 12

37. DB Shard
PHP JSON Search
DB Shard Denormalizer Database
DB Shard
Monday, May 14, 12

38. Full Apply
Install
Reindex Incremental
Monday, May 14, 12

39. Full Apply Apply
Install
Reindex Incremental Incremental
Monday, May 14, 12

40. r
Database
exe
Ind
Monday, May 14, 12

41. HBase +
Hadoop
Monday, May 14, 12

42. HBase + Hadoop
Why HBase?
Monday, May 14, 12

43. HBase + Hadoop
DB Shard
PHP JSON
DB Shard Denormalizer HBase
DB Shard
Monday, May 14, 12

44. HBase + Hadoop
listings_denormalized
{NAME => 'listings_denormalized', FAMILIES
=> [{NAME => 'listing_data', BLOOMFILTER =>
'ROW', REPLICATION_SCOPE => '0',
COMPRESSION => 'SNAPPY', VERSIONS => '1',
TTL => '-1', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE =>
Monday, May 14, 12

45. HBase + Hadoop
listings_denormalized_m
odified_index
{NAME =>
'listings_denormalized_modified_index',
FAMILIES => [{NAME => 'pks', BLOOMFILTER
=> 'ROW', REPLICATION_SCOPE => '0',
COMPRESSION => 'SNAPPY', VERSIONS => '1',
TTL => '-1', BLOCKSIZE => '65536',
Monday, May 14, 12

46. HBase + Hadoop
SOLR-1301
https://issues.apache.org/jira/browse/
SOLR-1301
Monday, May 14, 12

47. HBase + Hadoop
Disk •Solr
Solr
Output Format Document
HDFS
Converter
•Solr Requires
Monday, May 14, 12

48. HBase + Hadoop
•Not Great with
Multi-Core Configs
•Added Solr Multi-Core
Support
• Solr Config Issues
•Added ENV support
Monday, May 14, 12

49. HBase + Hadoop
SolrInputDocume
ntWritable
public class SolrInputDocumentWritable extends SolrInputDocument
implements org.apache.hadoop.io.Writable {
Monday, May 14, 12

50. HBase + Hadoop
Oozie
Monday, May 14, 12

51. HBase + Hadoop
Oozie + HBase?
Monday, May 14, 12

52. HBase + Hadoop
ScanStringGenera
tor
http://blog.ozbuyucusu.com/2011/07/21/
using-hbase-tablemapper-via-oozie-workflow/
Monday, May 14, 12

53. HBase + Hadoop
Hadoop Indexer
Oozie Start
Map HBase Copy
Reduce HDFS Merge
Solr
Disk Install
Output
Monday, May 14, 12

54. HBase + Hadoop
IndexerActionMai
n
Monday, May 14, 12

55. HBase + Hadoop
Deployinator
Monday, May 14, 12

56. HBase + Hadoop
IndexCompare
Monday, May 14, 12

57. HBase + Hadoop
$ ./compare
ERROR: please provide two index directories
example: ./compare -p 0.1 -i user_id ./index ./index-1332867952588
options:
-p --percent= percent of the index to check
-i --id= primary key id field in the index
-h --hash= comparison or hash field in the index
<index> <index>
Monday, May 14, 12

58. HBase + Hadoop
$ ./compare
/search/data/person/index-1332867952588/
/search/data/person/index-1335378487672
id field: user_id
hash field: hash
percentage: 0.0010
files: /search/data/person/index-1332867952588/ /search/
data/person/index-1335378487672
/search/data/person/index-1332867952588 contains 1515512 docs
/search/data/person/index-1335378487672 contains 14837972 docs
1516 of 1516 documents are the same
Monday, May 14, 12

59. HBase + Hadoop
Copy and Merge
Monday, May 14, 12

60. HBase + Hadoop
Open Source
Monday, May 14, 12

61. Replication
Monday, May 14, 12

62. Replication
Monday, May 14, 12

63. Replication
Slaves
Master
+n slaves
Monday, May 14, 12

64. Monday, May 14, 12

65. BitTorrent
Replication
Monday, May 14, 12

66. Bit Torrent
Using BitTornado:
Monday, May 14, 12

67. Replication
Bit Torrent + Solr
Monday, May 14, 12

68. Replication
Bit Torrent + Solr
Monday, May 14, 12

69. Monday, May 14, 12

70. Monday, May 14, 12

71. Replication
Fork of TTorent: https://github.com/
etsy/ttorrent
Multi-File Support
Large File Support
Fork BitTorrent: Comming Soon
Monday, May 14, 12

72. Need a job?
Monday, May 14, 12

73. Monday, May 14, 12

74. Thanks!
Monday, May 14, 12

75. david@etsy
Monday, May 14, 12