Day 1 General Session RedisConf
•Download as PPTX, PDF•
2 likes•825 views
Salvatore Sanfilippo Bill Anderson TIm Savage Chris Richardson Itamar Haber
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Day 1 General Session RedisConf
Similar to Day 1 General Session RedisConf (20)
More from Redis Labs
More from Redis Labs (20)
Recently uploaded
Recently uploaded (20)
Day 1 General Session RedisConf
- 1. Salvatore Sanfilippo Creator of Redis
- 2. Salvatore Sanfilippo Creator of Redis
- 4. Salvatore Sanfilippo Creator of Redis
- 5. Tim Savage Armakuni Co-Founder and Cloud Native Developer
- 6. has raised over £1 , ,
- 8. 90% of annual public donations during a 7 hour window, once a year 90%
- 9. HELP! Tiny budget No 2nd chance No operational usage Operates at significant scale Brand Confidence & Reliability
- 12. 100,000 Concurrent Web Donors
- 15. Key performance metrics up to 400 donations per second up to 800,000 donations in total 7 hour window, ONCE a year with no 2nd chance
- 16. Our Solution
- 17. Internet DNS US EAST COAST Cloud Foundry (BOSH/EC2) Presentation Layer Service Layer Workers =
- 18. Internet DNS US EAST COAST Cloud Foundry (BOSH/EC2) DUBLIN Cloud Foundry (BOSH/EC2) Service Layer Workers Presentation Layer Service Layer Workers LONDON Cloud Foundry (BOSH/VMWare) Presentation Layer Service Layer Workers Presentation Layer
- 19. 2 1 3 (in no particular order)
- 20. LONDON Insight Layer Internet DNS US EAST COAST Cloud Foundry (BOSH/EC2) DUBLIN Cloud Foundry (BOSH/EC2) Service Layer Workers View API Insight Presentation Layer Service Layer Workers MGMT LONDON Cloud Foundry (BOSH/VMWare) Presentation Layer Service Layer Workers Shared Services Logging Metrics Alerting = +Postcodes Presentation Layer
- 21. LONDON Insight Layer Internet DNS US EAST COAST Cloud Foundry (BOSH/EC2) DUBLIN Cloud Foundry (BOSH/EC2) Service Layer Workers View API Insight Presentation Layer Service Layer Workers MGMT LONDON Cloud Foundry (BOSH/VMWare) Presentation Layer Service Layer Workers Shared Services Logging Metrics Alerting = +Postcodes Presentation Layer
- 23. Load testing
- 25. Card Details Stored Success Email Export Data Payment Taken
- 30. WP Insight Shard API Reporting Apps Management Apps £ @ D
- 31. WP Insight Shard API Reporting Apps Management Apps £ @ D
- 32. WP Insight Shard API Reporting Apps Management Apps £ @ D
- 33. WP Insight Shard API Reporting Apps Management Apps £ @ D
- 34. WP Insight Shard API Reporting Apps Management Apps £ @ D
- 35. WP Insight Shard API Reporting Apps Management Apps £ @ D
- 38. Can you trust Redis to save a life? to save 10,000?
- 40. Salvatore Sanfilippo Creator of Redis
- 41. Itamar Haber Chief Developer Advocate at RedisLabs
- 43. 43
- 44. AND v3.2! 44
- 45. 45
- 46. BUT MODULES! 46
- 47. 47
- 49. https://groups.google.com/forum/#!msg/redis-db/Z0aiVSRAnRU/XezAFFtgyPUJ 49 “Another interesting idea is to add support for plugins implementing specific commands and associated data types, and the embedding of a scripting language.”
- 50. 3 STEPS TO YOUR REDIS MODULE 50
- 55. Uploads by anyone. Open source modules. Automated checks. Community filtered. Being developed. 55
- 57. Uploads by anyone. OS and commercial. Certified by Redis Labs for open source Redis, Redis Cloud & RLEC. 57
- 58. 58
- 59. MODULES! 59
- 60. 60
- 61. Thank you and bon appetit @itamarhaber @redislabs
- 62. Enjoy RedisConf Updates • Your morning break is located downstairs in the Sponsor Showcase • Three concurrent Breakout Sessions will begin at 11:15am • The Developer Workshop begins at 1:00pm in the Developer Café • Save your Redis Community Raffle ticket for tomorrow’s drawing for a chance to win a Phantom Drone
- 63. Salvatore Sanfilippo Creator of Redis
- 64. 11:15am-12:00pm A Hacker’s Guide To A New Era in Redis 1:00pm-1:45pm Redis as a Message Bus 2:00pm-2:45pm Solving Redis Latency Issues with Transparent Huge Pages: Make Copy-On-Write Great Again 3:15pm-4:00pm Using Redis as a Distributed Cache for ASP.NET Apps with IIS 4:15pm-5:00pm Scalable Streaming Data Pipelines with Redis 64 BREAKOUTS: TUESDAY, MAY 10
- 65. Salvatore Sanfilippo Creator of Redis
Editor's Notes
- 25 years - GREW UP with it vision: JUST WORLD FREE FROM POVERTY OVER £1 Billion They primarily operate 2 campaigns, Red Nose Day and Sport Relief on alternating years, which culminate in telethon events. We support these by handling the credit card payments. Given its a UK thing, heres a little video to get a taste.....
- Video holding slide
- Last few campaigns have raised in excess of £80m each 90% of the public donations are taken during that night of tv - 7 hour window So we are processing 8 digit numbers in 7 hours, once a year
- Its a problem!! tiny budgets - its a charity! no 2nd chance - BUY TV - come back. it's an emotional event for a donor no operational usage - the events happen once a year giving them an annual feedback cycle - not particularly helpful Operates at significant scale - the events take over BBC1 for a Friday evening. A large portion of the U.K. Public will be watching. the numbers can be huge. Comic Relief's brand is one of its most valuable assets. It's instilled in the UKs awareness and has very positive brand image. Protecting this is vital.
- What do we deliver - what are the requirements?
- Red Nose Day has always been about being FUNNY FOR MONEY DAVID TENNANT / CELEBRITIES people EITHER GO ONLINE <<NEXT>>
- People start going online Provide a simple donation page, cross device That is SIMPLE FOR THE DONOR handles 100,000
- or they pick up a phone Interface supporting the 100 or so call centres running 10-14,000 operators
- Typical evenings TRAFFIC PROFILE SHOWS DONATIONS PER MINUTE 10k peak at 10pm This event peaked at about 200 per second
- Platform KPMs agreed with Comic Relief DONATIONS PER SECOND IS KEY METRIC - PROBLEMS IN PAST WITH DEFINITION OF A TRANSACTION CONTEXT TO NUMBERS UK Card Industry figures for the UK quote an average of 375 transactions per second across the entire UK
- I am going to give you an overview of the solution we operate
- Quick run through of event night platform FE SHARD Bosh provisioned (BY US) cloud foundry install in AWS US East WE RUN micro services architecture - all deployed,only uses those it needs RATED TO 400 DPS On this shard there is a mix of ruby apps, queues and workers. In addition there are some redis instances, but I will come back to that....
- NOW ADD BOSH PROV SHARD IN DUBLIN BOSH PROV SHARD ON A PRIVATE CLOUD IN LONDON ON VSPHERE or USING A SERVICE SUCH AS BLUEMIX NOW HAVE RESILIENCY THROUGH REDUNDANCY ACROSS 3 GEOGRAPHIC LOCATIONS & 2 INFRA STRUCTURE PROVIDERS
- Payment handling. USE 3rd PARTY PSPs minimal PCI footprint Creates CRITICAL DEPENDANCY LOAD BALANCE ACROSS 3 PROVIDERS TO GET REDUNDANCY AND PERFORMANCE In addition, whilst we try to process all transactions SYNCHRONOUSLY (better success rates) We can process ASYNC, affording us greater processing numbers (STORE the card with the PSP, and then process the actual payment LATER)
- BACK - ADD INSIGHT SHARD ASYNC tasks: emails, exports, payments, config switches for functionality Mgmnt Reporting transaction insight
- WE CAN LOSE A LOT OF INFRASTRUCTURE AND STILL PROCESS 400 DONATIONS PER SECOND
- How do we MANAGE all this? Everything is code. Operate Full continuous deployment for code and infra through to production. All Day, Every day, all year, so we are very comfortable pushing changes to production Our tests include code tests, security test and load tests CONFIDENCE BLUE GREEN DEPLOY WHILST PROCESSING 30-40 DPS
- LOAD TESTING KEY NO OPERATIONAL USAGE - NO CONFIDENCE We load test US (we have mocked out every 3rd party we use so we can scale them to the level we need) We load test the PSPs Typically load test will involve Our platform scaled Scaled mocks Platform to generate the load Takes 30 mins to bring all that up
- So that's a quick insight into the platform we operate. Where does Redis come into all this?
- Architecturally WE CHOSE AVAILABILITY OVER CONSISTENCY The data is EVENTUALLY CONSISTENT TRANSACTION EVENTS OR STATE IS STORED AS DELTAS THAT MOVE THROUGH THE SYSTEM AND ARE RECONSTRUCTED WITHIN OUR INSIGHT LAYER EXAMPLE ABOVE SHOWS DELTAS FOR AN ASYNC PAYMENT Redis plays a key part in how we move these deltas round the platform - let me show you....
- This is an overview of our architecture... i’m not really expecting you to actually see the detail, But I am going to drill in a little....
- This should isolate and highlight the redis logos a bit more
- And now you can see just data stores and workers (the little sea urchin shaped stars are our workers....)
- FRONT END SHARD On completion of a transaction, a delta (remember earlier) is created and stored in memory backed redii, and queued At this point, this is the only copy of this data, so we ensure its written to 2 separate redii (diff AZs for example). A worker will then pick up the deltas, pushing them to the insight layer api. If the apps can’t push data locally to 2 redii, then they will push direct to the api, and the shard will be marked as failing. Using redis as our data store locally to our front end shards helps us ensure high availability.
- Insight Shard Within the insight shard, we have a processing loop that we nicknamed “the circle of trust.” This allows multiple processes to be horizontally scaled independently of each other. I am going to quickly walk you through a delta going round. A delta will hit the api, and a job will be added to the system input queue
- It will then get picked up by a worker which will .....
- add the delta to mongo, & then add it to the next logical queue
- In this case, its an async payment - we have taken card data at the front end shard and need to process the actual payment So a payment job is created. The worker will then pick this up, process the payment, create a delta and add it to....
- The local redii stores. We duplicate this data because at this point in time, its the only record of that event. (again, dual AZs) (Previously, we have multiple copies)
- Another worker will pick this delta up and push it back into the system input queue and it will go round the loop again In this case, this will result in a success or fail email being generated, Then an export of the data is pushed to Comic Relief Throughout we are using the Ruby Resque library to handle our queues, and simple document storage.
- One of our dashboards from the night
- Shows Q totals, Success in the top half, and fails in the bottom half - for example, the export api was unavailable
- So the title of the talk talked about saving lives I don’t think what Comic Relief does can be converted into numbers that simply But I do know that the organisation TRUSTS US to deliver this and WE TRUST Redis as a FUNDAMENTAL PART of this very successful platform. I am very proud to be part of that.
- When were Redis plugins/extensions/modules first discussed?
- The possibilities are truly endless.
- The tools are available now. Redis v3.4 will support the v1 Modules API: https://github.com/antirez/redis/modules Modules SDK: https://github.com/RedisLabs/RedisModulesSDK More modules talks by Yiftach, Dvir and myself Stop by the Redis community & Redis Labs booths and ask us
- Work in progress. Contributors are welcome.
- Work in progress. Contributors are welcome.