Scaling Teams, Processes and Architectures
•
21 j'aime•41,217 vues
Generic presentation about scalability challenges. First London Scalability Meetup. Quick overview of the DataSift architecture.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (6)
Similaire à Scaling Teams, Processes and Architectures
Similaire à Scaling Teams, Processes and Architectures (20)
Dernier
Dernier (20)
Scaling Teams, Processes and Architectures
- 1. Lorenzo Alberton @lorenzoalberton Scaling Teams, Processes and Architectures Managing growth London Scalability Group, Innovation Warehouse, 16th April 2012 1
- 2. Scalability Is About... People Processes Technology 2
- 3. People Staffing, Roles, Management, Teams 3
- 4. Staffing Never compromise. Only hire people smarter than you. http://www.earthrangers.com/content/wildwire/toxic_spill.jpg 4
- 5. Staffing Hire people who can fit the company culture. Promote fun in your working environment. http://www.earthrangers.com/content/wildwire/toxic_spill.jpg 4
- 6. Staffing Beware of toxic people http://www.earthrangers.com/content/wildwire/toxic_spill.jpg 4
- 7. Team Size and Structure Micromanaging managers Poor communication too small Overworked team members Low morale too big Can’t accomplish much Low productivity 5
- 8. Team Size and Structure Micromanaging managers Poor communication too small Overworked team members Low morale too big Can’t accomplish much Low productivity CTO functional PM PM PM Designer Developer Tester Designer Developer Tester Designer Developer Tester Designer Developer Tester Designers Developers Testers 5
- 9. Team Size and Structure Micromanaging managers Poor communication too small Overworked team members Low morale too big Can’t accomplish much Low productivity CTO functional matrix PM PM PM Proj 1 PM Designer Developer Tester Proj 2 PM Designer Developer Tester Proj 3 PM Designer Developer Tester Proj 4 PM Designer Developer Tester Designers Developers Testers 5
- 10. Processes 6
- 11. Why are processes critical? Improve management of teams and employees Standardise actions in repetitive tasks Reduce mundane decisions to focus on grander ideas Allow the team to react quickly to crisis Determine system capacity and scalability needs 7
- 12. Why are processes critical? Improve management of teams and employees Standardise actions in repetitive tasks Reduce mundane decisions to focus on grander ideas Allow the team to react quickly to crisis Determine system capacity and scalability needs Challenge 7
- 13. Why are processes critical? Improve management of teams and employees Standardise actions in repetitive tasks Reduce mundane decisions to focus on grander ideas Allow the team to react quickly to crisis Determine system capacity and scalability needs Challenge right amount 7
- 14. Why are processes critical? Improve management of teams and employees Standardise actions in repetitive tasks Reduce mundane decisions to focus on grander ideas Allow the team to react quickly to crisis Determine system capacity and scalability needs Challenge right amount right process 7
- 15. Why are processes critical? Improve management of teams and employees Standardise actions in repetitive tasks Reduce mundane decisions to focus on grander ideas Allow the team to react quickly to crisis Determine system capacity and scalability needs Challenge right amount right process right time 7
- 16. Determining Headroom Capacity Current Load 8
- 17. Determining Headroom Why? Capacity Planning annual budget Hiring plan Current Load Prioritisation 8
- 18. Controlling Change: Determine Risk http://dilbert.com/strips/comic/2008-05-08/ 9
- 19. Controlling Change: Determine Risk http://dilbert.com/strips/comic/2008-05-08/ 9
- 20. Risk Management Risk is cumulative Determine limits and tolerance 10
- 21. Load / Stress Testing Load testing - identify, document and eliminate bottlenecks through a strict controlled process of measurement and analysis - measure system’s response and stability - verify the app can meet the desired performance objectives (SLA) Stress testing - determine the app’s stability when subjected to above-normal loads - verify the app’s behaviour when close to the breaking point - test the application recoverability (negative testing) 11
- 22. Barrier Conditions Code reviews Manual and automated QA processes Performance and stress testing Release documentation checks (runbook) Dev, Test, Stage and Live environments Instrumentation checks Protection from significant failures 12
- 24. Architectural Principles 14
- 25. Architectural Principles +1 N + 1 design 14
- 26. Architectural Principles +1 N + 1 design for rollback 14
- 27. Architectural Principles +1 N + 1 design for rollback to be disabled 14
- 28. Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored 14
- 29. Architectural Principles +1 N + 1 design for rollback to be disabled to be for multiple monitored live sites 14
- 30. Architectural Principles +1 N + 1 design for rollback to be disabled to be for multiple use mature monitored live sites technology 14
- 31. Architectural Principles +1 N + 1 design for rollback to be disabled to be for multiple use mature monitored live sites technology asynchronous design 14
- 32. Architectural Principles +1 N + 1 design for rollback to be disabled to be for multiple use mature monitored live sites technology asynchronous stateless design systems 14
- 33. Architectural Principles +1 N + 1 design for rollback to be disabled to be for multiple use mature monitored live sites technology asynchronous stateless buy when design systems non core 14
- 34. Stateless, Asynchronous Systems http://upload.wikimedia.org/wikipedia/commons/4/46/Synchronized_swimming_-_Russian_team.jpg 15
- 35. Fault Isolative Structures 16
- 36. Fault Isolative Structures Increase availability Limit impact of failures Easier debugging 16
- 37. Fault Isolative Structures Increase availability Limit impact of failures Easier debugging First 16
- 38. Fault Isolative Structures Increase availability Limit impact of failures Easier debugging Functions causing repetitive problems First 16
- 39. Fault Isolative Structures Increase availability Limit impact of failures Easier debugging Functions Natural layout causing or topology repetitive of the site problems First 16
- 40. Caching for Performance and Scale 17
- 41. Caching for Performance and Scale Object Caches Usually serialized (marshalling / unmarshalling) get() / set() / replace() APC, Memcached 17
- 42. Caching for Performance and Scale Object Caches Application Caches Usually serialized Proxy caches (marshalling / Reverse proxy unmarshalling) caches get() / set() / HTTP headers replace() ISP/Uni proxies APC, Memcached Squid, Varnish, mod_cache 17
- 43. Caching for Performance and Scale Object Caches Application Caches CDNs Usually serialized Proxy caches Multiple locations (marshalling / / backbones Reverse proxy unmarshalling) caches get() / set() / HTTP headers CNAME entries replace() ISP/Uni proxies Akamai, Coral, APC, Memcached Squid, Varnish, Limelight... mod_cache 17
- 44. Managing “Big Data” storage costs people and software power and space processing power backup time and costs 18
- 45. Managing “Big Data” The more storage ...the more storage management storage costs people and software power and space processing power backup time and costs 18
- 46. Managing “Big Data” The more storage ...the more storage management storage costs people and software power and space processing power backup time and costs Evaluate data retention policy Consider multi-tiered storage Distribute data/ work (Hadoop, M/R) 18
- 48. Monitoring: Measure Everything 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors 19
- 49. Monitoring: Measure Everything 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors Keep Signal vs. Noise ratio high 19
- 50. Monitoring: Measure Everything StatsD 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors Keep Signal vs. Noise ratio high 19
- 51. DataSift Architecture Some Architecture Pr0n 20
- 52. DataSift Architecture http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 21
- 53. DataSift Architecture SOA - loosely coupled, independently scalable services. Simple APIs http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 21
- 54. DataSift Architecture SOA - loosely coupled, independently scalable services. Simple APIs example http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 21
- 55. SOA - Scale Each Component 22
- 56. Our Stack Languages: C++, PHP, Java, Scala, Ruby, Node.JS Storage: MySQL, HBase Cache: Memcached, APC, Redis Queues: ZeroMQ, Kafka, Redis Development/Deployment: GIT, Jenkins CI, RPM, Chef Monitoring: StatsD + Graphite, Zenoss 23
- 57. Our Stack Languages: C++, PHP, Java, Scala, Ruby, Node.JS Storage: MySQL, HBase Cache: Memcached, APC, Redis Queues: ZeroMQ, Kafka, Redis Development/Deployment: GIT, Jenkins CI, RPM, Chef Monitoring: StatsD + Graphite, Zenoss Secret recipe: amazing people and working environment 23
- 58. Messaging ZeroMQ: PUSH-PULL, REQ-REP, PUB-SUB (multicast, broadcast) Internal communication: pass messages to the next processing stage, control events, monitoring Kafka/Redis: PUSH-PULL with persistence Internal message / workload buffering and distribution Node.js: WebSockets / HTTP Streaming Message delivery (output) 24
- 59. 0mq PUSH-PULL (workload distribution) Consumer 1 Consumer 2 Consumer 3 [Round-Robin-ish] 25
- 60. 0mq PUB-SUB (High Availability) Listener 1 Publisher 1 Listener 2 Publisher 2 Listener 3 [Broadcast] [Dynamic Subscriptions] 26
- 61. 0mq PUB-SUB (High Availability) DC 1 Publisher 1 Publisher 2 DC 2 27
- 62. Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 28
- 63. Instrumentation https://play.google.com/store/apps/details?id=net.networksaremadeofstring.rhybudd 29
- 65. References M. L. Abbot, M. T. Fisher, “The Art Of Scalability”, Addison Wesley http://theartofscalability.com/ http://www.slideshare.net/quipo/the-art-of-scalability-managing- growth http://www.slideshare.net/postwait/scalable-internet-architecture http://bit.ly/IJKwuc http://agile.dzone.com/news/approaches-organizational https://bitly.com/vCSd49 31
- 66. Lorenzo Alberton @lorenzoalberton Thank you! lorenzo@alberton.info http://www.alberton.info/talks Questions? 32
Notes de l'éditeur
- \n
- Let’s start by focusing on the true foundation: people and process, without which true scalability cannot be built.\nPeople are the most important element of scalability, as without people there are no processes and no technology.\n
- \n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Leadership is the influencing of an organisation to accomplish a specific objective (down to personal characteristics + skills + experience + actions).\nLook for solid software engineers with good understanding of CS topics, and exceptional devops. Create fun working environment. We solve serious, challenging problems. We also want to have fun. Avoid rockstars. "Hard work beats talent when talent doesn't work hard." - Tim Notke\nFocus and dedication.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- Overlapping responsibilities create wasted effort and value-destroying conflicts.\nKey scale-related responsibilities for any organisation include:\n- setting measurable goals; - staffing the team with the appropriate skills; - defining and implementing a scalable architecture.\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- There’s a link between organisational structure and scalability, it has a big impact in personal productivity.\nThe goal is to minimise the friction caused by organisational or team boundaries, without limiting the throughput, and at the same time making innovation and the work flow easy.\nThe team can be organised into 2 structures:- functional (employees divided by their primary function; homogeneity, simplicity of responsibilities, adherence to standards; Drawbacks: no single project owner, poor cross-functional communication).\n- matrix (similar, but with a second dimension that includes a new management structure; better communication, project ownership; Drawbacks: multiple bosses, distraction from a person’s primary discipline).\n
- \n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- Processes serve 3 general purposes:\n- they augment the management of our teams and employees\n- they standardise employee’s actions while performing repetitive tasks\n- they free employees up from daily mundane decisions to concentrate on grander ideas.\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- In order to navigating your way out of the woods, you need to know the point from which you are starting.\nHeadroom = amount of free capacity that exists within your system before you start having problems such as degradation of performance or an outage. Because your app is a system that involves many different components as a db, a firewall, application servers, in order to truly understand headroom you need to first understand the headroom of each of these.\n1) Identify major components. 2) Identify responsible team. 3) Determine usage and capacity. 4) Determine growth rate. Work together for a better analysis and to find the best solution\n
- The intent of change management is to limit the impact of changes by controlling them through their release into the production environment and logging them as they are introduced to production. Gut feeling / finger in the air: thanks to experience + innate ability: fast, but not accurate.\nSemaphore: Assign a risk level of green/yellow/red to each small component, then assign an overall colour: methodical, repeatable, documentable, no longer relying on a single person, accurate. Better: Failure Mode and Effect Analysis.\n\n\n
- Risk is cumulative. You might want to establish some limits to the amount of risk that you are willing to allow at a particular time of the day or customer volume.\nAlso consider the human factor, i.e. the level of risk tolerance that a person can have within a certain time frame.\n\n
- The purpose of load testing is to identify, document and eliminate bottlenecks in the system through a strict controlled process of measurement and analysis. Load testing is the process of putting load or user demand on a system to measure its response and stability, to verify that the app can meet the desired performance objectives (SLA: service level agreement).\n Establish success criteria (concurrent usage, response time, ...)\n Establish the test environment (as close as possible to the production environment)\n Define the tests (Pareto rule 20% - 80%) to cover different things (endurance, most used, most visible, different components)\n Identify what needs to be monitored / what data needs to be collected\n Run, Analyse, Report to Engineers\n Repeat Tests and Analysis\nStress testing is a process used to determine an application’s stability when subjected to above-normal loads, to verify the behaviour when close to the breaking point of the application.\nPositive testing is where the load is progressively increased to overwhelm the system’s resources.\nNegative testing takes away resources such as memory, threads, connections, testing the application recoverability.\nThe way to insure that the headroom calculations remain accurate is to conduct performance testing on all your releases to insure you are not introducing unexpected load increases.\n
- Good processes for the promotion of systems into the production environment have the capability of protecting you from significant failures. Developing effective barrier conditions and coupling them with a process and capability to roll back production changes are necessary components within any highly available service and are critical to the success of your scalability goals.\n
- \n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- - N+1 design (ensure that everything you develop has at least one additional instance of that system in the event of failure)\n- Designing the capability to roll back into an app helps limit the scalability impact of any given release.\n- Designing to disable features adds the flexibility of keeping the most recent release in production while limiting / containing the impact of offending features or functionality.\n- Design to be monitored: you want your system to identify when it’s performing differently than it normally operates in addition to telling you when it’s not functioning properly.\n- Design for multiple live sites: it usually costs less than the operation of a hot site and a cold disaster recovery site.\n- Use mature technology: early adopters risk a lot in finding the bugs; availability and reliability are important.\n- Asynchronous design: asynchronous systems tend to be more fault tolerant to extreme load.\n- Stateless Systems (if necessary, store state with the end users)\n- Buy when non-core\n- Scale out not up (with commodity hardware; horizontal split in terms of data, transactions and customers).\n- Design for any technology, not for a specific product/vendor\n
- Synchronous calls, if used excessively or incorrectly cause undue burden on the system and prevent it from scaling.\nSystems designed to interact synchronously have a higher failure rate than asynchronous ones. Their ability to scale is tied to the slowest system in the chain of communications. It’s better to use callbacks, and timeouts to recover gracefully should they not receive responses in a timely fashion.\nSynchronisation is when two or more pieces of work must be in a specific order to accomplish a task. Asynchronous coordination between the original method and the invoked method requires a mechanism that the original method determines when or if a called method has completed executing (callbacks). Ensure they have a chance to recover gracefully with timeouts should they not receive responses in a timely fashion.\nA related problem is stateful versus stateless applications. An application that uses state relies on the current condition of execution as a determinant of the next action to be performed. \nThere are 3 basic approaches to solving the complexities of scaling an application that uses session data: 1) Avoidance (using no sessions or sticky sessions) avoid replication: Share-nothing architecture; 2) Decentralisation (store session data in the browser’s cookie or in a db whose key is referenced by a hash in the cookie); 3) Centralisation (store cookies in the db / memcached).\n\n
- You must be able to isolate and limit the effects of failures within any system, by segmenting the components. Decouple decouple decouple! A swim lane represent both a barrier and a guide (ensure that swimmers don’t interfere with each other. Help guide the swimmer toward their objective with minimal effort). AKA Shard.\nThey increase availability by limiting the impact of failures to a subset of functionality, make incidents easier to detect, identify and resolve. The fewer the things are shared between lanes, the more isolative and beneficial the swim lane becomes to both scalability and availability. They should not have lines of communication crossing lane boundaries, and should always move in the direction of the communication. When designing swim lanes, always address the transactions making the company money first (e.g. Search&Browse vs Shopping Cart), then move functions causing repetitive problems into swim lanes; finally consider the natural layout or topology of the site for opportunities to swim lanes (e.g. customer boundaries within an app / environment. If you have a tenant who is very busy, assign it a swim lane; other tenants with a low utilisation can be all put into another swim lane).\n
- You must be able to isolate and limit the effects of failures within any system, by segmenting the components. Decouple decouple decouple! A swim lane represent both a barrier and a guide (ensure that swimmers don’t interfere with each other. Help guide the swimmer toward their objective with minimal effort). AKA Shard.\nThey increase availability by limiting the impact of failures to a subset of functionality, make incidents easier to detect, identify and resolve. The fewer the things are shared between lanes, the more isolative and beneficial the swim lane becomes to both scalability and availability. They should not have lines of communication crossing lane boundaries, and should always move in the direction of the communication. When designing swim lanes, always address the transactions making the company money first (e.g. Search&Browse vs Shopping Cart), then move functions causing repetitive problems into swim lanes; finally consider the natural layout or topology of the site for opportunities to swim lanes (e.g. customer boundaries within an app / environment. If you have a tenant who is very busy, assign it a swim lane; other tenants with a low utilisation can be all put into another swim lane).\n
- You must be able to isolate and limit the effects of failures within any system, by segmenting the components. Decouple decouple decouple! A swim lane represent both a barrier and a guide (ensure that swimmers don’t interfere with each other. Help guide the swimmer toward their objective with minimal effort). AKA Shard.\nThey increase availability by limiting the impact of failures to a subset of functionality, make incidents easier to detect, identify and resolve. The fewer the things are shared between lanes, the more isolative and beneficial the swim lane becomes to both scalability and availability. They should not have lines of communication crossing lane boundaries, and should always move in the direction of the communication. When designing swim lanes, always address the transactions making the company money first (e.g. Search&Browse vs Shopping Cart), then move functions causing repetitive problems into swim lanes; finally consider the natural layout or topology of the site for opportunities to swim lanes (e.g. customer boundaries within an app / environment. If you have a tenant who is very busy, assign it a swim lane; other tenants with a low utilisation can be all put into another swim lane).\n
- You must be able to isolate and limit the effects of failures within any system, by segmenting the components. Decouple decouple decouple! A swim lane represent both a barrier and a guide (ensure that swimmers don’t interfere with each other. Help guide the swimmer toward their objective with minimal effort). AKA Shard.\nThey increase availability by limiting the impact of failures to a subset of functionality, make incidents easier to detect, identify and resolve. The fewer the things are shared between lanes, the more isolative and beneficial the swim lane becomes to both scalability and availability. They should not have lines of communication crossing lane boundaries, and should always move in the direction of the communication. When designing swim lanes, always address the transactions making the company money first (e.g. Search&Browse vs Shopping Cart), then move functions causing repetitive problems into swim lanes; finally consider the natural layout or topology of the site for opportunities to swim lanes (e.g. customer boundaries within an app / environment. If you have a tenant who is very busy, assign it a swim lane; other tenants with a low utilisation can be all put into another swim lane).\n
- You must be able to isolate and limit the effects of failures within any system, by segmenting the components. Decouple decouple decouple! A swim lane represent both a barrier and a guide (ensure that swimmers don’t interfere with each other. Help guide the swimmer toward their objective with minimal effort). AKA Shard.\nThey increase availability by limiting the impact of failures to a subset of functionality, make incidents easier to detect, identify and resolve. The fewer the things are shared between lanes, the more isolative and beneficial the swim lane becomes to both scalability and availability. They should not have lines of communication crossing lane boundaries, and should always move in the direction of the communication. When designing swim lanes, always address the transactions making the company money first (e.g. Search&Browse vs Shopping Cart), then move functions causing repetitive problems into swim lanes; finally consider the natural layout or topology of the site for opportunities to swim lanes (e.g. customer boundaries within an app / environment. If you have a tenant who is very busy, assign it a swim lane; other tenants with a low utilisation can be all put into another swim lane).\n
- What is the best way to handle large volumes of traffic? Answer: “Establish the right organisation, implement the right processes and follow the right architectural principles”. Correct, but the best way is not to have to handle it at all. The key to achieving this is through pervasive use of caching. The cache hit ratio is important to understand its effectiveness. The cache can be updated/refreshed via a batch job or on a cache-miss. If the cache is filled, some algorithms (LRU, MRU...) will decide on which entry to evict. When the data changes, the cache can be updated through a write-back or write-through policy. There are 3 cache types:\n- Object caches: used to store objects for the app to be reused, usually serialized objects. The app must be aware of them. Layer in front of the db / external services. Marshalling is a process where the object is transformed into a data format suitable for transmitting or storing.\n- Application caches: A) Proxy caches, usually implemented by ISPs, universities or corporations; it caches for a limited number of users and for an unlimited number of sites. B) Reverse proxy caches (opposite): it caches for an unlimited number of users and for a limited number of applications; the configuration of the specific app will determine what can be cached. HTTP headers give much control over caching (Last-Modified, Etag, Cache-Control).\n- Content Delivery Networks: they speed up response time, off load requests from your application’s origin server, and usually lower costs. The total capacity of the CDN’s strategically placed servers can yield a higher capacity and availability than the network backbone. The way it works is that you place the CDN’s domain name as an alias for your server by using a canonical name (CNAME) in your DNS entry\n
- What is the best way to handle large volumes of traffic? Answer: “Establish the right organisation, implement the right processes and follow the right architectural principles”. Correct, but the best way is not to have to handle it at all. The key to achieving this is through pervasive use of caching. The cache hit ratio is important to understand its effectiveness. The cache can be updated/refreshed via a batch job or on a cache-miss. If the cache is filled, some algorithms (LRU, MRU...) will decide on which entry to evict. When the data changes, the cache can be updated through a write-back or write-through policy. There are 3 cache types:\n- Object caches: used to store objects for the app to be reused, usually serialized objects. The app must be aware of them. Layer in front of the db / external services. Marshalling is a process where the object is transformed into a data format suitable for transmitting or storing.\n- Application caches: A) Proxy caches, usually implemented by ISPs, universities or corporations; it caches for a limited number of users and for an unlimited number of sites. B) Reverse proxy caches (opposite): it caches for an unlimited number of users and for a limited number of applications; the configuration of the specific app will determine what can be cached. HTTP headers give much control over caching (Last-Modified, Etag, Cache-Control).\n- Content Delivery Networks: they speed up response time, off load requests from your application’s origin server, and usually lower costs. The total capacity of the CDN’s strategically placed servers can yield a higher capacity and availability than the network backbone. The way it works is that you place the CDN’s domain name as an alias for your server by using a canonical name (CNAME) in your DNS entry\n
- What is the best way to handle large volumes of traffic? Answer: “Establish the right organisation, implement the right processes and follow the right architectural principles”. Correct, but the best way is not to have to handle it at all. The key to achieving this is through pervasive use of caching. The cache hit ratio is important to understand its effectiveness. The cache can be updated/refreshed via a batch job or on a cache-miss. If the cache is filled, some algorithms (LRU, MRU...) will decide on which entry to evict. When the data changes, the cache can be updated through a write-back or write-through policy. There are 3 cache types:\n- Object caches: used to store objects for the app to be reused, usually serialized objects. The app must be aware of them. Layer in front of the db / external services. Marshalling is a process where the object is transformed into a data format suitable for transmitting or storing.\n- Application caches: A) Proxy caches, usually implemented by ISPs, universities or corporations; it caches for a limited number of users and for an unlimited number of sites. B) Reverse proxy caches (opposite): it caches for an unlimited number of users and for a limited number of applications; the configuration of the specific app will determine what can be cached. HTTP headers give much control over caching (Last-Modified, Etag, Cache-Control).\n- Content Delivery Networks: they speed up response time, off load requests from your application’s origin server, and usually lower costs. The total capacity of the CDN’s strategically placed servers can yield a higher capacity and availability than the network backbone. The way it works is that you place the CDN’s domain name as an alias for your server by using a canonical name (CNAME) in your DNS entry\n
- \n
- \n
- \n
- \n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- Logging at scale is useless. Too much noise. Instrumentation is essential.\nYou need to identify bottlenecks quickly or suffer prolonged and painful outages. The question of "How come we didn't catch that earlier?" addresses the incident, not the problem. The alternative question "What in our process is flawed that allowed us to launch the service without the appropriate monitoring to catch such an issue?" addresses the people and the processes that allowed the event you just had and every other event for which you didn't have appropriate monitoring.\nDesigning to be monitored is an approach wherein one builds monitoring into the application rather than around it. "How do we know when it's starting to behave poorly?" First, you need to answer the question "Is there a problem?" with user experience and business metrics monitors (lower click-through rate, shopping cart abandonment rate, ...). Then you need to identify where the problem is with system monitors (the problem with this is that it's usually relying on threshold alerts - i.e. checking if something is behaving outside of our expectations - rather than alerting on when it's performing significantly differently than in the past). Finally you need to identify what is the problem thanks to application monitoring. \nNot all monitoring data is valuable, too much of it only creates noise, while wasting time and resources. It's advisable to only save a summary of the reports over time to keep costs down while still providing value. In the ideal world, incidents and crises are predicted and avoided by a robust monitoring solution.\n
- \n
- \n
- \n
- \n
- Use queues and workers to make processes asynchronous, distribute data to parallel workers. \n
- happy to talk about any of them\n
- \n
- \n
- \n
- listeners can only subscribe to one or more topics. Different output channels.\nZeroMQ v3: filtering done on the publisher side\n
- An interesting idea if you have a highly dynamic site / service, with each update affecting several other users / pages, is to have an internal data bus that carries all the information, with updates labelled with topics, and all the services/users subscribing to the relevant topics.\n
- We collect millions of events every second.\nThe importance of people: devops who know what to monitor, how, how to use and write tools, and have 100% dedication.\nWe use different technologies. It’s very easy to set up a new ZeroMQ listener.\nWe use StatsD (from Flickr / Etsy), Zenoss, Graphite\n
- shameless plug\n
- \n
- \n