Building scalable and resilient internet applications

MillionEyes Healthcare Technologies
Building Scalable and
Resilient Internet
Applications
Gaurav Bhalotia
Foundation Day, Persistent Systems Nagpur

About Me
● Founder @ MillionEyes Healthcare, AI and wellness
● Was SVP Engineering Trusting Social, Credit Scoring for unbanked.
● Was VP Engineering @ Flipkart. Ran the Customer Platform (website),
Relevance Platform (search/recommendations) and Data Platform
● Was Head of India Dev center Retrevo (review aggregation and sentiment
analysis online)
● Director of Engineering @ Kosmix (categorisation and search, seeded
WalmartLabs)

Internet Applications are the new Electricity

Root cause?

Traffic Spikes1

Failures2

Site/App Robustness
With more 9s, the cost and complexity grows
Measured as availability (represented in 9s)1
2

Robust architecture isn’t all about software
It starts at the infrastructure layer
Progresses to the network and data
Influences application design
Extends to people and culture.Scalable
Resilient+

Resilience
“Failures are a given, and everything will
eventually fail over time.”

What is resiliency?

What is resiliency?
Ability to recover from failures and
continue to function

What is resiliency?
Not about avoiding failures,
but responding to failures “Well”

Scaleup
Prevent Failures
Traditional Enterprise

Optimise mean time between failures

Vs Internet Applications
Scaleout
Minimise Impact

“Optimise mean time to restore”

Why care about Resiliency
1. Financial Losses
2. Losing/Affecting Customers
3. Potentially Affecting Livelihood of Customers

Faults vs Failures

Faults
Incorrect Internal State in a
system

Database slowdown
Memory leaks
Blocked threads
Dependency failure
Bad Data
Faults

Failure
Inability of the system to do its
intended job

Resiliency
Preventing Faults from turning into
Failures

Network is unreliable
Multiple Dependencies which can Fail
Users are unpredictable
Resiliency in distributed systems is Hard

Resiliency is of Three Types
High Availability (HA)
Fault Tolerant
Disaster Recovery (DR)

Patterns for Resiliency

Resiliency Pattern #1
Redundancy
availability set → Multi zone/multi-region deployment
Let N be the composite SLA for the application deployed in one region. The expected chance that the application will fail in
both regions at the same time is (1 − N) × (1 − N). Therefore,
● Combined SLA for both regions = 1 − (1 − N)(1 − N) = N + (1 − N)N

Composition of services
The composite SLA 99.94%. Application that
relies on multiple services has more potential
failure points.
composite SLA for combined path is 99.99999%
more complex, you are paying for the queue, and there
may be data consistency issues to consider.

Retries
Transient failures can be caused by momentary loss of network connectivity, a dropped database connection, or a timeout
when a service is busy. Often, a transient failure can be resolved simply by retrying the request.

Requirements for Retries
Idempotent Operations
For retry mechanism to be safe, need ability to repeat without side effects. use unique traceable identifiers in
requests to your application and reject those that have been processed successfully.
Backoff Algos
To avoid network flooding and congestion, gradually increase the rate at which retries are performed

Avoid Cascading Failure
Prevent systems from entering a stressed states they cannot recover from

Mechanism to avoid Cascading failure
Throttling
When a single client makes an excessive number of requests, the application throttles the client for a certain period
of time, refusing some or all of the requests from that client. Helps minimise impact on other users, and thereby
avoiding an impact to overall availability of our application
Timeouts
With one request holding a resource,pool of connections quickly runs out. Timeouts prevent this from cascading. The
importance of thinking, planning and implementing timeouts is frequently underestimated.
Rejection
Final act of ‘self-defense’: start dropping requests deliberately when the service begins to overload. Can happen
server side, on load-balancers or even on the client’s side.

Graceful degradation
Instead of failing, your application degrades to a lower-quality service.

Strategies for degrading gracefully
Offering a variant of the service which is easier to compute
and deliver to the user
● Return an estimated value.
● Use locally cached data.
● Put a work item on a queue, to be handled later.
or Dropping unimportant traffic.

Circuit breakers
Applying circuit breakers to potentially-failing method calls, prevent an application from
repeatedly trying an operation that is likely to fail

Semantic Logging
Application logs are an important source of diagnostics data, to monitor the error rate. Generate structured logs that enable
automated analysis and actioning

Logging Best practices
Log in production.
Otherwise, you lose insight where you need it most.
Log events at service boundaries.
Include a correlation ID that flows across service boundaries.
Use asynchronous logging
Non blocking log writes, preventing any request backup

Automated Deployments
Manual deployments are prone to error. Need an automated, idempotent process → run on demand, and re-run if
something fails. Immutable infrastructure: Avoid modifying infrastructure after production deployment - hard to track and
reason (e.g. automate LB inclusion)

Best Practices
Blue Green Deployment: deploy updates into a production environment separate from the live application. Switch to
updated deployment after validation. This can further be done in a rolling manner to control the impact of an erroneous
build, this becomes a Canary Deployment. Rolling Deployments can also be done using feature flags

Resiliency Testing
Test and Break: Find failure modes, create a test harness to break callers, inject failures

Summary
Resiliency does not happen by accident. It must be
designed and built in from the start.
Resiliency leads to higher availability, and lower mean time
to recover from failures.
1
2
Resiliency touches every part of the application lifecycle,
from planning and coding to operations
3

Scaling
“A scalable web application is one that works
equally well with 1 user or 1,000,000 users”

Scaling: Dimensions

Load Balancers + Shared nothing
Units
Reduce any kind of contention among nodes as there is no scope for data or any other kind of resource sharing
Scalability Pattern #1

Examples?

LB + Stateless Nodes + Scalable
Storage
Decouple compute and data. Several stateless nodes talking to a scalable
storage, and a load balancer distributes load among the nodes

Scalable Storage
Use DB slaves to improve throughput
Segregate reads and writes.
1
2
NoSQL or KV store where applicable3

Asynchronous Constructs
Decouple request and response to allow for flattening spikes

Prefetching
Predict user behaviour and get data beforehand to reduce latency.

Caching
All static resources → images, stylesheets, javascripts, etc. Also dynamic content
that is not user specific (e.g. landing pages)

Event driven stream processing
Optimise for reads and write only when something changes

Offload work to client
Not every user action should require a request to the server, handle locally
anything that doesn’t need new data.

Lazy Loading
Load low resolution images and lazily upgrade

In Conclusion ...

Patterns are no silver bullet

War Story
Systems fail, deal with it!

Building scalable and resilient internet applications

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Building scalable and resilient internet applications

Similaire à Building scalable and resilient internet applications (20)

Dernier

Dernier (20)

Building scalable and resilient internet applications