3. WHY?
Counter/respond to disruptors
Stay ahead of competition
Keep up with evolving customer
expectations
Respond to changes & threats
e.g. regulatory, economic,
security
22. “Controlled experimentation is becoming
the norm in advanced software
companies”
The evolution of continuous experimentation in software
development by Fabijan et al ICSE '17
42. If you want to build a ship, don’t drum up the
people to gather wood, divide the work, and give
orders. Instead, teach them to yearn for the vast
and endless sea.
- Antoine de Saint-Exupéry, the author of The Little Prince
43. Dream Team
Freedom and Responsibility
Informed Captains
Disagree Openly
Context, not Control
Highly Aligned, Loosely Coupled
Seeking Excellencehttps://jobs.netflix.com/culture
59. Simulate special behaviors from server
Validate app response
Triggered externally by app developers
Transparent to app
Simone - A Distributed Simulation Service
Simulation Testing
63. Quality of Experience
Time to interact with UI take on a given device
Data throughput for members in a certain region
Delay between pressing Play and playback start
Rebuffer rates
etc.
Reflecting on innovation strategies and creating an environment that fosters it
Let’s start with they Why. Why focus on innovation?
To stay relevant and grow, businesses need to
Defend against disruptors - think Nordstrom competing with newer online only retailers
Stay ahead of competition - no time to rest on laurels, amazon, netflix
Ever changing customer needs - Netflix, Google, FB, Spotify
Regulatory & compliance threats -
It’s clear that innovation is has become a priority for more and more companies. Businesses have recognized innovation as a key to staying relevant and growing in a fast changing world.
There are 3 ingredients for innovation
Three Must-Haves for innovation to succeed:
Ability to move fast
Three Must-Haves for innovation:
Ability to move fast
Appetite for smart risks
Three Must-Haves for innovation:
Ability to move fast
Appetite for smart risks
Comfort with ambiguity
Now we are living in a digital world, therefore technology is a powerful driver of innovation.
And the people who leverage technology to generate business value are developers.
It follows that developers play a key role in executing a successful innovation strategy.
So there are 3 ingredients to innovation and developers play a lead role.
It stands to reason that businesses should empower developers to move fast, take calculated risks and navigate ambiguity.
What can we learn from companies who have been successful at empowering their developers to innovate?
A common thread is using combination of culture and technology to support an innovation focus.
Let’s dig into these aspects for a bit.
Starting with culture. Most companies, teams and communities have a culture. The question is whether it has been shaped intentionally or has developed organically over time.
Culture is more than a set of value statements. The real values of a company are shown by the behaviors and actions that get rewarded and those that are penalized. Being intentional about what those are is a great way to encourage the types of behaviors you want to see in your company, organization, team or community. Successful companies make it a priority to shape their culture.
To underscore the importance of culture, a study to answer the question: “What makes a team effective at Google?” found that the #1 factor that high performing teams had in common was Psychological safety.
Code-named Project Aristotle - a tribute to Aristotle’s quote, "the whole is greater than the sum of its parts" (as the Google researchers believed employees can do more working together than alone) - the goal was to answer the question: “What makes a team effective at Google?”
This insight helps teams within Google optimize for an environment that creates and promotes psychological safety. That’s an example of being intentional about the culture.
Additionally, as we saw earlier, the ability to take risks is a key must have for innovation. Unsurprisingly, Google is a great example of a company that nurtures innovation.
Questions to consider when assessing the culture of a company.
How much autonomy do people have?
Are they empowered to take risks?
What is the appetite for failure?
Are diverse opinions encouraged?
Answers to these questions will help assess how well a company is set up for innovation.
Businesses whose culture encourages people to be nimble, take calculated risks and be comfortable navigating ambiguity put themselves in the best position to succeed at innovating.
On to Technology. How do those innovation ingredients apply to technology?
Successful companies take an iterative approach to innovation as opposed to aiming for instant perfection. It is very hard to get things right the first time. An iterative approach requires the ability to support experimentation, which in turn puts an emphasis on the velocity of software delivery.
Image credit: https://svgsilh.com/image/576604.html
At the same time, we know that change is inherently risky. A study of contributing factors and root causes of incidents across the industry will typically look like the above (example from Netflix). You can see that the leading cause of issues is change.
If you needed further proof...
This of course poses an interesting and hard challenge - how can software organizations support innovation by moving fast while maintaining reliability that customers expect?
A study by the authors of Accelerate shows that high performing tech orgs are able to move FAST and stay SAFE most of the time. When things go wrong, they are able to recover quickly.
Businesses who invest in technology that encourages people to be nimble, take calculated risks and be comfortable navigating ambiguity put themselves in the best position to succeed at innovating.
Let’s look at a case study from Netflix
Edge Engineering provides functionality and metadata to power core Netflix product experiences
Edge provides functionality and metadata to power core Netflix product experiences
Edge provides functionality and metadata to power core Netflix product experiences
Edge provides functionality and metadata to power core Netflix product experiences: Signup, Discovery & Playback
We are constantly dealing with the challenge of balancing creating a compelling product while maintaining a reliable service.
Doing it at scale adds to the challenge because it doesn’t leave much room for error. How do we use a combination of Culture & Technology to support to create an environment for our developers that fosters innovation?
We are very intentional about our culture. It is a core part of our hiring process and permeates all aspects of working at Netflix.
The core philosophy of Netflix culture
Our CEO Reed Hastings was inspired by the above quote, aspects of which made their way into our culture. Empowering people yields far better results than constraining them with processes and policies.
If you want to build a ship, don’t drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.
Giving people the freedom to make decisions, as opposed to telling them what to do. Balancing that out by giving them full responsibility of their actions. Equipping them with vast amounts of context in the form of data, strategy choices and any information. This allows them to build the muscle needed to take smart risks and navigate a world of ambiguity.
Our technical decisions support our commitment to innovation by supporting risk taking and rapid iteration.
How do we empower our developers to support innovation?
By placing a major emphasis on Developer Experience
For us it means two things:
Give developers the shortest, safest path from idea to production
Maximize the time and energy spent by a developer on their core mission - art and science of building a great product, vs. the process of developing, deploying, operating and supporting it.
A word on DevEx (tangent). Who is a developer? With cloud, microservices, mobile, everyone is a developer. All communication happens through APIs
You provision a server through an API
You get data through APIs
You page someone through an API
So, who is a developer? EVERYONE
Why is DevEx important?
We think of developers as different from end users. The philosophy for Dev Tools is often function over form because developers are considered power users.
The reality is that they *could* be power users, but *should* they be? Their time is best spent focusing on innovation, rather than learning tools. Any cycles spent on the latter is energy taken away from the former.
As such, we should be placing an equal emphasis on all the other aspects of our tools and processes, in addition to function. How easy and quick is it to get onboarded, make a change, gain confidence in the change and roll it out? What does the feedback loop look like?
Don’t make developers spend time on gathering and understanding context and learning tools.
With that context in mind, let’s look at some examples of how we try to shorten the path to production for our developers and free up their cycles to focus on making an amazing service for Netflix members worldwide.
https://www.flickr.com/photos/16210667@N02/25300692725
The examples are in 3 categories
It is important to make DevEx a first class consideration as part of your architecture. Bolting things on top will only take you so far.
Remember we support well over 1000 device types? Form factors, input modes, network and device constraints vary widely across devices. Powering the UI takes a lot of data because of the highly personalized experience. How do we optimize the API to service the varying data needs of all these devices?
Zuul: Internet facing service, provides routing, traffic shaping, security and more
NQ: Translation layer to optimize experience for each device
API: Orchestration and abstraction over other Netflix services
PBL: Handles content licensing
DP & OI: Tools and infrastructure to enable engineers to develop and operate complex systems at scale
The NQ tier allows each UI team to have a dedicated service to customize data based on the needs of their device/app. They have complete control over the interactions between client and server. In effect, it acts as a translation layer between our API and UI tiers.
This pattern is also referred to as the BFF pattern.
Resources:
https://samnewman.io/patterns/architectural/bff/
https://nordicapis.com/building-a-backend-for-frontend-shim-for-your-microservices/
This architecture gives us a highly customizable API which frees up our UI developers to innovate on the experience. At the same time gives us the operational controls to maintain safety.
I’ve talked a lot about pushing out changes rapidly. As a change makes its way through the system, we want to provide quick feedback on the quality of the change so we can increase our confidence of it succeeding in production.
We want our apps to be able to handle any response that the server sends.
One technique we use is Simulation testing. The idea is to trigger a particular type of behavior in a service by asking it to override its normal behavior with something that’s specified by the caller. This allows app developers to test various scenarios.
Imagine a client developer wanted to test their app’s behavior if the server responded with this check. They can easily simulate this behavior by asking the server to return this response. The server advertises the various types of behavior that it supports for overrides. Clients use that to customize the specific behavior they want returned. This allows client developers to exercise various edge cases and scenarios that might otherwise be hard to create and maintain.
The third category I’d like to touch upon is related to insights.
App developers are responsible about the user experience on their platforms. They need to be able to access key business and operational metrics in a quick and easy to consume fashion. For instance they might want to know the uptake of a newly launched feature or they might be interested in understanding the performance profile of their app. And of course, failure is a part of life in a distributed environment. Ease of debugging is a critical piece as well. Let’s look at a couple of examples oh how we provide visibility into the environment for our developers.
Examples:
What is the QoE on various devices?
How long before the user can interact with the app?
Playback statistics and alerts on specific titles.
Helps answer questions such as “What’s the error distribution of members experiencing playback issues watching The Triple Frontier?” If a specific UI version or device type is seeing large errors, we can dig into the issues.
Server side telemetry is a rich source of information. Surface actionable insights derived from that data to client developers, with context is another way we help them troubleshoot problems. Making it self-service is another aspect we place high emphsis on.
This was an example of how we combine culture and technology to empower our developers at Netflix and create an environment that fosters innovation.
What is the culture of your Dev community?
How well does your Developer Experience support innovation?