NETFLIX AND CHILL— using AWS CLOUD.
In today’s world I don’t think there is a single individual who is unaware about Netflix’s existence.
Netflix is an OTT platform that provides video services to different part of the world seamlessly without any issue.
But have you ever thought how in the hell , this company is doing all these stuff!?
The answer is cloud computing.And the cloud platform that Netflix majorly uses is Amazon Web sevices(AWS).
Online content provider Netflix can support seamless global service by using Amazon Web Services (AWS). AWS enables Netflix to quickly deploy thousands of servers and terabytes of storage within minutes. Users can stream Netflix shows and movies from anywhere in the world, including on the web, on tablets, or on mobile devices such as iPhones.
HISTORY:
Everybody knows about the great success that Netflix is enjoying right now but very few know about the failures it faced during its initial years of establishment.
Back in 2008, Netflix was majorly working on DVD-by-mail service. Due to the above mentioned database corruption incident, DVD shipping was disrupted for three days. Netflix management decided to move to the cloud, away from relational systems in their data centers. The shift happened from vertical scaling of particular failure points to horizontal scaling of distributed systems which were highly reliable. The cloud was that of AWS (Amazon Web Services) which offered the company the ability to scale as much as they needed. Previously, Netflix team had to sit with their IT team to implement the scale up whenever their demand increased. Scalability was a huge issue with physical data warehousing. After shifting to AWS, scaling became seamless as petabytes of data could be used to stream videos within minutes, thanks to elasticity of the cloud. Based on user demand and with the help of AWS, Netflix could scale-up or down their data warehousing.In early January 2016, Netflix shut down its last data center which was used by their streaming service. Now, there are eight times more users for Netflix as compared to those present in 2008. This represents the phenomenal growth of Netflix over the years. The company currently streams about 150,000,000 hours of video content per day. It serves around 86,000,000 members from 190 countries across the globe.Amazing right!!!
Let’s look at the sevices that NETFLIX uses provided by AWS.
AWS SERVICES USED BY NETFLIX:
System design of Netflix:
The above picture gives an idea about the services that AWS provides to netflix.
- AMAZON EC2 and LAMBDA:
Amazon Elastic Compute Cloud (EC2) is a part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), that allows users to rent virtual computers on which to run their own computer applications. EC2 encourages scalable deployment of applications by providing a web service through which a user can boot an Amazon Machine Image (AMI) to configure a virtual machine, which Amazon calls an "instance", containing any software desired. A user can create, launch, and terminate server-instances as needed, paying by the second for active servers – hence the term "elastic". EC2 provides users with control over the geographical location of instances that allows for latency optimization and high levels of redundancy.
EC2 instances can be resized and the number of instances scaled up or down as per our requirement. These instances can be launched in one or more geographical locations or regions, and Availability Zones (AZs). Each region comprises of several AZs at distinct locations, connected by low latency networks in the same region.
Benefits of using EC2-
-Complete control with access
-Reliable
-Flexible cloud hosting service
-Secure
-Ability to select the platform of your choice
-Inexpensive.
Lambda:
AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume.
With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.
NETFLIX is planning to use AWS Lambda to build rule-based self-managing infrastructure and replace inefficient processes to reduce the rate of errors and save valuable time.
- AMAZON S3 :
Amazon Simple Storage Service (S3) - It is unlimited storage "bucket". It is perfect place for storage of all the media files. All the video files and their multiple versions of Netflix are stored in S3.
It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.
Benifits-
- Industry-leading performance, scalability, availability, and durability.
- Wide range of cost-effective storage classes
- Unmatched security, compliance, and audit capabilities
- Easily manage data and access controls
- Query-in-place services for analytics
- Most supported cloud storage service.
- ELASTIC TRANSCODER:
Amazon Elastic Transcoder is media transcoding in the cloud. It is designed to be a highly scalable, easy to use and a cost effective way for developers and businesses to convert (or “transcode”) media files from their source format into versions that will playback on devices like smartphones, tablets and PCs.
This service is used by NETFLIX very extensively.It helps Netflix to convert a video into different formats of resolution and provide service to the user with respect to the device that is being used.
Amazon says that Elastic Transcoder is a one-stop shop. “There’s no need to administer software, scale hardware, tune performance, or otherwise manage transcoding infrastructure. You simply create a transcoding ‘job’ specifying the location of your source video and how you want it transcoded,” it writes on the site. “Amazon Elastic Transcoder also provides transcoding presets for popular output formats, which means that you don’t need to guess about which settings work best on particular devices. All these features are available via service APIs and the AWS Management Console.”
The analogy given by Kevin McEntee, Netflix Vice President says it all. The Ferrari(in house) approach, i.e using high speed expensive compute capacity fails to meet the performance standards; as compared to the more economical Toyota Priuses(cloud) which is ideal for bulk transcoding. Netflix’s Feb 2010 video player launch for Apple’s ipad was another success story because of the ease to scale up compute capacities.
- ELASTIC LOAD BALANCER:
Elastic Load Balancing distributes incoming application or network traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses, in multiple Availability Zones. Elastic Load Balancing scales your load balancer as traffic to your application changes over time. It can automatically scale to the vast majority of workloads.
Features of load balancer:
-Detection of unhealthy Elastic Compute Cloud (EC2) instances.
-Spreading instances across healthy channels only.
-Flexible cipher support.
-Centralized management of Secure -Sockets Layer (SSL) certificates.
-Optional public key authentication.
-Support for both IPv4 and IPv6.
-High availability
-Automatic scaling
ELB offers two different load balancer features as shown above, which help provide scalable cloud computing capacity. The Application Load Balancer handles advanced traffic routing from other services or containers at the application level, while the Classic Load Balancer spreads app or network traffic across EC2 instances.
- AMAZON MSK (KAFKA) :
Netflix embraces Apache Kafka® as the de-facto standard for its eventing, messaging, and stream processing needs. Kafka acts as a bridge for all point-to-point and Netflix Studio wide communications. It provides us with the high durability and linearly scalable, multi-tenant architecture required for operating systems at Netflix. Our in-house Kafka as a service offering provides fault tolerance, observability, multi-region deployments, and self-service. This makes it easier for our entire ecosystem of microservices to easily produce and consume meaningful events and unleash the power of asynchronous communication.
A typical message exchange within Netflix Studio ecosystem looks like this:
Producers ➝ Kafka ➝ Enrichment ➝ Kafka ➝ Consumers
We can break them up as three major sub-components.
With Amazon MSK, you can use native Apache Kafka APIs to populate data lakes, stream changes to and from databases, and power machine learning and analytics applications.
Benefit:
With a few clicks in the Amazon MSK console you can create highly available Apache Kafka clusters with settings and configuration based on Apache Kafka’s deployment best practices. Amazon MSK automatically provisions and runs your Apache Kafka clusters. Amazon MSK continuously monitors cluster health and automatically replaces unhealthy nodes with no downtime to your application. In addition, Amazon MSK secures your Apache Kafka cluster by encrypting data at rest.
- SECURITY( AWS IAM):
- CloudTrail security
Among the primary services offered on AWS is the CloudTrail logging service, which Netflix uses to gain insight into how things are running.
Bengston said CloudTrail can be used to track event history for AWS account activity. Netflix has CloudTrail configured to send logs to Amazon S3 storage buckets so further analysis can be performed.
Figuring out if something has been misused starts with a requirement of knowing what is in use. For Netflix, that means being aware of all the IP addresses that are in use, a challenging task given the scale and ephemeral nature of some of the cloud services.
By comparing each IP found in CloudTrail to the list of IPs that Netflix knows it has, it can spot potential anomalous behavior and credential misuse. While that approach can work, Bengston said Netflix is now taking additional steps to identify credential misuse.
One such step is to look through CloudTrail logs to identify the "GetCallerIdentity" function, which is similar to the Linux "whoami" command. Bengston said an attacker would use that function to know what account they are in. He added that Netflix's legitimate systems never need to use that function as the credentials and access are already known.
AWS Identity and Access Management (IAM) is a powerful service that allows you to securely configure access to AWS cloud resources. With over 2,500 permissions and counting, IAM gives users fine-grained control over which actions can be performed on a given resource in AWS. However, this level of control introduces complexity, which can make it more difficult for developers. Rather than focusing on getting their application to run correctly they have to switch context to work on knowing the exact AWS permissions the system needs. If they don’t grant necessary permissions, the application will fail. Overly permissive deployments reduce the chances of an application mysteriously breaking, but create unnecessary risk and provide attackers with a large foothold from which they may further penetrate a cloud environment.
Right-sizing permissions-
In an ideal world every application would be deployed with the exact permissions required. In practice, however, the effort required to determine the precise permissions required for each application in a complicated production environment is prohibitively expensive and doesn’t scale. At Netflix we’ve adopted an approach that we believe balances developer freedom and velocity and security best-practices: access profiling and automated and ongoing right-sizing. We allow developers to deploy their applications with a basic set of permissions and then use profiling data to remove permissions that are demonstrably not used. By continually re-examining our environment and removing unused permissions, our environment converges to least privilege over time.
Like this there are many more AWS services that netflix uses.
- MICROSERVICES:
Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. These services are owned by small, self-contained teams.
Microservices architectures make applications easier to scale and faster to develop, enabling innovation and accelerating time-to-market for new features.
Monolithic vs. Microservices Architecture
With monolithic architectures, all processes are tightly coupled and run as a single service. This means that if one process of the application experiences a spike in demand, the entire architecture must be scaled. Adding or improving a monolithic application’s features becomes more complex as the code base grows. This complexity limits experimentation and makes it difficult to implement new ideas. Monolithic architectures add risk for application availability because many dependent and tightly coupled processes increase the impact of a single process failure.
With a microservices architecture, an application is built as independent components that run each application process as a service. These services communicate via a well-defined interface using lightweight APIs. Services are built for business capabilities and each service performs a single function. Because they are independently run, each service can be updated, deployed, and scaled to meet demand for specific functions of an application.
And it is thoroughly used by NETFLIX.
With that example I would like to conclude the above topic.
CONCLUSION :
In 2016 the vice president of Netflix declared that now they are completely based on cloud , majorly AWS cloud and this is the reason behind the quality and quantity of content that they have been providing us for years now.
And after Netflix many more OTT platforms have come up to provide video contents which uses cloud in a very similar fashion and brings the best out of it.
.