쿠버네티스에 어플리케이션을 손쉽게 배포하는 방법은 무엇일까요? 복잡하게 배포된 어플리케이션의 파드들은 어떻게 디버깅하고 로깅해야 할까요? 또한 요즘 자주 이야기 되는 클라우드 네이티브 아키텍처로 설계된 어플리케이션은 어떻게 만들고 배포해야하는 걸까요?삼성전자 무선사업부에서 삼성헬스를 EKS 에 배포한 사례를 살펴보며, 이러한 문제를 어떻게 해결했는지 알아봅니다. 또한 복잡하게만 느껴졌던 쿠버네티스의 어플리케이션 배포와 클라우드 네이티브 아키텍처의 베스트 프렉티스를 EKS 에 어플리케이션을 배포하고, 관리하는 예제를 통하여 간편하게 이해할 수 있게 도와드립니다.
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키텍트, 이상호 삼성전자 Health서비스팀 선임
1.
2. DevOps 개발자가 되기 위한 쿠버
네티스 핵심 활용 예제 알아보기
정영준
솔루션즈 아키텍트
AWS
이상호
선임
삼성전자 Health서비스팀
3. Agenda
• Container Service Overview
• EKS Tenets and Core Concept
• Samsung Health Use Case
• EKS Best Practices & Sample Deploy Architecture
4.
5. We’re making AWS the best place to run contain
ers and Kubernetes
6. AWS container services landscape
Management
Deployment, Scheduling,
Scaling & Management of
containerized applications
Hosting
Where the containers run
Amazon Elastic
Container Service
Amazon Elastic
Kubernetes Service
Amazon EC2 AWS Fargate
Image Registry
Container Image Repository
Amazon Elastic
Container Registry
7. Balancing flexibility and simplicity:
Workload-by-workload
Flexibility focused
Low level of opinion
Low level of abstraction
Focus on infrastructure
and configuration
Installing, configuring, and
managing my compute environment
is critical to achieving my goals
Value simplicity
High level of opinion
High level of abstraction
Focus only on app
and primitive
Having a standardized and
on-demand compute environment
is critical to achieving my goals
8. We give you the power to choose
Amazon ECS Amazon EKS
Amazon
EC2
AWS
Fargate
Amazon
EC2
AWS
Fargate
1. Choose your
orchestration tool
2. Choose your
launch type
We’re
working
on it #32
9. EKS tenets
1. Amazon EKS is a platform to run production-grade workloads. Security and reliability are
our first priority. After that we focus on doing the heavy lifting for you in the control
plane, including life cycle-related things like version upgrades.
2. Amazon EKS provides a native and upstream Kubernetes experience. Amazon EKS
provides vanilla, un-forked Kubernetes. In keeping with our first tenant, we ensure the
Kubernetes versions we run have security-related patches, even for older, supported
versions as quickly as possible. But there’s no special sauce and no lock in.
3. If you want to use additional AWS services, integrations are as seamless as possible.
4. The Amazon EKS team in AWS actively contributes to the upstream Kubernetes project
and the wider CNCF activities, both on the technical level as well as community, from
communicating good practices to participation in SIGs and working groups.
10. How are customer using Amazon EKS?
Microservices
PaaS
Platform as a service Enterprise App
Migration
Machine Learning
13. VPC
Kubernetes control plane
Highly available and single
tenant infrastructure
All “native AWS” components
Fronted by a Network Load
Balancer
NLB
Amazon
EKS
Availability Zone 1 Availability Zone 2 Availability Zone 3
Etcd
API Servers
15. eksctl - a CLI for Amazon EKS
• Single command cluster creation
eksctl create cluster --nodes=4
• Open source and on GitHub
• Built by Weave and AWS
• Official Amazon EKS CLI
16. Standard EC2 compute instance types
P and G type accelerated instances
i3 bare metal
Spot Instances
Bring your own instances
Instance flexibility
17. Bring your own OS
Amazon EKS AMI build scripts
https://github.com/awslabs/amazon-eks-ami
Amazon
Amazon
Amazon
18. Windows containers
Run Windows containers and Windows Server nodes with Amazon EKS
Supports heterogeneous (mixed) clusters.
Kubernetes version 1.11+
Available in all Amazon EKS Regions
Developer preview:
https://github.com/aws/containers-roadmap
21. Support Kubernetes version
Latest Kubernetes: 1.14
Amazon EKS will support up to three versions of Kubernetes at once
Deprecation in line with the community stopping support for older
versions
Version 1.11 deprecation on Nov 4th, 2019
22. Amazon EKS platform version
Platform version revisions represent API server configuration
changes or Kubernetes patches
Platform versions increment within a Kubernetes version only
23. Amazon EKS update life cycle
May 21, 2019
Blog: https://aws.amazon.com/blogs/compute/updates-to-amazon-eks-version-lifecycle/
24. Amazon EKS support sophisticated and scalable infrastructure
[mycluster].eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl
VPC
Instance
Auto Scaling group for m4.large Spot Instances
Auto Scaling group for t2.medium Spot Instances
Auto Scaling group for On-Demand Instances
Cluster Autoscaler Daemonset
Spot Interruption handler
Daemonset
https://eksworkshop.com/spot/
managespot/deployhandler
33. Reverse Proxy with ES
in case of vpc access
NGINXNGINX
KIBANAKIBANA
COGNITOCOGNITO
② if no authentication then 302 redirect_to_cognito
?redirect_uri={vpc_kibana_domain}
① kibana access via nginx
③ sign up then redirect to kibana
with redirect_uri
proxy_redirect https://{cognito_host} https://$host;
proxy_redirect https://{kibana_host} https://$host;
user
40. Challenges in Using & Deploying Containers
As cloud native technologies change the way companies are designing an
d building applications, challenges are inevitable. The top challenges that
respondents face are:
• Cultural Changes with Development Team (41%)
• Complexity (40% up from 35%)
• Lack of Training (40%)
• Security (38% down from 43%)
• Monitoring (34% down from 38%)
• Storage (30% down from 41%)
• Networking (30% down from 38%)
https://www.cncf.io/blog/2018/08/29/cncf-survey-use-of-cloud-native-technologies-in-production-has-grown-over-200-percent/
41.
42. Container security onion model: Defense in depth
• full blown distro (Ubuntu, AL) vs. minimal
environment (container-optimized
distribution)
• multi-tenancy requirements
• gotchas: Linux packages/CVEs,
leaks, GDPR (in Europe)
• runtime/standards (OCI)
• immutability of images
• all containers share a kernel (mitigation: Firecracker)
• gotchas: unnecessary privileged users, no scans, trust
• code analysis
• source available?
• gotchas: big surface,
many languages
{}
}
• sanitizing user input
• static code analysis
• gotchas: log-leaking }
• sensitive config (passwords,
API keys, etc.)
• gotchas: commits-to-source,
non-separated access (dev has
cleartext password)
{
• business core data
• Personal Identifiable
Information (PII)
• gotchas: leaks, GDPR
(in Europe)
{
host
container
dependencies
code
config
user data
45. API-server endpoint access control
Worker VPC (your account)
Kubectl
Master VPC (AWS account)
etcd
AZ 1
API Server
etcd
API Server
prod-cluster-123.eks.amazonaws.com
EKS-owned ENI
Kubelet
AZ 1
Worker
node
EKS-owned ENI
Kubelet
AZ 2
Worker
node
Public == true
AZ 2
Kube-proxy Kube-proxy
46. API-server endpoint access control
Worker VPC (your account)
Kubectl
Master VPC (AWS account)
etcd
AZ 1
AZ 2
API Server
etcd
API Server
prod-cluster-123.eks.amazonaws.com
EKS-owned ENIs
Public == true
Private == true
prod-cluster-123.eks.amazonaws.com
Private hosted zone
Kubelet
AZ 1
Worker
node
Kube-proxy
Kubelet
AZ 2
Worker
node
Kube-proxy
47. API-server endpoint access control
Worker VPC (your account)
Kubectl
Master VPC (AWS account)
etcd
AZ 1 AZ 2
API Server
etcd
API Server
EKS-owned ENIs
Public == false
Private == true
prod-cluster-123.eks.amazonaws.com
Private hosted zone
Kubelet
AZ 1
Worker
node
Kube-proxy
Kubelet
AZ 2
Worker
node
Kube-proxy
48. AWS Identity and Access Management (IAM) Aut
hentication
Kubectl
3) Authorizes AWS identity with RBAC
K8s API
1) Passes AWS identity
2) Verifies AWS identity
4) K8s action
allowed/denied
49. Great integration responsibility – IAM roles
Frontend pod
Node
Backend pod
Amazon S3
UI customization
bucket
Billing reports
bucket
Role
IAM
Log DaemonSet
Kinesis
Kinesis Data
Streams
ElastiCache
ElastiCache for
Redis
50. Great integration responsibility – IAM roles
Frontend pod
Node
Backend pod
Amazon S3
UI customization
bucket
Billing reports
bucket
IAM Kinesis
Kinesis Data
Streams
Log DaemonSet
ElastiCache
ElastiCache for
RedisCredential
Credential
Credential
52. Service type LoadBalancer
Service
(LoadBalancer)
ELB Users
Pod selector
(type = nodejs)
Frontend pod 1
Node
NodePort
Frontend pod 2
Node
NodePort
Frontend pod 3
Node
NodePort
Backend pod 1
Node
NodePort
Generic pod selector
53. Great responsibility – Service type LoadBalancer
Service
(LoadBalancer)
ELB Users
Pod selector
(app = frontend)
Frontend pod 1
Node
NodePort
Frontend pod 2
Node
NodePort
Frontend pod 3
Node
NodePort
Backend pod 1
Node
NodePortService
(LoadBalancer)
Pod selector
(app = backend)
ELB Users
54.
55. Amazon CloudWatch Container Insights
Gives you complete visibility into your cloud resources and applications
so you can monitor, troubleshoot, and remediate issues
Collect logs &
metrics
Monitor,
troubleshoot &
set alarms
Act AnalyzeAmazon
CloudWatch
56. Collects, aggregates, and summarizes
Reliable, secure metrics and logs collection
Automated dashboards and analysis
Observability experience across metrics, logs, traces
Ad hoc analytics
CloudWatch Container Insights
A fully managed observability service for monitoring, troubleshooting
, and alarming on your containerized applications and microservices
58. Container Insights available now
1. Fully managed, AWS native observability service providing autom
ated summary and analysis of compute capacity
2. Reliable and secure collection of application logs with built-in an
alytics capabilities
3. Prebuilt visualization to summarize cluster and node errors
4. Application & microservice tracing - Troubleshoot and debug ap
plication & microservice
59.
60. Container storage interface (CSI)
A flexible standard for orchestration
and storage provider connections
We support the CSI standard through following drivers:
Amazon Elastic Block Store: Amazon EBS CSI Driver
Amazon Elastic File System: Amazon EFS CSI Driver
Amazon FSx for Lustre: Amazon FSx CSI Driver
61. Storage volume lifecycle
Provisioning Binding Using Reclaiming
• Static
• Dynamic*
• Control loop watches
for PVC requests and
satisfies if PV is
available.
• For Dynamic, PVC will
provision PV
• PVC to PV binding is
one-to-one mapping
• Cluster mounts
volume based on
PVC
• Retain (default)
• Recycle
• Delete
62. What if I need specific volume type?
StorageClass
gp2 io1 sc1 encrypted
io1
st1
1) Admin pre-provisions
StorageClass based
on workload needs
2) End user requests for
specific volume types
(e.g., encrypted io1
volume)
3) Control loop watches
PVC request and
allocates volume if PV
exists
MySQL Pods
4) User creates stateful
workload
65. Load balancing
All three Elastic Load Balancing products are supported
NLB and CLB supported by Kubernetes Service
type=LoadBalancer
Internal and External Load Balancer support
https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/
66. • Exposes the service externally using a cloud
provider’s load balancer
• NodePort and ClusterIP services (to which LB
will route) automatically created
• Each service exposed with a LoadBalancer (ELB
or NLB) will get its own IP address
• Exposes L4 (TCP) or L7 (HTTP) services
Kubernetes ServiceType: LoadBalancer
67. Load balancing
Want to use an Internal Load Balancer? Use annotation:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
Want to use an NLB? Use annotation:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
69. Service load balancer: Network Load Balancer (NLB)
• NLB supports forwarding the client’s IP through to the node
.spec.externalTrafficPolicy = Local client ip passed to pod
• Nodes with no matching pods will be removed by specified NLB’s health check
.spec.healthCheckNodePort
• Use DaemonSet or pod anti-affinity to verify even traffic split
70. • exposes HTTP/HTTPS routes
to services within the cluster
• Many implementations: ALB,
NGINX, F5, HAProxy etc.
• Default service type: ClusterIP
Kubernetes Ingress object
71. ALB Ingress controller
AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort
https://docs.aws.amazon.com/en_pv/eks/latest/userguide/alb-ingress.html
74. Implementing logging with EFK
fluentd is an open source
data collector providing a
unified logging layer
elasticsearch is a
distributed, RESTful search
and analytics engine
kibana lets you visualize
your Elasticsearch data
76. Logging with FluentBit
• New AWS FluentBit container
plugin
• Optimize costs. Route logs fr
om Amazon EKS and Amazon
ECS clusters directly to Amaz
on S3 and query with Amazo
n Athena
• Open source
• More resource-efficient than
Fluentd. Tests show Fluentd u
ses 4x more CPU and 6x more
memory
https://aws.amazon.com/blogs/opensource/centra
lized-container-logging-fluent-bit/
77. Logging performance compare
Log Lines Per second Data Out Fluentd CPU Fluent Bit CPU Fluentd Memory Fluent Bit Memory
100 25 KB/s 0.013 vCPU 0.003 vCPU 146 MB 27 MB
1000 250 KB/s 0.103 vCPU 0.03 vCPU 303 MB 44 MB
10000 2.5 MB/s 1.03 vCPU 0.19 vCPU 376 MB 65 MB
Log Lines Per second Data Out Fluentd CPU Fluent Bit CPU Fluentd Memory Fluent Bit Memory
100 25 KB/s 0.006 vCPU 0.003 vCPU 84 MB 27 MB
1000 250 KB/s 0.073 vCPU 0.033 vCPU 102 MB 37 MB
10000 2.5 MB/s 0.86 vCPU 0.13 vCPU 438 MB 55 MB
78.
79. Why AWS App Mesh?
http/tcp
Service
team A
Service
team B
Common need: Manage interservice traffic
• How to observe (logs, metrics, traces)
• How to load balance E/W traffic
• How to shift traffic between deployments
• How to decouple service teams
• How to minimize impact to app code
80. App Mesh uses Envoy proxy
OSS community project
Wide community support, numerous integrations
Stable and production-proven
Graduated Project in Cloud Native Computing Foundation
Started at Lyft in 2016
81. Why App Mesh?
HTTP / TCP
Service
team A
Service
team B
Control plane
Translates logical intent to proxy config
Distributes proxy config
Proxy
Sits between all services
Manages and observes traffic
Control plane
82. App Mesh: App-level communication across AWS
Amazon ECS
AWS Fargate
Amazon EKS
Amazon EC2
AWS App Mesh
Kubernetes on EC2
83. App Mesh: Application observability
Logging
HTTP access logging
Amazon CloudWatch Logs
Available as container logs on
Amazon ECS, Amazon EKS, AWS Fargate
Metrics
CloudWatch metrics
StatsD (with tags)
Prometheus
Tracing
AWS X-Ray
Other Envoy tracing drivers
89. Adding business data (Node.js)
//Example showing how to add business data to traces
app.use(function(req, res, next){
if (req.session !== undefined) {
let segment = AWSXRay.getSegment()
// User sessionID as userID
segment.addAnnotation(‘userID', req.sessionID);
}
next();
})
//Example showing how to add business data to traces
app.use(function(req, res, next){
if (req.session !== undefined) {
let segment = AWSXRay.getSegment()
// User sessionID as userID
segment.addAnnotation(‘userID', req.sessionID);
}
next();
})
95. To conclude…
Clearly understand managed systems’ ownership boundaries and your sys
tems to build the own K8s Platform.
“With great power
comes great responsibility.”