A deep dive of the metrics exposed in Kubernetes.

1. Reveal Your Deepest
Kubernetes Secrets Metrics
Kubernetes Colorado - 10/25/2017
Bob Cotton - FreshTracks.io

2. About Me
● Co-Founder - FreshTracks.io - A CA Accelerator Incubation
● bob@freshtracks.io
● @bob_cotton

3. Agenda
● Short overview of Prometheus
● Sources of metrics
○ Node
○ kubelet and containers
○ Kubernetes API
○ etcd
○ Derived metrics (kube-state-metrics)
● The new K8s metrics server
● Horizontal pod auto-scaler
● K8s cluster hierarchies and metrics aggregation

4. Prometheus

5. Prometheus - A Short Overview
● Based on monitoring patterns at Google
● Open sourced by SoundCloud in early 2015
● Second project admitted to the Cloud Native Computing Foundation after Kubernetes
● Adoption surge is tracking Kubernetes
○ 63% of teams using Kubernetes use Prometheus
● Metric discovery and collection
● High-performance timeseries database
● Alert definition and generation

6. Prometheus - Unique Features
● Supports “Metrics 2.0” - i.e. metrics contain any number of label/values
● “Pull based” metrics collection
● Service discovery mechanism
● Rich set of “exporters”
● Extremely high-performance TSDB
● PromQL
● Alert Manager
● Easily installable from Helm
● Grafana typically used for visualization

7. Labeled Series - aka Metrics 2.0
APIServiceRegistrationController_adds{
instance="172.20.107.203:443",
job="kubernetes-apiservers"} 1
apiserver_request_count{
client="kubectl/v1.7.5 (darwin/amd64)kubernetes/17d7182",
code="200",
contentType="application/json",
instance="172.20.78.248:443",
job="kubernetes-apiservers",
resource="pods",
verb="GET"} 20
apiserver_request_count{
client="kubectl/v1.7.5 (darwin/amd64) kubernetes/17d7182",
code="200",
contentType="application/json",
instance="172.20.78.248:443",
job="kubernetes-apiservers",
resource="pods",
verb="LIST"} 30
172_20_78_248.apiserver.pods.get.request_count = 20

8. Prometheus
K8s API Server
TSDB
Kublet
(cAdvisor)
node-exporter
kube_state_metrics
App containers
other exporters
node_exporter
App containers
Kublet
(cAdvisor)
Service Discovery

9. Prometheus Exposition Format and Exporters
● The Prometheus exposition format - Text over http
○ Efforts to make it a standard
● Supported by Sysdig and the TICK collector
● Close to 100 exporters for various technologies
● The jmx_exporter can cover any Java/JMX application
● https://prometheus.io/docs/instrumenting/exporters/
Demo
● “Official Exporters”
○ node_exporter
○ jmx_exporter
○ snmp_exporter
○ haproxy_exporter
○ cloudwatch_exporter
○ collectd_exporter
○ mysql_exporter
○ memcached_exporter

10. Sources of Metrics in Kubernetes

11. Host Metrics from the node_exporter
● Standard Host Metrics
○ Load Average
○ CPU
○ Memory
○ Disk
○ Network
○ Many others
● ~1000 Unique series in a typical node
Demo
Node
node_exporter
/metrics

12. Container Metrics from cAdvisor
● cAdvisor is embedded into the kubelet, so we
scrape the kubelet to get container metrics
● These are the so-called “core” metrics
● For each container on the node:
○ CPU Usage (user and system) and time throttled
○ Filesystem read/writes/limits
○ Memory usage and limits
○ Network transmit/receive/dropped
Demo
Node
node_exporter
/metrics
kubelet
cAdvisor

13. Kubernetes Metrics from the K8s API Server
● Metrics about the performance of the K8s API Server
○ Performance of controller work queues
○ Request Rates and Latencies
○ Etcd helper cache work queues and cache performance
○ General process status (File Descriptors/Memory/CPU
Seconds)
○ Golang status (GC/Memory/Threads)
Node
node_exporter
kubelet
cAdvisor
Any other Pod
/metrics
API Server

14. Etcd Metrics from etcd
● Etcd is “master of all truth” within a K8s cluster
○ Leader existence and leader change rate
○ Proposals committed/applied/pending/failed
○ Disk write performance
○ Network and gRPC counters

15. K8s Derived Metrics from kube-state-metrics
● Counts and meta-data about many K8s types
○ Counts of many “nouns”
○ Resource Limits
○ Container states
■ ready/restarts/running/terminated/waiting
○ _labels series just carries labels from Pods
● cronjob
● daemonset
● deployment
● horizontalpodautoscaler
● job
● limitrange
● namespace
● node
● persistentvolumeclaim
● pod
● replicaset
● replicationcontroller
● resourcequota
● service
● statefulset

16. Sources of Metrics in Kubernetes
● Node via the node_exporter
● Container metrics via the kubelet and cAdvisor
● Kubernetes API server
● etcd
● Derived metrics via kube-state-metrics

17. Scheduling and Autoscaling
i.e. The Metrics Pipeline

18. The New “Metrics Server”
● Replaces Heapster
● Standard (versioned and auth) API aggregated into the K8s API Server
● In “beta” in K8s 1.8
● Used by the scheduler and (eventually) the Horizontal Pod Autoscaler
● A stripped-down version of Heapster
● Reports on “core” metrics (CPU/Memory/Network) gathered from cAdvisor
● For internal to K8s use only.
● Pluggable for custom metrics

20. Feeding the Horizontal Pod Autoscaler
● Before the metrics server the HPA utilized Heapster for it’s Core metrics
○ This will be the metrics-server going forward
● API Adapter will bridge to third party monitoring system
○ e.g. Prometheus

21. Kubernetes
Hierarchy and Aggregation

22. Core Metrics Aggregation
● K8s clusters form a hierarchy
● We can aggregate the “core” metrics to any level
● This allows for some interesting monitoring
opportunities
○ Using P8s “recording rules” aggregate the core metrics
at every level
○ Insights into all levels of your Kubernetes cluster
● This also applies to any custom application metric
Demo
Namespace
Deployment
Pod
Container

23. FreshTracks.io is Hiring!
https://searchjobs.ca.com/go/Accelerator/3279000/

24. Questions?

25. Resources
● Prometheus.io
● Core Metrics in Kubelet
● Kubernetes monitoring architecture
● What is the new metrics-server?