Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Paris Container Day 2016 : Deep dive dc-os ci-cd (Mesosphere & Container Solutions)
Next
Download to read offline and view in fullscreen.

Share

Highly efficient container orchestration and continuous delivery with DC/OS and Jenkins presentation

Download to read offline

Continuous delivery is all the rage these days, but without self-healing, highly available, and fault-tolerant infrastructure to deploy your applications to, it’s really only one piece of a much larger picture. Apache Mesos was born at UC Berkeley and grew into a robust, highly scalable cluster orchestrator while running thousands of nodes at Twitter. Support for Docker containers was added in 2013, and since then, it’s been adopted by companies like Netflix and Apple to run their critical infrastructure. Mesosphere has built the open source Datacenter Operating System (DC/OS) around Apache Mesos to provide all the supplementary tooling necessary to take Mesos to a production environment. Jenkins with DC/OS allows you to spin up build agents dynamically, an approach which has allowed companies like PayPal to cut the footprint of their build farms by hundreds of nodes, saving money on infrastructure by increasing utilization and reducing obstacles to providing teams with the resources they need when they need them.

Sunil Shah and Roger Ignazio introduce DC/OS and demonstrate how to integrate it with the stalwart continuous integration server Jenkins, allowing you to set up a continuous delivery pipeline that takes an application composed of microservices from code repository to Docker Hub to a staging or production server with seamless automation. Sunil and Roger walk attendees through setting up their own pipeline using Jenkins on a DC/OS cluster, from installation and configuration of Jenkins to setting up a build to actually deploying it to a live environment where it can serve traffic, and also cover the internals of practical microservice architecture, including component-level deployment, application-level persistence, and intraprocess communication via service discovery.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Highly efficient container orchestration and continuous delivery with DC/OS and Jenkins presentation

  1. 1. © 2016 Mesosphere, Inc. All Rights Reserved. VELOCITY NYC | SEPTEMBER 20, 2016 CONTINUOUS DELIVERY WITH DC/OS AND JENKINS
  2. 2. © 2016 Mesosphere, Inc. All Rights Reserved. SUNIL SHAH Product Manager at Mesosphere @ssk2 ROGER IGNAZIO Tech Lead at Mesosphere @rogerignazio WHO WE ARE
  3. 3. © 2016 Mesosphere, Inc. All Rights Reserved. mesosinaction.com Use the code vecymes for 42% off! BUY MY BOOK!
  4. 4. © 2016 Mesosphere, Inc. All Rights Reserved. AGENDA Lecture ● Introduction to Apache Mesos and DC/OS ● Components that make up modern infra. ● Running Jenkins as a service on DC/OS ● Continuously deploying applications to DC/OS Demos & Lab ● Installing and configuring Jenkins ● Installing and configuring a load balancer ● Creating a new CI/CD pipeline ● Putting it all together (CD in practice)
  5. 5. © 2016 Mesosphere, Inc. All Rights Reserved. DEVELOPER AGILITY, DEFINED
  6. 6. © 2016 Mesosphere, Inc. All Rights Reserved. Developer agility empowers developers to ● ship their apps to production ● leverage the power of Mesos and DC/OS ● fix bugs rapidly without downtime! DEVELOPER AGILITY, DEFINED
  7. 7. © 2016 Mesosphere, Inc. All Rights Reserved. Continuous integration is not the same as continuous delivery or continuous deployment. DEVELOPER AGILITY, DEFINED
  8. 8. © 2016 Mesosphere, Inc. All Rights Reserved. An artifact repo stores built binaries, whereas an image registry stores templates for deployment. DEVELOPER AGILITY, DEFINED
  9. 9. © 2016 Mesosphere, Inc. All Rights Reserved. DEVELOPER AGILITY, DEFINED
  10. 10. © 2016 Mesosphere, Inc. All Rights Reserved. INTRO TO APACHE MESOS AND DC/OS
  11. 11. © 2016 Mesosphere, Inc. All Rights Reserved. A QUICK PRIMER ON CONTAINERS Physical Server Virtual Machine–Based Application Deployment Container–Based Application Deployment Hypervisor OS VM VM OS App App Physical Server Hypervisor OS VM VM OS App App Physical Server Operating System App App Physical Server Operating System App App Mesos Isolate apps by running multiple VMs per physical server; still need to manage each guest OS! Isolate apps using features of the host OS, such as Linux cgroups.
  12. 12. © 2016 Mesosphere, Inc. All Rights Reserved. A QUICK PRIMER ON CONTAINERS Linux cgroup Dependencies Linux Host OS Application Docker Container Dependencies Host OS Application Linux cgroupsDocker Containers VM Dependencies Hypervisor Application Virtual Machines Guest OS Docker Engine
  13. 13. © 2016 Mesosphere, Inc. All Rights Reserved. A BIT OF CLARIFICATION https://mesos.apache.org https://dcos.io
  14. 14. © 2016 Mesosphere, Inc. All Rights Reserved. ● General purpose cluster resource manager ● Represents many machines as a single entity ● Advertises resources directly to frameworks ● Works at scale: Apple, Twitter, Airbnb, Netflix, … WHAT IS MESOS?
  15. 15. © 2016 Mesosphere, Inc. All Rights Reserved. ● Two-tier scheduling across resource types ● cpus, mem, disk, and ports by default ● Masters are highly available, agents are fault tolerant ● Checkpointing, agent recovery ● Resource isolation between processes ● Linux cgroups, Docker, … ● Language bindings: C++, Java, Python, Go, … WHAT IS MESOS? (CONTINUED)
  16. 16. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS ARCHITECTURE Launch task(s) Mesos Master Quorum Mesos Master (Standby) ZK Mesos Master (Standby) ZK Mesos Master (Leader) ZK Framework B Scheduler Agent 1 Framework A Executor Task(s) Agent n Framework A Executor Task(s) Framework B Executor Task(s) Resource offer Resource offer Framework A Scheduler Launch task(s)
  17. 17. © 2016 Mesosphere, Inc. All Rights Reserved. ANATOMY OF A RESOURCE OFFER (TWO-TIER SCHEDULING) Mesos Master(s) Mesos Agent 16 CPUs 128 GB RAM 1 TB disk Mesos Agent 16 CPUs 128 GB RAM 1 TB disk Mesos Agent 16 CPUs 128 GB RAM 1 TB disk Master offers resources to scheduler Scheduler Executor Available compute resources Resource offer accepted, launch executors/tasks Tasks Executor Tasks Executor Tasks Scheduler accepts or declines an offer
  18. 18. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB NEW (OLD) PROBLEMS
  19. 19. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB ● Monitoring and metrics collection ● Collectd, Nagios, Prometheus, Snap NEW (OLD) PROBLEMS
  20. 20. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB ● Monitoring and metrics collection ● Collectd, Nagios, Prometheus, Snap ● Persistent storage (filesystems, databases, etc) ● Ceph, HDFS, Amazon EBS / EFS / S3, NFS, Cassandra NEW (OLD) PROBLEMS
  21. 21. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB ● Monitoring and metrics collection ● Collectd, Nagios, Prometheus, Snap ● Persistent storage (filesystems, databases, etc) ● Ceph, HDFS, Amazon EBS / EFS / S3, NFS, Cassandra ● Administration: named URIs vs. ports, IPAM ● Nginx, HAProxy, Mesos-DNS, dhcpd, Minuteman NEW (OLD) PROBLEMS
  22. 22. © 2016 Mesosphere, Inc. All Rights Reserved. https://dcos.io https://github.com/dcos DC/OS: BUILT ON MESOS
  23. 23. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS: BUILT ON MESOS
  24. 24. © 2016 Mesosphere, Inc. All Rights Reserved. All of the benefits of Mesos, plus ● Built-in service discovery and load balancing ● Support for stateful services ● Turn-key installation of distributed systems ● Cloud-agnostic installer ● Web and command-line interfaces ● All components are integration tested and supported by Mesosphere, Inc. MESOS AND DC/OS: BETTER TOGETHER
  25. 25. © 2016 Mesosphere, Inc. All Rights Reserved. JENKINS ON DC/OS
  26. 26. © 2016 Mesosphere, Inc. All Rights Reserved. MULTIPLE JENKINS MASTERS Team A Jenkins . . . Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team B Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team X Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N
  27. 27. © 2016 Mesosphere, Inc. All Rights Reserved. MULTIPLE JENKINS MASTERS → STATIC PARTITIONING Team A Jenkins . . . Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team B Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team X Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N 90% Utilized (Normal) 140% Utilized (40 builds in queue) 0% Utilized (Idle)
  28. 28. © 2016 Mesosphere, Inc. All Rights Reserved. RUNNING JENKINS ON DC/OS → FAIR SHARING Team A Jenkins . . . DC/OS Agent 1 DC/OS Agent 4 DC/OS Agent N Team B Jenkins DC/OS Agent 2 DC/OS Agent 5 Team X Jenkins DC/OS Agent 3 DC/OS Agent 6 . . . DC/OS
  29. 29. © 2016 Mesosphere, Inc. All Rights Reserved. CONTINUOUSLY DEPLOYING APPLICATIONS TO DC/OS
  30. 30. © 2016 Mesosphere, Inc. All Rights Reserved. TRADITIONAL RELEASE PROCESS ● Planning & implementing new technologies ● Waiting for people & infrastructure ● Building environment specific CI/CD for each project ● Moving apps from dev to staging to prod DEV(OPS) TEAMS SPEND SIGNIFICANT TIME AND EFFORT ON: PROJECT PLANNING WEEK 1 WEEK 2 WEEK 3 WEEKS 4 TO 8 WEEK 9 CUSTOMER FEEDBACKCONFIGURATIONRESOURCE ALLOCATION Dev Test Stage Release
  31. 31. © 2016 Mesosphere, Inc. All Rights Reserved. MODERN RELEASE PROCESS WEEK 1 WEEK 2 WEEK 3 WEEK 4 PROJECT PLANNING CUSTOMER FEEDBACK CUSTOMER FEEDBACK CUSTOMER FEEDBACK RELEASE 1 RELEASE 2 RELEASE 3 RELEASE 4 Better products through a repeatable release cadence Happier customers through continuous feedback Dev Test Stage Release Dev Test Stage Release Dev Test Stage Release Dev Test Stage Release
  32. 32. © 2016 Mesosphere, Inc. All Rights Reserved. ● Scheduling — advertising available compute resources ● Deployments — getting an application onto a node ● Health checks — ensuring the app/service is healthy ● Service discovery — connecting to dependent services ● Persistence — running stateful services in containers DEPLOYING APPLICATIONS: BASIC REQUIREMENTS
  33. 33. © 2016 Mesosphere, Inc. All Rights Reserved. A sysadmin provisions one or more physical/virtual servers to host the app Mesos resource offers (two-tier scheduling) offers available resources directly to frameworks DEPLOYING APPLICATIONS: SCHEDULING Before DC/OS With DC/OS
  34. 34. © 2016 Mesosphere, Inc. All Rights Reserved. By hand or using Puppet / Chef / Ansible Jenkins SSHing to the machine and running a shell script Note: all dependencies must also be present! Marathon deploys containers, ideally using a CI/CD tool to create/update app definitions Docker containers packages app and dependencies DEPLOYING APPLICATIONS: DEPLOYMENTS Before DC/OS With DC/OS
  35. 35. © 2016 Mesosphere, Inc. All Rights Reserved. Nagios pages a sysadmin Marathon performs health checks, restarts unhealthy/failed instances DEPLOYING APPLICATIONS: HEALTH CHECKS Before DC/OS With DC/OS
  36. 36. © 2016 Mesosphere, Inc. All Rights Reserved. Static hostnames / IP addresses in a spreadsheet or config management A sysadmin configures a load balancer manually or with Puppet / Chef / Ansible Mesos-DNS provides DNS resolution for running services (hostname / IP address, ports, etc) Load balancer configs built dynamically using cluster state DEPLOYING APPLICATIONS: SERVICE DISCOVERY Before DC/OS With DC/OS
  37. 37. © 2016 Mesosphere, Inc. All Rights Reserved. Individual servers with RAID 1/5/6/10, expensive SANs, NFS, etc. Dedicated, statically partitioned Ceph or Gluster storage clusters Mesos external/persistent volumes (REX-Ray), HDFS, etc. Self-healing Ceph or Gluster on Mesos / DC/OS DEPLOYING APPLICATIONS: PERSISTENCE Before DC/OS With DC/OS
  38. 38. © 2016 Mesosphere, Inc. All Rights Reserved. Q: Given that we are all engineers, which tools would we like to work with?
  39. 39. © 2016 Mesosphere, Inc. All Rights Reserved. DEMOS & LAB
  40. 40. © 2016 Mesosphere, Inc. All Rights Reserved. 40 PIPELINE COMPONENTS Continuous Integration Container Registry Container Orchestrator Version Control System Load Balancer Production Environment Continuous Delivery Pipeline git push DC/OS
  41. 41. © 2016 Mesosphere, Inc. All Rights Reserved. 41 PIPELINE COMPONENTS Jenkins DockerHub MarathonGitHub Marathon-lbMesos Continuous Delivery Pipeline git push DC/OS
  42. 42. © 2016 Mesosphere, Inc. All Rights Reserved. 42 PIPELINE CONFIGURATION Jenkinsfile Dockerfile marathon.json Continuous Delivery Pipeline git push DC/OS
  43. 43. © 2016 Mesosphere, Inc. All Rights Reserved. A SNEAK PREVIEW
  44. 44. © 2016 Mesosphere, Inc. All Rights Reserved. ● Head over to github.com/mesosphere/vny ● Follow the exercises! ● Use the username/password: velocityuser/velocitypassword YOUR TURN
  45. 45. © 2016 Mesosphere, Inc. All Rights Reserved. Roger Ignazio roger@mesosphere.com @rogerignazio Sunil Shah sunil@mesosphere.com @ssk2 THANK YOU! Learn more by visiting DCOS.io and Mesosphere.com

Continuous delivery is all the rage these days, but without self-healing, highly available, and fault-tolerant infrastructure to deploy your applications to, it’s really only one piece of a much larger picture. Apache Mesos was born at UC Berkeley and grew into a robust, highly scalable cluster orchestrator while running thousands of nodes at Twitter. Support for Docker containers was added in 2013, and since then, it’s been adopted by companies like Netflix and Apple to run their critical infrastructure. Mesosphere has built the open source Datacenter Operating System (DC/OS) around Apache Mesos to provide all the supplementary tooling necessary to take Mesos to a production environment. Jenkins with DC/OS allows you to spin up build agents dynamically, an approach which has allowed companies like PayPal to cut the footprint of their build farms by hundreds of nodes, saving money on infrastructure by increasing utilization and reducing obstacles to providing teams with the resources they need when they need them. Sunil Shah and Roger Ignazio introduce DC/OS and demonstrate how to integrate it with the stalwart continuous integration server Jenkins, allowing you to set up a continuous delivery pipeline that takes an application composed of microservices from code repository to Docker Hub to a staging or production server with seamless automation. Sunil and Roger walk attendees through setting up their own pipeline using Jenkins on a DC/OS cluster, from installation and configuration of Jenkins to setting up a build to actually deploying it to a live environment where it can serve traffic, and also cover the internals of practical microservice architecture, including component-level deployment, application-level persistence, and intraprocess communication via service discovery.

Views

Total views

240

On Slideshare

0

From embeds

0

Number of embeds

6

Actions

Downloads

16

Shares

0

Comments

0

Likes

0

×