Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Empowering developers to deploy their
own data stores.
A story of Terraform, Puppet and rage
Tomas Doran
@bobtfish
• Iterate on the things you do often

• Hide complexity

• Empower others
2
Devops = Workflow
• A thing of the past (mostly)
• Need to be able to scale up and down in hours
• If not minutes

• Need to allow people to...
• ‘Infra’ layer
• DNS / puppet / apt - basic services
• A(WS)?nycast - failover / HA

• ‘App’ layer
• Smartstack - Service...
• Remembering the . on PTR records

• For some people!
• Why make them do this?
5
The hardest thing
• Datastore PAAS
• Elasticsearch clusters are the ‘easy’ case

• No ‘master’ - all machines are equal
• Automatic sharding...
• curl http://10.29.0.3:8142 (A(WS)nycast puppetmaster)

{
“habitat”: “uswest1aprod”
}

• “habitat”, “region”, “superregio...
• curl http://10.29.0.3:8142 (A(WS)nycast puppetmaster)

{
“habitat”: “uswest1aprod”
}

• “habitat”, “region”, “superregio...
• Hostname: search1-reviews-uswest1aprod
• Parse out cluster name

elasticsearch_cluster { ‘reviews’: }

puppet/modules/el...
• External Node Classifer
• Puppetmaster calls a script, returns node definition
• Create node definition from EC2 tags

p...
• Bad abstraction for contextual information
• Which db server is the master? Does it have ‘master’ in it’s FQDN?
• If it ...
• Got most of the pieces
• Machines auto-configure themselves after launch.
• Remaining step is actually launching machine...
13
14
15
• Terraform the most generic abstraction possible
• Map JSON (HCL) DSL => CRUD APIs
• Cannot do implicit mapping
• But pup...
17
Implicit mapping example - puppet AWS
18
Implicit mapping example - puppet AWS
19
Implicit mapping example - puppet AWS
20
Implicit mapping example - puppet AWS
• BUG - prefetch method eats exceptions (fixed now)
21
Implicit mapping example - puppet AWS
• BUG - prefetch method eats exceptions (fixed now)
22
Implicit mapping example - puppet AWS
• Reusable abstraction (in theory)

• Don’t try to use like puppet!
• Flat hierarchy (do not nest modules)
• Use version t...
• Why even is state?
• How to cope with state
• Atlas
• Workflow (locking!) is your problem
• Remote state
• Shard terrafo...
• Provides the workflow

• ‘awsadmin’ machine + IAM Role as slave

• Makefile based workflow

• Jenkins job builder to tem...
• Refresh state (upload refreshed state)
• Plan + save as artifact
• Filter plan!
• Approve plan
• Apply plan, save state
...
• Commit some files to git.
• Push to a branch
• Jenkins runs
• Gated approval/application process

• Abstract away the sc...
• Self service cluster provisioning
• Developers define their own clusters
• 1 click from OPs to approve

• Owning team ge...
P.S. We’re hiring!
@bobtfish
engineeringblog.yelp.com
github.com/Yelp
github.com/bobtfish
Upcoming SlideShare
Loading in …5
×

209

Share

Download to read offline

Empowering developers to deploy their own data stores

Download to read offline

Empowering developers to deploy their own data stores using Terrafom, Puppet and rage. A talk about automating server building and configuration for Elasticsearch clusters, using Hashicorp and puppet labs tool. Presented at Config Management Camp 2016 in Ghent

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Empowering developers to deploy their own data stores

  1. 1. Empowering developers to deploy their own data stores. A story of Terraform, Puppet and rage Tomas Doran @bobtfish
  2. 2. • Iterate on the things you do often
 • Hide complexity
 • Empower others 2 Devops = Workflow
  3. 3. • A thing of the past (mostly) • Need to be able to scale up and down in hours • If not minutes
 • Need to allow people to experiment • Cloud is expensive, unless you use it! 3 Artisanal hand-crafted servers
  4. 4. • ‘Infra’ layer • DNS / puppet / apt - basic services • A(WS)?nycast - failover / HA
 • ‘App’ layer • Smartstack - Service discovery + routing • Paasta (Mesos + Marathon) - Scheduling + Orchestration • search24-reviews-uswest1aprod - ugh! 4 2 Layer architecture
  5. 5. • Remembering the . on PTR records
 • For some people! • Why make them do this? 5 The hardest thing
  6. 6. • Datastore PAAS • Elasticsearch clusters are the ‘easy’ case
 • No ‘master’ - all machines are equal • Automatic sharding/replication
 • ASG + ELB • Zookeeper for discovery 6 Next logical step
  7. 7. • curl http://10.29.0.3:8142 (A(WS)nycast puppetmaster)
 { “habitat”: “uswest1aprod” }
 • “habitat”, “region”, “superregion”, “ecosystem” 7 Environment server
  8. 8. • curl http://10.29.0.3:8142 (A(WS)nycast puppetmaster)
 { “habitat”: “uswest1aprod” }
 • “habitat”, “region”, “superregion”, “ecosystem” 8 Environment server
  9. 9. • Hostname: search1-reviews-uswest1aprod • Parse out cluster name
 elasticsearch_cluster { ‘reviews’: }
 puppet/modules/elasticsearch_cluster/data/cluster/ reviews.yaml
 • Can locate the ‘data’ directory somewhere else! • Reuse the same YAML for service discovery + provisioning • Commit hook validation 9 puppet data in modules
  10. 10. • External Node Classifer • Puppetmaster calls a script, returns node definition • Create node definition from EC2 tags
 puppet::role::elasticsearch_cluster => cluster_name=reviews
 • Stop needing individual hostnames! • Pre-allocate names using GENERATE 10 puppet ENC
  11. 11. • Bad abstraction for contextual information • Which db server is the master? Does it have ‘master’ in it’s FQDN? • If it does, what happens when you promote another machine?
 • Need key => value for cattle not pets
 • Customize your monitoring system to actually tell you what’s wrong! • ‘The master db has crashed’ vs ‘A db has crashed’ • ‘10-46-11-54 is dead’ vs ‘zookeeper::10-46-11-54 is dead` 11 Hostnames
  12. 12. • Got most of the pieces • Machines auto-configure themselves after launch. • Remaining step is actually launching machines
 • Terraform is awesome… • IF you treat it as a low level abstraction • IF you keep things in composeable units • IF you add enough workflow to not run with scissors 12 Terraform
  13. 13. 13
  14. 14. 14
  15. 15. 15
  16. 16. • Terraform the most generic abstraction possible • Map JSON (HCL) DSL => CRUD APIs • Cannot do implicit mapping • But puppet / ansible / whatever can??? • ‘Name’ tag => namevar • Only works in some cases - not everything has tags! • Implicit mapping is evil • Duplicates will screw up your day 16 Low level
  17. 17. 17 Implicit mapping example - puppet AWS
  18. 18. 18 Implicit mapping example - puppet AWS
  19. 19. 19 Implicit mapping example - puppet AWS
  20. 20. 20 Implicit mapping example - puppet AWS
  21. 21. • BUG - prefetch method eats exceptions (fixed now) 21 Implicit mapping example - puppet AWS
  22. 22. • BUG - prefetch method eats exceptions (fixed now) 22 Implicit mapping example - puppet AWS
  23. 23. • Reusable abstraction (in theory)
 • Don’t try to use like puppet! • Flat hierarchy (do not nest modules) • Use version tags • Use other git repos
 • Or just generate resources as JSON
 • KISS 23 Terraform modules
  24. 24. • Why even is state? • How to cope with state • Atlas • Workflow (locking!) is your problem • Remote state • Shard terraform for (team) concurrency • S3 store • Many read, few write • Wrap it yourself (make, Jenkins, don’t install terraform in $PATH) 24 State
  25. 25. • Provides the workflow
 • ‘awsadmin’ machine + IAM Role as slave
 • Makefile based workflow
 • Jenkins job builder to template things 25 Jenkins
  26. 26. • Refresh state (upload refreshed state) • Plan + save as artifact • Filter plan! • Approve plan • Apply plan, save state 26 Split up the steps
  27. 27. • Commit some files to git. • Push to a branch • Jenkins runs • Gated approval/application process
 • Abstract away the scary parts • Enforce workflow 27 Cluster provisioning workflow
  28. 28. • Self service cluster provisioning • Developers define their own clusters • 1 click from OPs to approve
 • Owning team gets accounted • AWS metadata added as needed. • All metadata validated.
 • Clusters built around best practices • Can abstract further in future 28 Nirvana
  29. 29. P.S. We’re hiring! @bobtfish engineeringblog.yelp.com github.com/Yelp github.com/bobtfish
  • ConfidenceN

    Sep. 17, 2021
  • drrichardselden

    Jun. 21, 2021
  • DavidHanceColon

    May. 22, 2021
  • amw96

    Apr. 22, 2021
  • ssusercaee97

    Apr. 1, 2021
  • RoyalSbachl1

    Mar. 20, 2021
  • MonaAaa1

    Mar. 20, 2021
  • MichealCuthlow

    Feb. 17, 2021
  • leemcneill1

    Nov. 7, 2020
  • aboodi99

    Sep. 18, 2020
  • SeshuVarikallu

    Jul. 9, 2020
  • PattarawadeePradubmo

    Jul. 8, 2020
  • BaharBaabaa

    Jun. 26, 2020
  • JessicaLpez83

    Jun. 25, 2020
  • JunshanHe

    Jun. 13, 2020
  • HermaneSilue

    Jun. 8, 2020
  • DimasSaputra62

    Apr. 21, 2020
  • suraganigeetha

    Mar. 22, 2020
  • NaokiMikuni

    Mar. 14, 2020
  • AungSoePaing3

    Mar. 9, 2020

Empowering developers to deploy their own data stores using Terrafom, Puppet and rage. A talk about automating server building and configuration for Elasticsearch clusters, using Hashicorp and puppet labs tool. Presented at Config Management Camp 2016 in Ghent

Views

Total views

21,510

On Slideshare

0

From embeds

0

Number of embeds

302

Actions

Downloads

121

Shares

0

Comments

0

Likes

209

×