HashiCorp Vault Workshop：幫 Credentials 找個窩
•
21 j'aime•2,353 vues
Credentials 究竟該儲存在哪邊才算安全一直是個很尷尬的問題，外洩的消息更是時有耳聞；而一般的應用程式設計時，很少考量到如何定期去更新 Credentials，這時 DevOps 的救星 HashiCorp 發現了大家的需求，因而推出 Vault 來作為 Credentials 的歸宿，其主要最大的兩大功能就是讓大家可以將 Credentials 安心地存放在其中，並且針對一些特定的應用服務提供動態 Credentials 的功能，此 Workshop 主要分成三個主軸： - Vault 基本使用 - Dynamic Credentials 使用方式 - Vault 在生產環境使用的最佳準則
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à HashiCorp Vault Workshop：幫 Credentials 找個窩
Similaire à HashiCorp Vault Workshop：幫 Credentials 找個窩 (20)
Plus de smalltown
Plus de smalltown (20)
Dernier
Dernier (20)
HashiCorp Vault Workshop：幫 Credentials 找個窩
- 4. We’re Hiring !!! 4 Software Engineer in Test Site Reliability Engineer
- 5. Outline ◉ CH00 Environment Setup ◉ CH01 Vault Basics ◉ CH02 Dynamic Credentials ◉ CH03 Daily Operation ◉ CH04 Go Production 5
- 6. Environment Setup0 6 Setup Workshop Environment
- 7. Setup Cloud9 Following pictures of installation guide comes from: https://github.com/pahud/amazon-eks-workshop 7
- 10. 10
- 11. 11
- 14. 7. execute ‘aws configure‘ to configure the credentials for your IAM user. Make sure this IAM User has AdministratorAccess and run ‘aws sts get-caller-identity’ - you should be able to see the returned JSON output like this. 14
- 15. Create IAM Key if You Have No One (1/4) 15
- 16. Create IAM Key if You Have No One (2/4) 16
- 17. Create IAM Key if You Have No One (3/4) 17
- 18. Create IAM Key if You Have No One (4/4) 18
- 19. Run Command in Cloud9 $ git clone https://github.com/Taipei-HUG/workshop.git $ cd vault/CH00 $ ./step1.sh # get all binary $ ./step2.sh # setup eks cluster 19
- 21. Questions ● Where do you store credentials? ● How do you rotate credentials? 21
- 22. Where do you store credentials? 22
- 23. How do you rotate credentials? ● Create a new credential ● Rotate credentials one by one ● Delete old credential ● What if you have 10 components connect to db? 23
- 25. Where do you store credentials? 25
- 26. Credential 1 Credential 2 1. Get Credential by access token or other auth method 2. Access db via credential Basic secret management 0. Admin create credential 26
- 27. Vault Feature ● Web UI/CLI/API/SDK ● Centralized secret management ● Secure Secret Storage ● Dynamic Secrets ● Leasing and Renewal ● Revocation 27
- 28. Credential 1 User create credential Encrypt and save to Storage 28
- 30. Vault Feature ● Web UI/CLI/API/SDK ● Centralized secret management ● Secure Secret Storage ● Dynamic Secrets ● Leasing and Renewal ● Revocation 30
- 31. CMS cluster Dynamic secret endpoint 1 Dynamic secret endpoint 2User: vault-cms-1 Password: dynamic-pw-1 1 2 User: vault-cms-2 Password: dynamic-pw-2 Access via dynamic secret 31
- 32. Vault Feature ● Web UI/CLI/API/SDK ● Centralized secret management ● Secure Secret Storage ● Dynamic Secrets ● Leasing and Renewal ● Revocation 32
- 33. Credential 1 Credential 2 1. Get Credential by access token or other auth method 2. Access db via credential Basic secret management 0. User create credential 33
- 34. Setup Vault on docker-compose # Open cloud9 ide & see CH01/commands.txt $ docker-compose up -d $ export VAULT_ADDR=http://127.0.0.1:8080 $ export VAULT_TOKEN=my-root-token $ vault status 34
- 35. Manipulate vault kv $ vault kv list secret $ vault kv put secret/first-secret foo=bar $ vault kv list secret $ vault kv get secret/first-secret $ vault kv put secret/first-secret foo=bar test=true $ vault kv metadata get secret/first-secret $ vault kv delete secret/first-secret $ vault kv metadata delete secret/first-secret 35
- 36. Vault UI ● Cloud9 → Preview → Preview Running Application 36
- 37. 37
- 38. Dynamic Secret
- 39. CMS cluster Dynamic secret endpoint 1 Dynamic secret endpoint 2User: vault-cms-1 Password: dynamic-pw-1 1 2 User: vault-cms-2 Password: dynamic-pw-2 Access via dynamic secret 39
- 41. Request a dynamic credential Create a user with certain scope Returns a credentialReturns a credential Access database via the credential Revoke the credential Graceful shutdown Delete the user User deletedCredential deleted 41
- 42. Setup Dynamic Secret ● See init.sh ● Setup Vault Dynamic Secret ● Integrate into our service 42
- 43. $ vault secrets enable database 43
- 44. Supported engines ● Database ○ MySQL, PostgreSQL, MongoDB, etc. ● SSH ● AWS IAM ● Vault docs - secret engine 44
- 45. $ vault write database/config/my-database plugin_name=mysql-database-plugin connection_url="{{username}}:{{password}}@tcp(mysql_url:3306)/" allowed_roles=my-role username=${MYSQL_ROOT_USERNAME} password=${MYSQL_ROOT_PASSWORD} 45
- 46. $ vault write database/roles/my-role db_name=my-database creation_statements= "CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%';" default_ttl="1h" max_ttl="2h" 46
- 47. 1. Request credential --- $ vault read database/creds/my-role 2. Issue a dynamic credential with TTL by root credential 3. Get credential 4. Access db by dynamic credential CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%'; Execute by username=${MYSQL_ROOT_USERNAME} password=${MYSQL_ROOT_PASSWORD} 47
- 48. $ npm install $ node index.js 48
- 49. const credential = await vault.read("database/creds/my-role"); const { username: user, password } = credential.data; const conn = await mysql.createConnection({ host, port, user, password }); const result = await conn.query("SELECT USER() as user"); await vault.revoke({ lease_id: credential.lease_id }); 49
- 50. Revocation ● Manually revoke by cli/api ● Automatically revoke by TTL $ vault read database/creds/my-role Key Value --- ----- lease_id database/creds/my-role/IPUkANwU080vaJwARYm4S8NT lease_duration 1h lease_renewable true password A1a-5pV9iwoVWLmvh3Fu username v-token-my-role-wGzYgA6g8DozFW0k 50
- 51. Credential 1 Credential 2 1. Get Credential by access token or other auth method 2. Access db via credential Basic secret management 0. User create credential 51
- 52. Auth Methods ● Vault token ● Kubernetes ● AWS IAM ● LDAP ● … a lot 52
- 53. Kubernetes cluster Deployment A Deployment B Credential 1 Credential 2 Role A - Policy A Role B - Policy B Credential 1 Credential 2 See more: Vault 與 Kubernetes 的深度整合 Vault 53
- 54. 54
- 55. Put all together ● Dynamic Secret ● Kubernetes service authentication ● Limit permission scope 55
- 56. Ideal Credential Lifecycle 56 Service is Accessed Application 1. Request Access Credential (Running) 2. Use the Credential to Access Service 3. Revoke the Credential Credentials Only Exist in Memory
- 57. 57
- 58. Daily Operation3 58 Overview of How to Maintain Vault
- 59. ◉ When a Vault server is first initialized, Vault generates a master key ◉ Immediately splits this master key into a series of key shares following Shamir's Secret Sharing Algorithm Master Key 59
- 60. ◉ The master key is used to decrypt the underlying encryption key ◉ Vault uses the encryption key to encrypt data at rest in a storage backend like the filesystem or Consul Encryption Key 60
- 61. ◉ Vault never stores the master key, therefore, the only way to retrieve the master key is to have a quorum of unseal keys re-generate it. Seal/Unseal 61
- 62. Practice (1/3) # switch to CH03 folder ~$ cd vault/CH03 # boot vault server and login it ~$ ./start_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Seal Type shamir Initialized false Sealed true Total Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version n/a HA Enabled false 62
- 63. Practice (2/3) # initialize vault ~$ vault operator init # keep the unseal key and root token Unseal Key 1: QosKlf+rXJkDLZJX7tgpiKj8zDDNzGHrv4HrV2C9xlxH Unseal Key 2: 8dUP2J+1vPQcRM09QdK5Lo83YnHHjDe1nLdfAjESSKfa Unseal Key 3: t89QgyeOIC4W/7ZRCcMzrgPBYLhGeVOMtuHVNU2IQ5k2 Unseal Key 4: KPnn8uvGWu5DOpcuQA1/1DnjGCiakPWdP1ExtAcQm0L7 Unseal Key 5: LPCOtvaU944O2tbWXoeJGo7SL4d6pc0iFA8vrUv1gHyW Initial Root Token: s.96aLKTnD3WRiT0STWeLbLDDI 63
- 64. Practice (3/3) # unseal vault # repeat 3 times ~$ vault operator unseal Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce a0dfd3da-0fcb-0268-baba-ef4cbe 5550bc Version 1.1.2 HA Enabled false Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled false 64
- 65. ◉ AliCloud KMS, Amazon KMS, Azure Key Vault, and Google Cloud KMS Auto-Unseal 65
- 66. Practice (1/6) # exit vault server container by Ctrl+D # create kms key ~$ aws kms create-key { "KeyMetadata": { "AWSAccountId": "123456789012", "KeyId": "xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy", "Arn": "arn:aws:kms:us-west-2:123456789012:key/xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyy yyy", "CreationDate": 1559233248.825, ... 66
- 67. Practice (2/6) ◉ Append seal config section into config/default.hcl … seal "awskms" { region = "us-west-2" kms_key_id = "xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy" } 67
- 68. Practice (3/6) ◉ Add AWS AK/SK into .env file VAULT_ADDR=http://127.0.0.1:8200 AWS_ACCESS_KEY_ID=DFJLSFKJLD8358KJLJK8 AWS_SECRET_ACCESS_KEY=JioeuJek7+jgJLIUJWTYSfv3rr49JRoqt 68
- 69. Practice (4/6) # restart vault server ~$ ./restart_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed true Total Recovery Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Seal Migration in Progress true Version 1.1.2 HA Enabled false 69
- 70. Practice (5/6) # seal migration # repeat 3 times ~$ vault operator unseal -migrate Unseal Key (will be hidden): Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled false 70
- 71. Practice (6/6) # exit vault server container by Ctrl+D # restart vault server ~$ ./restart_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled false 71
- 73. Practice (1/2) # enable Userpass auth method ~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI" ~$ vault auth enable userpass Success! Enabled userpass auth method at: userpass/ ~$ vault write auth/userpass/users/smalltown password=12345678 policies=default Success! Data written to: auth/userpass/users/smalltown 73
- 74. Practice (2/2) # try to login vault by account & password ~$ vault login -method=userpass username=smalltown password=12345678 Key Value --- ----- token s.MXYyp2Q9OB1iVQJlhso3v3an token_accessor hPntyUJor6sMb1Iw3XwbW9qi token_duration 768h token_renewable true token_policies ["default"] identity_policies [] policies ["default"] token_meta_username smalltown 74
- 75. Authorization 75
- 76. Policies ◉ Vault store credentials like key/value database, e.g. ○ secret/stag/database/admin ○ secret/prod/database/admin ◉ Hence, predefined policy grant appropriate permission, e.g. path "secret/stag/database/admin" { capabilities = ["read"] } 76
- 77. Practice (1/3) # enable kv ~$ vault secrets enable -version=2 kv # put database credentials into vault ~$ vault kv put kv/stag/database/admin account=stag password=12345678 ~$ vault kv put kv/prod/database/admin account=prod password=87654321 77
- 78. Practice (2/3) # create policy ~$ vault policy write stag /vault/policy/stag.hcl ~$ vault policy write prod /vault/policy/prod.hcl # assign policy ~$ vault write auth/userpass/users/smalltown password=12345678 policies=stag 78
- 79. Practice (3/3) # login user and use the token ~$ unset VAULT_TOKEN ~$ vault login -method=userpass username=smalltown password=12345678 # try to get the database credentials ~$ vault kv get kv/stag/database/admin ~$ vault kv get kv/prod/database/admin 79
- 81. Practice (1/4) # exit vault server container by Ctrl+D # create dynamodb ~$ aws dynamodb create-table --table-name vault-workshop --attribute-definitions AttributeName=Path,AttributeType=S AttributeName=Key,AttributeType=S --key-schema AttributeName=Path,KeyType=HASH AttributeName=Key,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10 81
- 82. Practice (2/4) # migrate storage ~$ ./login_local_vault.sh ~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI" ~$ vault operator migrate -config /vault/config/migration.hcl … 2019-05-31T05:37:56.188Z [INFO] copied key: path=sys/token/id/hc99e5ba3d69e7cbaecf0489031eadbf11b55984d698d0f9784e5e1a28 25f93fe 2019-05-31T05:37:56.485Z [INFO] copied key: path=sys/token/salt Success! All of the keys have been migrated. 82
- 83. Practice (3/4) # modify the config file config/default.hcl storage "file" { path = "/vault/file" } api_addr = "https://vault-workshop.hub.internal" storage "dynamodb" { ha_enabled = "true" region = "us-west-2" table = "vault-workshop" } 83
- 84. Practice (4/4) # exit vault server container by Ctrl+D # restart vault server ~$ ./restart_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled true HA Cluster https://vault-workshop.hub.internal: 444 HA Mode active 84
- 85. Audit Devices 85
- 86. Practice # enable auditing device ~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI" ~$ vault audit enable file file_path=/vault/logs/audit.log Success! Enabled the file audit device at: file/ ~$ cat /vault/logs/audit.log {"time":"2019-05-31T06:13:54.1656812Z","type":"response","auth":{"client_token":"hmac-sha256:a5a7726a6e12c568dad5caf12 bec4841ca775d28bbd0f7683c09ebb260ae604c","accessor":"hmac-sha256:38c0629c4ecca5961e9954cbafc80cfd009d6c0 31516fc2f13da54b2eea9e1e0","display_name":"root","policies":["root"],"token_policies":["root"],"metadata":null,"entity_id":"","t oken_type":"service"},"request":{"id":"219eb2f0-a0a7-59a0-c2b8-e7eeac31f31c","operation":"update","client_token":"hmac-s ha256:a5a7726a6e12c568dad5caf12bec4841ca775d28bbd0f7683c09ebb260ae604c","client_token_accessor":"hmac-sha25 6:38c0629c4ecca5961e9954cbafc80cfd009d6c031516fc2f13da54b2eea9e1e0","namespace":{"id":"root","path":""},"path":"sys /audit/file","data":{"description":"hmac-sha256:3eaeeda1a043e3504c691abdb19982eac0dc42c6d9d3cffda9e7cceaa7e3a88 7","local":false,"options":{"file_path":"hmac-sha256:a0a483f16d313b0b882262eea4bb58358ef1323be63b68eb51da01ebf26b dd1a"},"type":"hmac-sha256:b26ec68f8abd74be4fc88990ebba948314f39b1b428091636b1e7fde2c0e9939"},"policy_override" :false,"remote_address":"127.0.0.1","wrap_ttl":0,"headers":{}},"response":{"headers":null},"error":""} 86
- 87. Clean # exit vault server container by Ctrl+D # shutdown vault container ~$ ./stop_local_vault.sh # delete kms key ~$ aws kms schedule-key-deletion --key-id xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy # delete dynamodb ~$ aws dynamodb delete-table --table-name vault-workshop 87
- 88. Go Production4 88 There are Things Need to Do Before Vault Online
- 90. Banzai Cloud - bank-vaults ◉ Bank-Vaults provides various tools for Hashicorp Vault to make its use easier 90
- 91. Practice (1/2) # install vault cluster ~$ cd vault/CH04 ~$ ./setup_vault_cluster.sh 91
- 92. Practice (2/2) # check vault cluster ~$ kubectl get pod NAME READY STATUS RESTARTS AGE etcd-cluster-vj8r8mwslr 1/1 Running 0 3m8s etcd-operator-588ccb98fd-twrs4 1/1 Running 0 8m46s vault-0 3/3 Running 6 3m42s vault-1 3/3 Running 8 3m42s vault-configurer-759d4679dd-8lsjx 1/1 Running 0 3m42s vault-operator-b8c5c566c-w59b5 1/1 Running 0 5m52s 92
- 93. What bank-vaults Do? bank-vaults operator bank-vaults cli Vault storage backend created by ectd operator Create everything needed by vault: bank-vaults cli container, vault, prometheus-exporter bank-vaults cli: 1. Initial and Unseal automatically 2. Configure policy, secret engine...etc 93
- 94. How to Monitor Vault 94
- 95. Telemetry ◉ The Vault server process collects various runtime metrics about the performance of different libraries and subsystems. ◉ To view the telemetry information, you must send a signal to the Vault process ◉ Also can be streamed directly from Vault to a range of metrics aggregation solutions 95
- 96. Monitoring Mechanism StatsD Exporter - Collect telemetry information - Expose to Prometheus - Prometheus scrape metrics - Grafana generate Dashboard 96
- 97. Practice (1/5) # install prometheus ~$ ./setup_prometheus.sh ~$ kubectl create -f vault/servicemonitor.yaml ~$ kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 8m42s alertmanager-main-1 2/2 Running 0 8m30s alertmanager-main-2 2/2 Running 0 8m19s grafana-74b6b56ddf-5k8mz 1/1 Running 0 8m37s kube-state-metrics-7c7979b6bc-zwrwz 4/4 Running 0 5m56s ... 97
- 98. Practice (2/5) # check prometheus scrape vault metric ~$ ./get_prometheus.sh Grafana http://a951eb90e83b511e99a350a7fef3486e-1999986850.us-west-2.elb.amazonaws. com:3000 Prometheus http://aa1ffcdb183b511e99a350a7fef3486e-1607220011.us-west-2.elb.amazonaws.co m:9090 ... 98
- 99. Practice (3/5) ◉ Visit Prometheus, then the vault telemetry can be acquired 99
- 100. ◉ Visit Grafana -> Click Import ◉ Paste JSON from grafana-vault.json ◉ Click Import Practice (4/5) 100
- 101. Practice (5/5) 101
- 102. Clean # Don’t forget to remove the resource create in CH04 ~$ ./uninstall.sh 102
- 103. All Thing Down !!!
- 104. Of Course Not!!!
- 105. Production Hardening (1/3) ◉ End-to-End TLS (O) ◉ Single Tenancy (X) ◉ Firewall traffic (X) ◉ Disable SSH / Remote Desktop (X) ◉ Disable Swap (X) ◉ Don't Run as Root (O) 105
- 106. Production Hardening (2/3) ◉ Turn Off Core Dumps (X) ◉ Immutable Upgrades (O) ◉ Avoid Root Tokens (X) ◉ Enable Auditing (O) ◉ Upgrade Frequently (X) ◉ Configure SELinux / AppArmor (X) 106
- 107. Production Hardening (3/3) ◉ Restrict Storage Access (X) ◉ Disable Shell Command History (X) ◉ Tweak ulimits (X) ◉ Docker Containers (X) Ref 107
- 109. We’re Hiring !!! 109 Software Engineer in Test Site Reliability Engineer