SlideShare une entreprise Scribd logo
1  sur  44
DevOps and HPC:
Saudi Aramco HPC use case
Walid A. Shaari 20th April 2016
Ahmed Bu-khamsin
2
References in this document to any specific commercial products, process, or
service by trade name, trademark, manufacturer, or otherwise, does not
necessarily constitute or imply its endorsement, recommendation, or favoring
by Saudi Aramco or Saudi Aramco HPC group. The ideas and findings of authors
expressed in any slides or other material should not be construed as an official
Saudi Aramco or HPC team position and shall not be used for advertising or
product endorsement purposes. Information contained in this document is
published in the interest of scientific and technical information exchange.
DISCLAIMER OF ENDORSEMENT
27/10/2014
3
DevOps
Cultural movement or practice that
emphasizes the collaboration and
communication of both Application
Developers and Operations
professionals.
Development
Business
Operations
adaptive
automated
agile
4
Business Drives
o Optimization
Effective data center(s) resources utilization:
• Utilization of systems, storage, network, or services.
• Better use of employees time and skills.
o Growth ( N x R x P )
Increasing Infrastructure scale
• N: number of managed nodes/clusters/environments
• R: number of applications(business roles)
• P: number of technical services (technology profiles)
5
Popular DevOps Tools
Docker
Mesos
GIT Puppet
6
Data Center blueprints
7
Script
Packages
Files
Services Mounts Security
Cluster Deployment
8
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
Script
Pack ag es
Files Servic
es
Mo
un
ts
Securi
ty
• Different Hardware
• Different Sizes
• Different Users
• Different Operating Systems
9
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Script
Packag
es
Fi l es Ser
vi c
es
M
o
u
n
t
s
Se c
u rit
y
Common Tasks:
Apply security patches
Add new storage
Upgrade the OS
Install new packages
Common Issues:
Scalability issue
Lack of history
No team collaboration
No drift control
Long development and
test cycle
10
• Do it DevOps way
- Infrastructure as code
• Definition of Infrastructure as code:
"Enable the reconstruction of the business from nothing but a source code
repository, an application data backup, and bare metal resources"
Solution
11
• Domain Specific Language:
- To describe the infrastructure desired state
• Data Store:
- To store the configuration specifications and other data
• Control System:
- To deploy the code and apply the required configuration changes
• Versioning Control System
- To keep history
- enforce workflow and peer review
- Team collaboration
Configuration Management Tools
12
Puppet
• Open-source IT automation framework
• Framework to simplify and automate system configuration and provisioning
• Replaces ssh-for loops and scripts
• Hundreds of configuration modules available for download
• Supports many Linux distributions, Windows, storage and network devices
13
• Hardware Delivery
• Power Up and Network Connectivity
• OS Installation
• Aramco Customization
• Benchmarking
• Application Testing
• Production
HP CMU . IBM xCat . Dell Bright
Where Puppet Fits
Cluster Deployment Project Plan
14
Benefits
• Speeds up clusters deployment From days to hours
- Shorter development cycle
- Same code is used for deployment and compliance
- Code Reuse
15
Benefits
Contribution During Puppet Deployment Project
Contribution During First Deployment Project
Contribution During Second Deployment Project
November 13 2014 - April 22 2015
Commits statistic for
production
697 commits during 160
days
Average 4.4 commits per day
Contributed by 9 authors
16
Benefits
• Automatic and continuous deployment
- Classify the cluster to the right type and Puppet does the rest
17
Benefits
• Advanced reporting capabilities
• Self healing and drift control
• Baseline configuration compliance
18
Benefits
• Version control and development workflow
• Team Collaboration
Production
Bug-fix
New feature
Merge
Request
Merge
Request
19
git Branches and Commits
20
How Pervasive is Configuration Management?
ASM
21
Traditional HPC Cluster Management tools
https://www.flickr.com/photos/vrogy/514733529
22
Provisioning
Workload
Scheduler
& Metrics
System
(user land, kernel modules, devices)
Bare metalBootstrapping
Coniguration
Orchestration
consistency
Provisioningactivity
puppet,
Ansible,
Chef
Grid Engine
SLURM
TORQUE/MOAB
Mesos /Swarm/Nomad
puppet,
Chef
Ansible
foreman
Razor
Digital-rebar
Ironic
Virtual
image
Container
HPC OPSWeb/Cloud OPS
HPC workload runs on
the cloud
25%
24
Which workloads and frameworks are running on
OpenStack?
Source : https://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf
25
HPC in non bare-metal Experimental? Is it Mature?
Vendor trends
26
Next Generation Provisioning
Puppet
Razor Ironic
• No vendor lock: Open Source availability
• Environments Agnostic
• bare-metal, virtual image, and containers
• Use open standards
• Ipmi2, ipxe, dhcp, REST, https
• Handles end to end application provisioning
• Better integration with other tools
• configuration management, CMDB, Monitoring
• Programmable
• On-demand provisioning
• Policy/Model based
27
Data Center current state
SchedulerSchedulerScheduler
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Cluster Management A
Cluster Management B
Cluster Management C
0%
50%
100%
28
Data Center
Breaking the Silos
SchedulerSchedulerScheduler
MetaScheduler
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
29
Data Center
Efficient Secure Allocation of Resources
VC3
BigData
VC1
Infra
VC2
HPC
SchedulerSchedulerScheduler
DataCenterScheduler
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
2nd Generation Cluster Management
30
Containers
Container encapsulate an application completely with all of its
software dependencies into a standardized unit for software portable
across different platforms*
https://www.docker.com/what-docker
31
Containers Potential Benefits to HPC
o High performing
o Lightweight
o Portable, could solve software packaging, configuration, and delivery
o Host Kernel and system drivers visibility
o Composable
o Targets better scalable monitoring, logging, and security
o Private in-house repositories
o Workforce Separation of concerns (e.g. Operations, Development, Security, Users)
o Builds on mature agile application lifecycle management
o Empowers application support, and developers
o Holistic, yet modular ECO system
o Schedulers, and cluster managers
(Traditional e.g. LSF, UGE, Moab, and Slurm)
(Modern: Mesos, Kubernetes, nextflow)
32
Docker Performance
http://www.theregister.co.uk/2014/08/18/docker_kicks_kvms_butt_in_ibm_tests
33
NVIDIA Example use case
https://github.com/NVIDIA/nvidia-docker
34
Host possible workload
Tiny Core Linux (VM)
Docker Engine
Bin/libs
Enterprise Linux Distribution
Service
RHEL7
HPCtask
HPCtask
HPCtask
HPCtask
AlpineMicroService
MicroService
MicroService
MicroService
Ubuntu
Bigdata
Alpine
Redis
Kibana
Logstash
Elasticsearc
35
HPC Host Reality
RHEL7
HPCTask
HPCTask
HPCTAsk
HPCTask
Bin/Libs
HPC service
Docker Engine
Docker capable OS
Bin/Libs
HPC service
Bin/Libs
HPC service
Docker Engine
Docker capable OS
Docker Engine
Docker capable OS
Bin/Libs
HPC Job 3
Docker Engine
Docker capable OS
Docker Engine
Docker capable OS
Bin/Libs
HPC Job 3
Bin/Libs
HPC Job 3
Container Cluster Management/orchestration
36
Possible HPC Challenges
o Change of processes, and mindset
o Linux kernel requirements
o Maturity of the cluster management and scheduling solution
o Keeping up with the containers eco system
o Extremely fast moving target
o Several architectural and fundamental decisions to make
o Memory deduplication
o Necessity of automated tool-chains
“development, integration, and delivery workflows”
o Security
Trusted container libraries
37
Thank you
38
Extra Slides
27/10/2014
39
• http://www.meetup.com/Docker-Riyadh/
• http://www.meetup.com/Docker-Dhahran/
Saudi Docker meetups
27/10/2014
40
Mesos
§ Mature, Open Source Apache Project
§ Cluster Resource Manager
§ Scalable to 10,000s of nodes
§ Fault tolerant, no single point of failure
§ Multi-tenancy with strong resource isolation
§ Improved resource utilization
41
Mesos workload schedulers “Frameworks”
42
43
File system Layers
44
Devil in the details
ssh
mpi
Scheduler
Init
musl glibc
Docker Engine
Docker capable OS
Bin/Libs
HPC service

Contenu connexe

Tendances

소프트웨어 공학의 사실과 오해
소프트웨어 공학의 사실과 오해소프트웨어 공학의 사실과 오해
소프트웨어 공학의 사실과 오해한 경만
 
初心者向けMongoDBのキホン!
初心者向けMongoDBのキホン!初心者向けMongoDBのキホン!
初心者向けMongoDBのキホン!Tetsutaro Watanabe
 
Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)
Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)
Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)Takashi Takizawa
 
Docker - un outil pour faciliter le développement et le déploiement informatique
Docker - un outil pour faciliter le développement et le déploiement informatiqueDocker - un outil pour faciliter le développement et le déploiement informatique
Docker - un outil pour faciliter le développement et le déploiement informatiquesdenier
 
Integrating microservices with apache camel on kubernetes
Integrating microservices with apache camel on kubernetesIntegrating microservices with apache camel on kubernetes
Integrating microservices with apache camel on kubernetesClaus Ibsen
 
フロントからバックエンドまで - WebAssemblyで広がる可能性
フロントからバックエンドまで - WebAssemblyで広がる可能性フロントからバックエンドまで - WebAssemblyで広がる可能性
フロントからバックエンドまで - WebAssemblyで広がる可能性IIJ
 
GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月
GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月
GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月VirtualTech Japan Inc.
 
Docker introduction
Docker introductionDocker introduction
Docker introductionGourav Varma
 
Docker Swarm Introduction
Docker Swarm IntroductionDocker Swarm Introduction
Docker Swarm Introductionrajdeep
 
Introduction à l’intégration continue avec Jenkins
Introduction à l’intégration continue avec JenkinsIntroduction à l’intégration continue avec Jenkins
Introduction à l’intégration continue avec JenkinsEric Hogue
 
Cookpad TechConf 2016 - DWHに必要なこと
Cookpad TechConf 2016 - DWHに必要なことCookpad TechConf 2016 - DWHに必要なこと
Cookpad TechConf 2016 - DWHに必要なことMinero Aoki
 
Terraform features(kr)
Terraform features(kr)Terraform features(kr)
Terraform features(kr)규석 이
 
EKS Anywhere on vSphere
EKS Anywhere on vSphereEKS Anywhere on vSphere
EKS Anywhere on vSphereMasanori Nara
 
コンテナの作り方「Dockerは裏方で何をしているのか?」
コンテナの作り方「Dockerは裏方で何をしているのか?」コンテナの作り方「Dockerは裏方で何をしているのか?」
コンテナの作り方「Dockerは裏方で何をしているのか?」Masahito Zembutsu
 
Intégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec JenkinsIntégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec JenkinsKokou Gaglo
 
AWS Glueを使った Serverless ETL の実装パターン
AWS Glueを使った Serverless ETL の実装パターンAWS Glueを使った Serverless ETL の実装パターン
AWS Glueを使った Serverless ETL の実装パターンseiichi arai
 
MHA for MySQLとDeNAのオープンソースの話
MHA for MySQLとDeNAのオープンソースの話MHA for MySQLとDeNAのオープンソースの話
MHA for MySQLとDeNAのオープンソースの話Yoshinori Matsunobu
 
Introduction to Docker - 2017
Introduction to Docker - 2017Introduction to Docker - 2017
Introduction to Docker - 2017Docker, Inc.
 
PenTesterが知っている危ないAWS環境の共通点
PenTesterが知っている危ないAWS環境の共通点 PenTesterが知っている危ないAWS環境の共通点
PenTesterが知っている危ないAWS環境の共通点 zaki4649
 

Tendances (20)

소프트웨어 공학의 사실과 오해
소프트웨어 공학의 사실과 오해소프트웨어 공학의 사실과 오해
소프트웨어 공학의 사실과 오해
 
初心者向けMongoDBのキホン!
初心者向けMongoDBのキホン!初心者向けMongoDBのキホン!
初心者向けMongoDBのキホン!
 
Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)
Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)
Unbound/NSD最新情報(OSC 2013 Tokyo/Spring)
 
Docker - un outil pour faciliter le développement et le déploiement informatique
Docker - un outil pour faciliter le développement et le déploiement informatiqueDocker - un outil pour faciliter le développement et le déploiement informatique
Docker - un outil pour faciliter le développement et le déploiement informatique
 
ROS.js の紹介
ROS.js の紹介ROS.js の紹介
ROS.js の紹介
 
Integrating microservices with apache camel on kubernetes
Integrating microservices with apache camel on kubernetesIntegrating microservices with apache camel on kubernetes
Integrating microservices with apache camel on kubernetes
 
フロントからバックエンドまで - WebAssemblyで広がる可能性
フロントからバックエンドまで - WebAssemblyで広がる可能性フロントからバックエンドまで - WebAssemblyで広がる可能性
フロントからバックエンドまで - WebAssemblyで広がる可能性
 
GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月
GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月
GPU Container as a Serviceを実現するための最新OSS徹底比較 - OpenStack最新情報セミナー 2017年7月
 
Docker introduction
Docker introductionDocker introduction
Docker introduction
 
Docker Swarm Introduction
Docker Swarm IntroductionDocker Swarm Introduction
Docker Swarm Introduction
 
Introduction à l’intégration continue avec Jenkins
Introduction à l’intégration continue avec JenkinsIntroduction à l’intégration continue avec Jenkins
Introduction à l’intégration continue avec Jenkins
 
Cookpad TechConf 2016 - DWHに必要なこと
Cookpad TechConf 2016 - DWHに必要なことCookpad TechConf 2016 - DWHに必要なこと
Cookpad TechConf 2016 - DWHに必要なこと
 
Terraform features(kr)
Terraform features(kr)Terraform features(kr)
Terraform features(kr)
 
EKS Anywhere on vSphere
EKS Anywhere on vSphereEKS Anywhere on vSphere
EKS Anywhere on vSphere
 
コンテナの作り方「Dockerは裏方で何をしているのか?」
コンテナの作り方「Dockerは裏方で何をしているのか?」コンテナの作り方「Dockerは裏方で何をしているのか?」
コンテナの作り方「Dockerは裏方で何をしているのか?」
 
Intégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec JenkinsIntégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec Jenkins
 
AWS Glueを使った Serverless ETL の実装パターン
AWS Glueを使った Serverless ETL の実装パターンAWS Glueを使った Serverless ETL の実装パターン
AWS Glueを使った Serverless ETL の実装パターン
 
MHA for MySQLとDeNAのオープンソースの話
MHA for MySQLとDeNAのオープンソースの話MHA for MySQLとDeNAのオープンソースの話
MHA for MySQLとDeNAのオープンソースの話
 
Introduction to Docker - 2017
Introduction to Docker - 2017Introduction to Docker - 2017
Introduction to Docker - 2017
 
PenTesterが知っている危ないAWS環境の共通点
PenTesterが知っている危ないAWS環境の共通点 PenTesterが知っている危ないAWS環境の共通点
PenTesterが知っている危ないAWS環境の共通点
 

En vedette

Docker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetupDocker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetupWalid Shaari
 
Streamlining HPC Workloads with Containers
Streamlining HPC Workloads with ContainersStreamlining HPC Workloads with Containers
Streamlining HPC Workloads with ContainersDustin Kirkland
 
[Container world 2017] The Questions You're Afraid to Ask about Containers
[Container world 2017] The Questions You're Afraid to Ask about Containers[Container world 2017] The Questions You're Afraid to Ask about Containers
[Container world 2017] The Questions You're Afraid to Ask about ContainersDustin Kirkland
 
HPC Top 5 Stories: March 22, 2017
HPC Top 5 Stories: March 22, 2017HPC Top 5 Stories: March 22, 2017
HPC Top 5 Stories: March 22, 2017NVIDIA
 
HPC in a Box - Docker Workshop at ISC 2015
HPC in a Box - Docker Workshop at ISC 2015HPC in a Box - Docker Workshop at ISC 2015
HPC in a Box - Docker Workshop at ISC 2015inside-BigData.com
 
HPC Storage Appliances for the Enterpris
HPC Storage Appliances for the EnterprisHPC Storage Appliances for the Enterpris
HPC Storage Appliances for the EnterprisIntel IT Center
 
Packaging Software, Puppet Labs Style - PuppetConf 2014
Packaging Software, Puppet Labs Style - PuppetConf 2014Packaging Software, Puppet Labs Style - PuppetConf 2014
Packaging Software, Puppet Labs Style - PuppetConf 2014Puppet
 
Cell phone safety by Waleed Al-Shemamry (ARAMCO)
Cell phone safety by Waleed Al-Shemamry (ARAMCO)Cell phone safety by Waleed Al-Shemamry (ARAMCO)
Cell phone safety by Waleed Al-Shemamry (ARAMCO)Dr Ghaiath Hussein
 
Reproducible Computational Pipelines with Docker and Nextflow
Reproducible Computational Pipelines with Docker and NextflowReproducible Computational Pipelines with Docker and Nextflow
Reproducible Computational Pipelines with Docker and Nextflowinside-BigData.com
 
Optimizing Lustre and GPFS with DDN
Optimizing Lustre and GPFS with DDNOptimizing Lustre and GPFS with DDN
Optimizing Lustre and GPFS with DDNinside-BigData.com
 
Dell Lustre Storage Architecture Presentation - MBUG 2016
Dell Lustre Storage Architecture Presentation - MBUG 2016Dell Lustre Storage Architecture Presentation - MBUG 2016
Dell Lustre Storage Architecture Presentation - MBUG 2016Andrew Underwood
 
GPFS - graphical intro
GPFS - graphical introGPFS - graphical intro
GPFS - graphical introAlex Balk
 
How a Business Executive Led the Implementation of Agile, Lean & CI/CD
How a Business Executive Led the Implementation of Agile, Lean & CI/CDHow a Business Executive Led the Implementation of Agile, Lean & CI/CD
How a Business Executive Led the Implementation of Agile, Lean & CI/CDEm Campbell-Pretty
 
APIs as your digital connector
APIs as your digital connectorAPIs as your digital connector
APIs as your digital connectorNuwan Bandara
 
Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...Walid Shaari
 
From Teams to Tribes: Creating a one team culture - #DOES15
From Teams to Tribes: Creating a one team culture - #DOES15From Teams to Tribes: Creating a one team culture - #DOES15
From Teams to Tribes: Creating a one team culture - #DOES15Em Campbell-Pretty
 
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014Amazon Web Services
 
Creating Packages that Run Anywhere with Chef Habitat
Creating Packages that Run Anywhere with Chef HabitatCreating Packages that Run Anywhere with Chef Habitat
Creating Packages that Run Anywhere with Chef HabitatNell Shamrell-Harrington
 
Trends towards the merge of HPC + Big Data systems
Trends towards the merge of HPC + Big Data systemsTrends towards the merge of HPC + Big Data systems
Trends towards the merge of HPC + Big Data systemsIgor José F. Freitas
 

En vedette (20)

Docker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetupDocker Dhahran Nov 2016 meetup
Docker Dhahran Nov 2016 meetup
 
Streamlining HPC Workloads with Containers
Streamlining HPC Workloads with ContainersStreamlining HPC Workloads with Containers
Streamlining HPC Workloads with Containers
 
[Container world 2017] The Questions You're Afraid to Ask about Containers
[Container world 2017] The Questions You're Afraid to Ask about Containers[Container world 2017] The Questions You're Afraid to Ask about Containers
[Container world 2017] The Questions You're Afraid to Ask about Containers
 
HPC Top 5 Stories: March 22, 2017
HPC Top 5 Stories: March 22, 2017HPC Top 5 Stories: March 22, 2017
HPC Top 5 Stories: March 22, 2017
 
HPC in a Box - Docker Workshop at ISC 2015
HPC in a Box - Docker Workshop at ISC 2015HPC in a Box - Docker Workshop at ISC 2015
HPC in a Box - Docker Workshop at ISC 2015
 
HPC Storage Appliances for the Enterpris
HPC Storage Appliances for the EnterprisHPC Storage Appliances for the Enterpris
HPC Storage Appliances for the Enterpris
 
Packaging Software, Puppet Labs Style - PuppetConf 2014
Packaging Software, Puppet Labs Style - PuppetConf 2014Packaging Software, Puppet Labs Style - PuppetConf 2014
Packaging Software, Puppet Labs Style - PuppetConf 2014
 
Docker for HPC in a Nutshell
Docker for HPC in a NutshellDocker for HPC in a Nutshell
Docker for HPC in a Nutshell
 
Cell phone safety by Waleed Al-Shemamry (ARAMCO)
Cell phone safety by Waleed Al-Shemamry (ARAMCO)Cell phone safety by Waleed Al-Shemamry (ARAMCO)
Cell phone safety by Waleed Al-Shemamry (ARAMCO)
 
Reproducible Computational Pipelines with Docker and Nextflow
Reproducible Computational Pipelines with Docker and NextflowReproducible Computational Pipelines with Docker and Nextflow
Reproducible Computational Pipelines with Docker and Nextflow
 
Optimizing Lustre and GPFS with DDN
Optimizing Lustre and GPFS with DDNOptimizing Lustre and GPFS with DDN
Optimizing Lustre and GPFS with DDN
 
Dell Lustre Storage Architecture Presentation - MBUG 2016
Dell Lustre Storage Architecture Presentation - MBUG 2016Dell Lustre Storage Architecture Presentation - MBUG 2016
Dell Lustre Storage Architecture Presentation - MBUG 2016
 
GPFS - graphical intro
GPFS - graphical introGPFS - graphical intro
GPFS - graphical intro
 
How a Business Executive Led the Implementation of Agile, Lean & CI/CD
How a Business Executive Led the Implementation of Agile, Lean & CI/CDHow a Business Executive Led the Implementation of Agile, Lean & CI/CD
How a Business Executive Led the Implementation of Agile, Lean & CI/CD
 
APIs as your digital connector
APIs as your digital connectorAPIs as your digital connector
APIs as your digital connector
 
Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...
 
From Teams to Tribes: Creating a one team culture - #DOES15
From Teams to Tribes: Creating a one team culture - #DOES15From Teams to Tribes: Creating a one team culture - #DOES15
From Teams to Tribes: Creating a one team culture - #DOES15
 
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
 
Creating Packages that Run Anywhere with Chef Habitat
Creating Packages that Run Anywhere with Chef HabitatCreating Packages that Run Anywhere with Chef Habitat
Creating Packages that Run Anywhere with Chef Habitat
 
Trends towards the merge of HPC + Big Data systems
Trends towards the merge of HPC + Big Data systemsTrends towards the merge of HPC + Big Data systems
Trends towards the merge of HPC + Big Data systems
 

Similaire à What HPC can learn from DevOps?

The DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitThe DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitMarco Ferrigno
 
The DevOps Paradigm
The DevOps ParadigmThe DevOps Paradigm
The DevOps ParadigmNaLUG
 
7 flavours of devops implementation
7 flavours of devops implementation7 flavours of devops implementation
7 flavours of devops implementationAspire Systems
 
Sviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE Manager
Sviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE ManagerSviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE Manager
Sviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE ManagerSUSE Italy
 
DevOps on Oracle Cloud
DevOps on Oracle CloudDevOps on Oracle Cloud
DevOps on Oracle CloudMee Nam Lee
 
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...Siva Rama Krishna Chunduru
 
Enabling Business Agility with SUSE CaaS Platform
Enabling Business Agility with SUSE CaaS PlatformEnabling Business Agility with SUSE CaaS Platform
Enabling Business Agility with SUSE CaaS PlatformSUSE
 
VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...
VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...
VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...VMworld
 
Cloud-native Application Lifecycle Management
Cloud-native Application Lifecycle ManagementCloud-native Application Lifecycle Management
Cloud-native Application Lifecycle ManagementNeil Gehani
 
CNCF Introduction - Feb 2018
CNCF Introduction - Feb 2018CNCF Introduction - Feb 2018
CNCF Introduction - Feb 2018Krishna-Kumar
 
Devops phase-1
Devops phase-1Devops phase-1
Devops phase-1G R VISHAL
 
Cluster-as-code. The Many Ways towards Kubernetes
Cluster-as-code. The Many Ways towards KubernetesCluster-as-code. The Many Ways towards Kubernetes
Cluster-as-code. The Many Ways towards KubernetesQAware GmbH
 
OpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software StackOpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software Stackinside-BigData.com
 
Transforming Application Delivery with PaaS and Linux Containers
Transforming Application Delivery with PaaS and Linux ContainersTransforming Application Delivery with PaaS and Linux Containers
Transforming Application Delivery with PaaS and Linux ContainersGiovanni Galloro
 
Oracle Developer Cloud - 소개 (신기능 포함)
Oracle Developer Cloud - 소개 (신기능 포함)Oracle Developer Cloud - 소개 (신기능 포함)
Oracle Developer Cloud - 소개 (신기능 포함)Mee Nam Lee
 
Srikanth_testing resume
Srikanth_testing resumeSrikanth_testing resume
Srikanth_testing resumesrikanth Burra
 
Journey to cloud engineering
Journey to cloud engineeringJourney to cloud engineering
Journey to cloud engineeringMd. Sadhan Sarker
 
Enabling multicloud in the enterprise with DevSecOps
Enabling multicloud in the enterprise with DevSecOpsEnabling multicloud in the enterprise with DevSecOps
Enabling multicloud in the enterprise with DevSecOpsJosh Boyd
 

Similaire à What HPC can learn from DevOps? (20)

The DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitThe DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkit
 
The DevOps Paradigm
The DevOps ParadigmThe DevOps Paradigm
The DevOps Paradigm
 
7 flavours of devops implementation
7 flavours of devops implementation7 flavours of devops implementation
7 flavours of devops implementation
 
Sviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE Manager
Sviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE ManagerSviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE Manager
Sviluppare velocemente applicazioni sicure con SUSE CaaS Platform e SUSE Manager
 
DevOps on Oracle Cloud
DevOps on Oracle CloudDevOps on Oracle Cloud
DevOps on Oracle Cloud
 
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
 
Enabling Business Agility with SUSE CaaS Platform
Enabling Business Agility with SUSE CaaS PlatformEnabling Business Agility with SUSE CaaS Platform
Enabling Business Agility with SUSE CaaS Platform
 
VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...
VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...
VMworld 2015: Build and Run Cloud Native Apps in your Software Defined Data C...
 
Cloud-native Application Lifecycle Management
Cloud-native Application Lifecycle ManagementCloud-native Application Lifecycle Management
Cloud-native Application Lifecycle Management
 
CNCF Introduction - Feb 2018
CNCF Introduction - Feb 2018CNCF Introduction - Feb 2018
CNCF Introduction - Feb 2018
 
Red hat cloud platforms
Red hat cloud platformsRed hat cloud platforms
Red hat cloud platforms
 
Devops phase-1
Devops phase-1Devops phase-1
Devops phase-1
 
Cluster-as-code. The Many Ways towards Kubernetes
Cluster-as-code. The Many Ways towards KubernetesCluster-as-code. The Many Ways towards Kubernetes
Cluster-as-code. The Many Ways towards Kubernetes
 
Ibm innovate ci for system z
Ibm innovate ci for system zIbm innovate ci for system z
Ibm innovate ci for system z
 
OpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software StackOpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software Stack
 
Transforming Application Delivery with PaaS and Linux Containers
Transforming Application Delivery with PaaS and Linux ContainersTransforming Application Delivery with PaaS and Linux Containers
Transforming Application Delivery with PaaS and Linux Containers
 
Oracle Developer Cloud - 소개 (신기능 포함)
Oracle Developer Cloud - 소개 (신기능 포함)Oracle Developer Cloud - 소개 (신기능 포함)
Oracle Developer Cloud - 소개 (신기능 포함)
 
Srikanth_testing resume
Srikanth_testing resumeSrikanth_testing resume
Srikanth_testing resume
 
Journey to cloud engineering
Journey to cloud engineeringJourney to cloud engineering
Journey to cloud engineering
 
Enabling multicloud in the enterprise with DevSecOps
Enabling multicloud in the enterprise with DevSecOpsEnabling multicloud in the enterprise with DevSecOps
Enabling multicloud in the enterprise with DevSecOps
 

Plus de Walid Shaari

Towards-cloud-native-HPC.pdf
Towards-cloud-native-HPC.pdfTowards-cloud-native-HPC.pdf
Towards-cloud-native-HPC.pdfWalid Shaari
 
Aws ug dxb 2021 container series iv
Aws ug dxb 2021 container series  ivAws ug dxb 2021 container series  iv
Aws ug dxb 2021 container series ivWalid Shaari
 
Okd wg kubecon marathon azure & vsphere
Okd wg kubecon marathon azure & vsphereOkd wg kubecon marathon azure & vsphere
Okd wg kubecon marathon azure & vsphereWalid Shaari
 
K8s architecture meetup2- k8saraby
K8s architecture  meetup2- k8sarabyK8s architecture  meetup2- k8saraby
K8s architecture meetup2- k8sarabyWalid Shaari
 
Pydata 2020 containers meetup
Pydata  2020 containers meetup Pydata  2020 containers meetup
Pydata 2020 containers meetup Walid Shaari
 
Dammam aws user group meetup
Dammam aws user group meetupDammam aws user group meetup
Dammam aws user group meetupWalid Shaari
 
Bahrain ch9 introduction to docker 5th birthday
Bahrain ch9 introduction to docker 5th birthday Bahrain ch9 introduction to docker 5th birthday
Bahrain ch9 introduction to docker 5th birthday Walid Shaari
 
IAU workshop 2018 day one
IAU workshop 2018 day oneIAU workshop 2018 day one
IAU workshop 2018 day oneWalid Shaari
 
Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...Walid Shaari
 
Network Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspectiveNetwork Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspectiveWalid Shaari
 
Kick starting Network Automation
Kick starting Network AutomationKick starting Network Automation
Kick starting Network AutomationWalid Shaari
 
Docker Dhahran November 2017 meetup
Docker Dhahran November 2017 meetupDocker Dhahran November 2017 meetup
Docker Dhahran November 2017 meetupWalid Shaari
 
Docker 101 @KACST Saudi HPC 2016
Docker 101  @KACST Saudi HPC 2016Docker 101  @KACST Saudi HPC 2016
Docker 101 @KACST Saudi HPC 2016Walid Shaari
 

Plus de Walid Shaari (14)

Towards-cloud-native-HPC.pdf
Towards-cloud-native-HPC.pdfTowards-cloud-native-HPC.pdf
Towards-cloud-native-HPC.pdf
 
Aws ug dxb 2021 container series iv
Aws ug dxb 2021 container series  ivAws ug dxb 2021 container series  iv
Aws ug dxb 2021 container series iv
 
Open hybrid cloud
Open hybrid cloudOpen hybrid cloud
Open hybrid cloud
 
Okd wg kubecon marathon azure & vsphere
Okd wg kubecon marathon azure & vsphereOkd wg kubecon marathon azure & vsphere
Okd wg kubecon marathon azure & vsphere
 
K8s architecture meetup2- k8saraby
K8s architecture  meetup2- k8sarabyK8s architecture  meetup2- k8saraby
K8s architecture meetup2- k8saraby
 
Pydata 2020 containers meetup
Pydata  2020 containers meetup Pydata  2020 containers meetup
Pydata 2020 containers meetup
 
Dammam aws user group meetup
Dammam aws user group meetupDammam aws user group meetup
Dammam aws user group meetup
 
Bahrain ch9 introduction to docker 5th birthday
Bahrain ch9 introduction to docker 5th birthday Bahrain ch9 introduction to docker 5th birthday
Bahrain ch9 introduction to docker 5th birthday
 
IAU workshop 2018 day one
IAU workshop 2018 day oneIAU workshop 2018 day one
IAU workshop 2018 day one
 
Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...Containers - Portable, repeatable user-oriented application delivery. Build, ...
Containers - Portable, repeatable user-oriented application delivery. Build, ...
 
Network Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspectiveNetwork Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspective
 
Kick starting Network Automation
Kick starting Network AutomationKick starting Network Automation
Kick starting Network Automation
 
Docker Dhahran November 2017 meetup
Docker Dhahran November 2017 meetupDocker Dhahran November 2017 meetup
Docker Dhahran November 2017 meetup
 
Docker 101 @KACST Saudi HPC 2016
Docker 101  @KACST Saudi HPC 2016Docker 101  @KACST Saudi HPC 2016
Docker 101 @KACST Saudi HPC 2016
 

Dernier

GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 

Dernier (20)

GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 

What HPC can learn from DevOps?

  • 1. DevOps and HPC: Saudi Aramco HPC use case Walid A. Shaari 20th April 2016 Ahmed Bu-khamsin
  • 2. 2 References in this document to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Saudi Aramco or Saudi Aramco HPC group. The ideas and findings of authors expressed in any slides or other material should not be construed as an official Saudi Aramco or HPC team position and shall not be used for advertising or product endorsement purposes. Information contained in this document is published in the interest of scientific and technical information exchange. DISCLAIMER OF ENDORSEMENT 27/10/2014
  • 3. 3 DevOps Cultural movement or practice that emphasizes the collaboration and communication of both Application Developers and Operations professionals. Development Business Operations adaptive automated agile
  • 4. 4 Business Drives o Optimization Effective data center(s) resources utilization: • Utilization of systems, storage, network, or services. • Better use of employees time and skills. o Growth ( N x R x P ) Increasing Infrastructure scale • N: number of managed nodes/clusters/environments • R: number of applications(business roles) • P: number of technical services (technology profiles)
  • 8. 8 Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty Script Pack ag es Files Servic es Mo un ts Securi ty • Different Hardware • Different Sizes • Different Users • Different Operating Systems
  • 9. 9 Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Script Packag es Fi l es Ser vi c es M o u n t s Se c u rit y Common Tasks: Apply security patches Add new storage Upgrade the OS Install new packages Common Issues: Scalability issue Lack of history No team collaboration No drift control Long development and test cycle
  • 10. 10 • Do it DevOps way - Infrastructure as code • Definition of Infrastructure as code: "Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources" Solution
  • 11. 11 • Domain Specific Language: - To describe the infrastructure desired state • Data Store: - To store the configuration specifications and other data • Control System: - To deploy the code and apply the required configuration changes • Versioning Control System - To keep history - enforce workflow and peer review - Team collaboration Configuration Management Tools
  • 12. 12 Puppet • Open-source IT automation framework • Framework to simplify and automate system configuration and provisioning • Replaces ssh-for loops and scripts • Hundreds of configuration modules available for download • Supports many Linux distributions, Windows, storage and network devices
  • 13. 13 • Hardware Delivery • Power Up and Network Connectivity • OS Installation • Aramco Customization • Benchmarking • Application Testing • Production HP CMU . IBM xCat . Dell Bright Where Puppet Fits Cluster Deployment Project Plan
  • 14. 14 Benefits • Speeds up clusters deployment From days to hours - Shorter development cycle - Same code is used for deployment and compliance - Code Reuse
  • 15. 15 Benefits Contribution During Puppet Deployment Project Contribution During First Deployment Project Contribution During Second Deployment Project November 13 2014 - April 22 2015 Commits statistic for production 697 commits during 160 days Average 4.4 commits per day Contributed by 9 authors
  • 16. 16 Benefits • Automatic and continuous deployment - Classify the cluster to the right type and Puppet does the rest
  • 17. 17 Benefits • Advanced reporting capabilities • Self healing and drift control • Baseline configuration compliance
  • 18. 18 Benefits • Version control and development workflow • Team Collaboration Production Bug-fix New feature Merge Request Merge Request
  • 20. 20 How Pervasive is Configuration Management? ASM
  • 21. 21 Traditional HPC Cluster Management tools https://www.flickr.com/photos/vrogy/514733529
  • 22. 22 Provisioning Workload Scheduler & Metrics System (user land, kernel modules, devices) Bare metalBootstrapping Coniguration Orchestration consistency Provisioningactivity puppet, Ansible, Chef Grid Engine SLURM TORQUE/MOAB Mesos /Swarm/Nomad puppet, Chef Ansible foreman Razor Digital-rebar Ironic Virtual image Container HPC OPSWeb/Cloud OPS
  • 23. HPC workload runs on the cloud 25%
  • 24. 24 Which workloads and frameworks are running on OpenStack? Source : https://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf
  • 25. 25 HPC in non bare-metal Experimental? Is it Mature? Vendor trends
  • 26. 26 Next Generation Provisioning Puppet Razor Ironic • No vendor lock: Open Source availability • Environments Agnostic • bare-metal, virtual image, and containers • Use open standards • Ipmi2, ipxe, dhcp, REST, https • Handles end to end application provisioning • Better integration with other tools • configuration management, CMDB, Monitoring • Programmable • On-demand provisioning • Policy/Model based
  • 27. 27 Data Center current state SchedulerSchedulerScheduler Jobs Jobs Jobs Jobs Jobs Jobs Jobs Jobs Jobs Cluster Management A Cluster Management B Cluster Management C 0% 50% 100%
  • 28. 28 Data Center Breaking the Silos SchedulerSchedulerScheduler MetaScheduler Jobs Jobs Jobs Jobs Jobs Jobs Jobs Jobs
  • 29. 29 Data Center Efficient Secure Allocation of Resources VC3 BigData VC1 Infra VC2 HPC SchedulerSchedulerScheduler DataCenterScheduler Jobs Jobs Jobs Jobs Jobs Jobs Jobs Jobs 2nd Generation Cluster Management
  • 30. 30 Containers Container encapsulate an application completely with all of its software dependencies into a standardized unit for software portable across different platforms* https://www.docker.com/what-docker
  • 31. 31 Containers Potential Benefits to HPC o High performing o Lightweight o Portable, could solve software packaging, configuration, and delivery o Host Kernel and system drivers visibility o Composable o Targets better scalable monitoring, logging, and security o Private in-house repositories o Workforce Separation of concerns (e.g. Operations, Development, Security, Users) o Builds on mature agile application lifecycle management o Empowers application support, and developers o Holistic, yet modular ECO system o Schedulers, and cluster managers (Traditional e.g. LSF, UGE, Moab, and Slurm) (Modern: Mesos, Kubernetes, nextflow)
  • 33. 33 NVIDIA Example use case https://github.com/NVIDIA/nvidia-docker
  • 34. 34 Host possible workload Tiny Core Linux (VM) Docker Engine Bin/libs Enterprise Linux Distribution Service RHEL7 HPCtask HPCtask HPCtask HPCtask AlpineMicroService MicroService MicroService MicroService Ubuntu Bigdata Alpine Redis Kibana Logstash Elasticsearc
  • 35. 35 HPC Host Reality RHEL7 HPCTask HPCTask HPCTAsk HPCTask Bin/Libs HPC service Docker Engine Docker capable OS Bin/Libs HPC service Bin/Libs HPC service Docker Engine Docker capable OS Docker Engine Docker capable OS Bin/Libs HPC Job 3 Docker Engine Docker capable OS Docker Engine Docker capable OS Bin/Libs HPC Job 3 Bin/Libs HPC Job 3 Container Cluster Management/orchestration
  • 36. 36 Possible HPC Challenges o Change of processes, and mindset o Linux kernel requirements o Maturity of the cluster management and scheduling solution o Keeping up with the containers eco system o Extremely fast moving target o Several architectural and fundamental decisions to make o Memory deduplication o Necessity of automated tool-chains “development, integration, and delivery workflows” o Security Trusted container libraries
  • 40. 40 Mesos § Mature, Open Source Apache Project § Cluster Resource Manager § Scalable to 10,000s of nodes § Fault tolerant, no single point of failure § Multi-tenancy with strong resource isolation § Improved resource utilization
  • 41. 41 Mesos workload schedulers “Frameworks”
  • 42. 42
  • 44. 44 Devil in the details ssh mpi Scheduler Init musl glibc Docker Engine Docker capable OS Bin/Libs HPC service