Microservices have emerged as a modern interpretation of service-oriented architectures where processes are small and communicate over a network using lightweight protocols to fulfill a goal.
In this talk I will present our work on microservices, and how they can be used to empower interoperable and scalable analysis services and pipelines in virtual infrastructures on cloud computing resources.
I will also give examples and experiences from the PhenoMeNal H2020 project where a developer community in metabolomics is moving to such architecture.
Pests of mustard_Identification_Management_Dr.UPR.pdf
Interoperability and scalability with microservices in science
1. Interoperability and scalability with
microservices
Ola Spjuth <ola.spjuth@farmbio.uu.se>
Science for Life Laboratory
Uppsala University
2. Service-Oriented Architectures (SOA) in
the life sciences
• Standardize
– Agree on e.g. interfaces, data formats,
protocols etc.
• Decompose and compartmentalize
– Experts (scientists) should provide
services – do one thing and do it well
– Achieve interoperability by exposing
data and tools as Web services
• Integrate
– Users should access and integrate
remote services
API
Scientist
service
Scientist
consume
3. Service-Oriented Architectures (SOA) in
the life sciences, ~2005
Scientist
downtime
API
changed
Not maintained
Difficult to sustain,
unreliable solutions
API
API
API
4. Cloud Computing
• Cloud computing offers advantages over
contemporary e-infrastructures in the life sciences
– On-demand elastic resources and services
– No up-front costs, pay-per-use
• A lot of businesses (and software development)
moving into the cloud
– Vibrant ecosystem of frameworks and tools, including for
big data
• High potential for science
5. Virtual Machines and Containers
Virtual machines
• Package entire systems (heavy)
• Completely isolated
• Suitable in cloud environments
Containers:
• Share OS
• Smaller, faster, portable
• Docker!
5
6. MicroServices
• Similar to Web services: Decompose functionality into smaller, loosely
coupled services communicating via API
– “Do one thing and do it well”
• Preferably smaller, light-weight and fast to instantiate on demand
• Easy to replace, language-agnostic
– Suitable for loosely coupled teams (which we have in science)
– Portable - easy to deploy and scale
– Maximize agility for developers
• Suitable to deploy as containers in cloud environments
10. Kubernetes: Orchestrating containers
• A declarative language for
launching containers
• Start, stop, update, and manage
a cluster of machines running
containers in a consistent and
maintainable way
• Origin: Google
• Suitable for microservices
Containers
Scheduled and packed containers on nodes
11. Virtual Research Environment (VRE)
• Virtual (online) environments for research
– Easy and user-friendly access to computational resources, tools and
data, commonly for a scientific domain
• Multi-tenant VRE – log into shared system
• Private VRE
– Deploy on your favorite cloud provider
11
12. • Horizon 2020-project, €8 M, 2015-2018
– “standardized e-infrastructure for the processing, analysis and information-
mining of the massive amount of medical molecular phenotyping and
genotyping data generated by metabolomics applications.”
• Enable users to provision their own virtual infrastructure (VRE)
– Public cloud, private cloud, local servers
– Easy access to compatible tools exposed as microservices
– Will in minutes set up and configure a complete data-center (compute
nodes, storage, networks, DNS, firewall etc)
– Can achieve high-availability, scalability and fault tolerance
• Use modern and established tools and frameworks supported by industry
– Reduce risk and improve sustainability
• Offer an agile and scalable environment to use, and a straightforward
platform to extend
http://phenomenal-h2020.eu/
17. Build and test
tools, images,
infrastructure
Docker Hub
PhenoMeNal
Jenkins
PhenoMeNal
Container Hub
Development: Container lifecycle
Source code repositories
18. Two proof of concepts so far
Kultima group Pablo Moreno
19. Implications
• Improve sustainability
– Not dependent on specific data centers
• Improve reliability and security
– Users can run their own service environments (VREs) within isolated
environments
– High-availability and fault tolerance
• Scalability
– Deploy in elastic environments
• Agile development
– Automate “from develop to deploy”
• Agile science
– Simple access to discoverable, scalable tools on elastic compute
resources with no up-front costs
• NB: Many problems of interoperability remains!
– Data
– APIs
– etc.
19
20. Ongoing research on VREs
20
Data
federation
Compute
federation
Privacy
preservation
Workflows
Big Data
frameworks
Data management and
modeling
21. Acknowledgements
Pharmb.io
Wesley Schaal
Maris Lapins
Jonathan Alvarsson
Arvid Berg
Samuel Lampa
Marco Capuccini
Martin Dahlö
Valentin Georgiev
Anders Larsson
Polina Georgiev
Staffan Arvidsson
Laeeq Ahmed
21
AstraZeneca
Lars Carlsson
Ernst Ahlberg-Helgee
University Vienna
David Kreil
Maciej Kańduła
SNIC Cloud
Salman Toor
Andreas Hellander
Caramba.clinic
Kim Kultima
Stephanie Herman
Payam Emami
ToxHQ team
Notes de l'éditeur
Idea with SOA (~2005)
Achieve interoperability by exposing data and functionality as Web services
Experts (scientists) should set up and host their own Web services
Users should integrate a multitude of distributed services, connect into workflows (e.g. Taverna), and share (parts of) workflows
What happened?
Users could not rely on Web services (downtime, API changes, abandoned) and they could not be mirrored
Workflows never gained widespread popularity
Today, stable web services mainly remain at large data and tool providers (EBI, NCBI etc)
Drop applications into VMs running Docker in different clouds.