SpringOne Platform 2017 Mark Pollack, Pivotal "In this session you will learn how you can create data integration and real-time data processing pipelines using Spring Cloud Data Flow and deploy them to multiple platforms – Cloud Foundry, Kubernetes, and YARN to name a few. Spring Cloud Data Flow’s DSL and Designer User Interface will be demonstrated to show how you can easily assemble data pipelines without writing any code for common use-cases. When the need for custom code arises, you can create new application components using the programming model offered by Spring Cloud Stream and Spring Cloud Task. The latest features of Spring Cloud Data Flow will be showcased, which include managing the entire lifecycle of data processing pipelines, allowing for a subset of the applications that compose a stream to be updated independently or rolled back to previous versions and also data provenance of data flowing through the pipelines."

1. Orchestrating Data Microservices
with Spring Cloud Data Flow
By Mark Pollack
@markpollack
1

2. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Speaker Background
● Spring Cloud Skipper
● Spring Cloud Data Flow
● Spring XD
● Spring Shell
● Spring Data
● Spring.NET
● Spring Framework
2

3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is Spring Cloud Data Flow
● Spring Cloud Data Flow lets you easily create data integration processing
pipelines for continuous streams of events or in finite batches.
● Pipelines consist of Spring Boot apps, using Spring Cloud Stream for events
or Spring Cloud Task for batch processes.
● The Data Flow server provides interfaces to compose and deploy pipelines
onto multiple Cloud Platforms using
○ DSLs
○ UI Designer
○ Stream and Task Administration UI
3

4. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Microservices are not just for web apps
● Spring Cloud Stream
○ Event driven based Microservices
○ Loose coupling via pub/sub messaging
○ Opinionated Primitives
○ For integration and stream processing use-cases
4

5. Spring Cloud Stream in a nutshell
Application Core
Messaging
Middleware
Binder
Inputs
Output
s
Spring Boot Configuration
Pluggable Messaging
Middleware:
RabbitMQ, Kafka,
Google PubSub, JMS,
Kinesis
Flexible programming
model:
Spring Integration,
KStream,
Reactor
Standardized
configuration
model

6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Microservices are not just for web apps
● Spring Cloud Task
○ Tasks are short lived Boot Microservices
○ System tracks invocations, exit-status
○ Spring Batch Jobs are wrapped as Tasks
○ Useful for ETL between databases and filesystems
6

7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Data Flow Architecture
7
SCDF Server
sour
ce
Deployer
Implementation

8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Don’t start from scratch!
● Choose from one of the app starters
● Or create a custom stream or task application from Spring Initializr
8

9. UI Tour

10. Let’s do a Twitter Sentiment
Analysis!

11. Ingest Scoring
Storag
e

12. 1 User submits a DSL to SCDF
2
Parse the DSL; resolve
Spring Boot Apps
3
Push Apps; bind services;
stage; deploy
4
Operational metrics and
monitoring
Inges
t
Filter
Predic
t
Stor
e
Data Flow Lifecycle

13. Java DSL

14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Tasks - File Ingest
14
FTP Server
sftpLister | batchJobTransformer
--uri=batch-ingest-1.0.0.jar > :filesAvailable
FTPPollStream
BatchIngestTask DB
LaunchStream
:filesAvailable > task-launcher-local

15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Some Feedback
● Changing deployment properties or app versions means deploying a
completely new stream
● Rely on runtime-platform’s blue-green deployment support for rolling-
upgrades
15

16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Stream Updates
16

17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Stream Updates
17

18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Introducing Spring Cloud Skipper
Skipper is a tool that allows you to
● Search for Spring Boot applications
● Manage their lifecycle on multiple Cloud
Platforms
Use standalone or within CI pipeline
Latest release 1.0 M3
http://cloud.spring.io/spring-cloud-skipper/
18

19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Data Flow Architecture with Skipper
19
SCDF Server
Deployer
Implementations
Skipper Server
sour
ce
Generate Skipper Package

20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Easy Upgrades and Rollbacks
20
app register --name http --type source
--uri maven://org.springframework.cloud.stream.app:http-source-rabbit:1.2.0.RELEASE
app register --name log --type sink
--uri maven://org.springframework.cloud.stream.app:log-sink-rabbit:1.1.0.RELEASE

21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Easy Upgrades and Rollbacks
21
stream create --name httptest --definition "http | log"
stream skipper deploy --name httptest
stream skipper update --name httptest --properties version.log=1.2.0.RELEASE
stream skipper rollback --name httptest
make improvement to log sink, release version 1.2.0

22. Data Flow With Skipper

23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Functions
● Write a java.util.Function for Stream Sources, Processors, and Sinks
23
public class CharCounter implements Function<String, Integer> {
@Override
public Integer apply(String word) {
return word.length();
}
}

24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Functions
24
http --server.port=9001 |
function-runner
--function.className=com.example.functions.CharCounter
--function.location=file:///<PATH/TO/FUNCTION>/myfunction.jar
| log
app register --name function-runner --type processor
--uri http://repo.spring.io/.../function-app-rabbit-1.0.0.jar

25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Roadmap: https://buzz.cfapps.io
25

26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Roadmap: Improved Samples
https://docs.spring.io/spring-cloud-
dataflow/docs/1.3.0.M3/reference/htmlsingle/#dataflow-samples
26

27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
https://cloud.spring.io/spring-cloud-dataflow
27

28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
http://cloud.spring.io/spring-cloud-skipper/
● https://gitter.im/spring-cloud/spring-cloud-skipper
28

29. Thanks!
29
#springone@s1p

30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
STOP! Download Fonts Now
PLEASE DOWNLOAD AND INSTALL PROXIMA NOVA FONTS BEFORE CREATING
YOUR PRESENTATION. You can download the fonts here…
https://brandfolder.com/pivotal Password: keepitsimple
Fonts included in the ZIP file:
Proxima Nova (headline and body text)
http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-mac
http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-windows-
pc
30

31. Java DSL

32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is
intended for information purposes only and may not be incorporated into any
contract. Any information regarding pre-release of Pivotal offerings, future updates
or other planned modifications is subject to ongoing evaluation by Pivotal and is
subject to change. This information is provided without warranty or any kind,
express or implied, and is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing decisions
regarding Pivotal's offerings. These purchasing decisions should only be based on
features currently available. The development, release, and timing of any features
or functionality described for Pivotal's offerings in this presentation remain at the
sole discretion of Pivotal. Pivotal has no obligation to update forward looking
information in this presentation.
32
If you are a Pivotal employee and your talk contains forward facing
information, please include this slide at the beginning or conclusion of
the presentation.
Remove this text box before presenting.

33. Slide Templates

34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Body Slide - Dark Background
All body text is Proxima Nova Regular
• Subhead (18pt)
• Level Two (18pt)
• Level Three (18pt)
• Level Four (18pt)
Use the “Decrease/Increase Indent”
tools to change bullet levels
• Click on the Home ribbon, Paragraph tab
Line spacing is set in master slides
34

35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Two Columns – Dark Background
Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Aenean
commodo ligula eget dolor. Aenean
massa. Cum sociis natoque penatibus
et magnis dis parturient montes,
nascetur ridiculus mus. Donec quam
felis, ultricies nec, pellentesque
Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Aenean
commodo ligula eget dolor. Aenean
massa. Cum sociis natoque penatibus et
magnis dis parturient montes, nascetur
ridiculus mus. Donec quam felis,
ultricies nec, pellentesque
35

36. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Body Slide - Light Background
36
All body text is Proxima Nova Regular
• Subhead (18pt)
• Level Two (18pt)
• Level Three (18pt)
• Level Four (18pt)
Use the “Decrease/Increase Indent”
tools to change bullet levels
• Click on the Home ribbon, Paragraph tab
Line spacing is set in master slides

37. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Two Columns – Light Background
Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Aenean
commodo ligula eget dolor. Aenean
massa. Cum sociis natoque penatibus
et magnis dis parturient montes,
nascetur ridiculus mus. Donec quam
felis, ultricies nec, pellentesque
Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Aenean
commodo ligula eget dolor. Aenean
massa. Cum sociis natoque penatibus et
magnis dis parturient montes, nascetur
ridiculus mus. Donec quam felis,
ultricies nec, pellentesque
37

38. Design Assets

39. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Brand Colors
SpringOne Platform 2017
Theme Colors
39
Spring
Brand Color
Pivotal
Brand Color

40. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Code Slide
40
// This is Andale Mono: 14pt or higher please
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}
…
}

41. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Pivotal Logos on Dark Background
41
Looking for more Pivotal logos, PCF services icons, or OSS logos?
Visit: brandfolder.com/pivotal-assets

42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Pivotal Logos on Light Background
42
Looking for more Pivotal logos, PCF services icons, or OSS logos?
Visit: brandfolder.com/pivotal-assets

43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Event Logos
43

44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Logo and Project Icons
44
Spring
Framewor
k
Spring
Securit
y
Sprin
g
Data
Sprin
g
Batch
Spring
Integratio
n
Project
Reacto
r
Sprin
g
AMQ
P
Spring
Hateoa
s
Spring
Mobil
e
Spring
Androi
d
Sprin
g
Social
Spring
Web
Services
Spring
Web
Flow
Sprin
g
XD
Sprin
g
Boot
Sprin
g
LDAP
Spring
Tool Suite
Spring
Cloud
Data Flow
Spring
Kafka
Spring
Cloud

45. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Sample Table
45
2007 2008 2009 2010
Category 1 2.4 6.4 4.1 6.6
Category 2 8.2 4.5 3.2 3.8
Category 3 4.6 3.2 1.9 9.6
Category 4 6.7 3.3 3.4 2.2
Category 5 4.3 5.6 7.1 3.4

46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Sample Table
46
2007 2008 2009 2010
Category 1 2.4 6.4 4.1 6.6
Category 2 8.2 4.5 3.2 3.8
Category 3 4.6 3.2 1.9 9.6
Category 4 6.7 3.3 3.4 2.2
Category 5 4.3 5.6 7.1 3.4

47. Learn More. Stay Connected.
<Your CTA>
<Related Session>
47
#springone@s1p