Reactive Streams 1.0 and Akka Streams

Konrad 'ktoso' Malawski
GeeCON 2014 @ Kraków, PL
Konrad `@ktosopl` Malawski
(Channeled by @deanwampler)
streams
Rise of the 1.0 :
reactive streams

Konrad `@ktosopl` Malawski
akka.io
typesafe.com
geecon.org
Java.pl / KrakowScala.pl
sckrk.com / meetup.com/Paper-Cup @ London
GDGKrakow.pl
lambdakrk.pl
(we’re renaming soon!)

@deanwampler
dean.wampler@typesafe.com
(we’re renaming soon!)

Streams
“You cannot enter the same river twice”
~ Heraclitus
http://en.wikiquote.org/wiki/Heraclitus

Streams
Real Time Stream Processing
When you attach “late” to a Publisher,
you may miss initial elements – it’s a river of data.
http://en.wikiquote.org/wiki/Heraclitus

Reactive Streams
Stream processing

Reactive Streams
Back-pressured
Stream processing

Reactive Streams
Back-pressured
Asynchronous
Stream processing

Reactive Streams
Back-pressured
Asynchronous
Stream processing
Standardised (!)

Reactive Streams: Goals
1. Back-pressured Asynchronous Stream processing
2. Standard implemented by many libraries

Reactive Streams
http://www.reactive-streams.org/

Reactive Streams - Who?
http://reactive-streams.org
Kaazing Corp.
RxJava @ Netﬂix,
Reactor @ Pivotal (SpringSource),
Vert.x @ Red Hat,
Twitter,
Akka Streams, Slick @ Typesafe,
Spray @ Spray.io,
Oracle,
OpenJDK (Java 9) – Doug Lea - SUNY Oswego
…

Reactive Streams - Inter-op
We want to make different implementations
co-operate with each other.

The different implementations “talk to each other”
using the Reactive Streams protocol.

The Reactive Streams SPI is NOT meant to be user-api.
You should use one of the implementing libraries.

Reactive Streams - Inter-op example
import ratpack.rx.RxRatpack
import ratpack.test.embed.EmbeddedApp
import ratpack.handling.Handler
import ratpack.handling.Context
import rx.Observable
import scala.collection.JavaConverters._
import akka.stream.scaladsl.Flow
import akka.stream.scaladsl.Source
import rx.RxReactiveStreams
import akka.stream.scaladsl.Sink
import akka.actor.ActorSystem
import akka.stream.FlowMaterializer
import ratpack.http.ResponseChunks
import java.util.function.Consumer
import ratpack.test.http.TestHttpClient

import ratpack.rx.RxRatpack
import ratpack.test.embed.EmbeddedApp
import ratpack.handling.Handler
import ratpack.handling.Context
import rx.Observable
import scala.collection.JavaConverters._
import akka.stream.scaladsl.Flow
import akka.stream.scaladsl.Source
import rx.RxReactiveStreams
import akka.stream.scaladsl.Sink
import akka.actor.ActorSystem
import reactor.rx.Streams

import reactor.rx.Streams
object ScalaMain extends App {
val system = ActorSystem("InteropTest")
implicit val mat =
FlowMaterializer()(system)
RxRatpack.initialize()
val handler = new Handler {
override def handle(ctx: Context): Unit ={
// RxJava Observable
val intObs =
Observable.from((1 to 10).asJava)

RxRatpack.initialize()
val handler = new Handler {
override def handle(ctx: Context): Unit ={
// RxJava Observable
val intObs =
Observable.from((1 to 10).asJava)
// Reactive Streams Publisher
val intPub =
RxReactiveStreams.toPublisher(intObs)
// Akka Streams Source
val stringSource =
Source(intPub).map(_.toString)
val stringPub =
stringSource.runWith(
Sink.fanoutPublisher(1, 1))

val intPub =
RxReactiveStreams.toPublisher(intObs)
// Akka Streams Source
val stringSource =
Source(intPub).map(_.toString)
val stringPub =
// Reactor Stream
val reactor.function.Function
val linesStream =
Streams.create(stringPub).map[String](
new Function[String, String] {
override def apply(in: String) = in+"n"
})

val stringPub =
// Reactor Stream
val reactor.function.Function
val linesStream =
Streams.create(stringPub).map[String](
new Function[String, String] {
override def apply(in: String) = in+"n"
})
// and now render the HTTP response
ctx.render(
ResponseChunks.stringChunks(linesStream))
}
})
EmbeddedApp.fromHandler(handler).test(

// and now render the HTTP response
ctx.render(
ResponseChunks.stringChunks(linesStream))
}
})
EmbeddedApp.fromHandler(handler).test(
new Consumer[TestHttpClient] {
override def accept(
client: TestHttpClient): Unit = {
val text = client.getText()
println(text)
system.shutdown()
}
})
}

public interface Subscriber<T> {
public void onSubscribe(Subscription s);
public void onNext(T t);
public void onError(Throwable t);
public void onComplete();
}
public interface Publisher<T> {
public void subscribe(Subscriber<? super T> s);
}
public interface Subscription {
public void request(long n);
public void cancel();
}

Back-pressure? Example Without
Publisher[T] Subscriber[T]

Back-pressure? Example Without
Fast Publisher Slow Subscriber

Back-pressure?
“Why would I need that!?”

Back-pressure? Push + NACK model

Subscriber usually has some kind of buffer.

What if the buffer overﬂows?

Back-pressure? Push + NACK model (a)
Use bounded buffer,
drop messages + require re-sending

Back-pressure? Push + NACK model (a)
Kernel does this!
Routers do this!
(TCP)
Use bounded buffer,
drop messages + require re-sending

Back-pressure? Push + NACK model (b)
Increase buffer size…
Well, while you have memory available!

Back-pressure? Push + NACK model (b)

NACKing is NOT enough!
Negative ACKnowledgement

Back-pressure? Example NACKing
Buffer overﬂow is imminent!

Telling the Publisher to slow down / stop sending…

NACK did not make it in time,
because M was in-ﬂight!

Back-pressure?
speed(publisher) < speed(subscriber)

Back-pressure? Fast Subscriber, No Problem
No problem!

Back-pressure?
Reactive-Streams
=
“Dynamic Push/Pull”

Just push – not safe when Slow Subscriber
Just pull – too slow when Fast Subscriber
Back-pressure? RS: Dynamic Push/Pull

Solution:
Dynamic adjustment
Just push – not safe when Slow Subscriber
Just pull – too slow when Fast Subscriber

Slow Subscriber sees it’s buffer can take 3 elements.
Publisher will never blow up it’s buffer.

Fast Publisher will send at-most 3 elements.
This is pull-based-backpressure.

Fast Subscriber can issue more Request(n),
before more data arrives!

Fast Subscriber can issue more Request(n),
before more data arrives.
Publisher can accumulate demand.

Back-pressure? RS: Accumulate demand
Publisher accumulates total demand per subscriber.

Back-pressure? RS: Accumulate demand
Total demand of elements is safe to publish.
Subscriber’s buffer will not overﬂow.

Back-pressure? RS: Requesting “a lot”
Fast Subscriber can issue arbitrary large requests,
including “gimme all you got” (Long.MaxValue)

Akka
• Fault tolerant
• Supervision hierarchies
• Failure detection
• Asynchronous data processing
• Optimised for high performance
• both in-jvm and across network
• Adaptive Cluster
• Load balancing among Cluster Nodes
Actor 131 Actor 132
Supervisor 1
Actor 12 Actor 13
Actor 111 Actor 112
Supervisor 11

Akka
Akka has multiple modules:
akka-actor: actors (concurrency abstraction)
akka-camel: integration
akka-remote: remote actors
akka-cluster: clustering
akka-persistence: CQRS / Event Sourcing
akka-streams: stream processing
…

Akka
Akka is a high-performance concurrency
library for Scala and Java.
At it’s core it focuses on the Actor Model:

An Actor can only:
• Send and receive messages
• Create Actors
• Change it’s behaviour
Akka
Akka is a high-performance concurrency
library for Scala and Java.
At it’s core it focuses on the Actor Model:

class Player extends Actor {
def receive = {
case NextTurn => sender() ! decideOnMove()
}
def decideOnMove(): Move = ???
}
Akka Actors

Akka
Actors are:
a distribution and concurrency abstraction.
Streams are:
a way to describe how data ﬂows through a system.

Akka Streams – Linear Flow
Flow[Double].map(_.toInt). [...]
No Source attached yet.
“Pipe ready to work with Doubles”.

implicit val sys = ActorSystem()
implicit val mat = ActorMaterializer()
Source(1 to 3).runWith(Sink.foreach(println))

// sugar for runWith
Source(1 to 3).foreach(println)

// sugar for runWith
Source(1 to 3).foreach(println)
Sink.fold
Sink.head
Sink.ignore
Sink.publisher
Sink.cancelled
// your own Sink
…

Akka Streams – Flows are reusable
sink.runWith(Source(1 to 10))
source.runWith(Sink.ignore)
source.runWith(Sink.foreach(println))
Multiple materializations
val ready = source.to(Sink.ignore)
ready.run()
ready.run()

Akka Streams <-> Actors – Advanced
val subscriber = ActorSubscriber(
system.actorOf(Props[SubStreamParent], ”parent”)
)
Source(1 to 100)
.map(_.toString)
.filter(_.length == 2)
.drop(2)
.conflate(seed => seed)((acc, i) => acc + i)
.groupBy(_.last)
.runWith(subscriber)
All the usual ops available for Linear Flows.

)
Source(1 to 100)
.map(_.toString)
.drop(2)
.groupBy(_.last)
Aggregating values until downstream demand comes.

)
Source(1 to 100)
.map(_.toString)
.drop(2)
.groupBy(_.last)
Creates a stream of streams:
Source[(Int, Source[String])]

Akka Streams: Graphs
val p: Publisher[String] =
FlowGraph.closed(out) { implicit b o
val merge = b.add(new StrictRoundRobin[String]) 
in1 ~> merge.in(0)
in2 ~> merge.in(1)
 
merge.out ~> o.inlet
}.run()
val in1 = Source(List("a", "b", "c", "d"))
val in2 = Source(List("e", "f"))
val out = Sink.publisher[String]

in1 ~> merge.in(0)
in2 ~> merge.in(1)
 
}.run()
Sink[String, Publisher[String]]
imports Graphs

in1 ~> merge.in(0)
in2 ~> merge.in(1)
 
}.run()
Sink[String, Publisher[String]]
materializes a Publisher[String]

Reactive Streams
Bigger than Scala-ecosystem - JDK-wide (and wider).
Inter-operable back-pressure protocol.
Future work: reactive-streams-io, reactive-streams-js
Akka Streams - one of the leading Reactive Streams impls.
Complex in-memory stream processing.
SlickR - provides Reactive Stream from DB queries
Akka Http - Akka Streams based; “Spray 2.0”

Available Sources
•FutureSource
•IterableSource
•IteratorSource
•PublisherSource
•SubscriberSource
•ThunkSource
•SynchronousFileSource
•TickSource (timer based)
•… easy to add your own!

Available operations
• drop / dropWithin
• take / takeWithin
• filter
• groupBy
• grouped
• transform
• buffer
• collect
• expand
• splitWhen / splitAfter
• map
• scan
• prefixAndTail
• … easy to add your own!
“Rate – detaching” operations:
• buffer
• collect
• concat
• conflate

Available Sinks
• BlackHoleSink
• FoldSink
• ForeachSink
• FutureSink
• OnCompleteSink
• UdpSink [next release]
• SynchronousFileSink
• PublisherSink / FanoutPublisherSink
• SubscriberSink
• FileSink

Available Junctions
• Broadcast
• Merge
• FlexiMerge
• Route
• FlexiRoute
• Zip
• ZipWith
• Unzip
• UnzipWith
• Concat

Community ecosystem
• Kafka
• Spark (soon?)
• Slick
• S3
• Files
• MongoDB
• …

Links
• The projects:
• http://akka.io
• http://reactive-streams.org 
• Akka User - mailing list:
• https://groups.google.com/group/akka-user

Tänan!
Dzięki!
Thanks!
ありがとう！
ktoso @ typesafe.com
t: ktosopl / g: ktoso
blog: project13.pl

Reactive Streams 1.0 and Akka Streams

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Reactive Streams 1.0 and Akka Streams

Similaire à Reactive Streams 1.0 and Akka Streams (20)

Plus de Dean Wampler

Plus de Dean Wampler (7)

Dernier

Dernier (20)

Reactive Streams 1.0 and Akka Streams