Scalaz-stream introduced the Scala ecosystem to purely functional streams. Although widely deployed in production at Verizon, SlamData, and elsewhere, the last release of scalaz-stream was October 23rd, 2016, and newer projects such as FS2, Monix Iterant, and others have risen to fill the gap, each addressing different use cases in the functional programming community. In this talk, John A. De Goes, architect of the Scalaz 8 effect system, unveils a new design for a next-generation version of scalaz-stream. Designed to embrace the powerful features of the Scalaz 8 effect system—features not available in any other effect type—this design features higher-performance, stronger guarantees, and far less code than previous incarnations. You are invited to discover the practicality of functional programming, as well as its elegance, beauty, and composability, in this unique presentation available exclusively at Scale By the Bay.
38. Stream
PERFORMANCEMOTIVATION API DESIGN SUMMARY
type StreamIO[A] = Stream[IOException, A]
val bytes: StreamIO[Byte] = …
val headersAndContent:
Managed[IOException,
(ContentType, StreamIO[Byte])] =
bytes.peel(parseContentType)
headersAndContent.use {
case (ContentType(Json), content) =>
content.transduce(toJson)...
case (ContentType(Xml), content) =>
content.transduce(toXml)...
}
stream.map(f)
stream.flatMap(f)
stream.filter(f)
stream1 ++ stream2
stream.transduce(what)
stream.scan(z)(f)
stream.take(n)
stream.takeWhile(f)
stream.drop(n)
stream.dropWhile(f)
stream.foreach(f)
stream1.zip(stream2)
stream1.merge(stream2)
stream1.joinWith(stream2)(f)
stream.peel(what)
39. Sink
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Sink[E, A0, A, B]
May fail with a checked error of type E
May effectfully produce a result of type B
Consumes values of type A
May possibly output a remainder of type A0
41. Sink
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Monoid
append
zero
Applicative
map
point
ap
Monad
bind
Bifunctor
leftMap
bimap
Alternative
alt
empty
Category
id
compose
Profunctor
lmap
rmap
dimap
Strong
first
second
Choice
left
right
60. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Start with stream of integers (chunked)
Filter for only even integers
Convert to longs
Sum all integers
61. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
def akkaChunkFilterMapSum = {
val chunks = (1 to chunkCount).flatMap(i =>
Array.fill(chunkSize)(i))
val program = AkkaSource
.fromIterator(() => chunks.iterator)
.filter(_ % 2 == 0)
.map(_.toLong)
.toMat(AkkaSink.fold(0L)(_ + _))(Keep.right)
Await.result(program.run, Duration.Inf)
}
62. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
63. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
def fs2ChunkFilterMapSum = {
val chunks = (1 to chunkCount).map(i =>
FS2Chunk.array(Array.fill(chunkSize)(i)))
val stream = FS2Stream(chunks: _*)
.flatMap(FS2Stream.chunk(_))
.filter(_ % 2 == 0)
.map(_.toLong)
.covary[CatsIO]
.compile
.fold(0L)(_ + _)
stream.unsafeRunSync
}
64. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
FS2
65. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
def scalazChunkFilterMapSum = {
val chunks = (1 to chunkCount).map(i =>
Chunk.fromArray(Array.fill(chunkSize)(i)))
val stream = StreamChunk
.fromChunks(chunks: _*)
.filter(_ % 2 == 0)
.map(_.toLong)
val sink = Sink.foldLeft(0L)((s, c) => c.foldl(s)(_ + _))
unsafeRun(stream.run(sink))
}
66. Filter · Map · Sum
PERFORMANCEMOTIVATION API DESIGN SUMMARY
FS2
67. CSV Tokenize
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Start with stream of characters (chunked)
Identify CSV separators and columns
Convert character stream into token stream
76. Initial Encoding
sealed trait Stream[+A] Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
PERFORMANCEMOTIVATION API DESIGN SUMMARY
77. Initial Encoding
sealed trait Stream[+A]
case class Emit[A](a: A) extends Stream[A]
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
78. Initial Encoding
sealed trait Stream[+A]
case class Emit[A](a: A) extends Stream[A]
case class Map[A0, A](s: Stream[A0],
f: A0 => A) extends Stream[A]
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
79. Initial Encoding
sealed trait Stream[+A]
case class Emit[A](a: A) extends Stream[A]
case class Map[A0, A](s: Stream[A0],
f: A0 => A) extends Stream[A]
case class Join[A](s: Stream[Stream[A]])
extends Stream[A]
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
80. Initial Encoding
sealed trait Stream[+A]
case class Emit[A](a: A) extends Stream[A]
case class Map[A0, A](s: Stream[A0],
f: A0 => A) extends Stream[A]
case class Join[A](s: Stream[Stream[A]])
extends Stream[A]
case class Merge[A](l: Stream[A],
r: Stream[A]) extends Stream[A]
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
81. Initial Encoding
sealed trait Stream[+A]
case class Emit[A](a: A) extends Stream[A]
case class Map[A0, A](s: Stream[A0],
f: A0 => A) extends Stream[A]
case class Join[A](s: Stream[Stream[A]])
extends Stream[A]
case class Merge[A](l: Stream[A],
r: Stream[A]) extends Stream[A]
case class Fold[A0, A](a: A,
s: Stream[A0],
f: (A, A0) => A) extends Stream[A]
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
82. Initial Encoding
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
sealed trait Stream[+A]
case class Emit[A](a: A) extends Stream[A]
case class Map[A0, A](s: Stream[A0],
f: A0 => A) extends Stream[A]
case class Join[A](s: Stream[Stream[A]])
extends Stream[A]
case class Merge[A](l: Stream[A],
r: Stream[A]) extends Stream[A]
case class Fold[A0, A](a: A,
s: Stream[A0],
f: (A, A0) => A) extends Stream[A]
case class Effect[A](io: IO[A]) extends Stream[A]
83. Initial Encoding
def drain[A](s: Stream[A]): IO[Unit] =
s match {
case Emit(a) => …
...
}
Interpreter
PERFORMANCEMOTIVATION API DESIGN SUMMARY
85. Final Encoding
case class Stream[+A](
run: (A => IO[Unit]) => IO[Unit])
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
86. Final Encoding
case class Stream[+A](
run: (A => IO[Unit]) => IO[Unit])
object Stream {
def emit[A](a: => A): Stream[A] =
Stream[A](read => read(a))
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
87. Final Encoding
case class Stream[+A](
run: (A => IO[Unit]) => IO[Unit]) {
def map[B](f: A => B): Stream[B] =
Stream(r => run(r.compose(f)))
}
object Stream {
def emit[A](a: => A): Stream[A] =
Stream[A](read => read(a))
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
88. Final Encoding
case class Stream[+A](
run: (A => IO[Unit]) => IO[Unit]) {
def map[B](f: A => B): Stream[B] =
Stream(r => run(r.compose(f)))
}
object Stream {
def emit[A](a: => A): Stream[A] =
Stream[A](read => read(a))
def join[A](s: Stream[Stream[A]]) =
Stream(r => s.run(sa => sa.run(r)))
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
89. Final Encoding
case class Stream[+A](
run: (A => IO[Unit]) => IO[Unit]) {
def map[B](f: A => B): Stream[B] =
Stream(r => run(r.compose(f)))
def merge[A1 >: A](that: Stream[A1]) =
Stream[A1](r =>
run(r).par(that.run(r)).void)
}
object Stream {
def emit[A](a: => A): Stream[A] =
Stream[A](read => read(a))
def join[A](s: Stream[Stream[A]]) =
Stream(r => s.run(sa => sa.run(r)))
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
90. Final Encoding
case class Stream[+A](
run: (A => IO[Unit]) => IO[Unit]) {
def map[B](f: A => B): Stream[B] =
Stream(r => run(r.compose(f)))
def merge[A1 >: A](that: Stream[A1]) =
Stream[A1](r =>
run(r).par(that.run(r)).void)
def fold[S](s: S)(f: (S, A) => S) = ...
}
object Stream {
def emit[A](a: => A): Stream[A] =
Stream[A](read => read(a))
def join[A](s: Stream[Stream[A]]) =
Stream(r => s.run(sa => sa.run(r)))
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Type
Emission
Mapping
Joining
Merging
Folding
Effect
...
93. Constant space processing
Discovered in 2003
Aggressively lazy
ITERATEES
PERFORMANCEMOTIVATION API DESIGN SUMMARY
Final encoding-friendly
Automatically leak-free
“Left folds”
94. Iteratees 101
sealed trait Iteratee[-A, +B]
case class More[A, B](f: A => Iteratee[A, B]) extends Iteratee[A, B]
case class Done[B](value: B) extends Iteratee[Any, B]
sealed trait Enumerator[+A] { self =>
def run[A1 >: A, B](iteratee: Iteratee[A1, B]): IO[B]
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
95. Iteratees 101
sealed trait Enumerator[+A] { self =>
final def run[A1 >: A, B](iteratee: Iteratee[A1, B]): IO[B] =
fold(...)(...)
def fold[B, S](s: S)(f: (S, A) => Either[S, B]): IO[B]
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
96. Iteratees 101
sealed trait Enumerator[+A] { self =>
final def run[A1 >: A, B](iteratee: Iteratee[A1, B]): IO[B] =
fold(...)(...)
def fold[B, S](s: S)(f: (S, A) => Either[S, B]): IO[B]
final def map[B](f0: A => B): Enumerator[B] =
new Enumerator[B] {
def fold[C, S](s: S)(f: (S, B) => Either[S, C]): IO[C] =
self.fold(s)((s, a) => f(s, f0(a)))
}
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
97. Iteratees 101
sealed trait Enumerator[+A] { self =>
final def run[A1 >: A, B](iteratee: Iteratee[A1, B]): IO[B] =
fold(...)(...)
def fold[B, S](s: S)(f: (S, A) => Either[S, B]): IO[B]
final def map[B](f0: A => B): Enumerator[B] =
new Enumerator[B] {
def fold[C, S](s: S)(f: (S, B) => Either[S, C]): IO[C] =
self.fold(s)((s, a) => f(s, f0(a)))
}
final def filter(f0: A => Boolean): Enumerator[A] =
new Enumerator[A] {
def fold[B, S](s: S)(f: (S, A) => Either[S, B]): IO[B] =
self.fold(s)((s, a) => if (f0(a)) f(s, a) else Left(s))
}
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
103. Iteratees 101
sealed trait IterateeT[F[_], -A, +B]
case class More[F[_], A, B](f: A => IterateeT[F, A, B]) extends IterateeT[F, A, B]
case class Done[F[_], B](value: B) extends IterateeT[F, Any, B]
case class Effect[F[_], B](fx: F[B]) extends IterateeT[F, Any, B]
PERFORMANCEMOTIVATION API DESIGN SUMMARY
104. Iteratees 101
sealed trait IterateeT[F[_], -A, +B]
case class More[F[_], A, B](f: A => IterateeT[F, A, B]) extends IterateeT[F, A, B]
case class Done[F[_], B](value: B) extends IterateeT[F, Any, B]
case class Effect[F[_], B](fx: F[B]) extends IterateeT[F, Any, B]
type Parallel[F[_], A1, A2, B] =
IterateeT[IterateeT[F, A1, ?], A2, B]
def lift[F[_], A1, A2, B](it: IterateeT[F, A1, B]): Parallel[F, A1, A2, B] =
Effect[IterateeT[F, A1, ?], B](it)
PERFORMANCEMOTIVATION API DESIGN SUMMARY
105. Iteratees 101
sealed trait IterateeT[F[_], -A, +B]
case class More[F[_], A, B](f: A => IterateeT[F, A, B]) extends IterateeT[F, A, B]
case class Done[F[_], B](value: B) extends IterateeT[F, Any, B]
case class Effect[F[_], B](fx: F[B]) extends IterateeT[F, Any, B]
type Parallel[F[_], A1, A2, B] =
IterateeT[IterateeT[F, A1, ?], A2, B]
def lift[F[_], A1, A2, B](it: IterateeT[F, A1, B]): Parallel[F, A1, A2, B] =
Effect[IterateeT[F, A1, ?], B](it)
def readLeft[F[_], A1, A2]: Parallel[F, A1, A2, A1] =
lift(More(a => Done(a)))
def readRight[F[_], A1, A2]: Parallel[F, A1, A2, A2] =
More(a => Done(a))
PERFORMANCEMOTIVATION API DESIGN SUMMARY
106. Iteratees 101
sealed trait IterateeT[F[_], -A, +B]
case class More[F[_], A, B](f: A => IterateeT[F, A, B]) extends IterateeT[F, A, B]
case class Done[F[_], B](value: B) extends IterateeT[F, Any, B]
case class Effect[F[_], B](fx: F[B]) extends IterateeT[F, Any, B]
type Parallel[F[_], A1, A2, B] =
IterateeT[IterateeT[F, A1, ?], A2, B]
def lift[F[_], A1, A2, B](it: IterateeT[F, A1, B]): Parallel[F, A1, A2, B] =
Effect[IterateeT[F, A1, ?], B](it)
def readLeft[F[_], A1, A2]: Parallel[F, A1, A2, A1] =
lift(More(a => Done(a)))
def readRight[F[_], A1, A2]: Parallel[F, A1, A2, A2] =
More(a => Done(a))
PERFORMANCEMOTIVATION API DESIGN SUMMARY
107. ZIO Stream
trait Stream[E, A] {
def fold[S](s: S)(f: (S, A) => IO[E, Step[S]]): IO[E, Step[S]]
}
object Stream {
sealed trait Step[S]
object Step {
case class Stop[S](value: S) extends Step[S]
case class Cont[S](value: S) extends Step[S]
}
}
PERFORMANCEMOTIVATION API DESIGN SUMMARY
108. ZIO Stream
trait Stream[E, A] {
def fold[S](s: S)(f: (S, A) => IO[E, Step[S]]): IO[E, Step[S]]
}
object Stream {
sealed trait Step[S]
object Step {
case class Stop[S](value: S) extends Step[S]
case class Cont[S](value: S) extends Step[S]
}
}
Continuation Monad!
PERFORMANCEMOTIVATION API DESIGN SUMMARY
117. Fusion, Queue &
complex combinators
Performance
Outstanding Work
Comprehensive test
suites and great docs
Tests & Docs
Combinators for
every reason
Combinators
PERFORMANCEMOTIVATION API DESIGN SUMMARY
118. Eric Torreborre — Beautiful Folds
The FS2 team — Michael Pilquist, Fabio Labella, Pavel Chlupacek, et al
Special thanks to all people who made ZIO Stream possible.
Credits
Oleg Kiselyov — Iteratees 2003
PERFORMANCEMOTIVATION API DESIGN SUMMARY
119. Any questions?
Thanks!
John A. De Goes
@jdegoes - degoes.net
Itamar Ravid
@itrvd - iravid.com
PERFORMANCEMOTIVATION API DESIGN SUMMARY