Quark is a new Scala DSL for data processing and analytics that runs on top of the Quasar Analytics compiler. Quark is adept at processing semi-structured data and compiles query plans to operations that run entirely inside a target data source. In this presentation, John A. De Goes provides an overview of the open source library, showing several use cases in data processing and analytics. John also demonstrates a powerful technique that every developer can use to create their own purely-functional, type-safe DSLs in the Scala programming language.
2. Apache Spark
Apache Spark is a fast and general engine for big data
processing, with built-in modules for streaming, SQL,
machine learning and graph processing.
val textFile = sc.textFile("hdfs://...")
val counts =
textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
4. Why Does Spark Have to Suck?
Computation
val textFile = sc.textFile("hdfs://...")
val counts =
textFile.flatMap(line => line.split(" ")) <---- Where Spark goes wrong
.map(word => (word, 1)) <---- Where Spark goes wrong
.reduceByKey(_ + _) <---- Where Spark goes wrong
5. WWFPD?
— Purely functional
— No exceptions, no casts, no nulls
— No global variables
— No serialization
— Safe type-safe programs
— First-class support for databases
— Few dependencies
— Better debugging
— Implementation-independent performance
6. Rule #1 in Functional
Programming
Don't solve the problem, describe the solution.
AKA the "Do Nothing" rule
=> Don't compute, embed a compiled language into
Scala
7. Quark
Compilation
Quark is a Scala DSL built on Quasar Analytics, a general-
purpose compiler for translating data processing over
semi-structured data into efficient plans that execute
100% inside the target infrastructure.
val textFile = Dataset.load("...")
val counts =
textFile.flatMap(line => line.typed[Str].split(" "))
.map(word => (word, 1))
.reduceByKey(_.sum)
8. More Quark
Compilation
val dataset = Dataset.load("/prod/profiles")
val averageAge = dataset.groupBy(_.country[Str]).map(_.age[Int]).reduceBy(_.average)
9. Quark Targets
One DSL to Rule Them All
— MongoDB
— Couchbase
— MarkLogic
— Hadoop / HDFS
— Add your connector here!
10. Both Quark and Quasar Analytics are purely-functional,
open source projects written in 100% Scala.
https://github.com/quasar-analytics/
11. How To DSL
Adding Integers
sealed trait Expr
final case class Integer(v: Int) extends Expr
final case class Addition(v: Expr, v: Expr) extends Expr
def int(v: Int): Expr = Integer(v)
def add(l: Expr, r: Expr): Expr = Addition(l, r)
add(add(int(1), int(2)), int(3)) : Expr
def interpret(e: Expr): Int = e match {
case Integer(v) => v
case Addition(l, r) => interpret(l) + interpret(r)
}
def serialize(v: Expr): Json = ???
def deserialize(v: Json): Expr = ???
12. How To DSL
Adding Strings
sealed trait Expr
final case class Integer(v: Int) extends Expr
final case class Addition(l: Expr, r: Expr) extends Expr // Uh, oh!
final case class Str(v: String) extends Expr
final case class StringConcat(l: Expr, r: Expr) extends Expr // Uh, oh!
13. How To DSL
Phantom Type
sealed trait Expr[A]
final case class Integer(v: Int) extends Expr[Int]
final case class Addition(l: Expr[Int], r: Expr[Int]) extends Expr[Int]
final case class Str(v: String) extends Expr[String]
final case class StringConcat(l: Expr[String], r: Expr[String]) extends Expr[String]
def interpret[A](e: Expr[A]): A = e match {
case Integer(v) => v
case Addition(l, r) => interpret(l) + interpret(r)
case Str(v) => v
case StringConcat(l, r) => interpret(l) ++ interpret(r)
}
def serialize[A](v: Expr[A]): Json = ???
def deserialize[Z](v: Json): Expr[A] forSome { type A } = ???
14. How To DSL
GADTs in Scala still have bugs
SI-8563, SI-9345, SI-6680
FRIENDS DON'T LET FRIENDS USE GADTS IN SCALA.
16. How To DSL
Finally Tagless
type Id[A] = A
def interpret: Expr[Id] = new Expr[Id] {
def int(v: Int): Id[Int] = v
def str(v: String): Id[String] = v
def add(l: Id[Int], r: Id[Int]): Id[Int] = l + r
def concat(l: Id[String], r: Id[String]): Id[String] = l + r
}
add(int(1), int(2)).apply(interpret) // Id(3)
final case class Const[A, B](a: A)
def serialize: Expr[Const[Json, ?]] = ???
def deserialize[F[_]: Expr](json: Json): F[A] forSome { type A } = ???
17. Quark 101
The Building Blocks
— Type. Represents a reified type of an element in a dataset.
— **Dataset[A]**. Represents a dataset, produced by successive
application of set-level operations (SetOps). Describes a directed-
acyclic graph.
— **MappingFunc[A, B]**. Represents a function from A to B that is
produced by successive application of mapping-level operations
(MapOps) to the input.
— **ReduceFunc[A, B]**. Represents a reduction from A to B, produced
by application of reduction-level operations (ReduceOps) to the input.
19. Mini-Quark
Type System
sealed trait Type
object Type {
final case class Unknown() extends Type
final case class Timestamp() extends Type
final case class Date() extends Type
final case class Time() extends Type
final case class Interval() extends Type
final case class Int() extends Type
final case class Dec() extends Type
final case class Str() extends Type
final case class Map[A <: Type, B <: Type](key: A, value: B) extends Type
final case class Arr[A <: Type](element: A) extends Type
final case class Tuple2[A <: Type, B <: Type](_1: A, _2: B) extends Type
final case class Bool() extends Type
final case class Null() extends Type
type UnknownMap = Map[Unknown, Unknown]
val UnknownMap : UnknownMap = Map(Unknown(), Unknown())
type UnknownArr = Arr[Unknown]
val UnknownArr : UnknownArr = Arr(Unknown())
type Record[A <: Type] = Map[Str, A]
type UnknownRecord = Record[Unknown]
}