A discussion about databases, how they fit into the world, and how we might be able to leverage certain models to beat the speed of light.

1. Databases
Sargun Dhillon
@Sargun

2. What is a database?
A database is an organized collection of data

3. What are databases
for?
Applications

4. Internet Applications
Experiencing exploding growth

5. Internet Traffic vs. Penetration
0
25
50
75
100
0
10000
20000
30000
40000
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
IP Traffic (PB/mo) Global Penetration (%)

6. Number of Internet Users in 2012

7. Average Distance to Every Human

8. Extrapolating
We have not yet reached Peak “Web” and we won’t see
it for some time

9. Applications
How are they built?

10. Basic Application

11. Useful Application
Add Persistence

12. Scale Out

13. Scale Out with Correctness

14. What is a Transaction?
A Unit of Work

15. Transaction Scheduling
Concurrent Operations

16. Non-Conflicting Concurrency
Parallel Execution

17. ACID

18. ACID = Atomicity
A transaction executes or it does not

19. ACID = Consistency
Correctness; Require the database to follow set of
invariants

20. ACID = Isolation
Prevent inter-actor visibility during concurrent operations

21. ACID = Durability
Once you write, it will survive

22. Lifecycle of a Transaction

23. Vertically Scalability
Moore’s Law can take us places

24. Biggest AWS Database
• vCPUs: 32
• Memory: 244
• Storage: 3TB
• IOPs: 30,000 IOPs
• Networking: 10 Gigabit
• Resiliency: Multi-AZ
• SLA: 99.95%
• Backend: Postgresql

25. $141,052.66/yr

26. Scaling Beyond

27. Sharding?

29. Do we have a natural
sharding key?

30. Add a Coordinator?

31. Two-phase commit?
Three-phase commit?
Paxos?
Enhanced Three-phase commit?
Wat?
Egalitarian Paxos?

32. Do we really want to
run NxM databases?

33. Partial Availability

34. Failure detectors are
hard

35. Database Failure

36. Cascading App Failure

37. Recovery

38. Hotspots?
(The “Bieber” problem)

39. Scaling SSI databases
is a hard problem

40. What if want
multidatacenter?

44. No latency win for
mutable data

45. Must sacrifice recency
for latency win

46. Complex Routing
Semantics

47. Multi-master requires
at least 1 RTT

48. -F1: A Distributed SQL Database That Scales, Google
“Because the data is synchronously replicated
across multiple datacenters, and because
we’ve chosen widely distributed datacenters,
the commit latencies are relatively high (50-150
ms).”

49. -Kohavi and Longbotham 2007
“Every 100 ms increase in load time of
Amazon.com decreased sales by 1%.”
(~$120M of losses per 100 ms)

50. “Average partition duration ranged from 6 minutes for
software-related failures to more than 8.2 hours for
hardware-related failures (median 2.7 and 32 minutes;
95th percentile of 19.9 minutes and 3.7 days,
respectively).”
-The Network is Reliable
WANs Fail

51. Is there another way?

52. Eventually
Consistent
Systems

53. -F1: A Distributed SQL Database That Scales, Google
“We also have a lot of experience with eventual
consistency systems at Google. In all such
systems, we find developers spend a
significant fraction of their time building
extremely complex and error-prone
mechanisms to cope with eventual consistency
and handle data that may be out of date. We
think this is an unacceptable burden to place
on developers and that consistency problems
should be solved at the database level. ”

54. CAP Theorem

55. “A shared-data system can have at most
two of the three following properties:
Consistency, Availability, and tolerance to
network Partitions.”
-Dr. Eric Brewer

56. On Consistency
• ACID Consistency: Any transaction, or operation
will bring the database from one valid state to
another
• CAP Consistency: All nodes see the same data at
the same time (synchrony)

57. On Partition Tolerance
• The network will be allowed to lose arbitrarily many
messages sent from one node to another.
• Databases systems, in order to be useful must
have communication over the network
• Clients count

58. There is no such thing as
a 100% reliable network:
Can’t choose CA
http://codahale.com/you-cant-sacrifice-partition-tolerance

59. We Can Have Both*
(*Just not at the same time)

60. PNUTS
• Paper released by Yahoo! research in 2008
• Operations:
• Read-Any
• Read-Critical(Required-Version)*
• Read-Latest
• Write
• Test-and-set-write(Required-Version)
* Will fall back to CP operation

61. Weak Consistency

62. Weak Consistency

63. “This is a specific form of weak
consistency; the storage system
guarantees that if no new
updates are made to the object,
eventually all accesses will
return the last updated value.”
Definition of “Eventual Consistency” from “Eventually
Consistency Revisited” - Werner Vogels

65. Eventual Consistency
in the LAN

66. Less Relevant Today

67. Good at Building
LANs at Scale

68. Facebook Fabric

69. Microsoft VL2

70. Google Jupiter

71. Less Interesting

72. Eventual Consistency
in the WAN

73. Low-latency
everywhere

74. Write Anywhere
Beat the speed of the light

75. Build for WAN locality

76. Typical Pattern
with
COTS EC Store

77. System Model

78. Use Case:
Social Network

79. Models:
Users, Posts, Friends

80. Schema
CREATE TABLE test.users (
user_name text PRIMARY KEY,
friends set<text>,
posts set<text>
)

81. State
*****:test> SELECT * FROM users;
user_name | friends | posts
-----------+----------+-------
sargun | {'BOSS'} | null

82. Let’s Post!
(But First)

83. Remove Boss
*****:test> UPDATE users SET
friends = friends - {'BOSS'}
WHERE user_name = 'sargun' ;

84. Hidden Failure

85. Dropped Unfriending

86. State at DC2 & DC3
*****:test> SELECT * FROM users;
user_name | friends | posts
-----------+----------+-------
sargun | {'BOSS'} | null

87. Post Message
*****:test> UPDATE users SET
posts = posts + {'PARTY'} WHERE
user_name = 'sargun' ;

88. State at DC2 & DC3
*****:test> SELECT * FROM users;
user_name | friends | posts
-----------+----------+-----------
sargun | {'BOSS'} | {'PARTY'}

89. Worse Than Banking
Unbounded Financial Loss

90. No
Happens-Before (h.b.)
Relationship

91. Solution: Wait For Acks

92. Very Little Benefit
Over
CP system
Quorum Systems

93. RYOW at an
Incredible Cost

94. Why not just do
Paxos*?
Single-Decree Paxos Variant such as EPaxos, Cheap Paxos, or
Multi-Paxos

97. Quorum

98. Quorum

99. Participating Quorums
Must Overlap

100. Just Perform
Paxos Reconfiguration
to
Recover from Failure

104. Is there an alternative?

105. Strong
Eventual
Consistency

106. Strong Eventual Consistency
“Any set of nodes that have received
the same (unordered) set of updates
will be in the same state.”

107. How do you even use this?

108. Vector Clocks

109. Vector Clocks
• Extension of Lamport Clocks
• Used to detect cause and effect in distributed
systems
• Can determine concurrency of events, and
causality violations
• Preserves h.b. relationships

110. CRDTs

111. • CRDTs:
• Convergent Replicated Data Types
• Commutative Replication Data Types
• Enables data structures to be always writeable on both sides of a partition,
and replay after healing a partition
• Enable distributed computation across monotonic functions
• Two Types:
• CvRDTs
• CmRDTs
CRDTs

112. CvRDTs
• State / value based CRDTs
• Minimal state
• Don’t require active garbage collection

113. Set CvRDT

114. CmRDTs
• Op / method based CRDTs
• Size grows monotonically
• Uses version vectors to determine order of
operations

115. Counter CmRDT

116. CRDTs in the Wild
• Sets
• Observe-remove set
• Grow-only sets
• Counters
• Grow-only counters
• PN-Counters
• Flags
• Maps

117. Data structures that are
CRDTs
• Probabilistic, convergent data structures
• Hyper log log
• Bloom filter
• Co-recursive folding functions
• Maximum-counter
• Running Average
• Operational Transform

118. CRDTs
• Incredibly powerful primitive
• Not only useful for in-database manipulation but
client-database interaction
• You can compose them, and build your own
• Garbage collection is tricky

119. Riak
In Action

120. Model
curl -s http://localhost:8098/types/test/buckets/test/
datatypes/sargun |python -mjson.tool
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}

121. “Primary Key”
curl -s http://localhost:8098/types/test/buckets/test/
datatypes/sargun |python -mjson.tool
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}

122. Causal Context
curl -s http://localhost:8098/types/test/buckets/test/
datatypes/sargun |python -mjson.tool
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}

123. Update
curl -XPOST http://localhost:8098/types/test/buckets/
test/datatypes/sargun
-H "Content-Type: application/json"
-H "X-Riak-Vclock: g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq"
-d '
{
"update": {
"friends_set": {
"remove": "Boss"
}
}
}'

124. Updated Entries
(during partition)
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQtq",
"type": "map",
"value": {
"friends_set": [],
"posts_set": []
}
}

125. Updatecurl -XPOST http://localhost:8098/types/test/buckets/
test/datatypes/sargun
-H "Content-Type: application/json"
-H "X-Riak-Vclock: g2wAAAABaAJtAAAACBjtDYuvG6A4YQtq"
-d '
{
"update": {
"posts_set": {
"add": "Party"
}
}
}'

126. Updated Entries
(After Healing)
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQ5q",
"type": "map",
"value": {
"friends_set": [],
"posts_set": [
"Party"
]
}
}
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQ5q",
"type": "map",
"value": {
"friends_set": [],
"posts_set": [
"Party"
]
}
}

127. Currently:
Replicates entire value

128. Future Work:
δ-CRDT

129. Ship only Deltas

130. Eventual Consistency
In Summary

131. SEC Enables

132. Distributed

133. Scalable
Scalability
Processors

134. Fault-Tolerant

135. Applications

136. Eventual Consistency (CAP)
Without Consistency (ACID)
Gives EC a Bad Name

137. Invariant Operation AP / CP
Specify unique ID Any CP
Generate unique ID Any AP
> INCREMENT AP
> DECREMENT CP
< INCREMENT CP
< DECREMENT AP
Secondary Index Any AP
Materialized View Any AP
AUTO_INCREMEN
T
INSERT CP
Linearizability CAS CP
Operations Requiring
Weak Consistency
vs.
Strong Consistency

138. BASE not ACID
•Basically Available: There will be a response
per request (failure, or success)
•Soft State: Any two reads against the system
may yield different data (when measured
against time)
•Eventually Consistent: The system will
eventually become consistent when all
failures have healed, and time goes to infinity

139. Brand New Technology
Still being invented

140. Technology Timeline
• 1996 - Log structured merge tree
• 2000 - CAP Theorem
• 2007 - Amazon Dynamo Paper
• 2011 - INRIA CRDT Technical Report
• 2014 - Riak DT map: a composable, convergent
replicated dictionary

141. Further Reading
• Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area
Storage with COPS
• PNUTS: Yahoo!’s Hosted Data Serving Platform
• F1: A Distributed SQL Database That Scales
• Spanner: Google's Globally-Distributed Database
• The Network is Reliable: An informal survey of real-world communications
failures
• A comprehensive study of Convergent and CommutativeReplicated Data
Types
• Riak DT Map: A Composable, Convergent Replicated Dictionary

142. Get in Touch
• If you’re interested in cheating the speed of light
• Come use our software
• If you’re interested in solving today’s computer science
problems
• Come work for us
• If you’d like to learn more about distributed systems at
scale
• Maybe you have a better idea

143. Sargun Dhillon
@Sargun
sdhillon@basho.com
The Case
for
Eventual Consistency