1. A Storage Engine for
Amazon S3
How to store a hundred billion BLOBs
without buying a single disk.

2. Who is this guy?

3. http://fallenpegasus.com/code

4. Storage Engine

5. Protocol Translator

6. Traditional
Storage Engines
use the local disk

7. Network Storage Engines

8. Federated
ODBC
HTTP
MemCacheD
and...

9. S3

10. What is S3?

11. Petabytes?!

12. This can't be free

13. Why do I want to use it?

14. “An empty disk costs the same as
a full one.”

15. Can I move my
existing database over?

16. Translating S3 to MySQL
AWS Account → CREATE SERVER
•
S3 Bucket → Table
•
S3 Item → Row
•
S3 Item Key → VARCHAR PRIMARY KEY
•
S3 Item Contents → BLOB or VARCHAR
•

17. CREATE SERVER 'MyAWSAcct'
FOREIGN DATA WRAPPER 'AWS'
OPTIONS
(USER 'aws id string',
PASSWORD 'aws secret string');

18. CREATE TABLE 'bierce' (
'word' VARCHAR(255) NOT NULL PRIMARY KEY,
'defn' BLOB)
CHARSET=utf-8
ENGINE=AWSS3
CONNECTION='awss3 DevilDictionary $server MyAWSAcct';

19. SELECT defn FROM bierce WHERE word='WIT';

20. SELECT defn FROM bierce WHERE word='WIT';
WIT, n. The salt with which the American humorist spoils his
intellectual cookery by leaving it out.

21. INSERT INTO bierce (word, defn) VALUES
('AUTHOR',
'One noted for confusing bitterness with humor.');

22. DELETE FROM bierce WHERE word='AUTHOR';

23. So, what's it good for?

24. “If you build it,
they will come.”

25. Saving EC2 work
•
SQL CMS for S3 virtual hosting
•
Huge list of persistent primary keys
•
Big slow BLOBs, to join against fast local tables.
•
“The Image Server Problem”
•
... and more
•

26. Transactions?

27. What do you mean,
“No temporal guarantees”?!

28. No WHERE clause
means
“MAKE MONEY FAST”
for Amazon

29. The Future
(watch me wave my hands)

30. Code Improvement
•
S3 & HTTP Metadata
•
Multiple Data Columns
•
Information Schema
•
Security & Authentication
•
Transfer & Storage Compression
•
Streaming BLOB
•
More Storage Engines
•

31. Code Improvement

32. S3 & HTTP
Metadata

33. Multiple Data Columns

34. Sharing a solution with
HTTP and MemCacheD
Engines

35. Information Schema

36. Security &
Authentication

37. Transfer & Storage
Compression

38. (new slide)
Streaming BLOBs
like PBXT

39. More storage engines

40. More storage engines
“A Storage Engine for AWS SQS”
•
There are more AWS services rolling out
•
Replication distribution via S3
•
Cluster persistent storage via S3
•
... and the EC2 holy grail:
•

41. “A Generic Schema
Storage Engine for S3”

42. Back to the present...

43. “Where do I get it?”

44. “Where do I get it?”
MySQL 5.1
•
GPL
•
http://fallenpegasus.com/code
•

45. “You never write,
you never call...”

46. Suggestions
•
Use Cases
•
Bug Reports
•
Patches
•
Feature Requests
•
Support Requests
•
Gigs & Proposals
•
mark@fallenpegasus.com

47. Questions?