In this presentation, we will present the performance measurement metrics of leading cloud providers - AWS, Google Cloud, Microsoft Azure, and Digital Ocean. We’ll give you useful tools to measure your own cloud performance and a handy guide on how to calculate cloud TCO (total cost of ownership). In addition, you’ll learn how to estimate correctly your market positioning and perform better than the cloud giants. Boyan Krosnov is a Co-Founder and Chief Product Officer of StorPool Storage. He has been part of the technical teams building 5 service providers from scratch in 4 countries. In most of these projects, he has designed the architecture, led the technical teams, and managed the implementation of projects in the millions.

1. Benchmarking your cloud performance with top 4
global public clouds
Boyan Krosnov
data://disrupted
2020

2. ● Chief of Product & co-founder at StorPool
● 20+ years in ISPs, SDN, SDS
● IT Infrastructure with a focus on invention,
performance & efficiency
About me
https://www.linkedin.com/in/krosnov/
bk@storpool.com

3. About StorPool
● NVMe software-defined storage for VMs and containers
● Scale-out, HA, API-controlled
● Since 2011, in commercial production use since 2013
● Based in Sofia, Bulgaria
● Mostly virtual disks for KVM
● … and bare metal Linux hosts
● Also used with VMWare, Hyper-V, XenServer
● Integrations into OpenStack/Cinder, Kubernetes Persistent
Volumes, CloudStack, OpenNebula, OnApp
3

4. Why performance
● Better application performance -- e.g. time to load a page, time to
rebuild, time to execute specific query
● Happier customers in cloud and multi-tenant environments
● ROI, TCO - Lower cost per delivered resource (per VM) through
higher density
● Public cloud - win customers over from your competitors
● Private cloud - do more with less; win applications / workloads /
teams over from public cloud

5. 1. Understanding performance
2. Benchmarks of public clouds
3. How to measure measure and optimize your own cloud
4. What's in a TCO
5. Conclusion
Agenda

6. Latency
ops
per
second
best service
6

7. Latency
ops
per
second
best service
lowest cost per
delivered resource
7

8. Latency
ops
per
second
best service
lowest cost per
delivered resource
only pain
8

9. Latency
ops
per
second
best service
lowest cost per
delivered resource
only pain
9
benchmarks

10. Benchmarks

11. Real load

12. ?

13. ?

14. 1. Understanding performance
2. Benchmarks of public clouds
3. How to measure and optimize your own cloud
4. What's in a TCO
5. Conclusion
Agenda

15. * - ramdisk used to reduce usable RAM to 16 GB
VMs and block storage
Provider Instance name region
monthly cost
(with 12 month
commitment)
vCPUs RAM free -m
AWS Compute optimized: c5.2xlarge us-east-2 $245 8 16GB 15,437
Google Cloud General purpose: n2-8vcpu-16gb us-central1 $197 8 32GB 32,116*
Microsoft Azure
Compute optimized:
Standard_F8s_v2 - 8 vcpus, 16
GiB memory
East US 2 $235 8 16GB 15,962
Digital Ocean CPU Optimized Droplet: 16GB sfo2 $160 8 16GB 16,039
Katapult ROCK-24 London $120 8 24GB 23,458*
Storage volume
Size of volume
[GiB]
IOPS limit Monthly cost
AWS - EBS gp2 1024 3,072.00 $102
Google Cloud - SSD persistent disk 1T 1024 15,000.00 $174
Microsoft Azure - Premium SSD 1T 1024 3,500.00 $123
DigitalOcean - Block Storage 1T 1024 10,000.00 $102
Katapult Shared disk NVMe (StorPool-based) 1024 unlimited $154

16. ● Storage heavy, a little CPU
○ FIO, rsync
● Storage & CPU
○ pgbench, sysbench
● CPU, RAM*
○ coremark
● Network*
* - future additions to our suite
Tools used

17. Results - FIO Storage type
FIO rand r/w
QD1 latency
[ms]
FIO QD1
random r/w
IOPS
FIO QD64
random r/w
IOPS
Katapult 1T ($153) StorPool-based 0.10 ms 10,101 IOPS 113,447 IOPS
AWS EBS gp2 1T ($102) 0.36 ms 2,762 IOPS 3,123 IOPS
Google Cloud SSD Persistent Disk 1T
($174)
0.72 ms 1,386 IOPS 15,436 IOPS
Azure Premium SSD 1T ($124) 8.18 ms 122 IOPS 5,100 IOPS
DO Block Storage 1T ($102) 3.34 ms 299 IOPS 1,044 IOPS

18. Results - rsync
storage type seconds to re-sync
Katapult 1T ($153)
StorPool-based
85
AWS EBS gp2 1T ($102) 176
Google Cloud SSD Persistent
Disk 1T ($174)
281
Azure Premium SSD 1T ($124) 431
DO Block Storage 1T ($102) 1,303

19. Results - pgbench

20. Results - pgbench
Unfit for databases!

21. Results - pgbench

22. Results - pgbench
2.3x transactions
with same CPU,
RAM / same cost

23. Results - pgbench
2.5-3x lower latency at
fixed load

24. 1. Understanding performance
2. Benchmarks of public clouds
3. How to measure and optimize your own cloud
4. What's in a TCO
5. Conclusion
Agenda

25. ● Design benchmarks which reflect your use-case and application
● Measure what matters. Examples:
○ developer productivity - simple SQL database for up to X users, so no
need to pay for complexity of clusters; runs CI/tests in half the time
○ Efficiency - $ per user, $ per features
● If you can't measure what matters directly, find good proxies. Example:
○ "I can't run my entire stack as a benchmark, but I know it consists of
a load balancer and a transaction-heavy database, so I'll use a load
balancer and a DB benchmark"
Benchmarks

26. Storage benchmarks
Beware: lots of snake oil out there!
● performance numbers from hardware configurations totally
unlike what you’d use in production
● synthetic tests with high iodepth - 10 nodes, 10 workloads *
iodepth 256 each. (because why not)
● testing with ramdisk backend
● synthetic workloads don't approximate real world

27. ● Previous version of our tools and methodology:
○ https://storpool.com/storage-performance-and-resilience-
testing
● We'll be releasing updated tools and method with the write-up
in the next month
○ coremark, fio, rsync, pgbench, sysbench
● Until then drop us an email at info@storpool.com
Benchmarks

28. 1. Your existing hardware can give you more
a. See Venko's talk on KVM optimization (tomorrow 11am)
b. fast networking (OVS-DPDK), fast storage (StorPool)
2. If you are building a new cloud - optimize for your use-case
a. per-rack power limit
b. per-core performance, per-core memory, per-core storage
c. per-core cost
Hardware

29. Hardware optimization example: Xeon scalable selection

30. ● Hardware
● Host OS and hypervisor (KVM)
● Virtual networking, service mesh
● Storage
Optimization areas

31. 1. Understanding performance
2. Benchmarks of public clouds
3. How to measure and optimize your own cloud
4. What's in a TCO
5. Conclusion
Agenda

32. ● Define minimum service level
● When comparing options use TCO tool (large spreadsheet) to find
lowest cost per delivered unit of infrastructure (fixed-size
VM/container with associated storage and networking)
● 100s of parameters
● Usable for both public and private scenarios
TCO approach

33. https://storpool.com/public-and-private-cloud-roi-calculator
TCO example:

34. 1. Datacenter
- power, cooling, max power per rack, remote hands
2. Compute
- servers, CPUs, RAM, minimum core performance, cloud
orchestration, management cost
3. Storage
- storage servers, drives, software, management cost
4. Network
- virtual network, CPU/RAM allocation, software, management
cost
- public/wide area network, IP transit cost
What to include

35. 1. Understanding performance
2. Benchmarks of public clouds
3. How to measure and optimize your own cloud
4. What's in a TCO
5. Conclusion
Agenda

36. 1. You can't judge a VM by its vCPUs and vRAM
2. Measure what matters to you
3. If you are a public or private cloud 2x,3x, higher application
performance (per $ !) than hyperscalers is within reach. Half price
for the same workload!
4. On your next project work with partners who understand
performance. You can gain a lot!
Conclusions

37. Questions?

38. StorPool
Storage
@storpool StorPool
Storage
StorPool
Storage
StorPool
Storage
StorPool
Storage
Follow StorPool Online

39. Talk to us:
The virtual booth at Data://Disrupted
info@storpool.com
https://storpool.com
Thank you!