Lyon jug-how-to-fail-at-benchmarking

How to fail at benchmarking ?
Pierre Laporte
Performance engineer - Datastax
@pingtimeout

@pingtimeout
Agenda
• Intro
• Why this talk?
• How to fail a benchmark ?
– What to measure ?
– How to measure ?
– "Coordinated Omission" ...?
• Available tools ?
• Mindset ?
• JMH ?
2

@pingtimeout 5
Cassandra
• NoSQL Database
• Linear Scalability
• Always On

@pingtimeout 6
Datastax Enterprise (DSE)
• Full featured database platform
• Certified for production
• Enhanced search
• Analytics w/ Hadoop/Spark
• Security
• OpsCenter

@pingtimeout 7
Datastax Enterprise (DSE)
OpsCenter

@pingtimeout 9
I have not failed.
I have just found 10,000 ways
that won’t work
— Thomas Edison

To learn
Make mistakes
Learn from them
Do *not* blame

@pingtimeout 13
Example
• User login benchmark (POST /login)
• 100.000 logins
• 1 user
• Measure each login
• Measure from first login
• Perform one (long) run
• On a dev laptop
• Average the results
• CPU utilization => Capacity Planning

@pingtimeout 15
Cache
• « 100.000 logins of 1 user »
• CPU caches L1, L2, L3 (and L4)
• TLB cache
• Database cache
• Hibernate cache
• …

@pingtimeout 16
Clock resolution
• « Measure every single login latency »
• login ~> 5ms
• System.currentTimeMillis() ~> 1ms
• ±1 unit <=> ±20%
• System.nanoTime() "provides
nanosecond precision, but not
necessarily nanosecond resolution"

@pingtimeout 17
Warmup
• « Start measuring at first login »
• Class loading
• JVM Warmup time

@pingtimeout 18
Precision
• « Perform one run »
• What if background noise?
• How do we build confidence?

@pingtimeout 19
Environment
• « On a dev laptop » vs. Production

@pingtimeout 20
Metrics
• « Average the results »

@pingtimeout
Capacity Planning
• CPU utilization ≈ 50%
• Remaining Capacity ≈ …?
21

@pingtimeout 23
Average
• Get rid of outliers
• What is the average of a and b ?
• Why do we keep hearing about it ?
• Is it enough ?

@pingtimeout 25
Useful metrics
• Graphs
• Percentiles
• 90th percentile
• 95th percentile
• 99th percentile
• 99.9th percentile
• …
• Max

@pingtimeout 26
How many 9’s in percentiles ?
• Depend on number of users
• If 500 users, 99th percentile means… ?

@pingtimeout 27
How many 9’s in percentiles ?
• If 1.000.000 users, 99th percentile
means… ?
• 10.000 unlucky folks

@pingtimeout 28
Maximum
• « Meh, probably a timeout, let’s
ignore » — Random dev facing a 60s
response time
• Is that be acceptable…
• For a video game?
• For a real-time application?
• For a mobile application?

@pingtimeout
Benchmark genesis
• A specific question
• Precise requirements
• Ex:
• e-commerce website
• « With my 5 application servers, 20.000
articles in DB and 1.000 simultaneous
clients, will my purchase take 50ms on
average, 100ms once every 10
attempts, and 300ms at worst? »
29

@pingtimeout
Environment
articles in DB and 1.000 simultaneous clients,
will my purchase take 50ms on average,
100ms once every 10 attempts, and 300ms at
worst? »
• POC:
• TomEE
• Glassfish
• Weblogic
30

@pingtimeout
Load
worst? »
31

@pingtimeout
Requirements
worst? »
32

@pingtimeout
Epilogue
• Answer
• Satisfactory… or not
• Contextual results
• Environment
• Load
• Constraints
• Compromises
33

@pingtimeout 35
99th percentile on average
• Typical (bad) application in 2012:
• 100 Web Objects/page
• ≈ 100 HTTP requests/page
• Probability of avoiding 99th percentile?
• 0.99n, considering n attempts
• 0.99100 ≈ 36.6%

@pingtimeout 36
99th percentile on average

@pingtimeout 37
Normal distribution
• Client gives those latency numbers
• Average = 100 ms
• Std dev = 20 ms
• Median = 120 ms
• Max = 1 s
• What is the 99th percentile ?

@pingtimeout 39
Anscombe’s quartet

@pingtimeout 40
Normal distribution ?!
• Latency never follows a normal
distribution
• Dropwizard metrics :

@pingtimeout 41
• Use HdrHistogram
• Arbitrary range
• Precision
• Constant memory footprint
• http://hdrhistogram.github.io/
HdrHistogram/
Normal distribution ?!

@pingtimeout 42
HdrHistogram graph'd

@pingtimeout 44
Puzzler
• Any issue?
boolean isCustomer(Integer id) {
for (Integer cid : customers)
if (cid == id)
return true;
return false;
}

@pingtimeout 45
Puzzler (complete)
import java.util.*;
public class Fixnum {
private List<Integer> customers;
public Fixnum() {
customers = new ArrayList<Integer>();
for(int i=0 ; i<50_000 ; i++)
customers.add((Integer) i);
}
boolean isCustomer(Integer id) {
for (Integer cid : customers)
if (cid == id)
return true;
return false;
}
public static void main(String... args) {
Fixnum f = new Fixnum();
for(int i=0 ; i<Integer.valueOf(args[0]) ; i++)
if(!f.isCustomer(i))
System.out.println("Failed for i=" + i);
}
}

@pingtimeout 46
Puzzler (oh shit…)
$ java Fixnum 10
$ java Fixnum 50
$ java Fixnum 100
$ java Fixnum 128
$ java Fixnum 129
Failed for i=128
$

@pingtimeout 47
Puzzler (run 2)
• Anyone knows -XX:+AggressiveOpts ?
• What is going to happen ?
• … What does it do ?

@pingtimeout 48
Puzzler (oh shit…)
$ java -XX:+AggressiveOpts Fixnum 129
$

@pingtimeout 49
Aggressive Opts
• Increase fixnum pool from 128 to … ?
• How much left as an exercise
• Hint: set_aggressive_opts_flags() in
arguments.cpp
• That’s pretty much it.

@pingtimeout 50
Puzzler 2
• What does 
this code do ?
public class Haddock {
public static void main(String [] args) {
new Haddock().first();
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
third();
}
void third() {
for(int i=0; i<1_000; i++)
new Sabord();
}
static class Sabord {}
}

@pingtimeout 51
Puzzler 2
• What does 
this code do ?
1. It swears
2. It does not do 
what we think
3. It measures
Hotspot
4. It creates ~10k
objects
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
third();
}
void third() {
for(int i=0; i<1_000; i++)
new Sabord();
}
}

@pingtimeout 52
Puzzler 2
• All of the above 
(below)
1. It swears
2. It does not do 
what we think
3. It measures
Hotspot
4. It creates ~10k
objects
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
third();
}
void third() {
for(int i=0; i<1_000; i++)
new Sabord();
}
}

@pingtimeout 54
Puzzler 2 - Run
$ time java Haddock
real 0m0.169s
user 0m0.110s
sys 0m0.055s
Test @3.5 GHz

@pingtimeout 55
It does not do what we think
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
third();
}
void third() {
for(int i=0; i<1_000; i++)
new Sabord();
}
}

@pingtimeout 56
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
third();
}
void third() {
for(int i=0; i<1_000; i++)
;
}
}

@pingtimeout 57
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
third();
}
void third() {
}
}

@pingtimeout 58
}
void first() {
for(int i=0; i<1_000; i++)
second();
}
void second() {
for(int i=0; i<1_000_000; i++)
;
}
}

@pingtimeout 59
}
void first() {
for(int i=0; i<1_000; i++)
;
}
}

@pingtimeout 60
}
void first() {
}
}

@pingtimeout 61
Puzzler 2
• After ~10k iterations, code is DCE'd
public static void main(String [] args) {}
void first() {}
void second() {}
void third() {}
}

@pingtimeout 63
Java 8 Enterprise REST API
public class Server {
public static void main(String[] args) {
new WebServer().configure(
routes -> routes.get("/", Server::sayHello)
).start();
}
private static String sayHello() {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
throw new IllegalStateException(e);
}
return "Hello World";
}
}

@pingtimeout 64
CO client
public class Client {
public static void main(String[] args) throws Exception {
ArrayList<Long> latencies = new ArrayList<>(100);
HttpGet get = new HttpGet("http://localhost:8080/");
try (CloseableHttpClient client = HttpClients.createDefault()) {
// First execution to load the classes
execute(client, get);
for (int i = 0 ; i < 100 ; i++) {
long requestSendTime = System.currentTimeMillis();
long requestTime = System.currentTimeMillis() - requestSendTime;
latencies.add(requestTime);
SECONDS.sleep(1);
}
}
Collections.sort(latencies);
System.out.println("Number of points : " + latencies.size());
System.out.println("Average : " + average(latencies));
System.out.println("80% : " + percentile(latencies, 0.8));
System.out.println("Max : " + percentile(latencies, 1));
}
}

@pingtimeout 65
STW Runner
• Simulate lonnnng GC pause
• or any other lonnnng blocking task
• ^Z
echo "# Freezing server during 11 seconds..."
kill -STOP $server_pid
sleep 11
echo "# Unfreezing server..."
kill -CONT $server_pid
sleep 40

@pingtimeout 66
Expectations - Timeline
SIGSTOP
SIGCONT

@pingtimeout
Pre-run check
• Throughput = 1 req/s
• Total duration = 100 s
• Longest pause = 11 s
• Typical response time ≈ 10 ms
67

@pingtimeout 68
Expectations - Throughput

@pingtimeout 69
Expectations - Latency

@pingtimeout
Expectations - Maths
• Expected percentiles
• Max ∈ ]10; 11]
• 99% ∈ ]9; 10]
• 98% ∈ ]8; 9]
• 95% ∈ ]5; 6]
• 90% ∈ ]0; 1]
70

@pingtimeout 71
Reality
Wait… What?

@pingtimeout 73
Why?
public class Client {
public static void main(String[] args) throws Exception {
ArrayList<Long> latencies = new ArrayList<>(100);
HttpGet get = new HttpGet("http://localhost:8080/");
try (CloseableHttpClient client = HttpClients.createDefault()) {
// First execution to load the classes
for (int i = 0 ; i < 100 ; i++) {
long requestSendTime = System.currentTimeMillis();
latencies.add(requestTime);
SECONDS.sleep(1);
}
}
Collections.sort(latencies);
System.out.println("Number of points : " + latencies.size());
System.out.println("Average : " + average(latencies));
System.out.println("Max : " + percentile(latencies, 1));
}

@pingtimeout 74
Reality - Throughput
100s
100th data point

@pingtimeout 75
Reality - Throughput
100s
100th data point

@pingtimeout 76
What is CO ?
• Interval between samples : x
• Usual measurements << x
• Outliers > x
• Subsequent measurements delayed
• After the bad period

@pingtimeout 77
What is measured ?
• Service Time's 99th percentile
• ≠ Response Time's 99th percentile

@pingtimeout 78
Is it common ?
I rarely see load testers that
don't suffer from it, including
common testers used in the
industry (JMeter, Grinder, HP
LoadRunner) and benchmarks
(YCSB, SPECeverything, etc.)
— Gil Tene

@pingtimeout
Test the tester
• Mandatory step
• Replace SUT by noop
• Response time = 0
• ^Z Test
• Check hypothesis
79

@pingtimeout 80
How to fix it ? 1/2
• Know your theoretical send time
• Use it as the start time
• Delayed requests will show the issue
long requestSendTime = expectedSendTimeOfSample(n);

@pingtimeout 81
How to fix it ? 2/2
• Interval between samples
• Use HdrHistogram
• http://giltene.github.io/HdrHistogram/
• class Histogram
• copyCorrectedForCoordinatedOmission(…)
• recordValueWithExpectedInterval(…)

@pingtimeout 83
JMH
• Written by Oracle Perf Eng Team
• @shipilev leading
• Code generator
• Multi options runner (# threads, groups,
iterations, …)
• Pluggable profiles
• Multi language
• …

@pingtimeout 84
Power == Responsibility

@pingtimeout 85
JMH Samples
• Code = documentation
• Run the doc
• Learn

@pingtimeout 86
DCE
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public class DCE {
@Benchmark
public void baseline() {
}
@Benchmark
public void measureWrong() {
Math.log(Math.PI);
}
@Benchmark
public double measureRight() {
return Math.log(Math.PI);
}
}

@pingtimeout 87
DCE
Benchmark Samples Score Error Units
f.p.b.DCE.baseline 5 3776,469 Â± 14,079 ops/us
f.p.b.DCE.measureRight 5 503,629 Â± 10,431 ops/us
f.p.b.DCE.measureWrong 5 3714,931 Â± 172,927 ops/us

@pingtimeout 88
CPU arithmetic
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class CPU {
long a;
long b;
// [...]
@Benchmark
public long addition() { return a+b; }
@Benchmark
public long subtraction() { return a-b; }
@Benchmark
public long multiplication() { return a*b; }
@Benchmark
public long division() { return a/b; }
}

@pingtimeout 89
CPU arithmetic
f.p.b.CPU.addition 5 2,058 ± 0,034 ns/op
f.p.b.CPU.subtraction 5 2,070 ± 0,031 ns/op
f.p.b.CPU.multiplication 5 2,087 ± 0,016 ns/op
f.p.b.CPU.division 5 9,931 ± 0,218 ns/op

@pingtimeout 90
Locks
public class Locks {
@State(Scope.Benchmark)
public static class BenchmarkParameters {
@Param({"1"})
public int burnCpuFactor;
}
@State(Scope.Group)
public static class BaselineState {
private int value = 0;
}
@State(Scope.Group)
public static class SynchronizedState {
private final Object lock = new Object();
}
@State(Scope.Group)
public static class ReentrantRWLState {
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
}

@pingtimeout 91
Locks
@Benchmark
@Group("baseline")
public void updateBaseline(BaselineState s, BenchmarkParameters p) {
Blackhole.consumeCPU(p.burnCpuFactor);
s.value++;
}
@Benchmark
@Group("baseline")
public int readBaseline(BaselineState s, BenchmarkParameters p) {
return s.value;
}

@pingtimeout
@Benchmark
@Group("synchronized")
public void updateSynchronized(SynchronizedState s, BenchmarkParameters p) {
synchronized (s.lock) {
s.value++;
}
}
@Benchmark
@Group("synchronized")
public int readSynchronized(SynchronizedState s, BenchmarkParameters p) {
synchronized (s.lock) {
return s.value;
}
}
92
Locks

@pingtimeout
@Benchmark
@Group("rrwl")
public void updateRrwl(ReentrantRWLState s, BenchmarkParameters p) {
s.lock.writeLock().lock();
try {
s.value++;
} finally {
s.lock.writeLock().unlock();
}
}
@Benchmark
@Group("rrwl")
public int readRrwl(ReentrantRWLState s, BenchmarkParameters p) {
s.lock.readLock().lock();
try {
return s.value;
} finally {
s.lock.readLock().unlock();
}
}
93
Locks

@pingtimeout 94
Locks
# 1 reader, 1 writer, burnCpuFactor=1
f.p.b.Locks.readBaseline 20 48632,823 ±1549,042 ops/ms
f.p.b.Locks.updateBaseline 20 232492,376 ±12527,203 ops/ms
f.p.b.Locks.readSynchronized 20 2384,911 ±162,020 ops/ms
f.p.b.Locks.updateSynchronized 20 9295,519 ±1017,941 ops/ms
f.p.b.Locks.readSynchronized 20 13133,933 ±164,549 ops/ms
f.p.b.Locks.updateSynchronized 20 13701,232 ±357,065 ops/ms

@pingtimeout 95
Locks
f.p.b.Locks.readBaseline 20 22573,454 Â±158,246 ops/ms
f.p.b.Locks.updateBaseline 20 82398,369 Â±1429,225 ops/ms
f.p.b.Locks.readSynchronized 20 2176,295 Â±232,285 ops/ms
f.p.b.Locks.updateSynchronized 20 9236,524 Â±1575,705 ops/ms

@pingtimeout 96
Locks

@pingtimeout 97
Locks
# 1 reader, 3 writers, burnCpuFactor=10
f.p.b.Locks.readRrwl 20 76,190 Â±3,637 ops/ms
f.p.b.Locks.updateRrwl 20 30327,966 Â±674,847 ops/ms

@pingtimeout 99
The successful man will profit
from his mistakes and try again
in a different way
— Dale Carnegie

@pingtimeout
Contextual results
100

@pingtimeout 101
Sources
• https://groups.google.com/d/msg/mechanical-sympathy/
• http://psy-lob-saw.blogspot.fr/
• http://latencytipoftheday.blogspot.fr/
• http://www.websiteoptimization.com/speed/tweak/
average-number-web-objects/
• Hotspot source code
• @kcpeppe, @giltene, @shipilev
• @pingtimeout ;-)

@pingtimeout
Thank you for your attention!
• Q&A ?
102

Lyon jug-how-to-fail-at-benchmarking

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (7)

Similaire à Lyon jug-how-to-fail-at-benchmarking

Similaire à Lyon jug-how-to-fail-at-benchmarking (20)

Dernier

Dernier (20)

Lyon jug-how-to-fail-at-benchmarking