Des benchmarks qui déchirent avec JMH
•Télécharger en tant que PPTX, PDF•
0 j'aime•344 vues
Présentation faite au Mars JUG - Novembre 2017
Recommandé
Contenu connexe
Similaire à Des benchmarks qui déchirent avec JMH
Similaire à Des benchmarks qui déchirent avec JMH (20)
Des benchmarks qui déchirent avec JMH
- 1. Des benchmarks qui déchirent, avec JMH ! Leonardo F. Gomes 0
- 2. 1 @lgomes 2008 – Présent 1980 – 2008 architecte @ amadeus
- 3. 2 15000+ employés 120+ compagnies aériennes Et aussi aéroports, hôtels, voitures, assurances, etc.
- 5. Premature optimization Donald Knuth, 1974 is the root of all evil 5
- 6. L’optimization prématurée Donald Knuth, 1974 est la racine de tout mal 6
- 7. Donald E. Knuth Professor Emeritus at Stanford University ACM Grace Murray Hopper Award Turing Award Author of The Art Of Computer Programming Creator of TeX 7
- 8. Donald E. Knuth Nous devrions oublier les petites efficacités, disons environ 97% du temps: l'optimisation prématurée est la racine de tout mal. Pourtant, nous ne devrions pas laisser passer notre opportunité dans ce 3% critique. 8
- 9. Donald E. Knuth L'expérience universelle des programmeurs qui ont utilisé des outils de mesure a été que les suppositions intuitives échouent. 9
- 10. Un bon développeur mesure avant d’optimiser 10
- 13. Pourquoi ? 13
- 15. Warmup phase 15
- 16. Java source code Bytecodes HotSpot Java VM compile execute _Ahead-of-time _Utilisant javac _Instructions pour une machine abstraite 16
- 17. Bytecodes HotSpot Java VM Interpreter Heap Stack Garbage collector execute access access manage vC1 C2 Machine code Debug info Compiled method Object maps compile produce Compilation system 17
- 19. ©2016AmadeusITGroupanditsaffiliatesandsubsidiaries Ne développez pas votre propre benchmarking harness 19
- 20. JMH est ton ami 20
- 26. JMH est pour le ce que JUnit est pour l’unit testing 26 benchmarking
- 28. Macro 1 … 1000s Milli 1 … 1000ms Micro 1 … 1000us Nano 1 … 1000ns 28 Granularité
- 29. Benchmark modes Throughput ops/time_unit AverageTime time/operation SampleTime percentiles SingleShotTime cold performance 29
- 33. Multithreading rendu facile les résultats sont agrégés 33
- 35. 35 table lock thread 0 get ( key0 ) thread 1 get ( key1 ) put ( key0, value0 ) put ( key1, value1 ) lock (thread 0)
- 38. 38 Segment Segment Segment Segment lock lock lock lock thread 0 put ( key0, value0) segmentFor ( hash0 ) thread 1 put ( key1, value1) segmentFor ( hash1 ) thread 2 put ( key2, value2) segmentFor ( hash2 ) lock (thread 0) lock (thread 1)
- 39. 39 Segment Segment Segment Segment lock lock lock lock thread 0 get ( key0 ) segmentFor ( hash0 ) thread 2 get ( key2 ) segmentFor ( hash2 ) read volatile read volatile
- 41. Profileurs intégrés Compilation en cours pendant qu’on mesure ? Class loading en cours pendant qu’on mesure ? Taux d’allocation d’objets par benchmark Quelles méthodes consomment du temps CPU 41
- 42. Profileurs externes Linux perf_events Windows xperf Java Flight Recorder (pluggable) Yourkit, etc. 42
- 44. Java Flight Recorder Enregistreur de données très efficace Intégré dans la JVM (Oracle) API disponible pour explorer les données
- 45. Java Mission Control GUI pour visualiser les données de JFR Pas toujours la meilleure façon de le faire Pourquoi ?
- 48. Flamegraphs Nouvelle visualisation pour la sortie des profileurs Crée par Brendan Gregg chez Netflix Disponible sur Github Normalement utilisée pour le CPU, mais pas que ...
- 51. Profondeur de la pile
- 53. Temps passé dans le CPU
- 59. Flamegraph
- 60. CPU profiling mais pas que …
- 62. Locks Nombre des fois qu’un thread a été bloqué
- 63. Allocation TLAB* / Outside TLAB Quantité de mémoire allouée
- 64. Allocation TLAB* / Outside TLAB * Lire la discussion pour comprendre les limites: https://github.com/chrishantha/jfr-flame-graph/pull/7
- 65. Retour à JMH 65
- 68. Vérifier que le code correspond aux attentes Vérifier qu'aucune régression n'est introduite Valider les idées d'optimisation Couvrir les performances fixes avec un test associé
- 73. Se soucier du réchauffement Reduire le bruit Definir ce qui est régression S’assurer que le backlog est traité
- 74. Key takeaways 74 Le benchmarking est difficile Mesurer avant d’optimiser Les outils dispo aident beaucoup
- 75. References 75 • JMH website • Nitsan Wakart • Aleksey Shiplev • Brendan Gregg • JMC & JFR • JMH visualizer • JMH Jenkins plugin
- 76. • Caliper: https://www.flickr.com/photos/andrewthecook/14026422669/sizes/l • Geometric forms: https://www.flickr.com/photos/internetarchivebookimages/14753972274/sizes/l • Metric tape: https://www.flickr.com/photos/ilianov/3345314090/sizes/l/ • Mountain: https://www.flickr.com/photos/pthread/8151096195/sizes/l • Friends: https://www.flickr.com/photos/livenature/13895494231/sizes/l • Root: https://www.flickr.com/photos/paperpariah/19937816358/sizes/l/ • Knuth: https://www.flickr.com/photos/ioerror/56360019/sizes/l • Warmup: https://www.flickr.com/photos/komunews/2085730526/sizes/o/ • Multithreading: https://www.flickr.com/photos/slimjim/4329655445/sizes/l • Stop: https://www.flickr.com/photos/thematthewknot/3924980314/sizes/l • Boats: https://www.flickr.com/photos/cuppini/8465318134/sizes/l • Next steps: https://www.flickr.com/photos/gebagia/22346547334/sizes/l • Marines: https://www.flickr.com/photos/dvids/14007373489/sizes/l • Artic ice: https://commons.wikimedia.org/wiki/File:ICESCAPE.jpg • Demo time: https://www.flickr.com/photos/abstractbynature/6111219203 • Blue sky: https://www.flickr.com/photos/foctavian/16371691937/ • Flight recorded: https://www.flickr.com/photos/lwpkommunikacio/15609306263/ 76
- 77. Questions? 77
Notes de l'éditeur
- La presentation sera faite en franglais
- Amadeus est le leader mondial de l'industrie du voyage.
- Quand vous êtes un développeur, vous voulez: Finir votre code Que ça marche Que ça marche vite! Donc que faites-vous ? Vous faites l'optimisation ! N'est-ce pas?
- ”L'optimisation prématurée est la racine de tout Mal” This catching phrase is usually used without much context. Just like biblical citations it can lead to religious wars. Let’s check the context around that phrase.
- ”L'optimisation prématurée est la racine de tout Mal” This catching phrase is usually used without much context. Just like biblical citations it can lead to religious wars. Let’s check the context around that phrase.
- Mettons du contexte autour de cette citation. Regardons ce qui est écrit avant et après cette phrase dans le papier où elle est apparue. Knuth is the guy who said that. There’s some polemic about whether the quote is originally from Knuth or if he was citing Tony Hoare. This article tries to ”prove” that it’s actually from Knuth: https://shreevatsa.wordpress.com/2008/05/16/premature-optimization-is-the-root-of-all-evil/
- A little bit of context: "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.” Page 268 Donald Knuth Structured programming with go to statements Computing Surveys, Vol. 6, No. 4, December 1974 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
- A little bit of context: "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.” Page 268 Donald Knuth Structured programming with go to statements Computing Surveys, Vol. 6, No. 4, December 1974 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
- Summary of what Knuth said
- https://commons.wikimedia.org/wiki/File:ICESCAPE.jpg
- Now you’re all convinced that you should be measuring the performance of your code. But wait, don’t just put timers on your unit-tests .
- Now you’re all convinced that you should be measuring the performance of your code. But wait, don’t just put timers on your unit-tests .
- Explain how code is initially interpreted; Then, compiled at runtime; Then, it runs in compiled mode.
- Explain how code is initially interpreted; Then, compiled at runtime; Then, it runs in compiled mode.
- Explain how code is initially interpreted; Then, compiled at runtime; Then, it runs in compiled mode.
- Branch prediction Loop unrolling Dead code elimination Autobox elimitation Constant propagation Null check elimination Algebraic simplification Devirtualisation Range check elimitation Etc.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you. The thread scope matches well the concept of application server, because usually Java app servers have scope per thread. This would be like processing 20 requests in parallel. Benchmark scope would be a cache that all your requests are accessing. It should be guarded by synchronization mechanisms to make sure that it remains consistent.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- Write to a volatile happens-before every subsequent read of that volatile
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- Instead of using a single lock for the shared data, the shared data is segmented with each segment having its own lock. Uncontended lock acquisition is very cheap; it's the contented locks that cause scalability issues. With a different lock for each partition, ConcurrentHashMap effectively reduces how often a lock is requested by the number of partitions. You can think of ConcurrentHashMap as made up of n separate hash tables.
- Write to a volatile happens-before every subsequent read of that volatile
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- API available (even if unofficial) means that the data is hackable; Lots of recorded events, e.g. CPU
- API available (even if unofficial) means that the data is hackable; Lots of recorded events, e.g. CPU
- La vue hiérarchique conduit à cliquer sans cesse et, parfois, ça bloque la GUI
- Les flamegraphs peuvent être utilisés pour n’importe quelle donnée qui se presente bien de façon hiérarchique, e.g. tout ce qui est stacktrace.
- Dans un flammegraph, les fonctions seront agrégées en fonction de leur stacktrace, donc réorganisées pour regrouper autant que possible la même stacktrace.
- Dans un flammegraph, les fonctions seront agrégées en fonction de leur stacktrace, donc réorganisées pour regrouper autant que possible la même stacktrace.
- Jfr-flame-graph : est utilisé pour convertir le format proprietaire de JFR en input texte pour le script perl de Brendan Gregg FlameGraph : script perl que converti le format text en fichier SVG
- Les flamegraphs peuvent être utilisés pour n’importe quelle donnée qui se presente bien de façon hiérarchique, e.g. tout ce qui est stacktrace.
- Jfr-flame-graph : est utilisé pour convertir le format proprietaire de JFR en input texte pour le script perl de Brendan Gregg FlameGraph : script perl que converti le format text en fichier SVG
- Les flamegraphs peuvent être utilisés pour n’importe quelle donnée qui se presente bien de façon hiérarchique, e.g. tout ce qui est stacktrace.
- Jfr-flame-graph : est utilisé pour convertir le format proprietaire de JFR en input texte pour le script perl de Brendan Gregg FlameGraph : script perl que converti le format text en fichier SVG
- Les flamegraphs peuvent être utilisés pour n’importe quelle donnée qui se presente bien de façon hiérarchique, e.g. tout ce qui est stacktrace.
- Nombre des fois qu’un thread a été bloque sur ce lock. La stack trace done le lock.
- Nombre des fois qu’un thread a été bloque sur ce lock. La stack trace done le lock.
- Nombre des fois qu’un thread a été bloque sur ce lock. La stack trace done le lock.
- JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
- http://netty.io/wiki/microbenchmarks.html https://github.com/grpc/grpc-java/tree/master/benchmarks/src/jmh https://github.com/akka/akka/tree/master/akka-bench-jmh https://github.com/SonarSource/sslr/blob/master/sslr-benchmarks/src/main/java/org/sonar/sslr/benchmarks/RecursiveRuleBenchmark.java https://github.com/droolsjbpm/kie-benchmarks/blob/master/drools-benchmarks/src/main/java/org/drools/benchmarks/session/InsertFireLoopBenchmark.java https://github.com/finagle/finagle-serial#benchmarks
- LMAX: Our micro-benchmarks currently take over an hour to run, though with more hardware we could run them in parallel to improve this. That's still not bad, but for comparison, our suite of ~11k acceptance tests only takes ~25mins...
- Reduce noise / Isolate your benchmarks as much as possible (using cpu isolation, sched_setaffinity); Care about a correct warmup phase / Give benchmarks enough time to run; Don't do nanosecond per operation benchmarks in Continuous integration; Define regression / Some variance is expected; Define well your baseline; Differentiate inter-version, intra-version regressions; Make sure issues backlog is tracked and handled.