SlideShare une entreprise Scribd logo
1  sur  62
Télécharger pour lire hors ligne
Attila Szegedi, Software Engineer
@asz




                                1
Everything I ever
learned about JVM
performance tuning
     @twitter

        2
Everything More
than I ever wanted
 to learned about
JVM performance
       tuning
      @twitter
        3
• Memory tuning
• CPU usage tuning
• Lock contention tuning
• I/O tuning


                   4
Twitter’s biggest enemy




           5
Twitter’s biggest enemy
       Latency




           5
Twitter’s biggest enemy
       Latency




           5
Twitter’s biggest enemy
                         Latency




CC licensed image from http://www.flickr.com/photos/dunechaser/
213255210/                          5
Latency contributors


• By far the biggest contributor is garbage collector
• others are, in no particular order:
   • in-process locking and thread scheduling,
   • I/O,
   • application algorithmic inefficiencies.
                             6
Areas of performance
           tuning

• Memory tuning
• Lock contention tuning
• CPU usage tuning
• I/O tuning
                           7
Areas of memory
      performance tuning


• Memory footprint tuning
• Allocation rate tuning
• Garbage collection tuning

                              8
Memory footprint tuning


• So you got an OutOfMemoryError…
   • Maybe you just have too much data!
   • Maybe your data representation is fat!
   • You can also have a genuine memory leak…
                          9
Too much data

• Run with -verbosegc
• Observe numbers in “Full GC” messages secs]
  [Full GC $before->$after($total), $time

• Can you give the JVM more memory?
• Do you need all that data in memory? Consider
  using:
   • a LRU cache, or…
   • soft references*   10
Fat data

• Can be a problem when you want to do wacky
  things, like
   • load the full Twitter social graph in a single
     JVM
   • load all user metadata in a single JVM
• Slimming internal data representation works at
  these economies of scale
                       11
Fat data: object header


 • JVM object header is normally two machine
     words.
 • That’s 16 bytes, or 128 bits on a 64-bit JVM!
 • new java.lang.Object() takes 16 bytes.
 •   new byte[0]   takes 24 bytes.


                         12
Fat data: padding

              class A {
                  byte x;
              }
              class B extends A {
                  byte y;
              }



•   new A()   takes 24 bytes.

•   new B()   takes 32 bytes.
                        13
Fat data: no inline structs

         class C {
           Object obj = new Object();
         }




  • new C() takes 40 bytes.
  • similarly, no inline array elements.
                         14
Slimming taken to
      extreme
• A research project had to load the full follower
  graph in memory
• Each vertex’s edges ended up being represented
  as int arrays
• If it grows further, we can consider variable-
  length differential encoding in a byte array



                      15
Compressed object
     pointers


• Pointers become 4 bytes long
• Usable below 32 GB of max heap size
• Automatically used below 30 GB of max heap

                    16
Compressed object
           pointers
                   Uncompressed Compressed            32-bit

      Pointer           8               4               4

  Object header         16             12*              8

   Array header         24              16             12

  Superclass pad        8               4               4

* Object can have 4 bytes of fields and still only take up 16 bytes
                                17
Avoid instances of
      primitive wrappers

• Hard won experience with Scala 2.7.7:
 • a Seq[Int] stores java.lang.Integer
 • an Array[Int] stores int
 • first needs (24 + 32 * length) bytes
 • second needs (24 + 4 * length) bytes
                        18
Avoid instances of
     primitive wrappers

• This was fixed in Scala 2.8, but it shows that:
 • you often don’t know the performance
   characteristics of your libraries,
 • and won’t ever know them until you run your
   application under a profiler.


                          19
Map footprints


•   Guava MapMaker.makeMap() takes 2272 bytes!

•   MapMaker.concurrencyLevel(1).makeMap()
    takes 352 bytes!
• ConcurrentMap with level 1 makes sense
    sometimes (i.e. you don’t want a
    ConcurrentModificationException)

                       20
Thrift can be heavy



• Thrift generated classes are used to encapsulate a
  wire tranfer format.
• Using them as your domain objects: almost never
  a good idea.



                         21
Thrift can be heavy


• Every Thrift class with a primitive field has a
  java.util.BitSet __isset_bit_vector       field.

• It adds between 52 and 72 bytes of overhead per
  object.



                            22
Thrift can be heavy




         23
Thrift can be heavy


• Thrift does not support 32-bit floats.
• Coupling domain model with transport:
 • resistance to change domain model
• You also miss oportunities for interning and N-to-1
  normalization.

                          24
class Location {
   public String city;
   public String region;
   public String countryCode;
   public int metro;
   public List<String> placeIds;
   public double lat;
   public double lon;
   public double confidence;




                        25
class SharedLocation {
   public String city;
   public String region;
   public String countryCode;
   public int metro;
   public List<String> placeIds;
class UniqueLocation {
   private SharedLocation sharedLocation;
   public double lat;
   public double lon;
   public double confidence;




                        26
Careful with thread locals

• Thread locals stick around.
• Particularly problematic in thread pools with m⨯n
  resource association.
 • 200 pooled threads using 50 connections: you end
   up with 10 000 connection buffers.
• Consider using synchronized objects, or
• just create new objects all the time.
                          27
Part II:
fighting latency



       28
Performance tradeoff

               Memory




                 Time



  Convenient, but oversimplified view.
                  29
Performance triangle

          Memory footprint




 Throughput                  Latency




                 30
Performance triangle

                     Compactness




       Throughput                   Responsiveness
                     C ⨯T ⨯ R = a
• Tuning: vary C, T, R for fixed a
• Optimization: increase a  31
Performance triangle

• Compactness: inverse of memory footprint
• Responsiveness: longest pause the application will
  experience
• Throughput: amount of useful application CPU work
  over time
• Can trade one for the other, within limits.
• If you have spare CPU, can be pure win.
                           32
Responsiveness vs.
   throughput




        33
Biggest threat to
responsiveness in the JVM
  is the garbage collector


            34
Memory pools

Eden       Survivor             Old



                         Code
       Permanent
                        cache




  This is entirely HotSpot specific!
                   35
How does young gen
           work?
          Eden          S1        S2      Old

• All new allocation happens in eden.
 • It only costs a pointer bump.
• When eden fills up, stop-the-world copy-collection
  into the survivor space.
 • Dead objects cost zero to collect.
• Aftr several collections, survivors get tenured into
  old generation.
                             36
Ideal young gen operation


• Big enough to hold more than one set of all
  concurrent request-response cycle objects.
• Each survivor space big enough to hold active
  request objects + tenuring ones.
• Tenuring threshold such that long-lived objects
  tenure fast.

                           37
Old generation collectors

• Throughput collectors
 •   -XX:+UseSerialGC

 •   -XX:+UseParallelGC

 •   -XX:+UseParallelOldGC

• Low-pause collectors
 •   -XX:+UseConcMarkSweepGC

 •   -XX:+UseG1GC   (can’t discuss it here)

                             38
Adaptive sizing policy

• Throughput collectors can automatically tune
  themselves:
 •   -XX:+UseAdaptiveSizePolicy

 •   -XX:MaxGCPauseMillis=…      (i.e. 100)
 •   -XX:GCTimeRatio=…   (i.e. 19)



                            39
Adaptive sizing policy at
         work




            40
Choose a collector


• Bulk service: throughput collector, no adaptive sizing
  policy.
• Everything else: try throughput collector with
  adaptive sizing policy. If it didn’t work, use
  concurrent mark-and-sweep (CMS).



                           41
Always start with tuning
 the young generation
• Enable -XX:+PrintGCDetails, -XX:+PrintHeapAtGC,
  and -XX:+PrintTenuringDistribution.

• Watch survivor sizes! You’ll need to determine
  “desired survivor size”.
• There’s no such thing as a “desired eden size”, mind
  you. The bigger, the better, with some
  responsiveness caveats.
• Watch the tenuring threshold; might need to tune it
  to tenure long lived objects faster.
                          42
-XX:+PrintHeapAtGC


Heap after GC invocations=7000 (full 87):
  par new generation    total 4608000K, used 398455K
   eden space 4096000K,    0% used
   from space 512000K, 77% used
   to   space 512000K,    0% used
  concurrent mark-sweep generation total 3072000K, used 1565157K
  concurrent-mark-sweep perm gen total 53256K, used 31889K
}




                                43
-XX:+PrintTenuringDistribution

Desired   survivor size   262144000 bytes, new threshold 4 (max 4)
- age     1: 137474336    bytes, 137474336 total
- age     2:   37725496   bytes, 175199832 total
- age     3:   23551752   bytes, 198751584 total
- age     4:   14772272   bytes, 213523856 total



 • Things of interest:
  • Number of ages
  • Size distribution in ages
    • You want strongly declining.
                                   44
Tuning the CMS

• Give your app as much memory as possible.
 • CMS is speculative. More memory, less punitive
   miscalculations.
• Try using CMS without tuning. Use -verbosegc and
  -XX:+PrintGCDetails.

 • Didn’t get any “Full GC” messages? You’re done!
• Otherwise, tune the young generation first.
                         45
Tuning the old generation


 • Goals:
  • Keep the fragmentation low.
  • Avoid full GC stops.
 • Fortunately, the two goals are not conflicting.

                          46
Tuning the old generation


 • Find the minimum and maximum working set size
   (observe “Full GC” numbers under stable state and
   under load).
 • Overprovision the numbers by 25-33%.
  • This gives CMS a cushion to concurrently clean
    memory as it’s used.

                           47
Tuning the old generation


 • Set -XX:InitiatingOccupancyFraction to
   between 80-75, respectively.
  • corresponds to overprovisioned heap ratio.
 • You can lower initiating occupancy fraction to 0 if
   you have CPU to spare.


                         48
Responsiveness still not
     good enough?
• Too many live objects during young gen GC:
 • Reduce NewSize, reduce survivor spaces, reduce
   tenuring threshold.
• Too many threads:
 • Find the minimal concurrency level, or
 • split the service into several JVMs.
                          49
Responsiveness still not
    good enough?
• Does the CMS abortable preclean phase, well,
  abort?
 • It is sensitive to number of objects in the new
   generation, so
   • go for smaller new generation
   • try to reduce the amount of short-lived garbage
     your app creates.

                         50
Part III:
let’s take a break from GC



            51
Thread coordination
        optimization

• You don’t have to always go for synchronized.
• Synchronization is a read barrier on entry; write
  barrier on exit.
• Sometimes you only need a half-barrier; i.e. in a
  producer-observer pattern.
• Volatiles can be used as half-barriers.
                            52
Thread coordination
        optimization

• For atomic update of a single value, you only need
  Atomic{Integer|Long}.compareAndSet().

• You can use AtomicReference.compareAndSet() for
  atomic update of composite values represented by
  immutable objects.


                          53
Fight CMS fragmentation
   with slab allocators

• CMS doesn’t compact, so it’s prone to fragmentation,
  which will lead to a stop-the-world pause.
• Apache Cassandra uses a slab allocator internally.


                           54
Cassandra slab allocator


 • 2MB slab sizes
 • copy byte[] into them using compare-and-set
 • GC before: 30-60 seconds every hour
 • GC after: 5 seconds once in 3 days and 10 hours

                       55
Slab allocator constraints
• Works for limited usage:
 • Buffers are written to linearly, flushed to disk and
   recycled when they fill up.
 • The objects need to be converted to binary
   representation anyway.
• If you need random freeing and compaction, you’re
  heading down the wrong direction.
• If you find yourself writing a full memory manager
  on top of byte buffers, stop!
                          56
Soft references revisited

• Soft reference clearing is based on the amount of
  free memory available when GC encounters the
  reference.
• By definition, throughput collectors always clear
  them.
• Can use them with CMS, but they increase memory
  pressure and make the behavior less predictable.
• Need two GC cycles to get rid of referenced objects.
                           57
Everything More
than I ever wanted
 to learned about
JVM performance
       tuning
      @twitter
   Questions?
        58
Attila Szegedi, Software Engineer
@asz




                               59

Contenu connexe

Tendances

Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureUnder The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureScyllaDB
 
Introduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission ControlIntroduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission ControlLeon Chen
 
Secrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on KubernetesSecrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on KubernetesBruno Borges
 
G1 collector and tuning and Cassandra
G1 collector and tuning and CassandraG1 collector and tuning and Cassandra
G1 collector and tuning and CassandraChris Lohfink
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBrendan Gregg
 
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020Jelastic Multi-Cloud PaaS
 
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkWhere is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkFlink Forward
 
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...DataStax
 
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016DataStax
 
Introduction to redis - version 2
Introduction to redis - version 2Introduction to redis - version 2
Introduction to redis - version 2Dvir Volk
 
The top 3 challenges running multi-tenant Flink at scale
The top 3 challenges running multi-tenant Flink at scaleThe top 3 challenges running multi-tenant Flink at scale
The top 3 challenges running multi-tenant Flink at scaleFlink Forward
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc.
 
Scalable and Available, Patterns for Success
Scalable and Available, Patterns for SuccessScalable and Available, Patterns for Success
Scalable and Available, Patterns for SuccessDerek Collison
 
MongoDB Sharding
MongoDB ShardingMongoDB Sharding
MongoDB ShardingRob Walters
 
Docker and Go: why did we decide to write Docker in Go?
Docker and Go: why did we decide to write Docker in Go?Docker and Go: why did we decide to write Docker in Go?
Docker and Go: why did we decide to write Docker in Go?Jérôme Petazzoni
 
Windows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance ComparisonWindows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance ComparisonSeungmo Koo
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent
 
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
Demystifying flink memory allocation and tuning - Roshan Naik, UberDemystifying flink memory allocation and tuning - Roshan Naik, Uber
Demystifying flink memory allocation and tuning - Roshan Naik, UberFlink Forward
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixBrendan Gregg
 

Tendances (20)

Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureUnder The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database Architecture
 
Introduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission ControlIntroduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission Control
 
Secrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on KubernetesSecrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on Kubernetes
 
G1 collector and tuning and Cassandra
G1 collector and tuning and CassandraG1 collector and tuning and Cassandra
G1 collector and tuning and Cassandra
 
MongodB Internals
MongodB InternalsMongodB Internals
MongodB Internals
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
 
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
 
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkWhere is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in Flink
 
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
 
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
 
Introduction to redis - version 2
Introduction to redis - version 2Introduction to redis - version 2
Introduction to redis - version 2
 
The top 3 challenges running multi-tenant Flink at scale
The top 3 challenges running multi-tenant Flink at scaleThe top 3 challenges running multi-tenant Flink at scale
The top 3 challenges running multi-tenant Flink at scale
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
 
Scalable and Available, Patterns for Success
Scalable and Available, Patterns for SuccessScalable and Available, Patterns for Success
Scalable and Available, Patterns for Success
 
MongoDB Sharding
MongoDB ShardingMongoDB Sharding
MongoDB Sharding
 
Docker and Go: why did we decide to write Docker in Go?
Docker and Go: why did we decide to write Docker in Go?Docker and Go: why did we decide to write Docker in Go?
Docker and Go: why did we decide to write Docker in Go?
 
Windows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance ComparisonWindows IOCP vs Linux EPOLL Performance Comparison
Windows IOCP vs Linux EPOLL Performance Comparison
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
 
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
Demystifying flink memory allocation and tuning - Roshan Naik, UberDemystifying flink memory allocation and tuning - Roshan Naik, Uber
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at Netflix
 

En vedette

Java performance tuning
Java performance tuningJava performance tuning
Java performance tuningJerry Kurian
 
Towards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainTowards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainAttila Szegedi
 
Understanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil TeneUnderstanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil Tenejaxconf
 
Java Performance Monitoring & Tuning
Java Performance Monitoring & TuningJava Performance Monitoring & Tuning
Java Performance Monitoring & TuningMuhammed Shakir
 
Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applicationspkoza
 
Pimp my gc - Supersonic Scala
Pimp my gc - Supersonic ScalaPimp my gc - Supersonic Scala
Pimp my gc - Supersonic ScalaPierre Laporte
 
Learn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesLearn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesMatt Harrison
 
F5 link controller
F5  link controllerF5  link controller
F5 link controllerJimmy Saigon
 
Regions Of Pennsylvania
Regions Of PennsylvaniaRegions Of Pennsylvania
Regions Of Pennsylvaniaiheart4th
 
Infrastructureless Wireless networks
Infrastructureless Wireless networksInfrastructureless Wireless networks
Infrastructureless Wireless networksGwendal Simon
 
Tropical seasonal forests
Tropical seasonal forestsTropical seasonal forests
Tropical seasonal forestsVinaya Shah
 
radioligand binding studies
radioligand binding studiesradioligand binding studies
radioligand binding studiesankit
 
Feature Story - Sample
Feature Story - SampleFeature Story - Sample
Feature Story - SampleCourtney Dunn
 
Guide to Construction Procurement Strategies
Guide to Construction Procurement StrategiesGuide to Construction Procurement Strategies
Guide to Construction Procurement StrategiesSarah Fox
 
Relationship marketing concept, process and importance
Relationship marketing concept, process and importanceRelationship marketing concept, process and importance
Relationship marketing concept, process and importancegaurav jain
 
Citing Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA formatCiting Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA formatkhornberger
 
Difference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leafletsDifference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leafletsInteractive Bees
 
How Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't knowHow Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't knowAmie Weller
 
Layouting Your School Paper
Layouting Your School PaperLayouting Your School Paper
Layouting Your School PaperJerry Noveno
 
Pharmaceutical packaging
Pharmaceutical packagingPharmaceutical packaging
Pharmaceutical packagingceutics1315
 

En vedette (20)

Java performance tuning
Java performance tuningJava performance tuning
Java performance tuning
 
Towards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainTowards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages Toolchain
 
Understanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil TeneUnderstanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil Tene
 
Java Performance Monitoring & Tuning
Java Performance Monitoring & TuningJava Performance Monitoring & Tuning
Java Performance Monitoring & Tuning
 
Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applications
 
Pimp my gc - Supersonic Scala
Pimp my gc - Supersonic ScalaPimp my gc - Supersonic Scala
Pimp my gc - Supersonic Scala
 
Learn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesLearn 90% of Python in 90 Minutes
Learn 90% of Python in 90 Minutes
 
F5 link controller
F5  link controllerF5  link controller
F5 link controller
 
Regions Of Pennsylvania
Regions Of PennsylvaniaRegions Of Pennsylvania
Regions Of Pennsylvania
 
Infrastructureless Wireless networks
Infrastructureless Wireless networksInfrastructureless Wireless networks
Infrastructureless Wireless networks
 
Tropical seasonal forests
Tropical seasonal forestsTropical seasonal forests
Tropical seasonal forests
 
radioligand binding studies
radioligand binding studiesradioligand binding studies
radioligand binding studies
 
Feature Story - Sample
Feature Story - SampleFeature Story - Sample
Feature Story - Sample
 
Guide to Construction Procurement Strategies
Guide to Construction Procurement StrategiesGuide to Construction Procurement Strategies
Guide to Construction Procurement Strategies
 
Relationship marketing concept, process and importance
Relationship marketing concept, process and importanceRelationship marketing concept, process and importance
Relationship marketing concept, process and importance
 
Citing Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA formatCiting Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA format
 
Difference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leafletsDifference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leaflets
 
How Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't knowHow Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't know
 
Layouting Your School Paper
Layouting Your School PaperLayouting Your School Paper
Layouting Your School Paper
 
Pharmaceutical packaging
Pharmaceutical packagingPharmaceutical packaging
Pharmaceutical packaging
 

Similaire à Everything I Ever Learned About JVM Performance Tuning @Twitter

Decima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero DawnDecima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero DawnGuerrilla
 
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_SummaryHiram Fleitas León
 
Memory Management & Garbage Collection
Memory Management & Garbage CollectionMemory Management & Garbage Collection
Memory Management & Garbage CollectionAbhishek Sur
 
Performance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen BorgersPerformance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen BorgersNLJUG
 
Spil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NLSpil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NLThijs Terlouw
 
NYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ SpeedmentNYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ SpeedmentSpeedment, Inc.
 
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...Chester Chen
 
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterSearch at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterLucidworks
 
NickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScaleNickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScaleKostas Mavridis
 
An Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in JavaAn Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in JavaAbhishek Asthana
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]Speedment, Inc.
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]Malin Weiss
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Christopher Curtin
 
Large scale computing with mapreduce
Large scale computing with mapreduceLarge scale computing with mapreduce
Large scale computing with mapreducehansen3032
 
Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?Saltmarch Media
 
7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth7. Key-Value Databases: In Depth
7. Key-Value Databases: In DepthFabio Fumarola
 
High Performance With Java
High Performance With JavaHigh Performance With Java
High Performance With Javamalduarte
 
In-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great TasteIn-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great TasteDataWorks Summit
 

Similaire à Everything I Ever Learned About JVM Performance Tuning @Twitter (20)

Decima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero DawnDecima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero Dawn
 
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
 
Memory Management & Garbage Collection
Memory Management & Garbage CollectionMemory Management & Garbage Collection
Memory Management & Garbage Collection
 
Performance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen BorgersPerformance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen Borgers
 
Spil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NLSpil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NL
 
Pulsar
PulsarPulsar
Pulsar
 
NYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ SpeedmentNYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ Speedment
 
Loom promises: be there!
Loom promises: be there!Loom promises: be there!
Loom promises: be there!
 
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
 
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterSearch at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, Twitter
 
NickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScaleNickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScale
 
An Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in JavaAn Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in Java
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
 
Large scale computing with mapreduce
Large scale computing with mapreduceLarge scale computing with mapreduce
Large scale computing with mapreduce
 
Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?
 
7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth
 
High Performance With Java
High Performance With JavaHigh Performance With Java
High Performance With Java
 
In-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great TasteIn-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great Taste
 

Dernier

Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 

Dernier (20)

Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 

Everything I Ever Learned About JVM Performance Tuning @Twitter

  • 1. Attila Szegedi, Software Engineer @asz 1
  • 2. Everything I ever learned about JVM performance tuning @twitter 2
  • 3. Everything More than I ever wanted to learned about JVM performance tuning @twitter 3
  • 4. • Memory tuning • CPU usage tuning • Lock contention tuning • I/O tuning 4
  • 8. Twitter’s biggest enemy Latency CC licensed image from http://www.flickr.com/photos/dunechaser/ 213255210/ 5
  • 9. Latency contributors • By far the biggest contributor is garbage collector • others are, in no particular order: • in-process locking and thread scheduling, • I/O, • application algorithmic inefficiencies. 6
  • 10. Areas of performance tuning • Memory tuning • Lock contention tuning • CPU usage tuning • I/O tuning 7
  • 11. Areas of memory performance tuning • Memory footprint tuning • Allocation rate tuning • Garbage collection tuning 8
  • 12. Memory footprint tuning • So you got an OutOfMemoryError… • Maybe you just have too much data! • Maybe your data representation is fat! • You can also have a genuine memory leak… 9
  • 13. Too much data • Run with -verbosegc • Observe numbers in “Full GC” messages secs] [Full GC $before->$after($total), $time • Can you give the JVM more memory? • Do you need all that data in memory? Consider using: • a LRU cache, or… • soft references* 10
  • 14. Fat data • Can be a problem when you want to do wacky things, like • load the full Twitter social graph in a single JVM • load all user metadata in a single JVM • Slimming internal data representation works at these economies of scale 11
  • 15. Fat data: object header • JVM object header is normally two machine words. • That’s 16 bytes, or 128 bits on a 64-bit JVM! • new java.lang.Object() takes 16 bytes. • new byte[0] takes 24 bytes. 12
  • 16. Fat data: padding class A { byte x; } class B extends A { byte y; } • new A() takes 24 bytes. • new B() takes 32 bytes. 13
  • 17. Fat data: no inline structs class C { Object obj = new Object(); } • new C() takes 40 bytes. • similarly, no inline array elements. 14
  • 18. Slimming taken to extreme • A research project had to load the full follower graph in memory • Each vertex’s edges ended up being represented as int arrays • If it grows further, we can consider variable- length differential encoding in a byte array 15
  • 19. Compressed object pointers • Pointers become 4 bytes long • Usable below 32 GB of max heap size • Automatically used below 30 GB of max heap 16
  • 20. Compressed object pointers Uncompressed Compressed 32-bit Pointer 8 4 4 Object header 16 12* 8 Array header 24 16 12 Superclass pad 8 4 4 * Object can have 4 bytes of fields and still only take up 16 bytes 17
  • 21. Avoid instances of primitive wrappers • Hard won experience with Scala 2.7.7: • a Seq[Int] stores java.lang.Integer • an Array[Int] stores int • first needs (24 + 32 * length) bytes • second needs (24 + 4 * length) bytes 18
  • 22. Avoid instances of primitive wrappers • This was fixed in Scala 2.8, but it shows that: • you often don’t know the performance characteristics of your libraries, • and won’t ever know them until you run your application under a profiler. 19
  • 23. Map footprints • Guava MapMaker.makeMap() takes 2272 bytes! • MapMaker.concurrencyLevel(1).makeMap() takes 352 bytes! • ConcurrentMap with level 1 makes sense sometimes (i.e. you don’t want a ConcurrentModificationException) 20
  • 24. Thrift can be heavy • Thrift generated classes are used to encapsulate a wire tranfer format. • Using them as your domain objects: almost never a good idea. 21
  • 25. Thrift can be heavy • Every Thrift class with a primitive field has a java.util.BitSet __isset_bit_vector field. • It adds between 52 and 72 bytes of overhead per object. 22
  • 26. Thrift can be heavy 23
  • 27. Thrift can be heavy • Thrift does not support 32-bit floats. • Coupling domain model with transport: • resistance to change domain model • You also miss oportunities for interning and N-to-1 normalization. 24
  • 28. class Location { public String city; public String region; public String countryCode; public int metro; public List<String> placeIds; public double lat; public double lon; public double confidence; 25
  • 29. class SharedLocation { public String city; public String region; public String countryCode; public int metro; public List<String> placeIds; class UniqueLocation { private SharedLocation sharedLocation; public double lat; public double lon; public double confidence; 26
  • 30. Careful with thread locals • Thread locals stick around. • Particularly problematic in thread pools with m⨯n resource association. • 200 pooled threads using 50 connections: you end up with 10 000 connection buffers. • Consider using synchronized objects, or • just create new objects all the time. 27
  • 32. Performance tradeoff Memory Time Convenient, but oversimplified view. 29
  • 33. Performance triangle Memory footprint Throughput Latency 30
  • 34. Performance triangle Compactness Throughput Responsiveness C ⨯T ⨯ R = a • Tuning: vary C, T, R for fixed a • Optimization: increase a 31
  • 35. Performance triangle • Compactness: inverse of memory footprint • Responsiveness: longest pause the application will experience • Throughput: amount of useful application CPU work over time • Can trade one for the other, within limits. • If you have spare CPU, can be pure win. 32
  • 36. Responsiveness vs. throughput 33
  • 37. Biggest threat to responsiveness in the JVM is the garbage collector 34
  • 38. Memory pools Eden Survivor Old Code Permanent cache This is entirely HotSpot specific! 35
  • 39. How does young gen work? Eden S1 S2 Old • All new allocation happens in eden. • It only costs a pointer bump. • When eden fills up, stop-the-world copy-collection into the survivor space. • Dead objects cost zero to collect. • Aftr several collections, survivors get tenured into old generation. 36
  • 40. Ideal young gen operation • Big enough to hold more than one set of all concurrent request-response cycle objects. • Each survivor space big enough to hold active request objects + tenuring ones. • Tenuring threshold such that long-lived objects tenure fast. 37
  • 41. Old generation collectors • Throughput collectors • -XX:+UseSerialGC • -XX:+UseParallelGC • -XX:+UseParallelOldGC • Low-pause collectors • -XX:+UseConcMarkSweepGC • -XX:+UseG1GC (can’t discuss it here) 38
  • 42. Adaptive sizing policy • Throughput collectors can automatically tune themselves: • -XX:+UseAdaptiveSizePolicy • -XX:MaxGCPauseMillis=… (i.e. 100) • -XX:GCTimeRatio=… (i.e. 19) 39
  • 44. Choose a collector • Bulk service: throughput collector, no adaptive sizing policy. • Everything else: try throughput collector with adaptive sizing policy. If it didn’t work, use concurrent mark-and-sweep (CMS). 41
  • 45. Always start with tuning the young generation • Enable -XX:+PrintGCDetails, -XX:+PrintHeapAtGC, and -XX:+PrintTenuringDistribution. • Watch survivor sizes! You’ll need to determine “desired survivor size”. • There’s no such thing as a “desired eden size”, mind you. The bigger, the better, with some responsiveness caveats. • Watch the tenuring threshold; might need to tune it to tenure long lived objects faster. 42
  • 46. -XX:+PrintHeapAtGC Heap after GC invocations=7000 (full 87): par new generation total 4608000K, used 398455K eden space 4096000K, 0% used from space 512000K, 77% used to space 512000K, 0% used concurrent mark-sweep generation total 3072000K, used 1565157K concurrent-mark-sweep perm gen total 53256K, used 31889K } 43
  • 47. -XX:+PrintTenuringDistribution Desired survivor size 262144000 bytes, new threshold 4 (max 4) - age 1: 137474336 bytes, 137474336 total - age 2: 37725496 bytes, 175199832 total - age 3: 23551752 bytes, 198751584 total - age 4: 14772272 bytes, 213523856 total • Things of interest: • Number of ages • Size distribution in ages • You want strongly declining. 44
  • 48. Tuning the CMS • Give your app as much memory as possible. • CMS is speculative. More memory, less punitive miscalculations. • Try using CMS without tuning. Use -verbosegc and -XX:+PrintGCDetails. • Didn’t get any “Full GC” messages? You’re done! • Otherwise, tune the young generation first. 45
  • 49. Tuning the old generation • Goals: • Keep the fragmentation low. • Avoid full GC stops. • Fortunately, the two goals are not conflicting. 46
  • 50. Tuning the old generation • Find the minimum and maximum working set size (observe “Full GC” numbers under stable state and under load). • Overprovision the numbers by 25-33%. • This gives CMS a cushion to concurrently clean memory as it’s used. 47
  • 51. Tuning the old generation • Set -XX:InitiatingOccupancyFraction to between 80-75, respectively. • corresponds to overprovisioned heap ratio. • You can lower initiating occupancy fraction to 0 if you have CPU to spare. 48
  • 52. Responsiveness still not good enough? • Too many live objects during young gen GC: • Reduce NewSize, reduce survivor spaces, reduce tenuring threshold. • Too many threads: • Find the minimal concurrency level, or • split the service into several JVMs. 49
  • 53. Responsiveness still not good enough? • Does the CMS abortable preclean phase, well, abort? • It is sensitive to number of objects in the new generation, so • go for smaller new generation • try to reduce the amount of short-lived garbage your app creates. 50
  • 54. Part III: let’s take a break from GC 51
  • 55. Thread coordination optimization • You don’t have to always go for synchronized. • Synchronization is a read barrier on entry; write barrier on exit. • Sometimes you only need a half-barrier; i.e. in a producer-observer pattern. • Volatiles can be used as half-barriers. 52
  • 56. Thread coordination optimization • For atomic update of a single value, you only need Atomic{Integer|Long}.compareAndSet(). • You can use AtomicReference.compareAndSet() for atomic update of composite values represented by immutable objects. 53
  • 57. Fight CMS fragmentation with slab allocators • CMS doesn’t compact, so it’s prone to fragmentation, which will lead to a stop-the-world pause. • Apache Cassandra uses a slab allocator internally. 54
  • 58. Cassandra slab allocator • 2MB slab sizes • copy byte[] into them using compare-and-set • GC before: 30-60 seconds every hour • GC after: 5 seconds once in 3 days and 10 hours 55
  • 59. Slab allocator constraints • Works for limited usage: • Buffers are written to linearly, flushed to disk and recycled when they fill up. • The objects need to be converted to binary representation anyway. • If you need random freeing and compaction, you’re heading down the wrong direction. • If you find yourself writing a full memory manager on top of byte buffers, stop! 56
  • 60. Soft references revisited • Soft reference clearing is based on the amount of free memory available when GC encounters the reference. • By definition, throughput collectors always clear them. • Can use them with CMS, but they increase memory pressure and make the behavior less predictable. • Need two GC cycles to get rid of referenced objects. 57
  • 61. Everything More than I ever wanted to learned about JVM performance tuning @twitter Questions? 58
  • 62. Attila Szegedi, Software Engineer @asz 59

Notes de l'éditeur

  1. Here it is all together\n
  2. Here it is all together\n
  3. Here it is all together\n
  4. Here it is all together\n
  5. Here it is all together\n
  6. Here it is all together\n
  7. Here it is all together\n
  8. Here it is all together\n
  9. Here it is all together\n
  10. Here it is all together\n
  11. Here it is all together\n
  12. Here it is all together\n
  13. Here it is all together\n
  14. Here it is all together\n
  15. Here it is all together\n
  16. Here it is all together\n
  17. Here it is all together\n
  18. Here it is all together\n
  19. Here it is all together\n
  20. Here it is all together\n
  21. Here it is all together\n
  22. Here it is all together\n
  23. Here it is all together\n
  24. Here it is all together\n
  25. Here it is all together\n
  26. Here it is all together\n
  27. \n
  28. \n
  29. Here it is all together\n
  30. Here it is all together\n
  31. Here it is all together\n
  32. Here it is all together\n
  33. Here it is all together\n
  34. Here it is all together\n
  35. Here it is all together\n
  36. Here it is all together\n
  37. Here it is all together\n
  38. Here it is all together\n
  39. Here it is all together\n
  40. Here it is all together\n
  41. Here it is all together\n
  42. Here it is all together\n
  43. Here it is all together\n
  44. Here it is all together\n
  45. Here it is all together\n
  46. Here it is all together\n
  47. Here it is all together\n
  48. Here it is all together\n
  49. Here it is all together\n
  50. Here it is all together\n
  51. Here it is all together\n
  52. Here it is all together\n
  53. Here it is all together\n
  54. Here it is all together\n
  55. Here it is all together\n
  56. Here it is all together\n
  57. Here it is all together\n
  58. Here it is all together\n
  59. Here it is all together\n
  60. Here it is all together\n
  61. Here it is all together\n