mjprof: Monadic approach for JVM profiling

mjprof :
A Monadic Approach
for JVM Profiling
Haim Yadid
Performize-IT
#Devoxx #Performance #AdoptOpenJDK @lifeyx

About Me: Haim Yadid
• 21 Years of SW development experience
• independent Performance Expert
• Consulting R&D Groups
• Training: Java Performance Optimization
• Community leader : Java.IL

Java.IL
• Israeli Java Community and User Group
• Meetup : www.meetup.com/JavaIL/
• Twitter: @java_il
• ~ 400 members
• Meeting every month

Background
&
Motivation

The Two Most Important Questions

My wife asks me

My wife asks me
What are you doing? When will you be back?

Profiler
• A Performance analysis tool
• Which answers questions about method calls:
• E.g. : Number of invocation, Average duration,CPU
consumptions Contention Memory allocation IO, Cache
misses etc…..

Statistical Profiler
• A.k.a. as Sampling Profiler
• Samples the treads activity every several milliseconds
• Not very accurate for short tasks
• Low/Medium overhead
• Do not provide the number of invocations

Instrumenting Profiler
• A.k.a Tracing
• Instruments the code
• Knows when a method starts and when it ends
• High overhead (depends on instrumentation)

Instrumenting Profiler
• A.k.a Tracing
• Instruments the code
• Knows when a method starts and when it ends
• High overhead (depends on instrumentation)
Public void foo(a) {
Measure start time
increase invocation count
bla bla;
Measure end time
}
Injected
code

jstack - A Poor Man’s Profiler
• The most trivial sampling profiler
• Part of the JDK
• Connects to a running JVM and samples all the threads
• Only once !
• Writes to standard output

And it looks like this
• For each thread we have the following block

And it looks like this
• For each thread we have the following block
Thread name Priority Id and native id
"pool-35-thread-1" prio=5 tid=10315b800 nid=0x12c1fe000 waiting on condition
[12c1fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <107ac0140> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
.....
Locked ownable synchronizers:
- None

Now do this a Million Times!

Now do this a Million Times!
• Tons of threads
• Deep stacks

mjprof

mjprof
• A command line tool
• Acts on jstack output
• Extends it (enriched thread dumps)
• Applies a sequence of operations (monads) to
• Filter
• Aggregate
• Manipulate

Prerequisites
• Java installed
• Set JAVA_HOME (JDK1.7 or later)
• Add ${JAVA_HOME}/bin to path
• Attaching to processes requires JDK and not JRE
• Can run on a headless machine

Installing mjprof
• Git repo :
https://github.com/AdoptOpenJDK/mjprof
• Download and unzip
unzip mjprof1.0-bin.zip
• Or clone and build
mvn install assembly:assembly
• Add mjprof directory to your path

Synopsis and Help (Smoke test)
• Running mjprof without parameters will print help
• It is a good way to see it is functioning

Synopsis and Help (Smoke test)
• Running mjprof without parameters will print help
• It is a good way to see it is functioning
➜ mjprof git:(master) ✗ mjprof
Synopsis
A list of the following monads concatenated with .
…
Data sources:
jstack/pid,[count],[sleep]/ -Generate dumps using stack
path/path/ -Read thread dump from file
stdin -Read thread dumps from standard input
visualvm/path/ -Read profiling session from xml export of VisualVM
….

Monads
• Pipeline Building blocks
• Monads can
• Collect data
• Filter out certain threads
• Transform threads information
• Aggregate threads information
• Snapshot write state to a file or GUI
• Concatenate with a . (period)

Data Sources
stdin
file
jstack
jmx

Data Sources
stdin • cat mythreaddump.txt | mjprof …
file
jstack
jmx

Data Sources
• cat mythreaddump.txt | mjprof …
• mjprof path/mythreaddump.txt/
stdin
file
jstack
jmx

Data Sources
• mjprof jstack/19873/
stdin
file
jstack
jmx

Data Sources
• mjprof jstack/19873/
• mjprof jmx/myhost:65327/
stdin
file
jstack
jmx

Monads
Filters Aggregations Stack frame level
contains -contains group
-prop -fn
merge
sort
-sort
frame
waiting -locks
count
mergedcalles
-frame
top
bottom
trimbelow
namesuf
-fn
-pkg
jvm -jvm
blocked
locks
running withstack
output
stdout
gui
snapshot

Extending mjprof
• mjprof is inherently extensible
• Macros
• A text file named macros.properties
• Contains a sequence of macros
• Plugins
• Annotate your extension with @Plugin
• Compile it and place your jar in the directory plugins under mjprof
installation

10 mjprof Recipes

Question: How many threads my process has?

path/stack.txt/
mjprof jstack/1971/
jstack/MainClass/

path/stack.txt/
mjprof jstack/1971/ . count
jstack/MainClass/
➜ distro mjprof jstack/1971/.count
2014-11-01 10:34:10
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):
Total number of threads is 1045

Question: How many jetty thread are working ATM?

mjprof jstack/MyApp/

mjprof jstack/MyApp/ . contains/name,qtp/

. contains/state,RUN/
➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/
2014-11-04 20:05:03
"qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable
[0x000000011bc5f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000780209080> (a sun.nio.ch.Util$2)
- locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet)

. contains/state,RUN/ . count
➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/
2014-11-04 20:05:03
"qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable
[0x000000011bc5f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000780209080> (a sun.nio.ch.Util$2)
- locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet)

Question: Which threads are running my logic ?

Question: Which threads are running my logic ?
mjprof jstack/MyApp/ . contains/stack,myapp/
2014-11-04 20:11:55
"Indexing daemon" prio=5 tid=0x00007fb306bf4800 nid=0x6307 waiting on condition
[0x000000011c171000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.myapp.IndexManagement$1.run(IndexManagement.java:148)
"main" prio=5 tid=0x00007fb303809000 nid=0x1303 in Object.wait()
[0x0000000105500000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.myapp.MyAppApp.main(MyAppApp.java:18)

Question: Stack-trace is cluttered, how do I shorten it ?

mjprof jmx/MyApp/ .

mjprof jmx/MyApp/ . frame/mycompany/

-frame/grizzly/

-frame/grizzly/
top/3/

Question: How can I group threads by state ?

Question: How can I group threads by state ?
mjprof jmx/MyApp/ . group/state/ . gui

Question: How do I aggregate several thread dumps

mjprof jmx/MyApp,10,1000/

mjprof jmx/MyApp,10,1000/ .
group/state/
merge

mjprof jmx/MyApp,10,1000/ .
group/state/
merge .

Question: How can I combine thread dumps from several processes?

Question: How can I combine thread dumps from several processes?
mjprof jmx/myhost1.com:5467/ .
jmx/myhost2.com:5467/ .
group/state/ . gui

How do I measure thread pool CPU consumption?

mjprof jmxc/MyApp/

mjprof jmxc/MyApp/ . contains/name,COFFEE/

mjprof jmxc/MyApp/ . contains/name,COFFEE/
. -pkg . group
"*" count=7 tid=* cpu_ns=1164536000 wall_ms=4804 %cpu=24.24
java.lang.Thread.State: TIMED_WAITING
100.00% [7/7] at Thread.run(Thread.java:744)
100.00% [7/7] at Sleeper.run(Sleeper.java:51)
100.00% [7/7] at Sleeper.loopForever(Sleeper.java:37)
100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:32)
100.00% [7/7] at Recurse.singleCycle(Recurse.java:16)
100.00% [7/7] at Recurse.doRecursion(Recurse.java:25)
100.00% [7/7] at Recurse.doRecursion(Recurse.java:25)

Question: Which threads are blocked by the lock I hold

Question: Which threads are blocked by the lock I hold
mjprof path/threaddump.txt/ .
blocked
.
contains/stack,0x00000007aab51b38/

Question: What is the overall locking state of my application

Question: What is the overall locking state of my application
mjprof path/threaddump.txt/ .
contains/stack,myapp/
locks
.
"CPU3_5" prio=5 tid=0x00007fc4f4889800 nid=0x5703 waiting for monitor…
java.lang.Thread.State: BLOCKED (on object monitor)
- waiting to lock <0x00000007aad98988> (a java.lang.Object)
"CPU3_4" prio=5 tid=0x00007fc4f404a800 nid=0x5503 waiting for monitor…
java.lang.Thread.State: BLOCKED (on object monitor)
- waiting to lock <0x00000007aad98988> (a java.lang.Object)
"CPU3_3" prio=5 tid=0x00007fc4f4049800 nid=0x5303 waiting on …
java.lang.Thread.State: TIMED_WAITING (sleeping)
- locked <0x00000007aad98988> (a java.lang.Object)

3#!

Thanks / Contact Me
lifey@performize-it.com
blog.performize-it.com
www.performize-it.com
https://github.com/lifey
http://il.linkedin.com/in/haimyadid
@lifeyx

mjprof: Monadic approach for JVM profiling

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (10)

Similar to mjprof: Monadic approach for JVM profiling

Similar to mjprof: Monadic approach for JVM profiling (20)

More from Haim Yadid

More from Haim Yadid (16)

Recently uploaded

Recently uploaded (20)

mjprof: Monadic approach for JVM profiling