Slides from JCON World 2023 - "The New Frontier in Open Source Java Compilers: Just-In-Time Compilation as a Service"

1. 2

2. Who am I?
• Software Engineer and Developer
Advocate at IBM
• Based in the Austin, TX USA
• Focus on OSS cloud native Java
technologies
Rich Hagarty
@rhagarty8

3. 2023 State of Java in the Enterprise report by Vaadin
6
Java is foundational to enterprise applications
“Organizations continue to expand their investments in Java, building new
Java applications and modernizing existing Java applications.”
Java is undergoing a cloud-native resurgence
“Java is undergoing a transformative resurgence as a cloud-native
technology. Enterprises continue to invest in new Java applications, while
also modernizing existing Java applications for the cloud.”
How do we get there?
Let’s discuss…

4. Agenda
REASON:
JVM and JIT
compiler – the good
and the bad
7
JIT-as-a-Service
SOLUTION:
JIT-as-a-Service
to the rescue
PROBLEM:
Java on Cloud - a
bad fit for
microservices

5. Java on Cloud
A Good Fit?
8

6. Legacy Java Apps
9
•Java monolith on
dedicated server
•Plenty of CPU
power and
memory
•Never went down
•Infrequent
upgrade/refresh
schedule

7. Moving to the Cloud
10
-Running in containers
-Managed by Cloud
Provider (and K8s)
-Auto-scaling to meet
demand
Cloud native App talking to Microservices

8. Main Motivators
11
•Less infrastructure to maintain and manage
•Flexible & scalable
•Easier to roll-out new releases more frequently
•Take advantage of latest-greatest Cloud
technologies
•Save money

9. But what about performance?
12
Need to balance cost vs performance
2 variables:
-Container size
-Number of instances (efficient scaling)

10. Performance vs Cost
13
Performance
Cost
Variables – Container Size and Container Instances
Containers too small – inefficient scaling

11. Performance vs Cost
14
Performance
Cost
Variables – Container Size and Container Instances
Containers too big

12. Performance vs Cost
15
Performance
Cost
Variables – Container Size and Container Instances
Containers too big, too many instances

13. Performance vs Cost
16
Performance
Cost
Variables – Container Size and Container Instances
Ahh – just right. Why is this so hard to do?

14. JVM and JIT compiler:
The Good and The
Bad 17

15. Java Virtual Machine (JVM)
18
The Good
• Device independent – write once, run anywhere
• > 25 years of improvements
• JIT produces optimized machine code through use of Profilers
• Efficient garbage collection
• Longer it runs, the better it runs (JVM collects more profile
data, JIT compiles more methods)

16. Java Virtual Machine (JVM)
19
The Bad
• Initial execution run is “interpreted”, which is relatively slow
• “Hot Spot” methods compiled by JIT can create CPU and
memory spikes
• CPU spikes cause lower QoS
• Memory spikes cause OOM issues, including crashes
• Slow start-up time
• Slow ramp-up time

17. Java Virtual Machine (JVM)
20
0
50
100
150
200
250
300
350
400
0 30 60 90
CPU
utilization
(%)
Time (sec)
Daytrader7 CPU consumption
CPU spikes caused
by JIT compilation
0
100000
200000
300000
400000
500000
600000
0 30 60 90
Resident
set
size
(KB)
Time (sec)
Daytrader7 memory footprint
Footprint spikes caused
by JIT compilation

18. Cost vs Performance - Revisited
21
Finding the ”Sweet Spot”
• Right-sized provisioned containers
• Well-behaved efficient auto-scaling
Let’s revisit our 2 variables:
Container Size + Container Instances

19. Container Size
22
Main pain-points:
•Need to over-provision
to avoid OOM
•Very hard to do – JVMs
have a non-
deterministic behavior
0
100000
200000
300000
400000
500000
600000
0 30 60 90
Resident
set
size
(KB)
Time (sec)
Daytrader7 memory footprint
Footprint spikes caused
by JIT compilation

20. Container Instances (Auto-Scaling)
23
Main pain-points:
•Slow start-up and ramp-up times
•CPU spikes can cause auto-scaler to
incorrectly launch additional instances

21. Cost vs Performance - Revisited
24
Goal:
• Minimize/eliminate CPU and memory spikes – reduce
container size
• Improve start-up and ramp-up times – improve
scaling efficiency
Solution:
• JIT-as-a-Service

22. JIT-as-a-Service
to the rescue
25

23. JIT-as-a-Service
Decouple the JIT compiler from the JVM and let it run as an independent process
Offload JIT
compilation to
remote process
Remote
JIT
Remote
JIT
JVM
JIT
JVM
JIT
Kubernetes
Control Plane
Treat JIT
compilation as a
cloud service
• Auto-managed by orchestrator
• A mono-to-micro solution
• Local JIT still available
27

24. Eclipse OpenJ9 JITServer (Semeru Cloud Compiler)
• JITServer feature is available in the Eclipse OpenJ9 JVM
• “Semeru Cloud Compiler” when used with Semeru Runtimes
• OpenJ9 combines with OpenJDK to form a full JDK
Link to GitHub repo: https://github.com/eclipse-openj9/openj9
28

25. Overview of Eclipse OpenJ9
Designed from the start to span all the
operating systems needed by IBM products
This JVM can go from small to large
Can handle constrained environments or
memory rich ones
Renowned for its small footprint, fast start-up
and ramp-up time
Is used by the largest enterprises on the planet
29

26. Eclipse OpenJ9 Performance
30

27. IBM Semeru Runtimes
“The part of Java that’s
really in the clouds”
IBM-built OpenJDK runtimes powered by the Eclipse OpenJ9
JVM
No cost, stable, secure, high performance, cloud optimized,
multi-platform, ready for development and production use
Open Edition
• Open source license (GPLv2+CE)
• Available for Java 8 (LTS), 11 (LTS), 16, 17 (LTS), 18, 19,
20 (21 available soon)
Certified Edition
• IBM license
• Java SE TCK certified.
• Available for Java 11, 17 (21 available soon)
31

28. IBM Semeru Runtimes
“The part of Java that’s
really in the clouds”
• The default Java for Liberty in all
environments
• New Semeru and Liberty innovations
together make Java part of the Cloud
32
Liberty
container
App
instance
Semeru JVM

29. IBM Semeru Runtimes
“The part of Java that’s really in the clouds”
Other OpenJDK Distributions:
• OpenJDK builds by Oracle
• Adoptium Eclipse Temurin
• Azul Zulu
• BellSoft Liberica
• Amazon Corretto
• Alibaba Dragonwell
• Microsoft Build of OpenJDK
• Red Hat Build of OpenJDK
• Oracle GraalVM
34

30. JITServer advantages for JVM Clients
35
Provisioning
Easier to size; only consider the needs
of the application
Performance
Improved ramp-up time due to JITServer
supplying extra CPU power when the JVM
needs it the most.
Reduced CPU consumption with
JITServer AOT cache
Cost
Reduced memory consumption means
increased application density and
reduced operational cost.
Efficient auto-scaling – only pay for
what you need/use.
Resiliency
If the JITServer crashes, the JVM can
continue to run and compile with its
local JIT

31. Performance graphs
36

32. JITServer value in Kubernetes
• https://blog.openj9.org/2021/10/20/save-money-with-jitserver-
on-the-cloud-an-aws-experiment/
• Experimental test bed
• ROSA (RedHat OpenShift Service on AWS)
• Demonstrate that JITServer is not tied to IBM HW or SW
• OCP cluster: 3 master nodes, 2 infra nodes, 3 worker nodes
• Worker nodes have 8 vCPUs and 16 GB RAM (only ~12.3 GB available)
• Four different applications
• AcmeAir Microservices
• AcmeAir Monolithic
• Petclinic (Springboot framework)
• Quarkus
• Low amount of load to simulate conditions seen in practice
• OpenShift Scheduler to manage pod and node deployments/placement
37

33. JITServer improves container density and cost
Default config
AM 500
B 550
C 550
F 450 P 450
P 450
B 550
F 450
AM 500
A 350
AM 500
M 200
Q 350
P 450
Q 350
D 600
D 1000
F 450
B 550
Q 350
AM 500
AM 500
AM 500
B 550
B 550
A 350
C 550
F 450
M 200
P 450
P 450
P 450
Q 350
Q 350
D 1000
AM 500 B 550
P 450
AM 500
B 550
B 550
C 550
C 550
F 450
F 450 P 450
Q 350
Q 350
D 1000
D 1000
Q 350
AM 250
AM 250
P 250
P 250
F 250
F 250
B 400 C 350
Q 150
Q 150
M 150
AM 250
AM 250
P 250
P 250
F 250
B 400
Q 150
Q 150
J 1200
A 250
B 400
B 400
C 350
D 1000 D 1000
D 600
AM 250
AM 250
P 250
P 250
F 250
F 250
B 400 C 350
Q 150
Q 150
M 150
AM 250
AM 250
P 250
P 250
F 250
B 400
Q 150
Q 150
J 1200
A 250
B 400
B 400
C 350
D 1000
D 1000
JITServer config
Legend:
AM: AcmeAir monolithic
A: Auth service
B: Booking service
C: Customer service
D: Database (mongo/postgres)
F: Flight service
J: JITServer
M: Main service
P: Petclinic
Q: Quarkus
Total=8250 MB Total=8550 MB Total=8600 MB
Total=9250 MB Total=9850 MB
6.3 GB less

34. Throughput comparison
0
200
400
600
800
1000
1200
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
AcmeAir Microservices
0
200
400
600
800
1000
1200
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
AcmeAir Monolithic
0
20
40
60
80
100
120
140
160
180
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
Petclinic
0
500
1000
1500
2000
2500
3000
3500
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
Quarkus
Machine load:
17.5% from apps
7% from OpenShift
39
è JITServer and default configuration achieve the same level of throughput at steady-state
JITServer Baseline

35. Conclusions from high density experiments
• JITServer can improve container density and reduce
operational costs of Java applications running in the cloud by
20-30%
• Steady-state throughput is the same despite using fewer nodes
40

36. Horizontal Pod Autoscaling in Kubernetes
• Better autoscaling behavior with JITServer due to faster ramp-up
• Less risk to trick the HPA due to transient JIT compilation overhead
41
Setup:
Single node Microk8s cluster (16 vCPUs, 16 GB RAM)
JVMs limited to 1 CPU, 500MB
JITServer limited to 8 CPUs and has AOT cache enabled
Load applied with JMeter, 100 threads, 10 ms think-
time, 60s ramp-up time
Autoscaler: scales up when average CPU utilization
exceeds 0.5P. Up to 15 AcmeAir instances
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 60 120 180 240 300 360 420 480
Throughput
(pages/sec)
Time (sec)
AcmeAir throughput when using Kubernetes autoscaling
Baseline JITServer+AOTcache

37. DEMO
42

38. Improve ramp-up time with JITServer
• Experiment in docker containers
• Show that JITServer improves ramp-up
• Show that JITServer allows a lower memory limit for JVM containers
43
OpenLiberty+
AcmeAir
1P, 400 MB
limit
OpenLiberty+
AcmeAir
1P, 200 MB
limit
OpenLiberty+
AcmeAir
1P, 200 MB
limit
MongoDB
JITServer
4P, 1GB limit
JMeter
JMeter
JMeter
InfluxDB
Grafana
Collect throughput
data from JMeter
Display throughput
data
Apply load
to AcmeAir
instances
Provide data
persistence
services
Run the AcmeAir
application
Provide JIT
compilation
services
Grafana
Prometheus
Scrape
metrics
Display
JITServer
metrics

39. 44

40. 45

41. How to use it
46

42. JITServer usage basics
• One JDK, three different personas
• Normal JVM: $JAVA_HOME/bin/java MyApp
• JITServer: $JAVA_HOME/bin/jitserver
• Client JVM: $JAVA_HOME/bin/java -XX:+UseJITServer MyApp
• Optional further configuration through JVM command line options
• At the server:
-XX:JITServerPort=… default: 38400
• At the client:
-XX:JITServerAddress=… default: ‘localhost’
-XX:JITServerPort=… default: 38400
• Full list of options: https://www.eclipse.org/openj9/docs/jitserver/
• Note: Java version and OpenJ9 release at client and server must match
47

43. JITServer usage in Kubernetes
• Typically we create/configure
• JITServer deployment
• JITServer service (clients interact with service)
• Use
• Yaml files
• Helm charts: repo https://github.com/eclipse-openj9/openj9-utils/tree/master/helm-chart
• Certified OpenShift/K8s Operators from Open Liberty
• Tutorial: https://developer.ibm.com/tutorials/using-openj9-
jitserver-in-kubernetes/
48

44. JITServer usage in Kubernetes
• Create JITServer service that clients connect to
• Use
• Yaml files
• Helm charts: repo https://raw.githubusercontent.com/eclipse/openj9-utils/master/helm-
chart/
• Certified OpenShift/K8s Operators from
Open Liberty
• Tutorial: https://developer.ibm.com/tutorials/using-openj9-
jitserver-in-kubernetes/
49
semeruCloudCompiler:
enable: true
Simple to deploy with
Liberty operator

45. JITServer encryption/authentication through TLS
• Needs additional JVM options
• Server: -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem
• Client: -XX:JITServerSSLRootCerts=cert.pem
• Certificates and keys can be provided using Kubernetes TLS Secrets
• Create TLS secret:
• kubectl create secret tls my-tls-secret --key <private-key-filename> --cert <certificate-
filename>
• Use a volume to map “pem” files
50
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container-name
image: my-image
volumeMounts:
- name: secret-volume
mountPath: /etc/secret-volume
volumes:
- name: secret-volume
secret:
secretName: my-tls-secret

46. Monitoring
• Support for custom metrics for Prometheus
• Metrics scrapping: GET request to http://<jitserveraddress>:<port>/metrics
• Command line options:
-XX:+JITServerMetrics -XX:JITServerMetricsPort=<port>
• Metrics available
• jitserver_cpu_utilization
• jitserver_available_memory
• jitserver_connected_clients
• jitserver_active_threads
• Verbose logging
• Print client/server connections
-XX:+JITServerLogConnections
• Heart-beat: periodically print to verbose log some JITServer stats
• -Xjit:statisticsFrequency=<period-in-ms>
• Print detailed information about client/server behavior
-Xjit:verbose={JITServer},verbose={compilePerformance},vlog=…
51

47. JITServer usage recommendations
• When to use it
• JVM needs to compile many methods in a relatively short time
• JVM is running in a CPU/memory constrained environment, which can worsen
interference from the JIT compiler
• The network latency between JITServer and client VM is relatively low (<1ms)
• To keep network latency low, use “latency-performance” profile for tuned and configure your VM
with SR-IOV
• Recommendations
• 10-20 client JVMs connected to a single JITServer instance
• JITServer needs 1-2 GB of RAM
• Better performance if the compilation phases from different JVM clients do not overlap
(stagger)
• Encryption adds to the communication overhead; avoid if possible
• In K8s use “sessionAffinity” to ensure a client always connects to the same server
• Enable JITServer AOT cache: -XX:+JITServerUseAOTCache (client needs to have shared
class cache enabled)
52

48. Open Liberty InstantOn
•InstantOn improves start-up times 10-18x
•InstantOn improves Semeru Cloud Compiler auto-scaling
53

49. 54
Built-in “InstantOn” Java without compromise
Characteristics
Semeru
InstantOn
Semeru
JVM
Graal
Native
Full Java support Yes Yes No
‘Instant on’ Yes No Yes
High throughput Yes Yes No
Low memory (under
load)
Yes Yes No
Dev-prod parity Yes Yes No
Dev Build
Prod
Prod
Prod
checkpoint
restore
https://openliberty.io/blog/2023/06/29/rapid-startup-instanton.html
Liberty and Semeru InstantOn innovation for near-
native startup time for Java application containers.
Up to 15x faster startup
With all the benefits of the JVM and none of the
compromises of Native Image.
Leverages Linux CRIU to perform checkpoint / restore
Ideal for cloud function and scale-to-zero workloads.
restore
restore

50. Final thoughts on JITServer (Semeru Cloud Compiler)
• JIT provides advantage, but compilation adds overhead
• Disaggregate JIT from JVM è JIT compilation as a service
• Eclipse OpenJ9 JITServer (a.k.a Semeru Cloud Compiler)
• Available now on Linux for Java 8, 11, 17 and 21 (IBM Semeru Runtimes)
• Especially good for constrained environments (micro-containers)
• Kubernetes ready (Helm chart, Prometheus, Operator)
• Can improve ramp-up, autoscaling and performance of short lived
applications
• Can reduce peak memory footprint, increase app density and reduce costs
• Get peak performance when combined with InstantOn
• Java solution to Java problem, with no compromise
55

51. Final thoughts on microservice solutions
56
JITServer + InstantOn
Full Java API support
Dynamic Java
Efficient GC
JIT Optimization
Minimal Container Size
Start-up Time
Ramp-up Time
Dev/Prod Parity
Native Image
Full Java API support
Dynamic Java
Efficient GC
JIT Optimization
Minimal Container Size
Start-up Time
Ramp-up Time
Dev/Prod Parity
Java EE
Full Java API support
Dynamic Java
Efficient GC
JIT Optimization
Minimal Container Size
Start-up Time
Ramp-up Time
* Dev/Prod Parity
Java – without compromise
* 12 factor-app methodology for cloud-native apps - https://12factor.net

52. Open Liberty
© 2021 IBM Corporation 57
Useful Links
Why choose Liberty
for Microservices
https://ibm.biz/6ReasonsW
hyLiberty
Choosing the right
Java runtime
https://ibm.biz/ChooseJava
Runtime
How to approach
application modernization
https://ibm.biz/ModernizeJa
vaApps
Open Liberty Site
https://www.openliberty.io
Open Liberty Guides
https://www.openliberty.io/
guides https://openliberty.io

53. Workloads:
Java EE
⎼ WebSphere
Application Server
⎼ Oracle WebLogic
⎼ Red Hat JBoss
⎼ Apache Tomcat
Integration
⎼ IBM MQ
⎼ App Connect
Enterprise (IIB)
Open Source SDK
for other extenders:
https://github.com/IBM/transfo
rmation-advisor-sdk
IBM Cloud Transformation Advisor
© 2021 IBM Corporation 58
Accelerates the modernization journey by quickly discovering and
analyzing on-premise Java EE and/or messaging workloads in the
enterprise to help in determining and executing the optimum
modernization steps for each.
Optimum modernization depends on workload needs! https://ibm.biz/6ReasonsWhyLiberty
Traditional
WebSphere
network
deployment
Traditional
WebSphere
‘base’
Liberty
OpenShift OpenShift OpenShift
Monolith Modernize
Operations
Modernize
Runtime
Modernize
Architecture
Containers, Java EE, monoliths & microservices…

54. 59
a free virtual consultation
with an expert
Liberty References
Read Watch
Explore the latest on WebSphere and
Liberty
ibm.biz/LibertyTV
View our recent Expert TV episode all
about Liberty https://ibm.biz/BdPn8Y
Learn more about Liberty in containers
and Operator-based deployment
ibm.biz/LibertyContainerOperator
Experience
Try Liberty as a beginner
openliberty.io/guides/getting-
started.html
Learn Liberty, MicroProfile,
Containers, Kubernetes,
Hands-on
openliberty.io/guides
Try dev mode in Containers
https://openliberty.io/guides/doc
ker.html
Why choose Liberty for
Microservices
ibm.biz/6ReasonsWhyLiberty
Choosing the right Java
runtime
ibm.biz/ChooseJavaRuntime
How to approach application
modernization
ibm.biz/ModernizeJavaApps

55. Thank You!
rich.hagarty@ibm.com
@rhagarty8
linkedin.com/in/rhagarty/
https://www.slideshare.net/RichHagarty
61