One way to speed up you application is to bring more of your data into memory. But how to do you handle hundreds of GB of data in a JVM and what tools can help you. Mentions: Speedment, Azul, Terracotta, Hazelcast and Chronicle.

1. Qcon London 2015.
Peter Lawrey
Higher Frequency Trading Ltd
Responding rapidly when you have
100+ GB data sets in Java

2. Reactive system design
• Responsive (predictable and sufficient response time)
• Resilient (can cope with failures)
• Elastic (can employ more hardware resources and grow cost
effectively).

3. Reactive system design
Java responds *much* faster if you can keep your data in memory
Standard JVMs scale well of a specific range of memory sizes. From
• 100 MB – it is hard to make a JVM use less memory than this
(with libraries etc)
• 32 GB – above this the memory efficiency drops and you see
significant GC pauses times.
With larger data sizes, you have to be more aware of how memory is
being used, the lifecycle of your objects and which tools will best
help you utilise more memory effectively.

4. Reactive system design (Resilient)
The larger the node, the longer the recovery time. It if can
gracefully rebuild at a rate of 100 MB/s (without impact production
too much) the recover time can look like this.
Data Set to recover Time to replicate at 100 MB/s
10 GB < 2 minutes
100 GB 17 minutes
1 TB 3 hours
10 TB 28 hours
100 TB 12 days
1 PB 4 months

5. Reactive Memory Design
Knowing the limitations of your address space.
• 31–bit memory sizes (up to 2 GB)
• 35-bit memory sizes (up to 32 GB)
• 36-bit memory sizes (up to 64 GB)
• 40-bit memory sizes (up to 1 TB)
• 48-bit memory sizes (up to 256 TB)
• Beyond.

6. 32 bit operating system (31-bit heap)

7. Speedment SQL Relector
• Incrementally replicates your SQL database into Java.
• Access to SQL data with in memory speeds

8. Java 7 memory layout

9. Compress Oops in Java 7 (35-bit)
Using the default of
–XX:+UseCompressedOops
• In a 64-bit JVM, it can use “compressed” memory references.
• This allows the heap to be up to 32 GB without the overhead of 64-
bit object references. The Oracle/OpenJDK JVM still uses 64-bit
class references by default.
• As all object must be 8-byte aligned, the lower 3 bits of the
address are always 000 and don’t need to be stored. This allows
the heap to reference 4 billion * 8-bytes or 32 GB.
• Uses 32-bit references.

10. Compressed Oops with 8 byte alignment.

11. Java 8 memory layout

12. Compress Oops in Java 8 (36 bits)
Using the default of
–XX:+UseCompressedOops
–XX:ObjectAlignmentInBytes=16
• In a 64-bit JVM, it can use “compressed” memory references.
• This allows the heap to be up to 64 GB without the overhead of 64-
bit object references. The Oracle/OpenJDK JVM still uses 64-bit
class references by default.
• As all object must be 8 or 16-byte aligned, the lower 3 or 4 bits of
the address are always zeros and don’t need to be stored. This
allows the heap to reference 4 billion * 16-bytes or 64 GB.
• Uses 32-bit references.

13. 64-bit references in Java (100 GB?)
• A small but significant overhead on main memory use.
• Reduces the efficiency of CPU caches as less objects can fit in.
• Can address up to the limit of the free memory. Limited to main
memory.
• GC pauses become a real concern and can take tens of second or
many minutes.

14. Concurrent Mark Sweep (100 GB?)

15. 64-bit references in Java with a Concurrent Collector
• A small but significant overhead on every memory access to
support concurrent collection.
• Azul Zing with a fully concurrent collector. They can get worst case
pauses down to 1 to 10 milli-seconds.
• RedHat is developing a most concurrent collector (….) which can
support heap sizes around 100 GB with sub-second pauses.

16. Azul Zing Concurrent Collector (100s GB)

17. Azul Zing Concurrent Collector (100s GB)
• Can dramatically reduce GC pauses.
• Makes it easier to see and solve non GC pauses.
• Reduced development time to get a system to meet your SLAs.
• Can have lower throughputs.

18. Terracotta BigMemory
• Scale to multiple machines. Can utilize more smaller machines.

19. Terracotta BigMemory
• Uses off heap memory to store the bulk of the data.

20. Hazelcast High Density Memory Store
• Advanced memory caching
solution.
• Utilizes all main memory.
• Supports a wide range of
distributed collections.

21. NUMA Regions (~40 bits)
• Large machine are limited in how large a single bank of memory
can be. This varies based on the architecture.
• Ivy and Sandy bridge Xeon processors are limited to addressing 40
bits of real memory.
• In Haswell this has been lifted to 46-bits.
• Each Socket has “local” access to a bank of memory, however to
access other bank it may need to use a bus. This is much slower.
• The GC of a JVM can perform very poorly if it doesn’t sit within one
NUMA region. Ideally you want a JVM to use just one NUMA
region.

22. NUMA Regions (~40 bits)

23. Virtual address space (48-bit)

24. Virtual address space (48-bit)

25. Memory Mapped files (48+ bits)
• Memory mappings are not limited to main memory size.
• 64-bit OS support 128 TiB to 256 TiB virtual memory at once.
• For larger data sizes, memory mapping need to be managed and
cached manually.
• Can be shared between processes.
• A library can hide the 48-bit limitation by caching memory
mapping.

26. Memory Mapped files.
• No need to have a monolithic JVM just because you want in
memory access to all the data.

27. Peta Byte JVMs (50+ bits)
• If you are receiving 1 GB/s down a 10 Gig-E line in two weeks you
will have received over 1 PB.
• Managing this much data in large servers is more complex than
your standard JVM.
• Replication is critical. Large complex systems, are more likely to
fail and take longer to recover.
• You can have systems which cannot be recovered in the normal
way. i.e. Unless you recover faster than new data is added, you will
never catch up.

28. Peta Byte JVMs (50+ bits)

29. Peta Byte JVMs (50+ bits)

30. Questions and Answers
peter.lawrey@higherfrequencytrading.com
@PeterLawrey
http://higherfrequencytrading.com