Let's dive under the hood of Java network applications. We plan to have a deep look to classic sockets and NIO having live coding examples. Then we discuss performance problems of sockets and find out how NIO can help us to handle 10000+ connections in a single thread. And finally we learn how to build high load application server using Netty.
https://github.com/kslisenko/java-networking
2. 2CONFIDENTIAL
ABOUT ME
Java Backend engineer
Speaker at Java Tech Talks, SEC Online,
CMCC Tech Talks, IT Week
I’m interested in
Complex Java backend, SOA, databases
High load, fault-tolerant, distributed systems
KANSTANTSIN SLISENKA
EPAM Systems, Lead Software Engineer
3. 3CONFIDENTIAL
WHY IS IMPORTANT TO DIG INTO JAVA NETWORKING
Classic Java project
Tomcat + Spring + Hibernate + …
High load, latency-sensitive
Custom TCP/UDP solution
7. 7CONFIDENTIAL
NETWORK SOCKETS
Socket is blocking API over OS and Network hardware
not a physical object
PERMANENT
KNOWN PORT
ANY UNUSED
PORT
SOCKET SOCKET Max ports = 65535
10. 10CONFIDENTIAL
Server
3
2 Client 2
Client 3
Client 1
SOCKET SERVER ARCHITECTURE
no data
no data
data transfer in progress
1
Socket
Socket
SocketThread
(blocked*)
Thread
(running)
Thread
(blocked*)
*until keep alive timeout
Fixed
thread pool
Requests
queue
11. 11CONFIDENTIAL
Server
3
2 Client 2
Client 3
Client 1
SOCKET SERVER ARCHITECTURE
no data
no data
data transfer in progress
1
Socket
Socket
SocketThread
(blocked*)
Thread
(running)
Thread
(blocked*)
Client 4
*until keep alive timeout
Fixed
thread pool
Requests
queue
new
connection
12. 12CONFIDENTIAL
Server
3
2 Client 2
Client 3
Client 1
SOCKET SERVER ARCHITECTURE
no data
no data
data transfer in progress
1
Socket
Socket
SocketThread
(blocked*)
Thread
(running)
Thread
(blocked*)
Client 4
*until keep alive timeout
Fixed
thread pool
Requests
queue
new
connection
drop by timeout
13. 13CONFIDENTIAL
SOCKET SERVER CONCLUSION
1. Ineffective thread utilization
– many threads in waiting/blocked state
– extra memory for stack and thread local
resources
2. Poor performance
– context switching overhead
3. Limited number of connections
– when thread pool is fully consumed
Benefits Drawbacks
1. Simple development
well known Input/OutputStream API
2. No check for partially received
messages
thread is blocked until fully reads message
29. 29CONFIDENTIAL
Server
BASIC NIO SERVER ARCHITECTURE
buffer
Client #2
Client #1
no data
data transfer in progress
socket
channel #3 Client #3
data transfer in progress
buffer
socket
channel #2
socket
channel #1
WORKER THREAD
SELECTOR
SELECTOR
WORKER THREAD
buffer
Business
logic
30. 30CONFIDENTIAL
Server
BASIC NIO SERVER ARCHITECTURE
buffer
Client #2
Client #1
no data
data transfer in progress
socket
channel #3 Client #3
data transfer in progress
buffer
socket
channel #2
socket
channel #1
WORKER THREAD
SELECTOR
SELECTOR
WORKER THREAD
buffer
Business
logic
31. 31CONFIDENTIAL
Server
BASIC NIO SERVER ARCHITECTURE
buffer
Client #2
Client #1
no data
data transfer in progress
socket
channel #3 Client #3
data transfer in progress
buffer
get ready channels
socket
channel #2
socket
channel #1
WORKER THREAD
SELECTOR
SELECTOR
WORKER THREAD
buffer
Business
logic
32. 32CONFIDENTIAL
Server
BASIC NIO SERVER ARCHITECTURE
buffer
Client #2
Client #1
no data
data transfer in progress
socket
channel #3 Client #3
data transfer in progress
get buffer
put
read
write
get ready channels
socket
channel #2
socket
channel #1
Business
logic
WORKER THREAD
SELECTOR
SELECTOR
WORKER THREAD
buffer
33. 33CONFIDENTIAL
Server
BASIC NIO SERVER ARCHITECTURE
new
connection
buffer
Client #2
Client #4
Client #1
no data
data transfer in progress
new connection
socket
channel #3 Client #3
data transfer in progress
get buffer
put
register in selector
read
write
get ready channels
socket
channel #2
socket
channel #1
WORKER THREAD
SELECTOR
SELECTOR
WORKER THREAD
Business
logic
buffer
34. 34CONFIDENTIAL
CAN WE GO TO PRODUCTION WITH A SINGLE THREAD?
https://www.pinterest.com/banjolele/one-man-band/
38. 38CONFIDENTIAL
SELECTOR
SELECTOR
Application
thread pools
(long running tasks)
PRODUCTION-READY NIO SERVER
ServerSocket
Channel
selector
thread
…
worker
thread
worker
thread
worker
thread
application
thread
acceptor
thread
Socket
Channel
SELECTOR
selector
threadSocket
Channel
SELECTOR
selector
threadSocket
Channel
new
connection
Selector threads
Multiple processing threads to
get benefit from multi core CPU
Acceptor threads
Separate thread to not lose incoming
connections when selector threads
are overloaded
Application threads
For long tasks (back-end, DB,
computation), to avoid blocking
of selector threads
39. 39CONFIDENTIAL
PARTIAL MESSAGE RECEIVING PROBLEM
Message reader splits/joins data blocks into messages
Data block
Message
Message Message
Message Message Message
message
reader
Data
block
Data
block
MessageMessage
Data
block
40. 40CONFIDENTIAL
NIO CONCLUSION
Complexity
• complex code
• learning curve for developers
• handling partially received
messages
• combining async with sync code
(need different thread groups)
Benefits Drawbacks
Single shared thread for many
connections
• less memory
• less context switching overhead
45. 45CONFIDENTIAL
transport
Boss
EventLoopGroup
channel
channel pipeline
Worker EventLoopGroup EventExecutorGroup
NIO
NETTY SERVER ARCHITECTURE
Inbound
Handler
Inbound
Handler
Inbound
Handler
Outbound
Handler
Outbound
Handler
Outbound
Handler
Network
Socket
Native JNI
Separate thread pool
for blocking logic
I/O thread pool
For non-blocking logic
Thread pool for
new connections
53. 53CONFIDENTIAL
WHAT IS FASTER: IO OR NIO?
May be yes, may be not *
?
?
?
?
NIO is faster
Context switching is
very slow
Threads take up too
much memory
Thread synchronization
will kill you
54. 54CONFIDENTIAL
WHAT IS FASTER: IO OR NIO?
May be yes, may be not *
* Always point to holy-wars, see
www.mailinator.com/tymaPaulMultithreaded.pdf
?
?
?
Context switching is
very slow
Threads take up too
much memory
Thread synchronization
will kill you
IO is 25-30% faster
(see benchmark*)
NIO is faster
55. 55CONFIDENTIAL
WHAT IS FASTER: IO OR NIO?
May be yes, may be not *
* Always point to holy-wars, see
www.mailinator.com/tymaPaulMultithreaded.pdf
?
?
Context switching is
very slow
Threads take up too
much memory
Thread synchronization
will kill you
IO is 25-30% faster
(see benchmark*)
NIO is faster
New POSIX library in Linux > 2.6
IDLE thread costs almost zero
Context switching is much much faster
56. 56CONFIDENTIAL
WHAT IS FASTER: IO OR NIO?
May be yes, may be not *
* Always point to holy-wars, see
www.mailinator.com/tymaPaulMultithreaded.pdf
?
Context switching is
very slow
Threads take up too
much memory
Thread synchronization
will kill you
IO is 25-30% faster
(see benchmark*)
NIO is faster
New POSIX library in Linux > 2.6
IDLE thread costs almost zero
Context switching is much much faster
Don’t overuse thread-local objects!
57. 57CONFIDENTIAL
WHAT IS FASTER: IO OR NIO?
May be yes, may be not *
* Always point to holy-wars, see
www.mailinator.com/tymaPaulMultithreaded.pdf
Context switching is
very slow
Threads take up too
much memory
Thread synchronization
will kill you
IO is 25-30% faster
(see benchmark*)
NIO is faster
New POSIX library in Linux > 2.6
IDLE thread costs almost zero
Context switching is much much faster
Don’t overuse thread-local objects!
Use non-blocking data structures
58. 58CONFIDENTIAL
CONCLUSION
• Huge number of connections
• Medium or slow data rate per connection
• Asynchronous world (event-based)
• Non blocking database
• Queues
• Web-sockets
• Callbacks, listeners
• Data streaming
• Don’t use standard NIO, use frameworks like Netty
• Highly skilled team because of complex code
Use sockets* Use NIO*
*Not a strict rule, always point to holy-wars
• Small number of connections
• Huge data rate per connection
• Synchronous world (procedure-based)
• Classic enterprise applications
• Request-response protocols
• Blocking database
• Blocking RPC or web-service calls
• Bottleneck is not server, but blocking resource
• Make servers stateless and scale horizontally
59. 59CONFIDENTIAL
Java Network
Programming, 4’th
edition
O’REILLY
Elliotte Rusty Harold
BUY A BOOK, MAKE THEM RICH
Netty in Action
Manning
Norman Maurer
Java NIO
O’REILLY
Ron Hitchens
• Code examples: github.com/kslisenko/java-networking
• NIO performance: www.mailinator.com/tymaPaulMultithreaded.pdf
• NIO tutorial: tutorials.jenkov.com/java-nio/index.html
• ITT 2015 - Heinz Kabutz - The Multi-threading, Non Blocking IO:
youtube.com/watch?v=uKc0Gx_lPsg
• Netty - One Framework to rule them all by Norman Maurer:
youtube.com/watch?v=DKJ0w30M0vg
• Scalable IO in Java: http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf