4. Mercari
•Download: 27M (JP+US)
•GMV: Several Billion per a Month
•Items: Several hundreds of thousand
or more new items in a Day
•Backend language: PHP, Go, lua, etc
7. Rhebok
• Rack Handler/Web Server
• 1.5x-2x performance when compared to
Unicorn
• Prefork Architecture same as Unicorn
• Rhebok is suitable for running HTTP application
servers behind a reverse proxy like nginx
• Ruby port of Perl’s Gazelle
8. What’s Gazelle?
• High Performance Plack Handler
• Plack is Perl’s Rack
• 2x~3x times faster than servers
commonly used like Starman, Starlet
• Production Ready
• Installed to dozen servers and has shown to
reduce their CPU usage by 1-3%
10. Who should use Rhebok?
•A Highly optimized high traffic
websites
• Gaming, Ad-tec, Recipe Site, Media or
massive scale SNS
• By using Rhebok, it is possible to improve
the response speed to higher level
•Can be applied to any website
12. Who should not use
Rhebok?
•Who want to use WebSocket or
Streaming
•Who can not setup the reverse proxy
in front of Rhebok
13. Rhebok Spec
•HTTP/1.1 Web Server
•Support full HTTP/1.1 features except
for KeepAlive
•Support TCP and Unix Domain Socket
•Hot Deployment using start_server
•OobGC
15. Recommended
configuration
RhebokAmazon Web Services LLC or its affiliates. All rights reserved.
Client Multimedia Corporate
data center
Traditional
server
Mobile Client
IAM Add-on Example:
IAM Add-on
Assignment/
Task
RequesterWorkers
Reverse Proxy
(Nginx,h2o)
HTTP/2
HTTP/1.1
TCP
Unix Domain Socket
http {
listen 443 ssl http2;
upstream app {
server unix:/path/to/app.sock;
}
server {
location / {
proxy_pass http://app;
}
location ~ ^/assets/ {
root /path/to/webapp/assets;
}
}
}
24. ISUCON benchmark
• ISUCON
• web application tuning contest
• Contestants compete with the scores of
benchmark created by organizers
• Web application that becomes the theme
of ISUCON is close to the service it is in
reality
28. Rack
•Rack is specification
• interface between webservers that support
ruby and ruby web frameworks
•Rack also is implementation
• eg. Rack::Request, Response and
Middlewares
34. Role of Rack Handler
•Create env from an HTTP request sent
from a client
•Call an application
•Create an HTTP response from array
and send back to the client
env app array
HTTP req HTTP res
38. This rack handler has
some problems
• Performance problem
• Handle only one request at once
• Stop the whole world when one request
lagged
• No TIMEOUT
• No HTTP request parser support HTTP/
1.1 spec
39. Increase concurrency
• Multi process
• simple and easy to scale
• Multi thread
• lightweight context switch compared to the
process
• IO Multiplexing
• Event driven, can handle many connections
40. Concurrency strategy
• Unicorn
• -> multi process
• PUMA
• -> multi thread + limited event model
(+ multi process)
• Thin
• event model (+ multi process)
48. IO timeout
•Unicorn does not have io timeout
• send SIGKILL to a long running process
• default timeout 30 sec
E, [2015-12-08T03:13:24.863287 #90217] ERROR -- : worker=0 PID:
90243 timeout (61s > 60s), killing
E, [2015-12-08T03:13:24.865764 #90217] ERROR -- : reaped
#<Process::Status: pid 90243 SIGKILL (signal 9)> worker=0
I, [2015-12-08T03:13:24.866176 #90217] INFO -- : worker=0
spawning...
49. Using select(2)
while true
connection = @server.accept
buf = self.read_timeout(connection)
if buf == nil
connection.close
next
end
parse_http_header(…)
--
def read_timeout(conn)
if !IO.select([conn],nil,nil,READ_TIMEOUT)
return nil
end
return connection.sysread(4096)
end
50. Rhebok supports IO timeout
•Implement read_timeout in C
• avoid strange behavior of nonblock +
sysread
• use poll(2) instead of select(2)
$ rackup -s Rhebok -O Timeout=60 config.ru
52. HTTP parser
• HTTP Parser is easy to cause security issue. It's
safer to choose an existing one that is widely used
• There are several fast implementation
• Mongrel based - Unicorn, PUMA
• Node.js based - Passenger 5
• PicoHTTPParser - Rhebok, h2o
• pico_http_parser in rubygems
• Ruby binding of PicoHTTPParser
54. PicoHTTPParser in Rhebok
•uses PicoHTTPParser directly
• does not use pico_http_parser.gem
•performs both of reading and parsing
the HTTP header in a C function
• reduce overhead of create Ruby’s string
contain HTTP header
56. TCP_NODELAY
•When data is written, TCP does not
send packets immediately. There are
some delays.
•TCP uses Nagle’s algorithm to collect
small packets in order to send them all
at once by default
•TCP_NODELAY disable it
59. Problem of TCP_NODELAY
• When TCP_NODEALY is enable, take
care of excessive fragmentation of tcp
packet
• causes increase network latency
• To prevent fragmentation
• concat data in application
• use writev(2)