1. VARNISH-CACHE *
Lessons Learnt from Performance Testing
* https://www.varnish-cache.org
Tuesday, 25 October 11

2. WHY? FAST!
• If your throughput is currently measured in the hundreds per
second, think of varnish in the thousands per second.
• Theoretical limits in hundreds of thousands per second.*
* http://kristianlyng.wordpress.com/2010/10/23/275k-req/
Tuesday, 25 October 11

3. GET OUT OF JAIL. FREE.
• Crappy cache control?
• Expensive CDNs? 20¢/GB?
• Not enough hardware?
• Backend struggling?
Tuesday, 25 October 11

4. IT WON’T
• Do HTTPS... terminate SSL with something else, hopefully
your load balancers.
Tuesday, 25 October 11

5. INSTALL THE LATEST
• RHEL / CentOS
rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-
release-3.0-1.noarch.rpm
yum install -y varnish
• Debian
apt-get install varnish
Tuesday, 25 October 11

6. USE 64BIT*
“It’s all about virtual memory: Things like
stack size suddenly matter on 32bit. If you
must use Varnish on 32-bit, you’re somewhat
on your own. However, try to fit it within
2GB. I wouldn’t recommend a cache larger
than 1GB, and no more than a few hundred
threads… (Why are you on 32bit again?)”
* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/
Tuesday, 25 October 11

7. SYSTEM CONFIG
• The minimum.
echo 'fs.file-max=1000000' >> /etc/sysctl.conf
echo 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.conf
sysctl -p
echo '* soft nofile 1000000' >> /etc/security/limits.conf
echo '* hard nofile 1000000' >> /etc/security/limits.conf
echo 'session required pam_limits.so' >> /etc/pam.d/login
ulimit -n 1000000
ulimit -n -H
• Do not set “tw_reuse” to 1 (sysctl). It will cause problems for
NAT based clients.
• Enable keepalives on load balancers.
Tuesday, 25 October 11

8. THREADS
• Adjust default thread* settings in /etc/sysconfig/varnish
thread_pool_add_delay=2
thread_pools = <Number of cpu cores>
thread_pool_min = <800/number of cpu cores>
thread_pool_max = 4000
session_linger = 50
sess_workspace = <16k to 5m>
“The only thing that made a real difference while tuning Varnish
was the number of threads. And making sure it actually caches.
Beyond that, it really doesn’t matter much. Our defaults are
good.”
* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/
Tuesday, 25 October 11

9. STORAGE
• Memory is fast, duh.
• Got enough RAM to store the cache?
Yes! then use -s malloc
Unsure? then use -s file
Tuesday, 25 October 11

10. VCL
• Varnish Configuration Language
• Looks like C and compiled into C but ...
~ “has regex tilda operator”
Assignments use set keyword.
Only has conditional if tests.
Has a bunch of variables and some built in functions.
• Defined by backends, probes, ACLs and subroutines which
hook into the varnish workflow.
Tuesday, 25 October 11

11. VCL
• Default VCL is defined in
/etc/varnish/default.vcl
• Then create your custom VCL in
-f /var/custom.vcl
so varnish can fall through to the default.vcl
Can also include "/var/other.vcl";
• Keep it simple within the VCL workflow*
Do most of the work in vcl_recv and vcl_fetch
* https://www.varnish-cache.org/trac/wiki/VCLExampleDefault
Tuesday, 25 October 11

12. LOOKUP AND MISS
• request received from client recv
• hash request hash pipe
• lookup object and miss hit miss pass
• fetch response from backend fetch
• deliver to client deliver
Tuesday, 25 October 11

13. LOOKUP AND HIT
• request received from client recv
• hash request hash pipe
• lookup object and hit hit miss pass
fetch
• deliver to client deliver
Tuesday, 25 October 11

14. PASS AND DELIVER
• request received from client recv
hash pipe
• pass through to backend hit miss pass
• fetch response fetch
• deliver to client deliver
Tuesday, 25 October 11

15. PIPE
• request received from client recv
• pipe to backend until client or hash pipe
backend closes
hit miss pass
fetch
deliver
Tuesday, 25 October 11

16. PROBES
• Define common health probe health {
.url = "/healthCheck?onFailure=true";
checks for multiple .expected_response = 200;
.interval = 15s;
backends. .timeout = 5s;
.window = 4;
.threshold = 4;
}
• window is how many of the
latest polls are probed for
health.
• threshold is how many
polls must succeed to be
healthy.
Tuesday, 25 October 11

17. BACKENDS
• Define backend host, port
and health checks inline.
{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }
• Or named.
backend mysite {
.host = "www.example.com";
.port = "http";
.probe = "health";
}
Tuesday, 25 October 11

18. DIRECTORS
• Group backends together using different algorithms.
Random
Client (based on identity)
Hash (based on url)
Round robin
Fallback
Tuesday, 25 October 11

19. YMMV
•I found it better to balance
in front of Varnish using a
hardware load balancer ...
• Random seemed a little too
random using Varnish
directors however was
probably an artefact of the
load generation.
Tuesday, 25 October 11

20. DESIGN DECISIONS
• Share nothing.
Easy to scale horizontally.
Great if your site is not ‘transactional’.
Backend session affinity via LB set cookie.
LB Set-Cookie=BackendID
varnish varnish varnish
app app app
Tuesday, 25 October 11

21. DESIGN DECISIONS
• Share everything.
Varnish loves RAM, scale vertically.
Varnish cache becomes the shared source of truth.
LB
varnish
app app app
Tuesday, 25 October 11

22. SHARE NOTHING. FALLBACK.
director varnishA fallback {
{ .backend = { .host = "a"; } } LB Set-Cookie=VarnishID
{ .backend = { .host = "b"; } }
{ .backend = { .host = "c"; } }
}
director varnishB fallback {
varnish varnish varnish
{ .backend = { .host = "b"; } }
{ .backend = { .host = "a"; } }
{ .backend = { .host = "c"; } }
}
director varnishC fallback {
{ .backend = { .host = "c"; } }
{ .backend = { .host = "b"; } }
app app app
{ .backend = { .host = "a"; } }
}
Tuesday, 25 October 11

23. TERMINATE SSL.
• Get the LB to set a Port header.
Useful when user switches from HTTP to HTTPS so varnish
knows which backend port to route to.
LB Set-Cookie=VarnishID
Port=https
varnish app
:80
:443
if (req.http.Port ~ "https") {
set req.backend = appa_80;
} else {
set req.backend = appa_443;
}
Tuesday, 25 October 11

24. VCL TRICKS
• Rewrite headers to/from production.com.au to
testdomain.com
• vcl_recv
set req.http.Host = regsuball(req.http.Host, "testdomain.com", "com.au");
set req.http.Referer = regsuball(req.http.Host, "testdomain.com", "com.au");
• vcl_deliver
set resp.http.Host = regsuball(resp.http.Host, ".com.au", ".testdomain.com");
set resp.http.Referer = regsuball(resp.http.Host, ".com.au", ".testdomain.com");
set resp.http.Location = regsuball(resp.http.Location, ".com.au", ".testdomain.com");
set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");
Tuesday, 25 October 11

25. VCL TRICKS
• Add some debug headers for hit/miss in firebug in
vcl_deliver
if (obj.hits > 0) {
set resp.http.X-Varnish-Hit = "++";
set resp.http.X-Varnish-Hits = obj.hits;
set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1");
} else {
set resp.http.X-Varnish-Hit = "--";
set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1");
}
Tuesday, 25 October 11

26. VCL TRICKS
• Normalize the Accept-Encoding headers to reduce cache
usage in vcl_recv
if (req.http.Accept-Encoding) {
if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
remove req.http.Accept-Encoding;
}
}
Tuesday, 25 October 11

27. VCL TRICKS
• Remove Google Analytics cookies in vcl_recv
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;s*)(__[a-z]+|has_js)=[^;]*", "");
set req.http.Cookie = regsub(req.http.Cookie, "^;s*", "");
• Remember, varnish will pass any request with cookies by
default. You will need to over-ride this behaviour if your web
app uses cookies.
• Allow lookup of requests with basic auth (in test) or cookies
in vcl_recv but beware of creating a per-user-session store.
if (req.http.Authorization || req.http.Cookie) { return (lookup); }
Tuesday, 25 October 11

28. VCL TRICKS
• Pass on the client IP to the backend servers using the X-
Forwarded-For header in vcl_recv
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
Tuesday, 25 October 11

29. VCL TRICKS
• Let varnish keep objects for 2 hours beyond their TTL in
vcl_fetch
set beresp.grace = 2h;
• Let varnish serve stale objects up to 30 secs old for healthy
servers, otherwise 2 hours in vcl_recv
if (! req.backend.healthy) {
set req.grace = 2h;
} else {
set req.grace = 30s;
}
• Can also use saint mode to disable backends for a period of
time, otherwise if all backends unavailable, use grace.
Tuesday, 25 October 11

30. VCL TRICKS
• Force a grace on error. Restart keyword can be called any
where and will return request all the way back to vcl_recv
sub vcl_error {
if (req.restarts == 0) {
set req.http.X-Serve-Graced = "1";
return(restart);
}
}
Tuesday, 25 October 11

31. TTL
• Varnish will obey cache control headers from backend
Cache-Control: s-maxage
Cache-Control: max-age
• Can also override TTL control in VCL, but best do it from the
backend
set obj.ttl = x
Tuesday, 25 October 11

32. REDUCING CACHE SIZE
• Normalize / sanitize request headers
• Set Content-Length on backends
• Serve (static) digital assets from VFS cache
• Pipe large objects like videos
Tuesday, 25 October 11

33. MONITORING VARNISH
• Handy command line tools
varnishstat
varnishlog
varnishtop
varnishadm
Tuesday, 25 October 11

34. VARNISHSTAT
• varnishstat
Hit rate % (x100)
Average per second
Average/sec since boot
Sum total
• Filtered
varnishstat -f
client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail
Tuesday, 25 October 11

35. VARNISHSTAT
• Log all statistics to disk ...
#!/bin/bash
logdir=/var/log/perf/`hostname`/latest
mkdir -p $logdir
while true
do
varnishstat -x >> "$logdir/varnishstat.xml"
sleep 60
done
• Then graph as you see fit ...
Tuesday, 25 October 11

36. VARNISHLOG
• varnishlog debugging (-d to dump shared memory)
12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80
12 TxRequest b GET
12 TxURL b /
12 TxProtocol b HTTP/1.1
12 TxHeader b Host:
12 TxHeader b Referer:
12 TxHeader b Cookie:
12 TxHeader b X-Forwarded-For: 1.2.3.4
12 TxHeader b X-Varnish: 2092287889
12 TxHeader b Accept-Encoding: gzip
12 RxProtocol b HTTP/1.1
12 RxStatus b 401
12 RxResponse b Authorization Required
12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT
12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only"
12 RxHeader b Vary: Accept-Encoding
12 RxHeader b Content-Encoding: gzip
12 RxHeader b Content-Length: 336
12 RxHeader b Connection: close
12 RxHeader b Content-Type: text/html; charset=iso-8859-1
12 Fetch_Body b 4 0 1
12 Length b 336
12 BackendClose b xx_80[0]
Tuesday, 25 October 11

37. VARNISHTOP
• Monitor top URLs hitting backend
varnishtop -i txurl
• Monitor top received URLs
varnishtop -i rxurl
Tuesday, 25 October 11

38. VARNISHADM
• varnishadm for administration
# varnishadm
CLI connected to 127.0.0.1 6082
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit
Type 'help' for command list.
Type 'quit' to close CLI session.
varnish> help
200
help [command]
ping [timestamp]
auth response
quit
banner
status
start
stop
vcl.load <configname> <filename>
vcl.inline <configname> <quoted_VCLstring>
vcl.use <configname>
vcl.discard <configname>
vcl.list
Tuesday, 25 October 11

39. MORE HELP?
• Talk to the devs, read their blogs.
http://kristianlyng.wordpress.com
http://ingvar.blog.redpill-linpro.com/
• Get some online training.
https://www.varnish-software.com/products-services/training/
varnish-administration-course
• Consider the Varnish ‘Moral’ License.
http://phk.freebsd.dk/VML/
Tuesday, 25 October 11