Chris Jackel Systems Engineer, Fastly at Altitude 2016 A/B testing is valuable, aggravating, and everyone has an opinion on the best way to do it, with a strong belief that all other methods are flawed. At Fastly, we want to provide a set of tools to help you with experiments you can integrate into your own analytics. This workshop covers: *Randomization with Fastly *Persistent segments with and without cookies *The power and limitations of HTTP vary *Using Edge Dictionaries for updating tests programmatically *Integration with external tools If your page traffic is delivered directly by Fastly and you add information to the real-time log stream, there may be no need for third-party trackers. Otherwise, you can easily add the custom variables to existing cookies. These techniques aren’t designed to replace your existing tools, but may give you new options to build something exciting.

1. A/B testing at the edge
Chris Jackel | Systems Engineer, Fastly

3. Complexity

4. Too
Basic
Complexity

5. Too
Basic
Too
Code-y
Complexity

6. Too
Basic
Complexity
Sweet spot for
maximal dissatisfaction
Too
Code-y

7. Why, How, and More How
• Why do A/B testing?
• How to implement with Fastly
• Non-traditional uses

8. Why
• Science!

9. https://blog.optimizely.com/2010/11/29/

10. What We Need
• ‘A’ and a ‘B’

11. What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation

12. What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)

13. What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)

14. Web ServerClients
GET /
Fastly

15. Segmentation
sub vcl_recv {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
}

16. Segmentation
sub vcl_recv {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
}

17. Web ServerClients Fastly
GET /
X-FastAB: A
(95%/5%)

18. Web ServerClients Fastly
200 OK
(version ‘A’)

19. Web ServerClients Fastly
200 OK
(version ‘A’)

20. Segmentation
sub vcl_deliver {
add resp.http.Set-Cookie = "FastAB="
req.http.X-FastAB ”; expires=Thu, 20-Jul-17
19:47:08;”;
return(deliver);
}

21. Segmentation
sub vcl_deliver {
if (!req.http.Cookie:FastAB){
add resp.http.Set-Cookie = "FastAB="
req.http.X-FastAB ”; expires=Thu, 20-Jul-17
19:47:08;”;
}
return(deliver);
}

22. Cookie?
No Yes
Segment A/B
Set Header to Server
Set Cookie on Response
Set Header to Server = Cookie

23. Segmentation
sub vcl_recv {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
}

24. Segmentation
sub vcl_recv {
if (!req.http.Cookie:FastAB) {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
}
}

25. Segmentation
sub vcl_recv {
if (!req.http.Cookie:FastAB) {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
} else {
set req.http.X-FastAB =
req.http.Cookie:FastAB;
}
}

26. What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)

27. HTTP Vary

35. HTTP Vary
- The chef (or server) tells us what is important information (e.g. flavor)
- The customer (or browser) tells us the flavor
- Max of 200 variations!
- Otherwise change the hash
- Don’t vary on too big a set (e.g. user-agent)

36. Get /lamb-curry.html
Host: www.example.com
X-Flavor: Mild

37. Get /lamb-curry.html
Host: www.example.com
X-Flavor: Mild

38. 200 OK

39. Get /lamb-curry.html
Host: www.example.com
X-Flavor: Spicy

40. 200 OK
(mild)
WTF?

41. Get /lamb-curry.html
Host: www.example.com
X-Flavor: Mild

42. 200 OK
Vary: X-Flavor

43. HTTP Vary
if (beresp.http.Vary) {
set beresp.http.Vary = beresp.http.Vary ", X-FastAB";
} else {
set beresp.http.Vary = "X-FastAB";
}

44. HTTP Vary
if (beresp.http.Vary) {
set beresp.http.Vary = beresp.http.Vary ", X-FastAB";
} else {
set beresp.http.Vary = "X-FastAB";
}
# e.g. Vary: Accept-Encoding, X-FastAB

45. HTTP Vary
• What is a hash?

46. HTTP Vary
• What is a hash?
{
set req.hash += req.url;
set req.hash += req.http.host;
set req.hash += "#####GENERATION#####";
return (hash);
}

47. Other Topics
• Backend uses
• Updating weights with Edge Dictionaries
• Cookie-less options

48. Backend AB
- DR Testing

49. Backend AB
if (randombool(5,100)) {
set req.backend = backend_b;
} else {
set req.backend = backend_a;
}

50. Backend AB
- DR Testing
- CMS Migration

51. Backend AB
sub vcl_recv {
if (randombool(5,100)) {
set req.url = “/v2“ req.url;
}
}

52. Edge Dictionary
Change weights via API

53. Edge Dictionary
Change weights via API
table page_segments {
"/" : “5”, # percentage ‘b’
”/page1" : "30",
"/page2" : “40",
}

54. Edge Dictionary
Change weights via API
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}

55. Edge Dictionary
set req.http.X-Weight = table.lookup(page_segments,
req.url.path);
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}

56. Edge Dictionary
set req.http.X-Weight = table.lookup(page_segments,
req.url.path);
if (randombool(req.http.X-Weight,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}

57. Edge Dictionary
set req.http.X-Weight = table.lookup(page_segments,
req.url.path);
if (randombool(std.atoi(req.http.X-
Weight),100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}

58. Cookie-less
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}

59. Cookie-less
if (randombool_seeded(5,100,seed))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}

60. Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip
req.http.User-Agent)
# e.g. 3aaf91f17e7c63b07f8490ab242f9335
if (randombool_seeded(5,100,seed))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}

61. Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip
req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)
# e.g. 9223372036854775807
if (randombool_seeded(5,100,seed))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}

62. Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip
req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)
# e.g. 9223372036854775807
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}

63. Cookie-less
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}
Still the seed

64. Cookie-less
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}
Still the seed

65. The data must flow

66. Thanks!