Presentation from Velocity NYC 2014 on setting up private WebPagetest instances
Video: https://www.youtube.com/playlist?list=PLWa0Ky8nXQTaFXpT_YNvLElTEpHUyaZi4
2. The Basics
● Presentation is available on slideshare
○ http://www.slideshare.net/patrickmeenan
○ Mostly contains links to everything I’ll be showing
● Video will be uploaded to YouTube
○ https://www.youtube.com/user/patmeenan
● I’ll send links and announcements of both to Twitter and Google+
○ https://twitter.com/patmeenan - @patmeenan
○ https://plus.google.com/u/0/+PatrickMeenan/posts
3. WPT Web Server
Sites being tested
System Architecture
Desktop Agent
Node.js Mobile Agent
Mobitest Agent
4. WPT Web Server
Sites being tested
Agent
Data Flow
Submit test (UI or API)
http://<server>/runtest.php
User
5. Server:
- Creates test ID
YYMMDD_<hash>_<num>
- Creates directory for test
results/YY/MM/DD/<hash>/<num>
- Writes test job to work directory for location
work/jobs/<location>/<ID>.<priority>
- Adds test to queue file (lossy, can be re-built from the work directory)
tmp/<location hash>.queue
- Redirects to result page
6. WPT Web Server
Sites being tested
Agent
Poll for test status
http://<server>/result/<ID>/
User
7. WPT Web Server
Sites being tested
Agent
User
Poll for work for location
http://<server>/work/getwork.php
8. WPT Web Server
Sites being tested
Agent
User
- Launch Browser
- Navigate to page
- Measure/Collect Data
9. WPT Web Server
Sites being tested
Agent
User
Upload large files individually
(images, tcpdump, timeline, etc)
http://<server>/work/resultimage.php
10. WPT Web Server
Sites being tested
Agent
User
Complete run
(or full test if all runs are done)
http://<server>/work/workdone.php
11. WPT Web Server
Sites being tested
Agent
View completed test
http://<server>/result/<ID>/
User
12. Server Config
● Any server that can run php
○ Linux tends to work best
○ Can run on the same PC as test agents if needed
● Copy www/ directory from distribution to site docroot
● Configure rewrites for friendly URLs (optional)
○ .htaccess included for apache
○ nginx.conf can be included into nginx server config
● Fix anything that http://<server>/install/ complains about
○ PHP ini settings: allow_url_fopen, memory and post file size limits
○ PHP modules: GD, zip, zlib, curl, apc, sqlite
○ Command-line utilities: ffmpeg, imagemagick, jpegtran, exiftool
○ Filesystem permissions (and directories)
13. Nginx config
server {
listen 80;
server_name webpagetest.example.com;
root /var/www;
location ~ .php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME /var/www$fastcgi_script_name;
fastcgi_param HTTP_MOD_REWRITE On;
include fastcgi_params;
}
include /var/www/nginx.conf;
}
15. Server WPT Config
● Copy/modify settings/*.sample
○ settings/settings.ini
■ Contact information
■ Maximum number of runs
■ Screen shot quality
■ Show Slow integration
■ Test archiving (covered later)
○ settings/locations.ini
■ Definition for all of the test locations
18. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
List of the group IDs in order to be displayed
19. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
“Location ID”s that are part of the group in the
order the browsers are to be displayed.
String displayed for the “Test Location”
20. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
These are the location ID’s the agents connect to
and that are used when submitting tests through
the API
21. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
All of the browsers from all of the locations in a
group are collected and displayed in the Browser
select box.
22. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
All of the agents for a given location ID must
support all of the browsers listed
23. [NYC]
browser=Chrome,Firefox
label=”New York”
[NYC_IE10]
browser=IE 10
label=”New York”
[NYC_IE11]
browser=IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
A single agent can pull from multiple locations
(configure it as a comma-separated list).
location=NYC,NYC_IE10
24. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
lat=40.712918
lng=-74.005469
group=North America
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
Add map=1 to settings.ini
25. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
lat=40.712918
lng=-74.005469
group=North America
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
26. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
When traffic shaping is not supported.
Shows up in the UI as the connectivity information.
30. [locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
Hide the location from the UI (still works through the API
and from http://<server>/?hidden=1 )
31. Desktop Agent Config
● Prepare Windows
○ Auto logon
○ Disable screen savers and screen power management (never sleep)
○ Disable UAC
○ Copy agent/* to local disk (c:/webpagetest/ in the examples)
○ Install dummynet (optional for traffic shaping)
○ Create startup shortcut to wptdriver.exe (and urlblast.exe if used)
● Install browsers
○ Firefox can self-install (necessary for updates)
● Configure wptdriver.ini
○ Server URL
○ Location ID
○ Browsers
33. EC2 Test Agents
● AMI’s available in all regions
○ IE8-11, Chrome, Firefox, Safari 5 (Windows)
● Pass configuration through user data when launching
○ wpt_server=www.webpagetest.org wpt_loc=EC2_East
● No need to ever touch/login to instance
● Kill/Start as needed
● c1.medium or better recommended
36. Sites being tested
WPT Web Server
Mobile Agent (Node)
Node.js Mobile Agent
37. Mobile Agents - Android
● Android 4.4+ recommended
● Root (Nexus and Moto phones are easiest)
● Install Chrome and Chrome Beta from Play store
● Configure Play to auto-update
● Disable screen lock (security settings)
● Configure developer mode
○ Enable USB debugging
○ Disable screen sleep
38. Mobile Agents - iOS (alpha)
● Jailbreak
● install/configure iOS webkit debug proxy
● build openurl ipa
○ Needs to be signed with your dev key
39. Mobile Agents - Tethered host config
● Linux or Windows (Mac may work)
○ Multiple phones per tethered agent
● Node.js
● imagemagick
● Launch wptdriver.bat (or wptdriver.sh)
● Config is passed by command-line
○ Device ID
○ Server URL
○ Location ID
● Windows - consider using adbwatch
○ Kills/restarts adb is it gets hung
○ Launches/restarts agents if they crash
40. Sites being tested
WPT Web Server
Mobile Agent (Node) - Cell Network
Node.js Mobile Agent
41. Sites being tested
WPT Web Server
Mobile Agent (Node) - RNDIS
Node.js Mobile Agent
IPFW
(Optional)
42. WPT Web Server
Sites being tested
Mobile Agent (Node) - WiFi
Node.js Mobile Agent
IPFW Bridge
WiFi Access (Optional)
Point
43. Priority Queues
● Tests are pulled from queues based on test priority
● 10 Priority levels (0-9)
○ 0 = highest, 9 = lowest
● Defaults
○ 0 - Tests submitted through the UI
○ 4 - Bulk tests submitted through the UI
○ 5 - Tests submitted through the API
● Queue lengths reported on location check UI
○ http://<server>/getLocations.php
45. API Keys
settings/keys.ini (disabled if file is missing)
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
description=API Key
contact=notme@example.com
limit=100
46. API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Arbitrary string
used for hashing keys sent to browser
(I usually use GUIDs)
47. API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Server API key
To be used by the Web UI
48. API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Each API key defined as
an ini file section
49. API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Contact info shows up in
usage stats
50. API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Allowed “Page Loads” per day
(tests x runs)
x 2 if also recording repeat view
51. API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Forced test priority
Defaults to 5 for API
(can be specified in request)
54. Server Lock-down
● Use keys for test locations
● Open access to /work/* for test agents
● Auth access (or IP rules) for rest of the UI
55. WPT Web Server
Firewalls...
Private Sites
Intranet
Desktop Agent
Public Desktop
Agent
Public Sites
WPT Relay
Server
56. Relay Server
● Configure locations for public agents
● Configure API keys for internal instance to use for relay
● add headless=1 to settings.ini
○ Disbles UI for submitting tests, requires API key for all tests
● Results are pulled back to internal instance
○ never visible on external
57. Relay Server - locations.ini
● Configure locations.ini like normal on the relay server
● Local server configures 1:1 mapping of locations
○ forwarded to the relay server
[Public_EC2_East]
browser=IE 11,Chrome,Firefox
label="EC2 East (Public)"
relayServer="http://<relay server>/"
relayKey=<your API key>
relayLocation=EC2_East
Config like normal location
API Key and Location ID
on the Relay Server
58. Results Storage
● Each test is self-contained in a directory under results/
● Grouped by year, month and day
○ results/YY/MM/DD/…
● Prune old tests? Delete directories.
59. Results Archiving
● WPT can seamlessly restore missing tests from “archive”
● Supports archiving to local path or S3-like storage
● Configure archive settings in settings.ini
● Test copied to archive after it completes (as 1 zip file per test)
● To delete old archived tests need to run/schedule script
○ php cli/archive.php
● Local Path:
○ zip files copied to specified directory
○ YY/MM/DD/<hash>/<id>.zip
● S3-like
○ zip files uploaded to bucket with <id>.zip filename
○ Can restore from URL or S3 API
60. Results Archiving - Local Path
● Settings.ini
○ archive_dir=/archive/
○ archive_days=2
● Local Path:
○ zip files copied to specified directory
○ YY/MM/DD/<hash>/<id>.zip
● 2nd Level archiving
○ settings.ini: archive2_dir=/mnt/archive/
○ archive zip files merged into daily zip files
○ cli/archive2.php archive script
61. Results Archiving - S3
● Settings.ini
○ archive_s3_server=s3.amazonaws.com
○ archive_s3_key=<access key>
○ archive_s3_secret=<secret>
○ archive_s3_bucket=<bucket>
● zip files uploaded to bucket with <id>.zip filename
● Can use URL access to restore
○ Defaults to using API
○ archive_s3_url=http://s3.amazonaws.com/
63. Bulk Testing in the UI (Private Instances)
● Can be disabled through settings.ini
○ noBulk=1
● Submit list of URLs to test
● UI options apply to each test
● Scripts can be run against each URL
○ %URL% in script gets replaced with test URL
○ %HOST% in script gets replaced with host from test URL
○ %HOSTR% in script gets replaced with host from URL after redirects
● Aggregate results
64. API - Submitting a test
● Anything you can do with the UI you can do with the API
○ If a field isn’t documented, just inspect the HTML on the UI to get the
field name :-)
● Make sure to encode parameters
○ urlencode if GET
○ form encode if POST
66. Node API Wrapper
Created by Marcel Duran (@marcelduran)
https://github.com/marcelduran/webpagetest-api
$ npm install webpagetest -g
webpagetest test -k <key> http://twitter.com/marcelduran
webpagetest status 140610_F0_T5K
webpagetest results 140610_F0_T5K
webpagetest test http://twitter.com/marcelduran --poll 5 --timeout 60
68. CI Integration cont.
Uses Mocha with reporters suitable for integration with:
● Jenkins
● Travis-CI
● Drone.io
● Just about anything else you can imagine
70. Results logging
● As each test comes in, full result data is written to log files
○ Page Data (load time, Speed index, requests, custom metrics, etc)
○ Request Data (details for every individual request)
● Each test run or request is on it’s own log file line
● JSON-formatted
● Config information is included
○ Test URL
○ Run #
○ First/Repeat View
○ Test label
○ Location
○ Browser
○ Connectivity
○ Test ID
○ URL to test result
71. Results logging - integration
● Splunk
● Logster
○ Logster -> StatsD = Arbitrary metrics trending automatically
● Track individual requests across all pages
○ ads JS performance
○ CDN response times
○ Effectiveness of image compression
○ so much more
72. Benchmarks (Private Instances)
● Still pretty experimental
● Recurring tests run automatically
○ Configuration is a bit ugly but flexible
● Tests are scheduled, run and aggregated automatically
○ Top-level trended aggregate view
○ All-metrics trended aggregate view
○ Per-page trended view
○ Scatter Plot for a given run
○ Filmstrip comparisons
73. Monitoring the System
● Hourly Agent Monitoring
● getLocations.php
● getTesters.php
● checkTesters.php
● Custom cron jobs/scripts