In the SEO industry, we obsess on everything Google says, from John Mueller dropping a hint in a Webmaster Hangout, to the ranking data we spend £1000s to gather. Yet we ignore the data Google throws at us every day, the crawling data. For the longest time, site crawls, traffic data, and rankings have been the pillars of SEO data gathering. Log files should join them as something everyone is doing. We'll go through how to get everything set-up, look at some of the tools to make it easy and repeatable and go through the kinds of analysis you can do to get insights from the data.

1. 2009

4. God it’s bad.

16. -$1.5 Billion

18. Why hasn’t Google seen the changes on my page?

19. How should I prioritise errors in Search Console?

20. Are my canonicals being respected?

21. Does Google think this page is important?

28. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

31. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
IP Address

32. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Timestamp

33. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Request type

34. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Homepage

35. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Protocol

36. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Status Code

37. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Size of the page (in bytes)

38. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html))"
User Agent

39. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

40. 5 things
2 3 4 51

41. 1 Diagnose crawling &
indexation issues
2 3 4 51

44. Number of
requests
Five folders Googlebot crawled the most

45. Five folders Googlebot crawled the most
Number of
requests

46. % of Organic sessions VS % of crawl budget
Sessions Crawl budget

47. 2 Prioritisation
2 3 4 51

49. example.com/article

50. Prioritizing
1
Full Print

51. example.com/article/full

52. example.com/article/print

53. Prioritizing
2

54. example.com/article/pdf

55. Prioritizing
3

56. Prioritizing
1
Full Print

57. 3 Spot bugs &
view site health
2 3 4 51

58. Delayed errors with a limit of 1000

60. 4 How important does Google
see parts of your site?
2 3 4 51

61. My SEO was as bad as my design

62. But at least my hair was better

63. teflsearch.com

64. teflsearch.com/job-results

65. teflsearch.com/job-results/country/china

66. teflsearch.com/jobadvert3455

67. Average number of times Googlebot crawled a template

68. 1. teflsearch.com
2. teflsearch.com/job-results
3. teflsearch.com/job-results/country/china
4. teflsearch.com/job-advert3455

69. 1. teflsearch.com
2. teflsearch.com/job-results
3. teflsearch.com/job-results/country/china
4. teflsearch.com/job-advert3455

70. teflsearch.com/job-results

71. Average number of times Googlebot crawled a template
35%

72. 5 How fresh does it think your
content is?
2 3 4 51

73. bit.ly/moz-fresh

74. Average number of times a page template is crawled by
Googlebot

75. ●Improve our internal linking
●Build trust with last modified date in
sitemap

76. 2 3 4 51

77. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

81. Talk to a developer
and ask for
information

82. Are all the logs in one place?

83. Hi x
I’m {x} from {y} and we’ve been asked to do some log analysis to understand better how Google is behaving on the website and I was hoping you could help with some questions about
the log set-up (as well as with getting the logs!).
What we’d ideally like is 3-6 months of historical logs for the website. Our goal is look at all the different pages search engines are crawling on our website, discover where they’re
spending their time, the status code errors they’re finding etc.
There are also some things that are really helpful for us to know when getting logs.
Do the logs have any personal informationin?
We’re just concerned about the various search crawler bots like Google and Bing, we don’t need any logs from users, so any logs with emails, or telephone numbers etc. can be
removed.
Do you have any sort of caching which would create separate sets of logs?
If there is anything like Varnish running on the server, or a CDN which might create logs in different location to the rest of your server? If so then we will need those logs as well as just
those from the server. (Although we’re only concerned about a CDN if it’s caching pages, or serving from the same hostname; if you’re just using Cloudflare for example to cache
external images then we don’t need it).
Are there any sub parts of your site which log to a different place?
Have you got anything like an embedded Wordpress blog which logs to a different location? If so then we’ll need those logs as well.
Do you log hostname?
It’s really useful for us to be able to see hostname in the logs. By default a lot of common server logging set-ups don’t log hostname, so if it’s not turned on, then it would be very useful
to have that turned on now for any future analysis.
Is there anything else we should know?
Best,
{x}
Email for a developer

84. So we might have something that looks like this

85. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

89. BigQuery

91. BigQuery

92. Google’s online database for
data analysis.

93. 1. Ask powerful questions
2. Repeatable
3. Scaleable
4. Combine with crawl data
5. Easy to set-up
6. Easy to learn
What do we want from analysing our logs?

99. 9,000,000 rows of data for 2
months.
400 - 800 queries

101. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

102. Format the logs so we can import them into
BigQuery
Separate the Googlebot logs from all the
other logs

103. Screaming Frog Log
Analyser
Code something

104. Screaming Frog Log Analyser

106. Code something

107. bit.ly/logs-code

108. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

109. Our data in BQ

110. We make sure we
got what we wanted

111. THE QUESTION:
What is the total number of requests
Googlebot makes each day to our site?

112. Our first SQL query
SELECT
timestamp
FROM
[mydata.log_analysis]

113. Our first SQL query
SELECT
timestamp
FROM
[mydata.log_analysis]

114. Our first SQL query
SELECT
DATE(timestamp)
FROM
[mydata.log_analysis]

115. Our first SQL query
SELECT
DATE(timestamp)
FROM
[mydata.log_analysis]

116. Our first SQL query
SELECT
DATE(timestamp) as date
FROM
[mydata.log_analysis]

117. Our first SQL query
SELECT
DATE(timestamp) as date
FROM
[mydata.log_analysis]

118. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*)
FROM
[mydata.log_analysis]

119. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*)
FROM
[mydata.log_analysis]
GROUP BY
date

120. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*) as number_of_requests
FROM
[mydata.log_analysis]
GROUP BY
date

121. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*) as number_of_requests
FROM
[mydata.log_analysis]
GROUP BY
date

122. Comparing logs to GSC crawl volume
Number of
requests

123. Run queries
Find something weird
Go look at crawl & website

124. Our data in BQ

125. 1 Diagnose crawling &
indexation issues

126. 2 Prioritisation

127. 3 Spot bugs &
view site health

128. 4 How important does Google
see parts of your site?

129. 5 How fresh does it think
your content is?

130. 1 Diagnose crawling &
indexation issues
4 How important does Google
see parts of your site?

131. What are the top 20 URLs crawled by
Google over our logs?

132. Login is my top crawled page and then search?

133. What are the top 20 page_path_1 folders
crawled by Google over our logs?

134. Location folders are taking more than 70% of my budget

135. Getting data by the day
Page Number of Googlebot Requests
page1 200,000
page2 120,000

136. Number of Googlebot requests day by day

137. 3 Spot bugs &
view site health

138. How many of each status code does
Google find per day over our logs?

139. Number of Googlebot requests day by day

140. What are most requested 404 URLs by
Googlebot over the past 30 days?

141. Boy does it want that ad-tech snippet

142. 5 How fresh does it think your
content is?

143. How many times on average is each page
in a page template crawled a day?

144. Average number of times a page template is crawled by
Googlebot

145. How long does it take for a page to be discovered after being published?

146. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?

147. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?

148. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?

149. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?

150. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?

151. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?

152. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?

153. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?
Which pages are only partly downloaded?
How many hits does each section get, when the sections are classified in an
external dataset?

154. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?
Which pages are only partly downloaded?
How many hits does each section get, when the sections are classified in an
external dataset?
What percentage of a directory was crawled over the past 30 days?

155. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?
Which pages are only partly downloaded?
How many hits does each section get, when the sections are classified in an
external dataset?
What percentage of a directory was crawled over the past 30 days?
What are the total number of requests across two different time periods?

156. That’s a lot of questions

157. bit.ly/logs-resource

158. bit.ly/logs-resource

159. bit.ly/logs-resource

160. bit.ly/logs-resource

161. In Summary

162. This is the thing you’re probably not doing

166. bit.ly/logs-resource
@dom_woodman

168. bit.ly/logs-resource
@dom_woodman