If your site is having trouble ranking well in search engines such as Google, you've lost ranking, or you've having trouble with a site move or migration, the trouble could be with the site's technical architecture. View checklists to help diagnose issues with crawling, indexing, and ranking your site's content.

1. Tools and Tactics for Diagnosing Technical Search Issues Vanessa Fox

4. Search Engine Tools Created by NineByBlue.com Google Webmaster Central http://www.google.com/webmasters Microsoft Live Search Webmaster Center http://webmaster.live.com Yahoo! Site Explorer http://siteexplorer.search.yahoo.com Google Analytics http://www.google.com/analytics Google Search http://www.google.com

5. Ranking and Diagnostic Tools Created by NineByBlue.com SEOBook Rank Checker http://tools.seobook.com/firefox/rank-checker/ Firefox Web Developer Toolbar https://addons.mozilla.org/en-US/firefox/addon/60 Firefox Firebug http://getfirebug.com/ Firefox Live HTTP Headers https://addons.mozilla.org/en-US/firefox/addon/3829 Google Search http://adlab.msn.com/Keyword-Forecast/default.aspx http://janeandrobot.com/resources

6. How Search Engines Work

7. Crawling Discover links Check robots rules Bandwidth considerations URLs Indexing Canonicalization Context extraction Topic association Web-wide value Ranking Relevance Value Uniqueness Display

8. Search Engine Crawlers Haven’t Quite Grown Up Yet

9. Crawling Lack of discovery Crawl inefficiency URL issues (infinite, redirects, dynamic) Inaccessible links Indexing Duplication Extraction issues Lack of exposed content Non-optimized media Ranking Display issues Lack of quality links Guidelines violations Non-focused content

11. Crawling Indexing Ranking Which pages have the search engines crawled? What kind of pages are they? Has the search engine indexed all of the crawled pages? How’s the search engine traffic?

12. Benchmarking Top ten queries that bring search traffic Search results position URL that ranks

14. Crawl Issues

15. Crawl Log Example: Apache Log Analyzer 2 Feed 1 /** 2 * @see ApacheLogAnalyzer2Feed 3 */ 4 require_once 'ApacheLogAnalyzer2Feed.php'; 5 6 // create a new instance, parse access. log and 7 // write test.xml 8 $tool = new ApacheLogAnalyzer2Feed('access. log ', 9 'test.xml'); 10 // select entries matching Googlebot useragent 11 $tool->addFilter('User-Agent', 'Mozilla/5.0 12 (compatible; Googlebot/2.1; 13 +http://www.google.com/bot.html)'); 14 // run 15 $tool->run(); 12 http://code.simonecarletti.com/wiki/apachelog2feed 1 /** 2 * @see ApacheLogAnalyzer2Feed 3 */ 4 require_once 'ApacheLogAnalyzer2Feed.php'; 5 6 // create a new instance, parse access. log and write test.xml 7 $tool = new ApacheLogAnalyzer2Feed('access. log ', 'test.xml'); 8 // select entries matching Googlebot useragent with a regular 9 expression pattern 10 $tool->addFilter('User-Agent', 'regexp:Googlebot'); 11 // select entries with Request matching a regular expression 12 // pattern 13 $tool->addFilter('Request', 'regexp:/site/profilephp'); 14 // run 15 $tool->run(); 16 All Pages Google’s Crawled All Profile Pages Google’s Crawled

16. Communicating with Search Robots

17. Extractable Link Issues: Flash

18. Extractable Link Issues: Images

19. Extractable Link Issues: AJAX

20. Extractable Link Issues: URL Errors

21. Extractable Link Issues: URLs That Expire

26. Indexing Issues

27. Indexing Example: XML Sitemaps http://sitemaps.org XML Sitemap <?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> <urlset xmlns=&quot;http://www.sitemaps.org/schemas/sitemap/0.9&quot;> <url> <loc>http://www.example.com/</loc> </url> <url> <loc>http://www.example.com/page1.php</loc> </url> <url> <loc>http://www.example.com/page2.php</loc> </url> </urlset> http://www.google.com/webmasters Pages Indexed From Sitemap

28. Duplicate Content Issues

29. Partner Content http://www.google.co.uk/search?q=%22The+Radisson+Edwardian+Vanderbilt+Hotel+stands+among+a+row+of+Victorian+townhouses+located+in+the+fashionable+Kensington+district+of+London,+England%22&hs=cN0&filter=0

34. Ranking Issues

35. How’s the Search Engine Traffic? Overall Percentage Percentage Non-Branded

36. Do You Rank For the Right Things? arbor snowboards snowboard Google 1 49 500+ Yahoo 1 80 500+ Live Search 3 128 500+

37. If ranking loss… Drop For All Keywords Does the site rank for different queries than before? Did you substantially change the site content? Did you change the underlying site infrastructure? Was there a large change in linking behavior? Could there be a penalty? Drop For Only Some Keywords Do different pages rank highest than used to rank before? Are the pages that used to rank still indexed?

39. The Webmaster Guidelines http://google.com/support/webmasters/bin/answer.py?answer=35769 Common Definition of Spam On page schemes Keyword stuffing Fake/ stolen content Hidden text Hidden links Cloaking Linking schemes Paid Links Link exchanges Doorway pages Deceptive redirects

41. Traffic Issues

42. Traffic Drop

43. Display Issues

44. Would you click this link?

45. Does the Result Inspire Clicks?

46. First step in diagnosis: find the root

47. Ninebyblue.com Twitter.com/vanessafox Jane and Robot Developer Summit June 12 th , 2009 – San Francisco FREE for SMX attendees! janeandrobot.com Twitter.com/janeandrobot