your link to better business solutions

How To Figure Out What Parts of Your Website Aren’t Being Crawled

When Google took away the supplemental index last year, they killed one of the key diagnostic tools in the SEO’s toolbox, the ability to identify which parts of a site were unimportant (and being infrequently crawled) in a search engines eyes. However with the use of some structured text and clever searches I’m going to give you back that valuable information.

To tell what pats of a site google (or any search engine for that matter) thinks are important what you need is a way of date tagging when the last time a search engine visited/indexed the page. It’s not important to know that it was crawled 16:42:03 on September 8, 2008, it’s important to know that it’s been over 30/60/90 days since a search engine visited that page. What you need to do is put a month and year time stamp somewhere on every page, I recommend the footer, since proximity isn’t a factor. I’m also going to recommend keeping punctuation and special characters out of the time stamp, in my experience Google gets a bit unpredictable when you introduce those elemnets. I’d suggest you aim for something simple like “Sep 2008″. Next you need to add something that will only appear on your site. The most unique thing will be the site’s proper name (again omit any punctuation or special characters). So you’ll end up with something like “Joes Widget World Sep 2008″.

Once you’ve got that in place the next thing to is wait … at least two full months before you’ll get any good data … yes really. Lets assume you put this change into place today, then on December 1st you’d go to Google (or any other search engine) and type in the following query ["Joes Widget World Sep 2008"] (minus the brackets but with the quotes). The search engine of your choice will then spit out a list of pages with an exact match of the phrase Joes Widget World Sep 2008, or a list of pages that haven’t been crawled since September of 2008, over 60 days days ago … hopefully you just had a lightbulb moment …

Tagged as: , , , , ,

Leave a Response

Please note: comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.