Thursday, September 18, 2008

Googlebot Crawling - an experience

We created a new website. Google crawled the website within a couple of days. It was realy very fast and we don't want the site to be crawled when it is in beta. We have added a robots.txt and added the below tag.
User-agent: *
Disallow: /

Once the site is ready, we have updated the robots.txt with the below tag
User-agent: *
Allow: /

Also we have submitted the page to Google for crawling. (http://www.google.com/addurl.html)
In addition, we have added Google site map and added all the pages into the list. Even after a week of the updates, the website is not yet crawled. Are we missing anything in the process ? or Adding Disallow tag and removing it within couple of weeks will delay the crawling frequency?

We have added the website to our Google Webmaster Tools account and submitted the reconsideration request. We followed the below step.
  • Added the new site to our Google Webmaster tools account.
  • Verified the site ownership.
  • Logged into the Google Webmaster tools account.
  • On the dashboard, under Tools selected the Request for Reconsideration.
  • Selected the site, and explained the above reason for reconsideration.
I hope our new site will be crawled in a couple of days.

No comments: