If you're new here, you may want to subscribe to my RSS feed 
Today I was doing the regular check the services routine (I’ve been into this since my blog went 404 to Google - there’s a rankings blow - teaches you small things -like a crappy plugin- can screw up your blog…). While looking at analytics reports and then webmaster tools options, I was pleasantly surprised to discover a new option there, one which I’ve been waiting for since I started the blog. The crawl rate option of faster just became available. Or, as Google put it:
We’ve detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn’t use too much of your server’s resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below.
I’ve noticed this as well, Google indexing posts slowly, and the homepage, really slow (about once or twice a month)… To get you up to speed on this, have a look at the “number of pages crawled per day”:
You may not understand it so let me explain. See how every other day there’s a spike in the graph? That means every time the spider came to the site it got some new content. Now this content also increased, so basically it did not only get new content, it also got more of it, all the time. However, looking at the graph below, regarding downloaded size, we see that it has usually take the same amount of data from the site every time it came:
What that basically means is that in order to be able to get this new content, at the same rate, the spider needs to come more often. Easy, right? 
Therefore the faster rate.
Now, one interesting thing is that the crawl rate is only set for 3 months (or 90 days as they put it). I guess it’s a safeguard for them, so that if you quit the blog tomorrow, they won’t waste resources on a dead blog… Also, 3 months is a decent period of time for you to see if the crawl rate really interferes with Google’s ability to index your site properly. If by then you see a distinct change you can reactivate it yourself.
I’ve been searching the web for tips on this crawl rate and how to change it but I haven’t come up with much. Some SEO site is claiming it’s due to external links and page titles and all that but it’s not really connected. Sure, links and pings and all that draw the GoogleBot to your site, but it’s content that makes it want to see content faster, the links just notify it of the presence and I personally feel sitemaps are a lot better at that (Google checks them every few hours and downloads a complete fresh copy of the sitemap every week or so).
Another interesting fact provided by Google is that you can only set crawl rate for root websites (so basically if you have the website at example.com, the photos at photos.example.com and the blog at example.com/blog, you can only change crawl rate for the first two of them). Useful thing to know when you’re thinking of whether to host the blog using a subdomain or a folder. (choose subdomain).
In my tireless quests I have found a nice WordPress plugin (I just love them) which gives you pretty stats regarding crawls by search engine bots. Get WordPress crawl rate tracker here. It tracks Google, Yahoo and MSN, and looks pretty cute. It’s WP2.3 compatible and once activated you can find it under your dashboard.
That went pretty well. You liked it, admit it, now 
get the rss Or 
subscribe by email 
so you don’t miss out on tomorrow’s hot topic (it’s a secret, shhh).
March 18th, 2008 at 3:51 pm
Another good article Eydryan. One thing I wonder about though is how useful updating a website with related news can be if Google only crawls a website ~2 times a month.
For example a website I’m currently working on has no option for “Faster Crawl Rate” - and yet we produce new content three to four times a week. Obviously if this content isn’t crawled then by the time Google gets around to us, its no longer NEW News.
I guess it’s probably time related, we’ve only been writing news for the past 30-40 days. Hopefully with time we’ll get the Faster Crawl option.
Still, it’s great to know it’s only a 90-day flag - have to keep an eye out for that!
March 18th, 2008 at 4:10 pm
thanks for the comment dan, it’s nice to see returning visitors
First of all you assume incorrectly that Google crawls twice a month. Google crawls everyday, every few hours actually, and if it finds new content it keeps coming back. However, seeing as how its resources are limited, it cannot do a sitewide index every day, and depending on the extent, it can differ.
So let me explain this. There are more degrees of depth with GoogleBot. Basically the full siterip happens about four times a year, especially with large sites. That time of the year, Google looks at your entire site and sees what links are dead, which are resurrected, and basically reindexes all the content (or not, if it doesn’t feel like it). Then, there are less comprehensive scans, which take in most of the website (including home page, etc) but lay less impact on little traffic, little change pages, and these happen about twice a month. And then there’s the daily, hourly, etc crawls where it takes links, from the sitemap, or from the site navigation and crawls them for new content. Also, if it encounters some link to say an article on an external site it follows through to yours but only indexes that page… And, if it sees that repeatedly when it comes back to your site, much content has changed, it decides to let you change crawl rate.
so don’t worry, it’s news to google, especially if someone links to you. news are crawled frequently, so that’s why you need to write a lot of new content.
i write posts every day (except weekends). well, i try. and then there’s comments, trackbacks, little modifications i make to the site so google always has new stuff to take every time it visits.
don’t worry about crawl rate, it will appear.
and the best way to monitor crawl stats for your website (the way i do it) is to set a google alert (comprehensive) for that keyword you’re number one (for me it’s eydryan) and set it to send you immediately any news about you. What that does is it tells you when Google has found new content about that keyword. It usually takes me under 15 minutes to get my new posts indexed (but that’s also due to pinging).
I’ll have to discuss this topic these days, hang around and you’ll get more info.
again thanks for the comment dan and i hope i’ve helped you
March 18th, 2008 at 4:16 pm
Hey no problem at all for the comments - I started out as an Applications Programmer and ended up getting involved in Web Design & SEO - As I said previously your blog is helpful for a newcomer to the SEO game
March 18th, 2008 at 4:20 pm
that was fast :))
it’s interesting how people go from one thing to another. just a few years ago i was a photographer shooting interviews and events for a magazine and now look at me
and i do try to be helpful, when i get the time I’ll start doing SEO consulting via skype and there will be some free conferences, at least in the beginning so there will be lots of nice stuff coming in the future (not so near future though…).
April 8th, 2008 at 5:37 pm
[...] Wordpress SEO: First Steps - SEOnoobGoogle secrets: How to speed up Google Crawl Rate [...]
April 12th, 2008 at 7:03 pm
Nice article. What I’ve found to point Google back to crawling your site is what I would call setting off a lot of alerts that your site has a lot of new content and needs to be crawled again. And let me know if you have tried this in combination:
1. Submitting site’s index
2. Registering your site with Google Analytics
3. Bookmarking your site in Google and Yahoo
Something did the trick perhaps it was Google Bookmarks because as soon as I did that Google was back crawling my site the next morning. It might have had something to do with me linking to a site google was going to crawl and then it found my link and crawled my site. Just some thoughts.
April 12th, 2008 at 7:36 pm
@ Anonymous:
The next morning? :)) that’s a bit much… i usually get my crawl alert (and the page itself crawled) within an hour or so after posting. Strange, maybe you’re not posting it in the right places.
As for Google bookmarks, you really shouldn’t bother, from what I know Google doesn’t care about those…
You have been penalized for using a generic name while commenting.