Faster crawl rate option October 22, 2006 3:44 PM   Subscribe

Google is offering an option for a faster crawl rate
posted by Sharcho to Feature Requests at 3:44 PM (8 comments total)

Is there any guarantee that a faster crawl would make for a deeper index? Or would it be the same lackluster index just done faster? Has anyone tried it on their sites before?
posted by mathowie (staff) at 4:07 PM on October 22, 2006

Most sites don't have a problem a partial crawl index. I don't work at Google, so I can't guarantee anything, but it only takes a minute to set up, so it's worth a try. When you set this up, it can also give a lot of extra information to see why it's not crawling everything.
posted by Sharcho at 4:27 PM on October 22, 2006

Matt, I asked this in the recent thread, but have you ever considered talking directly with Google about why its index of this site is so spotty? There are Googlites here as members, aren't there?
posted by mediareport at 5:09 PM on October 22, 2006

Maybe Matt doesn't want to use up his weekly question on the green asking about google. He needs a sockpuppet.
posted by Rumple at 6:30 PM on October 22, 2006

I know some google folks, but they all work on fun things, not just search. In the few interactions I've had with hardcore search engineers for my old job, they were usually short-tempered jerks that blamed everything wrong on the non-google parties involved.
posted by mathowie (staff) at 8:58 PM on October 22, 2006

Well, in this case it is mostly your fault. Googlebot doesn't think large parts of your site are interesting.

Yahoo's index is comprehensive mostly because it is really enthusiastic about simple crawling. Googlebot is making more value judgments about links. The AJAX crap on the front page doesn't help at all, but at least you didn't use it for the vanilla reverse chronological view.
posted by blasdelf at 9:55 PM on October 22, 2006

the front page is the least important page of the entire site. The archives of 50,000 discussions are where the real action is.
posted by mathowie (staff) at 10:20 PM on October 22, 2006

This is so patently ridiculous because the URLs are fucking sequential. Most sites don't have this problem, but yours does for some reason. How to fix this?

Use Google's Tools

Make an Index of Sitemaps. Because the URLs are so simple, this doesn't have to involve the database at all – unless you want to exclude deleted ones (which I don't think you should, except for the edge cases), or if you want to have finely grained lastmod or changefreq values.

It's also not something you'd particularly need to have updated frequently – most of the value of it is for getting old posts indexed.
posted by blasdelf at 11:30 PM on October 22, 2006

« Older AJAX: bringing functionality -and- problems   |   Please Clarify AskMe Guidelines Newer »

You are not logged in, either login or create an account to post comments