7 posts tagged with indexing.
Displaying 1 through 7 of 7. Subscribe:

Better Google searching for MeFi?

Has Google changed/improved the way it indexes Metafilter? When looking up answers for some recent AskMe questions, I've been surprised to see the questions themselves show up in Google's search results. [more inside]
posted by O9scar on Sep 23, 2007 - 12 comments

How do they do that?

A few hours ago I posted a question at ask.metafilter.com. An hour later nobody's answered so I resume my Googling for an answer. Number three in the list of results is my very question! And it's on a different site! I'm shocked Google updates its index so fast, but, more than that, why is my question on a different site?
posted by deeper red on Jul 18, 2007 - 67 comments

Suggested use of robots.txt for better searching of MeFi and related subdomains

Robots.txt for the <tagname>.metafilter.com domains should exclude all rebots. Currently the robots reindex the entire site for each subdomain.
posted by Sharcho on Jan 15, 2006 - 23 comments

Yahoo vs Google

If it's true that Yahoo! has a more complete index of Metafilter than Google does - see here, here and here (just below) for tantalizing discussion - wouldn't it make more sense to point folks to Yahoo! rather than Google on the site's various search and posting pages? Anyone up for a semi-scientific test?
posted by mediareport on Aug 15, 2005 - 23 comments

AskMe indexing?

Is there a chunk of AskMe that has not been indexed? If not, why can't I find a certain question? (more inside)
posted by gluechunk on Apr 23, 2005 - 3 comments

Userpages closed to indexing?

Revisiting this.

Metafilter is the biggest site that I can edit a page (my user page) on. I'd like to have "I am the [my name] that went to so and so school, and so and so high-school and did so-and-so" so that if someone googled my name looking for me, they could find that. However, userpages are marked NOINDEX for robots, for to stop villians from using google to harvest stuff, I assume. Given that it would be trivial to write a script to go through all the user pages and snarf anything, and given that these are The Internets, is there a reasonable expectation that an email address or other information you put on a publicly viewable page will *not* be harvested? Doesn't everyone have a spam-catcher address for this purpose?
posted by Capn on Jan 23, 2005 - 7 comments

To Index or Not?

The site is getting pummelled lately, so I ran stats on the past few days to see if there was a national news story or something. Of the 300k page views in the past four days, 100k, or 1/3 of the traffic was solely due to the googlebot.

It appears that having 13k threads filled with 200k comments of google-loving ascii is acting as some sort of honeypot, attracting the google indexers like mad. Broken down by day, the Googlebot appears to visit over 25k pages at metafilter.com PER DAY. If you look at browser/OS stats, the googlebot visits metafilter more often than all Netscape clients combined. Also, the googlebot exceeds all visits by people using Mac operating systems.

Although I'm impressed with the results (google searches are the #1 referrer), is it worth basically bringing down the machine and keeping humans from being able to access it? If I were to include a robots exlusion file and block all search bots, would the net community be at a loss for not being able to find information discussed here?

I guess the big question is, does the utility of having the site indexed outweigh the problems the indexing causes?
posted by mathowie on Jan 4, 2002 - 32 comments

Page: 1