RSS feeds in Google searches

August 1, 2007 1:39 AM

The new RSS feeds (which are wonderful) are starting to show up in Google searches. I doubt this is what anyone wants. I would suggest that they be disallowed in robots.txt, or set NOINDEX, NOFOLLOW.
posted by grouse to bugs at 1:39 AM (14 comments total)

Double.
posted by WCityMike at 5:25 AM on August 1, 2007

Bad robot.
posted by liquorice at 5:34 AM on August 1, 2007

Now that there are a lot more feeds, it comes up a lot more.
posted by grouse at 6:08 AM on August 1, 2007

*nods furiously*
posted by Plutor at 6:24 AM on August 1, 2007

Indeed, the results are a real mess.
posted by Alvy Ampersand at 6:42 AM on August 1, 2007

Another option would be if the referral is a google search, it automatically switches to normal format. Or is that a bad thing?
posted by Deathalicious at 6:51 AM on August 1, 2007

Yes, it still screws up the results to have twice as many pages. And that kind of referrer sniffing doesn't seem like a good idea.
posted by grouse at 7:04 AM on August 1, 2007

From the other thread, I think it could be solved with a:

User-agent: Googlebot
Disallow: /*rss$

in the robots.txt file. Although robots.txt doesn't allow wildcards, Googlebot does.
posted by WCityMike at 8:04 AM on August 1, 2007

I'll try the RSS wildcard thing now...
posted by mathowie at 8:15 AM on August 1, 2007

ok, all robots.txt files have been updated. Let's see if that does the trick.
posted by mathowie at 8:19 AM on August 1, 2007

While we're at it, can we block off favorites pages? I hate it when that shit gets in the way when I'm a-hunting.
posted by cortex at 8:19 AM on August 1, 2007

Until MeFi RSS is out of Google's index, you could try sharpening your queries a bit. Try adding -inurl rss or -inurl favorites to exclude some pages.
posted by pb at 8:41 AM on August 1, 2007

And, you can create an alias in Dave's Quick Search Deskbar to insert the search refining strings automatically..
I love pimping for Dave. Unfortunately, my Aliases are completely out of date.
posted by Chuckles at 9:00 AM on August 1, 2007

Why not just add the parameters to the search form on the page? There's nothing you can do about stuff that's already been indexed except wait (and wait, and wait). But for the users of MeFi, it would be nice if the search box had the exclusion parameters hard-coded in hidden fields on the search form.
posted by Civil_Disobedient at 5:23 AM on August 2, 2007