robots exclusion suggestion for deleted threads

February 22, 2006 8:35 AM

Just a followup to the Optical4Less spam and the robots exclusion suggestion for deleted threads: deleted AskMefi threads still do not have the exclusion meta tag, so that Mefi spam there can still be indexed by search engines. Anyone else spotted any other loopholes spammers might have slipped through?
posted by brownpau to Feature Requests at 8:35 AM (7 comments total)

done.
posted by mathowie (staff) at 9:14 AM on February 22, 2006

So do we have to exclude user pages from search engines as well?

Been that way for ages.
posted by mendel at 10:51 AM on February 22, 2006

I kind of have a problem with the user page thing actually, one link per user seems like a very reasonable thing.

My understanding is that user page URLs were blocked because some users were receiving commercial offers to add links in the customizable space. I don't suppose there is any way to allow some links on a page to be indexed, but not others... Perhaps a page generated from the homepage URL field of each user?
posted by Chuckles at 3:13 PM on February 22, 2006

one link per user seems like a very reasonable thing

Some users have multiple, essentially disjoint sites.
posted by cortex at 3:52 PM on February 22, 2006

I kind of have a problem with the user page thing actually, one link per user seems like a very reasonable thing.

Users can (and do) still have links! They're just not indexed by search engines. Why should they be? The point of adding links to profiles is for other users to see them, not to increase pagerank.
posted by mendel at 4:52 PM on February 22, 2006

Exactly.
posted by cortex at 9:13 AM on February 23, 2006

If you remove all the legitimate links, all a search engine is left to index is spam...
posted by Chuckles at 10:43 AM on February 23, 2006