DNS wildcard and google crawler

January 6, 2006 11:34 AM

It's possible that DNS wildcards on metafilter.com are causing problems for the Google crawler. I suggest deleting the redundant DNS entries, and using a 301 redirect for http://metafilter.com/ to go to http://www.metafilter.com/
posted by Sharcho to Feature Requests at 11:34 AM (18 comments total)

you sure it's not just cooke's use of the blink tag?
posted by shmegegge at 11:49 AM on January 6, 2006

I only see one entry for metafilter.com. There is an A record that points to 70.86.84.162

www.metafilter.com has a cname record that points to metafilter.com

When I go to metafilter.com it redirects with a 302 status code. Since 302 is temporary redirect, I agree that it should probably be changed to a 301, which is a permanent redirect.

What are you seeing as issues with DNS though?
posted by afflatus at 11:49 AM on January 6, 2006

e.g. http://blahblahblah.metafilter.com/
posted by Sharcho at 11:50 AM on January 6, 2006

You're right, for metafilter.com it does return 302 (it should be 301 permanent direct), but for the other subdomains it doesn't even do that.
posted by Sharcho at 11:56 AM on January 6, 2006

Ah, now I see what you mean. I thought you were talking about roundrobin dns or some such. I agree, 3rd level domains that don't exist probably should just return a regular dns not found error.
posted by afflatus at 12:07 PM on January 6, 2006

afflatus: "Ah, now I see what you mean. I thought you were talking about roundrobin dns or some such. I agree, 3rd level domains that don't exist probably should just return a regular dns not found error."

They're there for a reason, though: to help people at less-enlightened employers get around simple proxy rules.
posted by Plutor at 12:10 PM on January 6, 2006

metatalk.google.com

What exactly does it break?
posted by mzurer at 12:29 PM on January 6, 2006

One idea: Matt could make it so that it checks the current URL and tells the robot to exclude the page if it's not under metafilter.com. That would probably be a one-liner.
posted by smackfu at 1:25 PM on January 6, 2006

It's done this way so that people who work for companies who block "www.metafilter.com" or "metafilter.com", but not "*.metafilter.com". can still browse the site.

This was done intentionally.
posted by delmoi at 1:40 PM on January 6, 2006

Actually, I don't know of anyone still successfully getting around any workplace rules with this in place.

Every corp network I've seen blocked did the whole *.metafilter.com domain.

I'll change the redirect status and make an A record for the www, and I have a cool idea what to do with all the subdomains to make them useful.
posted by mathowie (staff) at 2:20 PM on January 6, 2006

and I have a cool idea what to do with all the subdomains to make them useful

Unfortunately, this margin is not large enough to contain it...

I mean, c'mon! This shit keeps me up nights! Dish!
posted by cortex at 2:54 PM on January 6, 2006

I'll change the redirect status and make an A record for the www, and I have a cool idea what to do with all the subdomains to make them useful.

Probably username.metafilter.com.

/registers the metatalk sockpupet.
posted by delmoi at 4:01 PM on January 6, 2006

I bet {tagname}.metafilter.com would be pretty useful.
posted by todbot at 4:31 PM on January 6, 2006

As far as google goes, with the billions they pour into their search engine it should be smart enough to know what a wildcard domain is. It's not up to us (well, Matt) to fix google.
posted by whir at 2:02 AM on January 8, 2006

is, for example, http://politics.bush.metafilter.com too much to ask? :-)
posted by null terminated at 5:11 PM on January 9, 2006

Matt said "Actually, I don't know of anyone still successfully getting around any workplace rules with this in place"

Works perfect for me, my company has just started blocking www.metafilter.com - but using www.bbc.metafilter.com lets me in - slightly circuitious route via archives to view front page, but it gets me in, is there anyway to use this method to view Ask or Meta though?
posted by lloyder at 7:26 AM on January 15, 2006

lloyder, take a look at this AskMe question
posted by Sharcho at 3:12 PM on January 15, 2006

mathowie, another thing that needs a 301 redirect is
http://www.metafilter.com/comments.mefi/12345 ->
http://www.metafilter.com/mefi/12345

http://www.metafilter.com/user.mefi/12345 ->
http://www.metafilter.com/user/12345
posted by Sharcho at 11:32 AM on January 16, 2006