Comments on: DNS wildcard and google crawler

DNS wildcard and google crawler

Sharcho — Fri, 06 Jan 2006 11:34:42 -0800

It's possible that DNS wildcards on metafilter.com are causing problems for the Google crawler. I suggest deleting the redundant DNS entries, and using a 301 redirect for http://metafilter.com/ to go to http://www.metafilter.com/

By: shmegegge

shmegegge — Fri, 06 Jan 2006 11:49:07 -0800

you sure it's not just cooke's use of the blink tag?

By: afflatus

afflatus — Fri, 06 Jan 2006 11:49:22 -0800

I only see one entry for metafilter.com. There is an A record that points to 70.86.84.162

www.metafilter.com has a cname record that points to metafilter.com

When I go to metafilter.com it redirects with a 302 status code. Since 302 is temporary redirect, I agree that it should probably be changed to a 301, which is a permanent redirect.

What are you seeing as issues with DNS though?

By: Sharcho

Sharcho — Fri, 06 Jan 2006 11:50:39 -0800

e.g. http://blahblahblah.metafilter.com/

By: Sharcho

Sharcho — Fri, 06 Jan 2006 11:56:07 -0800

You're right, for metafilter.com it does return 302 (it should be 301 permanent direct), but for the other subdomains it doesn't even do that.

By: afflatus

afflatus — Fri, 06 Jan 2006 12:07:36 -0800

Ah, now I see what you mean. I thought you were talking about roundrobin dns or some such. I agree, 3rd level domains that don't exist probably should just return a regular dns not found error.

By: Plutor

Plutor — Fri, 06 Jan 2006 12:10:22 -0800

afflatus: "Ah, now I see what you mean. I thought you were talking about roundrobin dns or some such. I agree, 3rd level domains that don't exist probably should just return a regular dns not found error."

They're there for a reason, though: to help people at less-enlightened employers get around simple proxy rules.

By: mzurer

mzurer — Fri, 06 Jan 2006 12:29:53 -0800

metatalk.google.com

What exactly does it break?

By: smackfu

smackfu — Fri, 06 Jan 2006 13:25:10 -0800

One idea: Matt could make it so that it checks the current URL and tells the robot to exclude the page if it's not under metafilter.com. That would probably be a one-liner.

By: delmoi

delmoi — Fri, 06 Jan 2006 13:40:50 -0800

It's done this way so that people who work for companies who block "www.metafilter.com" or "metafilter.com", but not "*.metafilter.com". can still browse the site.

This was done intentionally.

By: mathowie

mathowie — Fri, 06 Jan 2006 14:20:41 -0800

Actually, I don't know of anyone still successfully getting around any workplace rules with this in place.

Every corp network I've seen blocked did the whole *.metafilter.com domain.

I'll change the redirect status and make an A record for the www, and I have a cool idea what to do with all the subdomains to make them useful.

By: cortex

cortex — Fri, 06 Jan 2006 14:54:00 -0800

and I have a cool idea what to do with all the subdomains to make them useful

Unfortunately, this margin is not large enough to contain it...

I mean, c'mon! This shit keeps me up nights! Dish!

By: delmoi

delmoi — Fri, 06 Jan 2006 16:01:16 -0800

I'll change the redirect status and make an A record for the www, and I have a cool idea what to do with all the subdomains to make them useful.

Probably username.metafilter.com.

/registers the metatalk sockpupet.

By: todbot

todbot — Fri, 06 Jan 2006 16:31:55 -0800

I bet {tagname}.metafilter.com would be pretty useful.

By: whir

whir — Sun, 08 Jan 2006 02:02:55 -0800

As far as google goes, with the billions they pour into their search engine it should be smart enough to know what a wildcard domain is. It's not up to us (well, Matt) to fix google.

By: null terminated

null terminated — Mon, 09 Jan 2006 17:11:40 -0800

is, for example, http://politics.bush.metafilter.com too much to ask? :-)

By: lloyder

lloyder — Sun, 15 Jan 2006 07:26:32 -0800

Matt said "Actually, I don't know of anyone still successfully getting around any workplace rules with this in place"

Works perfect for me, my company has just started blocking www.metafilter.com - but using www.bbc.metafilter.com lets me in - slightly circuitious route via archives to view front page, but it gets me in, is there anyway to use this method to view Ask or Meta though?

By: Sharcho

Sharcho — Sun, 15 Jan 2006 15:12:35 -0800

lloyder, take a look at this AskMe question

By: Sharcho

Sharcho — Mon, 16 Jan 2006 11:32:57 -0800

mathowie, another thing that needs a 301 redirect is
http://www.metafilter.com/comments.mefi/12345 ->
http://www.metafilter.com/mefi/12345

http://www.metafilter.com/user.mefi/12345 ->
http://www.metafilter.com/user/12345