Who broke metafilter?

February 11, 2011 9:54 PM

Is something weird going on with metafilter today? Sometimes I can't get to any of the pages ("Oops! Google Chrome could not find www.metafilter.com"), and sometimes it's just metatalk or my own profile. I don't think I've got through to my own profile page in about six hours now, but the other pages are up and down. Is it just me?
posted by lollusc to Bugs at 9:54 PM (75 comments total)

There's some DNS shifting going on on our end. We're hoping it's as transparent as possible but clearly there are some small wrinkles. I don't know the specifics, but it's definitely not you, it's us.
posted by jessamyn (staff) at 9:57 PM on February 11, 2011

Had the same problem, couldn't access mefi for a few hours this evening. Welcome back!
posted by quinoa at 10:14 PM on February 11, 2011

No MeFi here in Key West. 68.87.74.166
posted by Mike Mongo at 10:17 PM on February 11, 2011

whoops 68.87.68.166
posted by Mike Mongo at 10:18 PM on February 11, 2011

> Is it just me?

You can always check that via an external link checker. Also, a quick fixt to remember is to go to a command prompt and type "ipconfig /flushdns" to clear out all DNS entries stored on your PC.
posted by Burhanistan at 10:19 PM on February 11, 2011 [3 favorites]

ipconfig /bringmeataco
posted by boo_radley at 10:26 PM on February 11, 2011 [13 favorites]

Error: Unrecognized or inedible meat product
posted by Burhanistan at 10:27 PM on February 11, 2011 [4 favorites]

Metafilter: incredible meat product.
posted by joe lisboa at 10:33 PM on February 11, 2011 [1 favorite]

I always use this website to see if it's just me.
posted by patheral at 11:05 PM on February 11, 2011

whoops, someone linked that already. Okay, off to bed for me.
posted by patheral at 11:06 PM on February 11, 2011

Same for me for a couple of nights now (so around 3am server time?) Can see profile page, can often see a post, but can't comment.
posted by obiwanwasabi at 11:28 PM on February 11, 2011

Oh, you're in Canberra too! Maybe it's a local thing. You're not with Internode, are you?
posted by obiwanwasabi at 11:28 PM on February 11, 2011

I was just about to post this. The DNS system claims that dns3.easydns.org, dns4.easydns.info, ns1.easydns.com, and ns2.easydns.com are authoritative for metafilter.com. However, dns3 and dns4 are returning 'refused' for metafilter.com. ns1 and ns2 seem to be working fine.

Overall, this means that you're losing about half your queries, and presumably a hell of a lot of traffic.
posted by Malor at 12:05 AM on February 12, 2011

Yeah, and OpenDNS picked up the SRVFAIL. Drat.

Matt mentioned the move on the Twitters when informed. Flag it and move on, and all that. I can live without Meffy ha for eight hours.

Seems fine now.
posted by genghis at 3:47 AM on February 12, 2011

Couldn't get to Mefi from Starbucks, today. Worked fine once I connected from elsewheres.
posted by Blazecock Pileon at 4:14 AM on February 12, 2011

Still a little wonky.
posted by fixedgear at 4:47 AM on February 12, 2011

Very weird, I can access all the subsites except Podcasts and the blue.
posted by Memo at 4:54 AM on February 12, 2011

Haven't had a problem for the last few hours. Think I'm using google's DNS servers.

Meffy

Nope nope nope.
posted by empyrean at 6:02 AM on February 12, 2011

Yep, gettin' intermittent DNS SERVFAIL shenanigans on Comcast's cns.manassaspr.va.dc02.comcast.net server as of minutes ago... shucks, I had just come here to post something about it, figuring that I'd found something else to blame Comcast for, but no such luck.

Tor provides a convenient way to reach other DNS servers and get around the problem.
posted by XMLicious at 6:57 AM on February 12, 2011

All subsites except ask seem to be working but for that DNS seems completely shot. Have tried using multiple DNS providers with no luck.
posted by turkeyphant at 6:59 AM on February 12, 2011

I got on ask once this morning, that was it. I tried a bunch because I was cursing my phone thinking it was (once again) AT&T. But nope, for once it's not. It's askmeefeye.
posted by nevercalm at 7:07 AM on February 12, 2011

I got it too, but thought it was work blocking it after they found out all the time I spent on MeFi. That or the Great Firewall. Happy to find it's neither.
posted by arcticseal at 7:09 AM on February 12, 2011 [1 favorite]

(Wikipedia article on Tor, though using one of the bazillion more general-purpose proxies out there like Guardster might work too. Guardster appears to be working for me right now too but be sure to un-tick the "No Scripts" box, i.e. you want scripts.)
posted by XMLicious at 7:10 AM on February 12, 2011

I'm in Toronto, on Teksavvy, and I couldn't get on any MeFi sites at all last night. I called tech support and switched my network settings to use the DNS servers at 208.67.222.222 and 208.67.222.220 (tech guy just got those from opendns.com) and everything worked perfectly from that point. Those of you with issues might try those.

EXCEPT: Safari on my iPhone still fails to load any of the sites when I'm using WiFi in the house, but it works perfectly when I'm on 3G.

EXCEPT EXCEPT: Opera can load all the sites on my iPhone even when on WiFi.

I've poked around the settings on my phone and have cleared cache, cookies and history from my Safari settings, I've powered down and up, and still -- nothing. Any ideas?
posted by maudlin at 7:53 AM on February 12, 2011

The weird thing was that I could get through on my iPad and in Firefox, but not in Chrome.

I was starting to have withdrawal symptoms from not being able to check my favorites.
posted by Joe Beese at 7:57 AM on February 12, 2011

I just updated the dns to roll back to old settings so it should go back to normal soon. I won't enable new nameservers until I'm sure they have the domain on them and working.
posted by mathowie (staff) at 8:59 AM on February 12, 2011

I glad you agreed we could work this out Meffy. We'll talk more this time, I promise!
posted by Brandon Blatcher at 9:36 AM on February 12, 2011

I'm having a lot of trouble connecting directly from my location in Austin, but it comes up lightning fast from a proxy.
posted by Xezlec at 9:38 AM on February 12, 2011

The reason it sometimes works and sometimes doesn't is because whatever DNS server you use gets a list of authoritative servers from the .com root. It picks one of them at random. If it chooses a bad one, you won't be able to resolve MeFi on that local server until its cache expires. How long that takes depends on how long Matt has the Time To Live (TTL) value set for. Typically that's about an hour. Until that time expires, your local DNS server will tell you that Metafilter doesn't exist, basically. After it expires, you have a 50% chance to hit a 'good' server... if you hit another bad one, it'll stay down. Changing DNS providers gives you another 50% chance of getting through. It's not that the new server is any better than the old one... you just get to roll the dice a second time.

Matt, you still have a problem in the DNS. You've set the global WHOIS record back to the old servers, but the old servers themselves refer to the NEW servers in their Authority section. At least at least some of the DNS clients out there will use those values. Of the new servers, 2 out of 3 seem to be presently working. 'dns3.easydns.ca' is not answering for metafilter.com.

If you set your TTL values very short at least one full timeout period before making the actual switch, you can then roll out changes and revert them very quickly. This increases your DNS traffic proportionally to how much you shortened the timeout, so you typically don't want to keep the shortened values any longer than you need to, but it can minimize client disruption.
posted by Malor at 10:00 AM on February 12, 2011

OpenDNS seems to work right now. Google DNS and my ISP's DNS don't. Weird how only some subdomains were affected though.
posted by turkeyphant at 10:01 AM on February 12, 2011

Yeah, your DNS is messed up. com authorities are reporting ns1.easydns.com. and ns2.easydns.com. as authoritive, but your record has dns1.easydns.com., dns3.easydns.ca., and dns2.easydns.net..
posted by cj_ at 10:21 AM on February 12, 2011

Getting a coldfusion session error on my end...
posted by schmod at 10:41 AM on February 12, 2011

DNS.
posted by dougrayrankin at 10:51 AM on February 12, 2011

schmod, getting that consistently? Does restarting your browser help?
posted by pb (staff) at 10:54 AM on February 12, 2011

Yeah, what you put into the global database (via your name registrar) should normally always agree with what your servers themselves are claiming as authorities. For advanced use, you can get tricky and report different authorities to clients based on, say, geographical location, but a normal website wants to report exactly the same values that are in the global DNS. It avoids lots of confusion and potential issues.

You should at least take dns3.easydns.ca out of your local server Authority sections until you get that server responding for metafilter, or people will continue to have issues.
posted by Malor at 10:56 AM on February 12, 2011

If you're still having DNS problems one option is to add all of our subdomains to your local hosts file. That will bypass DNS altogether. I found some quick guides for Windows XP/2000, Windows Vista, Mac OS X. (I'm on a Mac and I prefer sudo vi /private/etc/hosts from the terminal.) If you're on Linux you probably do this all the time.

You'd want to add these entries:

174.132.172.58 www.metafilter.com
174.132.172.58 ask.metafilter.com
174.132.172.58 projects.metafilter.com
174.132.172.58 music.metafilter.com
174.132.172.58 jobs.metafilter.com
174.132.172.58 irl.metafilter.com
174.132.172.58 podcast.metafilter.com
174.132.172.58 metatalk.metafilter.com

This should take care of itself in a few hours, but this could help in the meantime if you're still having problems.
posted by pb (staff) at 11:02 AM on February 12, 2011 [2 favorites]

Yay! The old settings work on my iPhone!
posted by maudlin at 11:02 AM on February 12, 2011

If you're trying to transfer the zones to dns3.easydns.ca, you can do that with 'also-notify' and 'allow-transfer' clauses in your configuration file, as opposed to just listing it as an authority in the DNS. I have no idea if EasyDNS gives you that kind of control, however.

DNS is one of the things I like to run inhouse. It's a very low-load service. It's a bit tricky to learn originally, but once you know it, it's pretty easy. The big providers don't give you nearly the same level of control over things like TTL, which matters a lot for transitions like these.

If you want high availability, you can set up a 'slave' service, where you maintain the master server(s), and big ISP or registrar servers transfer the zones whenever you update them, and serve all your clients. You configure the global WHOIS records to point at the big servers, but they're all slaves to your tiny master server, wherever it is.
posted by Malor at 11:05 AM on February 12, 2011

The hallucinations and shakes are starting to diminish.
posted by caddis at 11:10 AM on February 12, 2011

You can always check that via an external link checker.

I did that and it worked there, but not for me. That site should really check from multiple IPs.
posted by DU at 11:12 AM on February 12, 2011 [2 favorites]

I haven't been able to access the site at all until just now.

I kept trying to get on, and saying, "Why you dead, Ricky?"


:(
posted by louche mustachio at 12:11 PM on February 12, 2011 [2 favorites]

Oh, whew! I couldn't even access the status blog for the last 9-10 hours, never mind ANY of Metafilter.

I wondered what I had done that had caused the Powers-That-Be to bypass banhammer and unleash complete BANISHMENT.

(Just for the record, I was compiling a list of what I was willing to do in order to repent. Because of it's dramatic and drastic nature, that list will be nuked in 3...2...1...)
posted by jeanmari at 12:54 PM on February 12, 2011 [2 favorites]

its NOT it's...GAH!
posted by jeanmari at 12:57 PM on February 12, 2011

The sky is the most interesting color of blue today.
posted by Devils Rancher at 1:29 PM on February 12, 2011

In the old days, we would read Plastic and eat Munchos.
posted by clavdivs at 2:18 PM on February 12, 2011

Ok, it turns out in the end there was an issue moving from a low end cheap account at EasyDNS over to their Enterprise service (MeFi was exceeding the monthly DNS lookup limit by 20million lookups). But in the switch over, they forgot to update the enterprise servers with the info on the lower end servers, so the domain was in limbo the past 24hrs or so. The company founder personally updated all the servers and sounds like they are changing their internal documentation so this doesn't happen again. I just again updated the servers to a new list and everything should just work.

Being on some new nameservers, they offer some redundancy on other continents, so hopefully connecting to the site will be faster and smoother if you're in Europe or Asia in the future.
posted by mathowie (staff) at 2:41 PM on February 12, 2011 [2 favorites]

I had the same problem getting to MeFi from Brussels.

hopefully connecting to the site will be faster and smoother if you're in Europe or Asia in the future

Hooray!
posted by dhens at 2:43 PM on February 12, 2011

The sky is the most interesting color of blue today.

Wait, where are you? Here the sky is the color of television, tuned to a dead channel.
posted by XMLicious at 2:55 PM on February 12, 2011 [6 favorites]

so hopefully connecting to the site will be faster and smoother if you're in Europe or Asia in the future.

You're outsourcing us?
posted by Brandon Blatcher at 3:03 PM on February 12, 2011 [1 favorite]

Could the status blog be mirrored externally? When I couldn't get to the site, I went there to investigate, but couldn't get there either (presumably also because of DNS issues).
posted by Jahaza at 3:19 PM on February 12, 2011 [1 favorite]

It was us. We broke MeFi. The domain was manually upgraded to Enterprise level service and we didn't flip a few switches in the database (i.e. "start serving DNS on this domain *now*)

MeFi should now be carried on 4 anycast constellations spread over 25 nameservers worldwide (used to be on 9).

easyDNS apologizes profusely for breaking metafilter.

- mark (easyDNS guy)
posted by StuntPope at 3:21 PM on February 12, 2011 [17 favorites]

StuntPope: we forgive you! maybe tweet it next time or something.
posted by olya at 3:32 PM on February 12, 2011 [1 favorite]

That still calls for a moderate spanking.
posted by Burhanistan at 3:49 PM on February 12, 2011

Could the status blog be mirrored externally?

The status blog is actually on a different server/setup completely, it was just that *.metafilter.com was no longer responding, so you couldn't get to status.metafilter.com because nameservers had no idea where the domain should even be.
posted by mathowie (staff) at 3:54 PM on February 12, 2011

It was not being able to get to the status blog that was the freaky part. On the other hand, it turns out that the outside world isn't a myth. Unfortunately, the downtime coincided with the worst weather ever for a three day weekend. Without Metafilter, I was stuck watching romantic comedies starring Sarah Jessica Parker.
posted by Ghidorah at 4:13 PM on February 12, 2011 [1 favorite]

The whole site does feel extra-double-snappy this evening.
posted by Devils Rancher at 4:20 PM on February 12, 2011

What the HELL are you folks talking about?
posted by Bushrod Johnson at 4:23 PM on February 12, 2011 [1 favorite]

Ghidorah: "It was not being able to get to the status blog that was the freaky part. On the other hand, it turns out that the outside world isn't a myth. Unfortunately, the downtime coincided with the worst weather ever for a three day weekend. Without Metafilter, I was stuck watching romantic comedies starring Sarah Jessica Parker

Sorry about your suffering.
posted by Splunge at 4:26 PM on February 12, 2011

Thanks to the DNS issue today I did my laundry and cleaned the bathroom. Yay!
posted by localhuman at 4:47 PM on February 12, 2011

The tragic thing was that after posting this I was never able to get back to it to see the answer! Oh well. Thanks anyway.

And yes, in Canberra, but not with Internode. Looks like it was lots of people, anyway.
posted by lollusc at 4:51 PM on February 12, 2011

Man, I coulda used one of them DNS issues. Here everything's worked fine all day, the laundry stinks and the bathroom is filthy.
posted by nebulawindphone at 6:02 PM on February 12, 2011 [2 favorites]

I cleaned the sink, hung up laundry, wrote almost three pages of single-space final draft (about old pianos and %#!€& user configuration and script), repaired a model locomotive with spare parts that arrived today, banged the downstairs girl into submission, floor-wise (who was shrieking like a steam whistle), AND had full access to the old filter. Yay.
DNS: did nothing silly
posted by Namlit at 6:12 PM on February 12, 2011

Switched to google's 8.8.8.8 and 8.8.4.4 DNS and I have metafilter again!
posted by Mike Mongo at 9:15 PM on February 12, 2011

D.N.S... Desperate Network Snacks!
posted by Juffo-Wup at 9:31 PM on February 12, 2011

The sky is the most interesting color of blue today.

You know there are bears under that blue thing, right? Running around, doing whatever the hell they want.

BEARS.
posted by louche mustachio at 9:38 PM on February 12, 2011 [4 favorites]

Here the sky is the color of television, tuned to a dead channel.

Yes, blue.
posted by crossoverman at 9:59 PM on February 12, 2011 [3 favorites]

For those of you who switched to external DNS resolvers, you might want to switch back. Many services (Youtube in particular) now provide answers based on your network location, to try to point you at nearby, fast servers, instead of ones that are far away and thus slow for you.

By using an external DNS, you in essence 'move' to that location for DNS answers, which can impair your throughput quite substantially on high-bandwidth sites.
posted by Malor at 11:31 PM on February 12, 2011

Yeah, thought I got my account blocked on some ill-advisedly sarcastic comments last night. In my defense, I was drunk on Egyptian proto-democracy. That said, even though I was not blocked personally, it triggered some self-examination, which is always a net positive. Thank you, technology!
posted by joe lisboa at 12:30 AM on February 13, 2011

I say, we rush them, hive mind! Fire up the LOIC's.

We are MeFi.

We do not forget.

We do not forgive.

(wanders off)
posted by Samizdata at 2:17 AM on February 13, 2011 [1 favorite]

one option is to add all of our subdomains to your local hosts file

I believe you've just converted your temporary DNS problems into a more insidious, hard-to-diagnose issue somewhere down the line.
posted by ryanrs at 8:28 AM on February 13, 2011 [2 favorites]

ryanrs, true. But if we do anything as big as moving to a new IP address we'll provide instructions about checking your hosts file for people who have problems with it.
posted by pb (staff) at 9:12 AM on February 13, 2011

Better do it quick, before they run out!
posted by ryanrs at 11:52 AM on February 13, 2011

Many services (Youtube in particular) now provide answers based on your network location

This solved a jerky YouTube problem that I have been having for months. Thank you!
posted by jessamyn (staff) at 12:11 PM on February 13, 2011

louche mustachio: "
The sky is the most interesting color of blue today.

You know there are bears under that blue thing, right? Running around, doing whatever the hell they want.

BEARS"

Bears. Beets. Battlestar Galactica.
posted by Splunge at 6:12 AM on February 16, 2011

I'm not going back out there again -- there's honey badgers.
posted by Devils Rancher at 7:04 AM on February 16, 2011