Woot, we're back up. Sorry about the downtime. September 13, 2006 11:25 PM   Subscribe

Woot, we're back up. Sorry about the downtime.
posted by mathowie (staff) to Uptime at 11:25 PM (103 comments total) 1 user marked this as a favorite

Yay!, thanks Matt
posted by blueberry at 11:31 PM on September 13, 2006


What happened?
posted by clockworkjoe at 11:31 PM on September 13, 2006


Doh! I just emailed you about comments not working and now they are. In any case there was an additional bit of info there so don't automatically delete it, svp.

And now...after about 50 attempts today, I can get back to reading and posting in the bluegraygreen! Yay!
posted by Kickstart70 at 11:32 PM on September 13, 2006


Something about a hardware failure...
posted by Rhomboid at 11:32 PM on September 13, 2006


The downtime was due to a win2003 server update. I have the server set to automatically download hotfixes and install them in the middle of the night. At some point last night that happened, and the server restarted itself and couldn't come back to life. Techs at my hosting place said Windows would not get past the splash screen of loading windows, even in safe mode.

Eventually after 8 hours of phonecalls and back and forths, they installed a new drive with a fresh copy of win2003 on it, and I moved old mefi server data over to it, setup coldfusion and apache, and made it live just now.

About the only things I lost in the server wipe-n-reinstall were some coldfusion custom settings, stuff like generating new tag pages every hour for the various sites, etc. I'll set those back up in the next couple days.
posted by mathowie (staff) at 11:34 PM on September 13, 2006 [1 favorite]


What a bright, scary day it was outside without metafilter! Glad we're back - thanks for keeping us posted, Matt (and Jessamyn too).
posted by Iamtherealme at 11:38 PM on September 13, 2006


/obsessively reloads in joy
posted by namespan at 11:38 PM on September 13, 2006


And hey - Sept 13 - The Day Without a MeTa Post. Does that count?
posted by Iamtherealme at 11:38 PM on September 13, 2006 [1 favorite]


Nevar forget!
posted by team lowkey at 11:39 PM on September 13, 2006


Ah, just another demonstration of His Noodly Appendage's sense of irony, that Matt's wish for a day without metatalk posts would come in this form.

(Though I don't think it really counts as if your time zone is PST this post was made on the thirteenth.)
posted by Rhomboid at 11:43 PM on September 13, 2006


mathowie, the spell check gave me an error when posting.
posted by tellurian at 11:47 PM on September 13, 2006


Oh sweet gay zombie baby Jesus! *fixes deeply*
posted by loquacious at 11:49 PM on September 13, 2006


Matt, I hope you understand that because Metafilter was down today, I had to work. Let me say that again. I had to WORK.

Ugh. I feel so dirty. Please don't ever let Metafilter go down during a workday again.
posted by Effigy2000 at 11:50 PM on September 13, 2006 [1 favorite]


d'oh!

i was hoping that would reset my MeTa quota, & i gotta post a reminder about Sunday's Sydney meetup.

help me, mattamyn!

(or tellurian - details would be alfred, early arvo, say, 1-2ish...?)
posted by UbuRoivas at 11:53 PM on September 13, 2006


"Unable to connec" are four letter words.
posted by Cranberry at 11:57 PM on September 13, 2006


See? A whole day without MetaFix and I am unable to connect with the t key.
posted by Cranberry at 11:59 PM on September 13, 2006


The site was down today? Huh. Glad I have access to the cabal's backup server.
posted by eyeballkid at 12:05 AM on September 14, 2006


C'mon, eyeballkid, you know there's no cabal.
posted by cgc373 at 12:07 AM on September 14, 2006


w00t indeed. *takes deep deep hit from the crackpipe*
posted by stavrosthewonderchicken at 12:07 AM on September 14, 2006


Huzzah!
posted by pompomtom at 12:09 AM on September 14, 2006


Jesus, well, at least the outage allowed me to connect with another user on Vox (tm) about how much we were annoyed about not having Mefi.
posted by lackutrol at 12:09 AM on September 14, 2006


Phew - now I can go back to doing less work; yesterday was quite stressful
posted by TheDonF at 12:09 AM on September 14, 2006


*stops shaking
posted by caddis at 12:21 AM on September 14, 2006


both MeFi and MeCha down at the same time - aggghhh
posted by caddis at 12:23 AM on September 14, 2006


Win 2003? Dammit, Matt, get a Mac.
posted by armoured-ant at 12:23 AM on September 14, 2006


Mathowie, one of the missing coldfusion components is captcha. I couldn't log in from my non cookied box (or a different browser for that matter). I did however manage to stub my toe in the process of running to my other computer that stores cookies.
posted by |n$eCur3 at 12:25 AM on September 14, 2006


there's no captcha anymore (there shouldn't be one)
posted by mathowie (staff) at 12:26 AM on September 14, 2006


Yes, I've been struggling for some time to get a message up.... I had to go to another computer to get this posted. Can't log in to complain about being unable to log in. :-)

This is the error message:

Could not find the ColdFusion Component Captcha.
Please check that the given name is correct and that the component exists.

posted by Malor at 12:28 AM on September 14, 2006


ok, login form fixed.
posted by mathowie (staff) at 12:31 AM on September 14, 2006


Missed you all.
posted by evariste at 12:31 AM on September 14, 2006


Wow, a full software transplant? That's stressful and sucky. Thanks for getting things going again.
posted by Nelson at 12:39 AM on September 14, 2006


In the future, people checking out the sidebar "what happened 1, 2, 3, 4, 5 years ago" links will be very confused. "What the heck happened on 9/13/06?"

Glad that everythings moving again, I almost needed an IV.
posted by spacelux at 12:43 AM on September 14, 2006


armoured-ant: I realize you're being silly, but since I'm a big geek, I'll answer you seriously. :)

The Mac kernel is very bad at big webloads. It just chokes and dies when it's trying to serve a lot of threads. It has terrible performance with high-concurrency environments. It's fine for one user, but not so hot for a thousand at once.

The other major webhost choice, Linux, has been having floods of security patches and problems in the kernel... they're moving so fast, and have gone to such a short dev cycle, that it's hard to find a stable kernel. The distros can do a pretty fair job of fixing obvious bugs and giving you a kernel that won't crash, but they don't easily find the holes that let people break in. And each time the kernel gets patched, that requires downtime. There was a period a couple months ago where the kernel devs issued something like thirty patches in just a few weeks. That does not give me much confidence that anyone is really looking over new code carefully and thinking about security issues. They're moving way, way too fast to really consider the implications of what they're doing.... as smart as they are, they're missing stuff, I guarantee it.

FreeBSD would work well, being run by very cautious developers, but that's pretty hard for users who don't know Unix at least a little already. And of course Solaris would be excellent, but take all the FreeBSD problems and multiply them by 2,323. :)

Win2003 really is a pretty good solution here. It isn't, obviously, perfect, but it's faster than a Mac would be, and probably has less downtime than a Linux box would.

I think the real stability issues are caused by the fact that Coldfusion sucks. This particular issue was an outlier; most of the problems, I think, have been caused by CF, not Windows.

Ok, I'll drop out of geek mode long enough to say, "thanks, Matt! Glad to see MeFi back."

(Hmm. Did I actually turn off geek mode in that sentence?)
posted by Malor at 12:45 AM on September 14, 2006 [1 favorite]


GEEK MODE MANDATORY. COMMENCE AND RESUME GEEKING. GEEKING MANDATORY.
posted by loquacious at 12:55 AM on September 14, 2006


Thanks matt. I can now log in from anywhere. First post 9/13 pony request: can we please have this captcha as a replacement? It's a hotornot.com mashup. The website features such testamonials as "I met my wife on your captcha."
posted by |n$eCur3 at 12:57 AM on September 14, 2006


Phew! Thank god it's back. It was a cold, cold day of reloading yesterday.
posted by antifuse at 1:07 AM on September 14, 2006


I just kept thinking, "surely, there's been some mistake. something clogging the series of tubes." Then I'd click reload, and click reload and lurk on strange, foreign boards while I wondered if I'd ever get to waste my day the metafilter way again.

Welcome back. Thank you for knocking out them tech problems quick.
posted by EatTheWeek at 1:39 AM on September 14, 2006


...Am I the only one getting a default Apache server page for MeFi?
posted by duende at 1:40 AM on September 14, 2006


So when is MeFi getting some redundancy? :)
posted by knave at 1:43 AM on September 14, 2006


duende: try a reload or flush your browser cache.
posted by pjern at 2:13 AM on September 14, 2006


I felt so alone that I actually took my kid out of his Skinner box for a few hours. Of course, he wanted to color. So predictable. And he kept insisting that 'snark' was 'a bitey fish.'

Please, never leave me again.
posted by maryh at 2:53 AM on September 14, 2006 [2 favorites]


Dear Diary

Metafilter was down today. Discovered I had a girlfriend. Was going to accept her offer of sex when I discovered Metafilter was back up. Posted comment about it in Metatalk. Life is super.
posted by Second Account For Making Jokey Comments at 3:33 AM on September 14, 2006 [1 favorite]


Somehow, I doubt you enjoyed your day off, Matt.
posted by crunchland at 4:16 AM on September 14, 2006


Ugh. I feel so dirty. Please don't ever let Metafilter go down during a workday again.

Try living in the UK.
posted by teleskiving at 4:31 AM on September 14, 2006


Absence makes the heart grow fonder.
posted by leapingsheep at 4:35 AM on September 14, 2006


Malor, I'm just not getting your bias here. Any flavor of Debian linux is obsessively stable, to the point of being several major releases in certain pieces behind. But win2003 being a more secure choice? Not in my movie.
posted by toastchee at 4:50 AM on September 14, 2006


Dr. Crippen busted the interweb.
posted by Astro Zombie at 5:02 AM on September 14, 2006


During the downtime, I discovered that the Google and Yahoo caches of MeFi don't work when it's down. Is this something MeFi's doing wrong, or Google/Yahoo? It'd be good to be able to still get into AskMe answers when server's out.
posted by bonaldi at 5:09 AM on September 14, 2006


thank you, and whew.
posted by sdn at 5:35 AM on September 14, 2006


I expect all the new FPPs to be well thought out and correctly spelled because people have had a day to think about them and have not hit Post in a fit of drunken finality
posted by MonkeySaltedNuts at 5:37 AM on September 14, 2006


I actually worked on my thesis.
posted by signal at 5:46 AM on September 14, 2006


mathowie, the spell check gave me an error when posting.
posted by tellurian at 2:47 AM EST on September 14 [+] [!]


That's still the case, FYI.
posted by chuma at 6:09 AM on September 14, 2006


win2003 would have less downtime then a linux solution would? wha??
posted by rsanheim at 6:11 AM on September 14, 2006


Metachat was down, too. I actually got some work done. It was horrifying.
posted by jonmc at 6:12 AM on September 14, 2006


Thank god I can go back to not doing work again!
posted by echo0720 at 6:14 AM on September 14, 2006


More geek comments"

The Mac kernel is very bad at big webloads. It just chokes and dies when it's trying to serve a lot of threads.

True. It reminds me of Irix, which was optimized for workstation performance, and made a lousy server. Which sucked, because when you looked at the I/O bandwidth on an SGI box, they should have rocked.

FreeBSD would work well, being run by very cautious developers

Yes. I gave up on Linux entirely with the 2.6 debacle (the one that broke a bunch of hardware.) If I need faster, I throw hardware and optimization at the problem. I'm an elitist pig hardcore FreeBSD user now -- if it isn't Solaris or AIX, it's FreeBSD. Take it, Tux!

Linux vs. FreeBSD reminds me of the Supercomputer/Mainframe argument. The first is all performance. The latter is all reliability. Linux does have real performance advantages, but I run FreeBSD for the same reasons I drive a Honda Civic -- I'm a production sysadmin, speed is nice, but reliability is what keeps the pager from going off.

It isn't, obviously, perfect, but it's faster than a Mac would be, and probably has less downtime than a Linux box would.

A good distro that controls the patch madness of the kenel should be stable. My biggest problem with Windows is the way you keep getting screwed on patches.

BTW, this is why "autoinstall" of patches is a bad idea. You download them, have them beep at you, then you image the drive and install them. That way, when it goes pear shaped, you reimage back and try to figure out what went wrong.

Ideally, you have an identical server that you can test the patches against. This doesn't save you from hardware errors, but the right answer to that is spare parts -- or spare servers. If it isn't the HDDs, you move them to the spare servers (along with FC cards, if possible -- easier and quicker than rezoning) and power them up. Then you call for service.

Furthermore, If you're buying multiple Sun boxen, you buy lots of the same one. You buy the top line support on one of them, which costs you bunches of money, but comes with lots of good stuff. The rest get the "We'll fix this sometime this week" service contract. The real service contract is the spare box -- which will be far cheaper than half a dozen top tier support contracts.
posted by eriko at 6:27 AM on September 14, 2006


Indeed, FreeBSD rocks for those times when you absolutely need a box to stay function for weeks/months on end. Literally the only thing you have to worry about is the electricity functioning as reliably. OpenBSD is said to be equally solid (although I haven't tried it myself yet).
posted by clevershark at 6:38 AM on September 14, 2006


Look at it this way. Matt got his day without a Metatalk post.
posted by pjern at 7:06 AM on September 14, 2006


Y'know, if you wanted a MetaTalk free day, you could have just asked.
posted by graventy at 7:07 AM on September 14, 2006


BTW, this is why "autoinstall" of patches is a bad idea. You download them, have them beep at you, then you image the drive and install them. That way, when it goes pear shaped, you reimage back and try to figure out what went wrong.


good advice
posted by caddis at 7:07 AM on September 14, 2006


Thanks for the sweat Matt.

Iamtherealme writes "And hey - Sept 13 - The Day Without a MeTa Post. Does that count?"

The truth comes out.

Effigy2000 writes "Matt, I hope you understand that because Metafilter was down today, I had to work. Let me say that again. I had to WORK.

"Ugh. I feel so dirty. Please don't ever let Metafilter go down during a workday again."


You think that was bad, I was home sick in bed. And MeCha wasn't up either. Thank god for my PVR.
posted by Mitheral at 7:11 AM on September 14, 2006


I actually turned work in before the deadline. Please don't ever make me do that again.
posted by languagehat at 7:14 AM on September 14, 2006


I actually found out that there's a lot of other cool stuff on the internet. I thought about working, but it seemed too hard.
posted by OmieWise at 7:30 AM on September 14, 2006


Thanks matt. Glad to hear you unclogged the tubes. It so hard when people try to send you intarnets on tuesday, and you don't get it till thursday. I think we need to legislation to permanently unclog mefi ; I mean, come on, its not a dump truck.
posted by Merik at 7:47 AM on September 14, 2006 [1 favorite]


Seriously, rethink your software choices.
posted by fold_and_mutilate at 7:59 AM on September 14, 2006


Please, baby, I swear I won't ever smack you around again, or call you my bitch. Just don't go away again.

*presents a fake silk rose purchased at the corner store*
posted by moira at 8:00 AM on September 14, 2006


I dunno what distros you guys are running, but around here our Linux boxes don't have these problems. Yeah, 2.6 was pretty screwed up but you shouldn't be running bleeding-edge software on a production machine anyway *boggle*

With a few exceptions for windows-only apps, all of our production machines are linux and one of our highest-traffic runs RHAS 3 which is running 2.4.21.

Not to disparage the BSDs, Solaris or AIX, I love 'em (ok, nobody loves AIX but you know what I mean), but saying that Linux boxen are unstable is kind of silly.
posted by Skorgu at 8:21 AM on September 14, 2006


GEEK MODE MANDATORY. ... Well, okay then.

Malor, I'm just not getting your bias here. Any flavor of Debian linux ...

I was tempted to respond to his criticism of Linux, which is at least arguably sort of half-way almost reasonable, until seeing his dismissal of Solaris and BSD which is just silly and reveals a willingness to grasp at any straw to dismiss everything but Windows. But you know, they all suck in their own unique ways, so whatever.

Seriously, rethink your software choices.

If solid uptime were for some reason a serious requirement, a better place to start would be hardware, in the form of a backup server (which wouldn't be updated at the same time as the primary server).
posted by sfenders at 8:25 AM on September 14, 2006


OpenBSD is said to be equally solid (although I haven't tried it myself yet).

Speaking as an OpenBSD fanatic - no.

OpenBSD is awesome at the edge of the network - there isn't a better firewalling solution for x86, and in fact given Cisco's (or just about any other specialty hardware vendor you care to name) security record vs. OpenBSD's there might not be a better firewalling solution, period.

But in terms of using it as a webserver in an environment like Metafilter or, better example, Slashdot - absolutely not. OpenBSD fucking chokes and dies in such situations. It's much, much better than something like OS X, sure - but in the spectrum of server-centric operating systems (Win2k3, Linux, the BSDs), it's definitely on the lower end of the scale for handling X number of simultaneous requests. Ultimately the unwillingness of the development team to compromise anything for the sake of security is at odds with performance, which is a good thing since we need a solid specialty OS in this regard.

Set OpenBSD on the edge of your network and have it dedicated to first-pass firewalling and traffic normalization. But your high-performance web and DB servers need to be running FreeBSD/Win2k3/Linux.
posted by Ryvar at 8:35 AM on September 14, 2006


Did anybody else think they might have done something to cause this?
posted by Astro Zombie at 8:40 AM on September 14, 2006


No, I knew it was you.
posted by Ryvar at 8:42 AM on September 14, 2006


9/13, 2006: The day the internet cried.
posted by blue_beetle at 9:14 AM on September 14, 2006


I wondered if Matt have finally received that subpoena from the Government, desperate to lay their hands on quonsar's incriminating diatribes.
posted by NinjaTadpole at 9:32 AM on September 14, 2006


I was hoping when MetaFilter came back to life it would fly and shoot lasers from its eyes, like Jesus did.
posted by brain_drain at 10:08 AM on September 14, 2006


I am still getting a

"Seeing this instead of the website you expected?

This page is here because the site administrator has changed the configuration of this web server. blah blah"

message with firefox
but not with IE

is it just me?
posted by M Edward at 10:20 AM on September 14, 2006


M Edward, try a hard refresh; ctrl-f5.
posted by monju_bosatsu at 10:27 AM on September 14, 2006


the browser is not the television.
posted by quonsar at 10:32 AM on September 14, 2006 [1 favorite]


Many thanks Monju
posted by M Edward at 10:35 AM on September 14, 2006


Looks like CF only supports RedHat, which no one I know can keep secure. Would any linux nerds here want to help me admin one? I realized in this rebuild that nothing I'm doing on the web server requires windows and I could use a linux box instead.
posted by mathowie (staff) at 10:48 AM on September 14, 2006


I told my fiance today, in the most devastated tone of voice ever: "Metafilter was down ALL DAY yesterday." He said, "I guess you had to do something productive at work then?"

"Of course not, I just browsed kitty pictures over at Something Awful's Pet Island."
posted by chiababe at 10:54 AM on September 14, 2006


So when is MeFi getting some redundancy? :)

I can go double-post something if you like.
posted by kindall at 10:55 AM on September 14, 2006


"Supported" and "will run on" are two different things. You can run ColdFusion on Debian, and I heartily recommend Debian.
posted by evariste at 11:02 AM on September 14, 2006


Here's a howto.
posted by evariste at 11:02 AM on September 14, 2006


I was afraid Matt got a little visit from the Homeland Security thugs, who forced him to hand over IP addresses and account info for all the treasonous liberals who hang out here. Then they busted his servers and smashed his glasses.
posted by Quietgal at 11:09 AM on September 14, 2006


not the nerd glasses! that's just evil.
posted by dame at 11:13 AM on September 14, 2006


the browser is not the television.
posted by quonsar at 1:32 PM EST on September 14 [+] [!]


Oh, but it is my friend. The browser is the new TV, but with more channels.
posted by caddis at 11:29 AM on September 14, 2006


Yesterday was beyond inconvenient, closer to tragic, yet not a single msm reporter showed up to ask how I felt about my loss. No wonder newspapers are losing readers, and TV news is all about whitened teeth.
posted by Cranberry at 11:30 AM on September 14, 2006 [1 favorite]


To those joyous people who felt bereft without metafilter for 18 hours : please take a moment to re-assess your life's goals and priorities.
posted by crunchland at 11:36 AM on September 14, 2006


crunchland: Ok...gimme a sec...uuunnnghhh!

I WAS WITHOUT BLUE FOR 18 HOURS!!! WAAA!!!
posted by Kickstart70 at 11:47 AM on September 14, 2006


Looks like CF only supports RedHat, which no one I know can keep secure.

There's nothing particularly hard about securing Red Hat. Remember, they just put a bunch of stuff in a box; otherwise they're shipping the same software, and often even the same versions of the same software, as anything else.

But switching to a new OS is a pretty big deal because of one rebuild -- once it's running you still need to get it running well, and you've probably got a good pocket of knowledge going on for Windows CF tuning.
posted by mendel at 12:01 PM on September 14, 2006


Is the DB server running on the same box as the webserver? Because that's an easy way to throw hardware at the problem.
posted by Skorgu at 12:04 PM on September 14, 2006


mathowie Re: RedHat I mentioned above that a rather high-traffic, internationally recognized site that I have code running on runs an old version of Red Hat, specifically Application Server.

I don't administer that box, but RH can certainly be secure. They're usually pretty good about their update services.
posted by Skorgu at 12:07 PM on September 14, 2006


Am I the only one who isn't able to see ask.metafilter?
posted by defreckled at 12:23 PM on September 14, 2006


Just out of curiosity, why does the status blog have a 'christmas' theme?
posted by delmoi at 12:47 PM on September 14, 2006


Since the update, when I go to Metafilter using Firefox, I get the "Test Page for Apache Installation" page. Anyone else have this problem? Is this on the server end, or is there something I can do in Firefox to fix this?
posted by Nquire at 3:15 PM on September 14, 2006


It was as if thirty thousand voices all cried out and were suddenly silenced.
posted by dg at 3:47 PM on September 14, 2006 [1 favorite]


For some reason the test page for apache gets cached by firefox for a long time. You need to force a refresh Nquire. Shift-reload.
posted by mathowie (staff) at 4:14 PM on September 14, 2006


Doh! It worked. Thanks Matt!
posted by Nquire at 4:26 PM on September 14, 2006


please do not attempt to adjust your set...
posted by quonsar at 6:54 PM on September 14, 2006


*adjusts his set, sits more comfortably*
posted by stavrosthewonderchicken at 7:11 PM on September 14, 2006


*having trouble adjusting the browser
posted by caddis at 7:33 PM on September 14, 2006


AAAAAEEEERRRRGGGH.
posted by cgc373 at 1:23 AM on September 15, 2006


Ahem.
posted by cgc373 at 1:24 AM on September 15, 2006


« Older Reminder: Auckland, New Zealand meetup   |   Sydney meetup September 2006 Newer »

You are not logged in, either login or create an account to post comments