MetaFilter Offline

December 17, 2013 4:05 AM

MetaFilter was offline from around 9:45 PM Pacific Time to 4:00 AM Pacific Time. A hard drive failed in our database server, taking some of the database with it. We were able to restore from a backup that happened at 7:40 PM Pacific. So we lost data during those two hours. We apologize for that, the long downtime, and the bumpiness as we get things running again.

If you added a post, comment, mefi mail, or favorite after 7:40 PM Pacific last night that data was lost. If you added a MetaFilter post during that time we can recover that text. (The full text of posts are sent to admins vie email.) So drop us a line via the Contact Form and we'll send that to you so you can repost.

posted by pb (staff) to Uptime at 4:05 AM (180 comments total)

Thanks for all your hard work.
posted by turbid dahlia at 4:06 AM on December 17, 2013 [86 favorites]

i was working a double shift, and i literally almost died without it.
posted by empath at 4:06 AM on December 17, 2013 [21 favorites]

I'm just glad we're back. God, that was horrible news to which to wake up. How would I have survived my day?
posted by Mrs. Pterodactyl at 4:07 AM on December 17, 2013

Thank you all so much for putting out the fires. You have my thanks and the thanks of every byte you saved from the bitbucket.
posted by 23 at 4:09 AM on December 17, 2013 [2 favorites]

I see zarq's epic Get Smart post missed it by that much :-(
posted by flabdablet at 4:11 AM on December 17, 2013 [14 favorites]

Yay for backups!
posted by octothorpe at 4:12 AM on December 17, 2013 [1 favorite]

(puts on binocular glasses)

I see zarq's epic Get Smart post missed it by that much :-)
posted by flabdablet at 4:13 AM on December 17, 2013 [5 favorites]

Thanks so much pb.
posted by goodnewsfortheinsane (staff) at 4:13 AM on December 17, 2013 [2 favorites]

Many thanks for the crisis management, pb. (It's also interesting to learn what kind of redundancy systems the mods have in place.)
posted by Doktor Zed at 4:14 AM on December 17, 2013 [1 favorite]

If you did make a post (there were 2-3 between 7-10PM when the site went down) last night, feel free to repost it now.
posted by mathowie (staff) at 4:15 AM on December 17, 2013

I was so lonely
posted by NoraReed at 4:18 AM on December 17, 2013 [14 favorites]

I am amazed at how efficient you are and thank you for updating the status and getting everything working again. I had a comment zooming off just as the darkness descended so I quickly made a note of it to post later. I loved finding out just how important this place is to me. Night time is my time to play and tonight was a little sad and scary. Thank you MeFi for being here and not letting us down.
posted by Anitanola at 4:18 AM on December 17, 2013 [3 favorites]

Thanks! That looked like a lot of work late at night - hope that the bugs weren't just squashed but will never breed again...
posted by Devonian at 4:27 AM on December 17, 2013

Thanks, pb! I'm glad to see the site's back up. I'm also glad the site was restored to where it was, because it got rid of a fighty, deraily comment I made in a thread on the Blue. Score one for accidental, blameless moderation.
posted by Rustic Etruscan at 4:27 AM on December 17, 2013 [5 favorites]

Thanks for all your hard work pb, you're a champ.
posted by Brandon Blatcher at 4:29 AM on December 17, 2013 [1 favorite]

Thanks for working through the night on this, watching this all unfold this morning from the future in Europe all I could think about was the sleep pb must not be getting.
posted by Blasdelb at 4:29 AM on December 17, 2013 [7 favorites]

You all rock and I missed you.

My wife spilled a glass of water on my Macbook around the time MeFi went dark so you can blame her. Now begins 3 days of waiting to see if it dries out ok.
posted by arcticseal at 4:37 AM on December 17, 2013

You do realise I had to spend the morning at work actually, you know, working?! It's quite simply not good enough. I shall expect a partial refund of my £3.31 signup fee at your convenience.


By which I mean: thankyouthankyouthankyouohpleaseneverleavemeagain
posted by billiebee at 4:43 AM on December 17, 2013 [6 favorites]

Oh man, that must have been a total nightmare. Thank you, pb and everyone else who worked on getting things back up and running so quickly.
posted by zarq at 4:49 AM on December 17, 2013 [1 favorite]

Happy Christmas pb.
posted by adamvasco at 4:51 AM on December 17, 2013 [1 favorite]

Thanks everyone. Just a heads up—chat is still offline. We'll get that running again tomorrow.
posted by pb (staff) at 4:51 AM on December 17, 2013

Noooo the Get Smart post didn't make it??

Ah, it did survive. Whew!
posted by JHarris at 4:53 AM on December 17, 2013

We'll live somehow
posted by Blasdelb at 4:53 AM on December 17, 2013

Frustrating that we still have to rely on spinning disks for storage in 2013. Hard drives just suck.
posted by octothorpe at 4:58 AM on December 17, 2013 [1 favorite]

MetaFilter down, and we lost the Ashes as well. End of Days.

Kudos to PB for diligently pulling MetaFilter back. Cook, KP, Swann and Anderson (28 off one over, ffs!) should take lessons from his commitment.
posted by Wordshore at 5:00 AM on December 17, 2013 [2 favorites]

pb, you are awesome as are all the metafilter staff. Thanks for all the hard work!
posted by pointystick at 5:01 AM on December 17, 2013 [1 favorite]

Thank you. And sorry for refreshing the status blog a bazillion times.

I appear to have a little metafilter problem.
posted by you must supply a verb at 5:04 AM on December 17, 2013 [3 favorites]

pb: "Thanks everyone. Just a heads up—chat is still offline. We'll get that running again tomorrow."

Entirely understood! I hope you get some sleep, pb, and thank you very much for all your hard work!
posted by beryllium at 5:04 AM on December 17, 2013 [5 favorites]

I figured it was NSA payback.
posted by spitbull at 5:09 AM on December 17, 2013 [1 favorite]

Wow, I missed the blackout period completely. I'm impressed! So good of you all to schedule this so as to spare us fragile EST'ers.
posted by DingoMutt at 5:10 AM on December 17, 2013 [3 favorites]

i demand a $5 refund…......


......ummm....wait a sec.....nebba mind


carry on
posted by lampshade at 5:11 AM on December 17, 2013

Frustrating that we still have to rely on spinning disks for storage in 2013. Hard drives just suck.

Hilariously, this is all due to a SSD drive we used to build faster search indexes dying. But 1s and 0s still suck.
posted by mathowie (staff) at 5:12 AM on December 17, 2013 [37 favorites]

Any developer or sysadmin knows what miserable goddamn things computers are. Thanks for the hard work. On the bright side, I was super relieved to discover it wasn't my DSL connection again.
posted by middleclasstool at 5:23 AM on December 17, 2013 [2 favorites]

Thanks for all the great work. I rarely comment, but reading this fine, fine site has become such a daily ritual that apparently I get all skeevy and jittery like when I don't have that option.
posted by Phlegmco(tm) at 5:25 AM on December 17, 2013 [5 favorites]

I'm living in the Middle East right now and I was afraid it had been blocked. We're all so codependent:

"Metafilter is down--was it something I did or said? How can I change myself to get Metafilter back?"
posted by mecran01 at 5:28 AM on December 17, 2013 [20 favorites]

Thanks for your hard work! I just noticed an answer I gave in an AskMeta Q disappeared, and I was all, whut? At least I know I wasn't deleted for being a jerk :D
posted by ThePinkSuperhero at 5:28 AM on December 17, 2013 [1 favorite]

Thanks, pb!
posted by kimberussell at 5:28 AM on December 17, 2013 [1 favorite]

Hilariously, this is all due to a SSD drive we used to build faster search indexes dying.

Really? I thought they were less prone to doing that, what with having no moving parts. What circumstances will cause a SSD to die?
posted by JHarris at 5:29 AM on December 17, 2013

Just a heads up for future reference there is the metafilter status blog, which is like somehow hosted separately, that we can always check to see if a problem is from us or the site.
posted by Blasdelb at 5:29 AM on December 17, 2013 [5 favorites]

What circumstances will cause a SSD to die?

Long and deeply annoying story, but half a jar of Lingonberry Jam from Ikea and an ex-housemate with "issues" will do this.

Am hoping the MetaFilter situation was rather different.
posted by Wordshore at 5:32 AM on December 17, 2013 [26 favorites]

Whew. Glad you're back.
posted by rtha at 5:35 AM on December 17, 2013

Goddammit, I'd piss on a spark plug​ if I thought it'd do any good!

(In all seriousness, I've contributed $5 to Metafilter, for nearly a decade of use of the best place on the web. Can I do anything more to help keep it running smoothly?)
posted by DigDoug at 5:38 AM on December 17, 2013 [3 favorites]

The downtime coincided with the portion of my return commute where sitting and watching video on the train is no longer possible. That's the time I usually check Ask, and I was so, so alone. Thanks for putting everything back so quickly. I was sure it was just my iPhone being dumb until I remembered the status blog.
posted by Ghidorah at 5:38 AM on December 17, 2013 [1 favorite]

SSD - Silicon Soon Dies. Live fast, die young...
posted by Devonian at 5:39 AM on December 17, 2013

So, first off - Thank you, Matt and pb, for keeping this place running so well in general, and for getting it back online so quickly last night.

Question, though: Does it make sense to do more frequent backups? What would be the impact of backups, say, every 10-15 minutes?
posted by Chrysostom at 5:45 AM on December 17, 2013 [1 favorite]

What circumstances will cause a SSD to die?

There's a limit to the number of writes to each block but even that takes a long time and if it was used for an index, I'd assume a lot more reads than writes.
posted by octothorpe at 5:49 AM on December 17, 2013

I missed the black out completely so yay me. But I had to chuckle because, as is my usual pattern, I checked Recent Activity where a single comment had been posted and then clicked over to Ask Me where there were only a couple new questions and I was like "What the heck was the site down". Clicks over to Meta. "Yes, Yes it was."
posted by Mitheral at 5:51 AM on December 17, 2013

Partial restoration from screenshots:

7:40 PM: DTMFA
7:40 PM: DTMFA
7:40 PM: We've all dated that guy. DTMFA.
7:40 PM: eat it, it's fine
7:41 PM: I don't usually say DTMFA, but DTMFA.
7:41 PM: As long as it smells ok
7:41 PM: DTMFA
7:42 PM: Four weeks?! I've had some of that in the back of my fridge for four years. I plan on serving it to guests on Christmas Eve.
7:42 PM: People here are always so quick to say DTMFA, but in this case I think they're right.
7:43 PM: DTMFA
posted by Flunkie at 5:54 AM on December 17, 2013 [19 favorites]

Is chat still down for others, or is this a problem on my end?
posted by Now there are two. There are two _______. at 6:05 AM on December 17, 2013

12/16/13 NEVER FORGET
posted by Jacqueline at 6:06 AM on December 17, 2013 [6 favorites]

Headline from my tech "news" tab this morning: Don't Believe Backup is Broken? Think Again

Get used to data evaporating, the current best dirty little secret of the amazing 4TB drives and huge fast SSD, very hard and expensive to back up, and then restoring TB's of data, sloooow.
posted by sammyo at 6:06 AM on December 17, 2013

Thank you, pb, for being the dude who is on this shit so I can just go to bed and hope things are better when I wake up.
posted by cortex (staff) at 6:08 AM on December 17, 2013 [18 favorites]

wow

such offline


many sad happy now
posted by Namlit at 6:09 AM on December 17, 2013 [12 favorites]

Chat will be fixed tomorrow, pb has gone to get some sleep.

For those who want to help maintain the site, I've donated in the past when I've felt flush for Matt to buy doughnuts, beer and whatnot for the mods.
posted by arcticseal at 6:10 AM on December 17, 2013 [1 favorite]

Thanks pb, thanks arcticseal, and sorry for my lousy reading comprehension.
posted by Now there are two. There are two _______. at 6:12 AM on December 17, 2013 [1 favorite]

IT'S ALIIIIIIIVEE again!!

Thanks, from another sysadmin.

(Hey, do we get a double helping of Favorites today? Because I, uh, used a full day's worth just before the crash. Honestly.)
posted by wenestvedt at 6:14 AM on December 17, 2013 [1 favorite]

How frustrating that must have been. Server hard drive failures suck.

Glad you were able to get everything up and running again.
posted by Deathalicious at 6:25 AM on December 17, 2013

Backups: I know the process would be slow but can we submit a Freedom of Information request to the NSA for the missing posts?
posted by sammyo at 6:27 AM on December 17, 2013 [4 favorites]

Dear AskMe, I have a relationship with this special site. It educates, enlightens, and infuriates me. We sometimes run hot and cold, but I always come back, and the site is waiting for me. Then, suddenly, at 9:45 PM Pacific Time, the site went dark. Was it something I did? How can I fix this? Please do not tell me to see a therapist; I am seeing two.
posted by GenjiandProust at 6:29 AM on December 17, 2013 [8 favorites]

Thank you mods, pb, and all other behind the scenes peeps, for fixing everything so quickly.
posted by Faintdreams at 6:31 AM on December 17, 2013

Me: Metafilter is down, but status blog says they are fixing it.

Boyf: Oh well that's basically the entire internet broken for you then.

Me: Nope it's okay ::lip quivers::
posted by Faintdreams at 6:33 AM on December 17, 2013 [25 favorites]

I was reading on my phone in bed when the outage hit (11:45 my time). I figured it was a sign from God to go to sleep, so that's the reason I'm coherent this morning. Blessing in disguise.
posted by desjardins at 6:33 AM on December 17, 2013 [5 favorites]

We couldn't even make it one lousy year between outages. Honestly, what are we paying you people for?

Everyone needs a hug, my ass.
posted by Etrigan at 6:36 AM on December 17, 2013 [4 favorites]

This Is Just to Say

I have damaged
the SSD
that was in
the server rack

and which
you were probably
saving
for Metafilter

Forgive me
It was del.icio.us
so 1337
and so old
posted by chavenet at 6:43 AM on December 17, 2013 [60 favorites]

My morning routine includes looking at the Metafilters before I get ready for work in the morning. I was early to work today and I blame you guys.

Glad to see it is up and running again.
posted by sciencegeek at 6:47 AM on December 17, 2013 [1 favorite]

So, opening a new tab and checking Mefi is like my ingrained default action when on the web late at night, so much that "Oh no, Mefi's down! What should I do?" *new tab, check Mefi, like I've got goldfish memory* happened multiple times last night.
posted by jason_steakums at 6:48 AM on December 17, 2013 [16 favorites]

The terrifying thing about SSDs is that when they fail, they tend to fail suddenly and without warning. You don't get the clicking and searching that is the HDD's faithful warning to back up your data and replace the drive. Just... nothing.
posted by running order squabble fest at 6:50 AM on December 17, 2013 [2 favorites]

desjardins: "I was reading on my phone in bed when the outage hit (11:45 my time). I figured it was a sign from God to go to sleep, so that's the reason I'm coherent this morning. Blessing in disguise."

Heh. My daughter woke me up at 2:30am to tell me that she had taken off her socks because her feet were too hot and now her feet are cold and she can't find her socks anywhere in her bed and maybe they crawled out of her bed while she was asleep because sometimes her socks like to hide and could i please, please get her another pair from her sock drawer because it's too dark in her room and even though she opened the drawer she can't see in the drawer because it's dark and she wants mismatched socks not ones that match because that would be boring. *gasp for air* But they need to be pink, both of them pink but two different kinds.

So I got up, using my phone as a flashlight. And I gave her the #^&%$$! socks. And of course on the way back to bed I checked mefi and it was down. I also took this as a sign that my daughter is an evil, sockless insomniac it was time to go back to bed.
posted by zarq at 6:54 AM on December 17, 2013 [36 favorites]

"Less prone to failure" does not mean "never fails", yeah.

Boyf: Oh well that's basically the entire internet broken for you then.

Me: Nope it's okay ::lip quivers::

Even knowing it was down and that I should wait awhile, I found myself clicking on either Metafilter or AskMe about every 2-3 minutes until I went to bed last night, and even though I knew logically there was other stuff out there I could look at, I kept coming back to it.
posted by Sequence at 6:55 AM on December 17, 2013

Thanks to the team for the hard work.

Sometimes I wonder when these things happen what this place would be like if there had to be a complete reboot. Like if there was no database, no code, no graphics, and it had to reconstructed from scratch. New very first post, new signup process, only people still interested and active would be signing up. The archive of posts and questions gone, so no doubles, no one bragging about their low user numbers, etc.

How long before something was functioning again? What assumptions would be changed? What choices would be made. Which sections would get priority. What would never be revived?

Some things would be possible to reconstruct from people's caches (like the graphics), but pretend nothing was. What would be done differently?

Consider this a case study. I'll take my answer off the air.
posted by cjorgensen at 6:56 AM on December 17, 2013 [3 favorites]

pb, thanks for restoring my morning!
posted by Sophie1 at 7:00 AM on December 17, 2013

I'm not putting my 4 pun tennis movie comment back as a protest. I'm not sure what exactly I'm protesting, but I'm protesting.



Also, thanks for getting things back up and running. I hope somebody is getting a well deserved rest now.
posted by Nanukthedog at 7:09 AM on December 17, 2013

> 12/16/13 NEVER FORGET

You misspelled NEVAR.

Also, thanks to pb and all the hell-demons he contracts the work out to! (Come on, how else do you think he could possibly accomplish all that?)
posted by languagehat at 7:13 AM on December 17, 2013 [4 favorites]

Hah, I just checked the status.metafilter.com site and ... Metafilter is still "currently offline".
posted by chavenet at 7:14 AM on December 17, 2013

mathowie: 1s and 0s still suck.

I'll get you guys a big bucket of 2s and 3s for Northern Hemisphere Winter Holiday/Southern Hemisphere Summer Celebration!

And thanks to the mods for helping out with those of us who didn't read this MetaTalk and were baffled at the disappearance of their activities last night.
posted by filthy light thief at 7:15 AM on December 17, 2013

Hah, I just checked the status.metafilter.com site and ... Metafilter is still "currently offline".

Wait, am I a figment of your imagination, or are you a figment of mine?
posted by Etrigan at 7:36 AM on December 17, 2013 [1 favorite]

Thanks for fixing everything, pb! Boy, seeing this happen reminds me that MetaFilter being offline rarely happens anymore, it's not like the old days of random Jrun errors. So thanks, Matt, for hiring pb and organizing everything so that the site is such a dependable home to so many.
posted by onlyconnect at 7:39 AM on December 17, 2013

Good work getting it back, guys!
posted by Artw at 7:39 AM on December 17, 2013

Only after I re-started my phone for the second time this morning did it occur to me to check a website OTHER than metafilter, AskMeFi, or MetaTalk to see why nothing would load.

Thanks for your hard work, pb!
posted by Fig at 7:48 AM on December 17, 2013 [5 favorites]

*Flings self on MeFi to give it an embarrassingly snotty, tearful, hysterical hug*

Oh thank God you're baaaaack. I was so worried. And on the way to work I had to stare out of the bus window at the real world. Sniff snifffff...

*wipes nose on back of shaky hand, tries to pull self together*

And thanks pb and mathowie and everyone else. You guys...

*moves towards the guys for the next hysterical hug*
posted by penguin pie at 8:04 AM on December 17, 2013 [5 favorites]

Insomnia sucks. Insomnia with no job yet is worse. Insomnia with no job and no metafilter... what is the world coming to?

Thanks for fixing the last of those, pb. Any hope on the first two?
posted by nat at 8:11 AM on December 17, 2013 [1 favorite]

Try adding more decorative gears and copper plating.
posted by nathancaswell at 8:13 AM on December 17, 2013 [2 favorites]

Three cheers for pb as usual! What a night - hope you are getting some sleep today.
posted by LobsterMitten (staff) at 8:30 AM on December 17, 2013

I slept through the whole thing. Glad the loss was minimal. Thanks for doing the heavy overnight lifting getting it working.
posted by jessamyn (staff) at 8:41 AM on December 17, 2013 [1 favorite]

I'm not putting my 4 pun tennis

Your 4 ton p..., what!? That's gigantic!
posted by ambrosen at 8:44 AM on December 17, 2013 [2 favorites]

If only the site went down every night I would consistently get my full 8 hours of sleep. It was like, huh, MetaFilter's down... guess I'll... uh... go to bed or something.
posted by rabbitrabbit at 8:44 AM on December 17, 2013 [10 favorites]

My comments! My precious comments are lost! You heartless bastards! Gone like so many tears in the rain, I can never get them back! You have betrayed me for the last time MetaFilter!!!

*converts PST to EST*

Oh, I was sleeping the entire time. Sooo...hunh.

Thanks, guys!
posted by slogger at 8:53 AM on December 17, 2013

Yeah, I had the usual bout of insomnia 4:30-ish, grabbed the phone off the nightstand, hit Safari & panicked - was I going to have to risk waking up the entire household to go re-boot the router? Then I hit status.metafiler.com (which I probably typed wrong) and IT didn't load, & was all oh, shit! but Twitter worked, then status loaded, then the tweets nearly lulled me back to sleep, in time for the coffee maker to do its thing at 5:45.

But really! If you guys could give a little more advance notice so I can adjust my meds on nights when there's gonna be a server outage, I would really appreciate it. PROTIP: Have DiskWarrior email you when it detects a S.M.A.R.T. status irregularity! (hamburger, etc. sorry, I'm punchy due to lack of sleep)
posted by Devils Rancher at 9:04 AM on December 17, 2013

My comments! My precious comments are lost! You heartless bastards! Gone like so many tears in the rain, I can never get them back! You have betrayed me for the last time MetaFilter!!!

I can totally hear this in a Peter Lorre voice. You have to hold your hand out stiffly, like you're holding an invisible globe, though to make it really work.
posted by Devils Rancher at 9:10 AM on December 17, 2013 [2 favorites]

Thank you, pb for this and all the other services you perform. Just like Santa (in the Northern Hemisphere, anyway), you're doing amazing things while we sleep. ♥
posted by Lynsey at 9:11 AM on December 17, 2013 [1 favorite]

1s and 0s still suck

I'll get you guys a big bucket of 2s and 3s

I felt a great disturbance in the 4s... as if millions of favorites suddenly cried out in terror and were silenced.
posted by It's Raining Florence Henderson at 9:12 AM on December 17, 2013 [10 favorites]

Thank you for all your diligent work and care!
posted by Elsa at 9:13 AM on December 17, 2013

Awwww man, I wrote a detailed and well-researched comment for restless_nomad's AskMeta question and I'm not sure I'm up to recreating it. Ok deep breath, and yes, thank you, pb for all your hard work to restore the site. So grateful you're willing to work through the night on this kind of thing!
posted by librarylis at 9:13 AM on December 17, 2013

I blame Shia LaBoeuf.
posted by Pruitt-Igoe at 9:14 AM on December 17, 2013 [4 favorites]

Thank you, pb! And thanks for letting us know. I had been wondering why my comment was "deleted" until I saw this.
posted by salvia at 9:21 AM on December 17, 2013

I feel bad for wasting Jessamyn's time asking about why my comment in the Shia LaBeouf thread got deleted (because I apparently don't have anything more important on my plate this morning...?) but I'm sure glad I used the contact form instead of posting a question to MetaTalk.
posted by nanojath at 9:21 AM on December 17, 2013 [1 favorite]

chat should be back online.
posted by pb (staff) at 9:33 AM on December 17, 2013 [2 favorites]

Is this the place where we complain about the downtime because we paid for this service goddammit and there are expectations?

No?

Good.
posted by davejay at 9:42 AM on December 17, 2013 [1 favorite]

chat should be back online.

thankyouthankyouthankyou

based on the time period you had to be awake, i was expecting this maybe in the afternoon some time after you got a decent 6 hrs or something. above and beyond! as per usual.
posted by twist my arm at 9:43 AM on December 17, 2013 [1 favorite]

I slept through the whole thing.

It was horrible.
posted by arcticseal at 9:56 AM on December 17, 2013 [3 favorites]

chat should be back online.

Maybe if the guy in charge of the magic internet boxes knew what "tomorrow" meant, we wouldn't be having these problems.
posted by Etrigan at 9:57 AM on December 17, 2013 [3 favorites]

So is this related to why the search functionality was broken just before the site went down?
posted by carolr at 10:13 AM on December 17, 2013

I was busy doing rl stuff and missed it!
posted by fearfulsymmetry at 10:28 AM on December 17, 2013

What a relief. I thought you'd all up and moved somewhere else and not told me. Which is the STORY OF MY LIFE.
posted by Decani at 10:44 AM on December 17, 2013 [2 favorites]

On the plus side, I finally got an "It's not just you!" message from downforeveryoneorjustme.com.
posted by Celsius1414 at 10:47 AM on December 17, 2013 [6 favorites]

What a relief. I thought you'd all up and moved somewhere else and not told me. Which is the STORY OF MY LIFE.

What a coincidence! My one lost (recently restored) comment was that Ted Chiang's novella STORY OF YOUR LIFE is getting turned into a movie soon. I suspect the incredibleness of that news is what crashed MeFi. Apologies.
posted by Rhaomi at 10:50 AM on December 17, 2013 [2 favorites]

Hey, so when I hit "back" on my iPhone(iOS7), MetaFilter isn't remembering where on the page I was anymore. It takes me back up to the top instead. Is this related to your problems last night?
posted by Weeping_angel at 10:53 AM on December 17, 2013

Never mind, it's working normally now. Weird.
posted by Weeping_angel at 11:00 AM on December 17, 2013

It was like, huh, MetaFilter's down... guess I'll... uh... go to bed or something.

As our boys Bret and Jemaine suggest:

"Conditions are perfect: there's nothing good on TV Metafilter is down...conditions are perfect."
posted by maxwelton at 11:11 AM on December 17, 2013

Glad to know its done.
posted by Nanukthedog at 11:14 AM on December 17, 2013

I didn't even notice, I'm so happy I bought that hand cranked metafilter.
posted by mannequito at 11:15 AM on December 17, 2013 [3 favorites]

pb's a jolly good fellow, pb's a jolly good fellow, pb's a jolly good fellow, and so say all of us!
posted by homunculus at 11:52 AM on December 17, 2013 [3 favorites]

I had just posted a comment last night criticizing General Alexander to the NSA thread when suddenly I couldn't access metafilter. And upon checking it today, my comment (and at least one other) was gone. Feeling real paranoid for about 5 minutes!!

I'm going to repost the comment and see if your supposed "hard drive" fails on your supposed "database server" again.
posted by NiceKitty at 12:02 PM on December 17, 2013 [1 favorite]

What an NSA thread might look like.
posted by NiceKitty at 12:03 PM on December 17, 2013

Much respect, pb.

Also, yeah, yeah HD failures happen and then you have to restore from backup, but the fact that a huge, solidly performant site like MetaFilter does not have database clustering affirms my crazy idea that not every web app needs to have a backend with DB clustering. Sure, it's cool, but they have their costs, and just maybe they don't have to be gospel for everything.
posted by ignignokt at 12:20 PM on December 17, 2013

Whoa, I just subtweeted outside of Twitter.
posted by ignignokt at 12:21 PM on December 17, 2013

mannequito: I didn't even notice, I'm so happy I bought that hand cranked metafilter.

Ooh, the rare hand cranked MeFi, the precursor to the Wiki Reader.
posted by filthy light thief at 12:28 PM on December 17, 2013

When I first noticed MeFi was down, I had a moment of terror that the IT guys at work had finally cottoned on to how I spend the bulk of my workdays. I was so relieved when I checked the status blog.
posted by paleyellowwithorange at 12:45 PM on December 17, 2013 [4 favorites]

Many thanks for your work, pb.
posted by gauche at 12:51 PM on December 17, 2013 [1 favorite]

Thanks pb! Hope you are vaguely caught up on sleep!
posted by leslies at 1:05 PM on December 17, 2013

On the upside, the site is now statistically safe from a hardware outage in the middle of the holidays.
posted by Dr Dracator at 1:16 PM on December 17, 2013 [1 favorite]

Dr Dracator: "On the upside, the site is now statistically safe from a hardware outage in the middle of the holidays."

Are you out of your mind?!? Go outside right now, turn around three times and spit, curse, do all the things.

*grumble* Kids these days. Tempting the wrath of the whatever from high atop the thing. *grumble*
posted by zarq at 1:38 PM on December 17, 2013 [13 favorites]

Now he has to wear his pajamas forward.
posted by Etrigan at 1:40 PM on December 17, 2013

Thanks!

That was like quitting smoking all over again.
posted by From Bklyn at 1:41 PM on December 17, 2013 [2 favorites]

Relevant Confession Bear.
posted by paleyellowwithorange at 1:51 PM on December 17, 2013 [2 favorites]

I left a web browser window open on the AskMeFi index page when I went to sleep last night, so I have preserved the AskMeFi questions that were lost. They are as follows:

I Don't Know How to Dress Myself
My "Partner" is an Asshole
Look At This Thing I Do - I'm So Quirky and/or Adorkable
My "Partner" is a Bitch
Should I Yell at my Boss and Then Quit My Job?
My Pet is a Shithead
Where to Find Artisanal Toilet Paper in Williamsburg?
Can Anyone Translate This Novel For Me?
posted by Tanizaki at 2:07 PM on December 17, 2013 [4 favorites]

I feel like you didn't quite manage to keep the wheel on the right side of the joke/jerkiness divide, there.
posted by cortex (staff) at 2:10 PM on December 17, 2013 [12 favorites]

Where to Find Artisanal Toilet Paper in Williamsburg?

Let's answer that right here. Right beside the guy who makes boxwood rat traps on his turning lathe While You Wait, there's a corner shop where they sell handmade toilet paper arcs and a special spool mechanism for you to assemble your own rolls.

You have the choice between various qualities:
Beethoven ("Rough" and "Harmonious". "Deafening" is out of stock)
Cicero
Sigmund F.
Isadora Duncan
Bart Simpson,

There's a possibility to order custom-made, too; delivery three weeks (which is problematic).
posted by Namlit at 2:16 PM on December 17, 2013 [3 favorites]

I Don't Know How to Dress Myself

To be fair to Tanizaki, I have a couple of AskMes cued up and this pretty much describes one of them. (Thanks for the question title!)
posted by billiebee at 2:47 PM on December 17, 2013 [3 favorites]

On the upside, the site is now statistically safe from a hardware outage in the middle of the holidays.

That's not how statistical probabilities work...
posted by Jacqueline at 3:02 PM on December 17, 2013 [1 favorite]

You have the choice between various qualities:
Beethoven ("Rough" and "Harmonious". "Deafening" is out of stock)
Cicero
Sigmund F.
Isadora Duncan
Bart Simpson,

No Tilda Swinton?
posted by Tanizaki at 3:58 PM on December 17, 2013

I gave up and read a book until bedtime...but felt like I was somehow betraying an old friend.
posted by Greg_Ace at 4:14 PM on December 17, 2013

No Tilda Swinton?

It's a very small shop; they have to be selective. Nothing too obvious - it's not for the crowds.
Inside rumors have it, though, that a Sherlock version is on its way. High-performance and super fast, but abrasive.

(They tried Legolas, but it wasn't a hit. Neither was Samwise Gamgee. "More Lembas bread," I'm asking you)
posted by Namlit at 4:25 PM on December 17, 2013

SSDs actually kind of suck in a lot of server applications especially with RAID, they don't take a good beating for very long (2 years maybe) like a 15K SAS drive might (4-5 years). The flash doesn't hold up to being written all of the time due to the nature of the silicon / circuitry design and wear leveling algorithms are necessary to keep them from dying prematurely.

We're looking at a storage system that uses them for caching for fast reads and writes but ultimately stores all data on spinning SATA drives. You gotta replace those SSD suckers every 1-2 years, at least the grade we're seeing.

For me the outage happened immediately as I was clicking on the latest NSA story. I was also stoned out of my vulcan mind. You do the math
posted by lordaych at 5:08 PM on December 17, 2013 [5 favorites]

(on my next thoughts that is)

Then the Google Cache worked and I was like "oh yeah Google Cache baby" *cough*
posted by lordaych at 5:10 PM on December 17, 2013

Thanks for all the hard work!

And a . for the lost data.
posted by nubs at 6:10 PM on December 17, 2013

Where to Find Artisanal Toilet Paper in Williamsburg?

In Colonial Williamsburg there isn't any other kind.
posted by George_Spiggott at 6:12 PM on December 17, 2013 [3 favorites]

ooompf
posted by Namlit at 6:24 PM on December 17, 2013

Yeah! What the heck! In my 3:30am panic mode awake in the dark time I need to check my Metafilter!! Actually maybe I was checking it in my "bored at work" time. Anyway, at no time is it acceptable for MF to not be functioning! Ok?? Thank you! PS I must learn how to convert time zones...
posted by bquarters at 6:26 PM on December 17, 2013

Namlit: "No Tilda Swinton?

It's a very small shop; they have to be selective. Nothing too obvious - it's not for the crowds.
Inside rumors have it, though, that a Sherlock version is on its way. High-performance and super fast, but abrasive.

(They tried Legolas, but it wasn't a hit. Neither was Samwise Gamgee. "More Lembas bread," I'm asking you)"

Trust me from experience: stay away from the Captain Hook.
posted by double block and bleed at 6:27 PM on December 17, 2013 [2 favorites]

At least the 2372 Mefites who favourited my comment on the nature of life, christmas pastries, and the platonic ideal of cat fluffiness got their $5's worth before my contribution was lost in the ether.
posted by ersatz at 6:43 PM on December 17, 2013 [3 favorites]

I am just going to pretend that the grumpy probably-off-topic answer I left in AskMe fell into the data hole instead of being deleted. The timing is almost right. No great loss there.

Thanks, pb! (For restoring MeFi at great effort, not for providing me a cover story.)
posted by gingerest at 7:06 PM on December 17, 2013

You know those things you hear about where reddit decides to gang up and send a zillion pizzas to someone for some reason?

We should all find a way to have beers delivered to pb to thank him for his hard work on our behalf.
posted by hippybear at 7:38 PM on December 17, 2013 [1 favorite]

Jeff Atwood's hot/crazy analogy for SSDs is still pretty accurate.
posted by holgate at 8:08 PM on December 17, 2013

Is this why I'm getting weird results when I click on things in Inoreader? For example I see in Inoreader that I'm going to get a post about "Betty Dodson's Feminist Sex Wars," but clicking on it takes me to a post about Christmas cookies for singles (which looks great, I'm not complaining, but it's odd).
posted by The corpse in the library at 8:17 PM on December 17, 2013

That Dodson post was one that was deleted last night, and then its existence was erased in the failure... so I imagine its post number got reassigned?
posted by LobsterMitten (staff) at 8:24 PM on December 17, 2013

cjorgensen: "Sometimes I wonder when these things happen what this place would be like if there had to be a complete reboot. Like if there was no database, no code, no graphics, and it had to reconstructed from scratch. New very first post, new signup process, only people still interested and active would be signing up. The archive of posts and questions gone, so no doubles, no one bragging about their low user numbers, etc."

I'm pretty sure that I'm not the only one that would have a less stupid user name. Sadly, I'm too attached to my pre-$5 user number and favorites count to change the name that guarantees me multiple typos in every. single. fucking. thing, I post.
posted by stet at 8:25 PM on December 17, 2013

Yeah, the post ID would have been reused. So your newsreader must have picked up and stored the post before the site went offline and doesn't check for updates. There are probably four or five IDs that could happen with and then you'll be set.
posted by pb (staff) at 8:28 PM on December 17, 2013

Jeff Atwood's hot/crazy analogy for SSDs is still pretty accurate.

Neatest way I've seen for enjoying the hot without being hurt by the crazy is to one as an external journal for a disk filesystem.
posted by flabdablet at 8:36 PM on December 17, 2013

I was gobsmacked last night to discover Metafilter really was down and my cable company wasn't just screwing up again. Bright House, my ass. More like dim bulbs...grumble grumble.

Anyway, then I realized the only logical explanation was that after months of attempting to win us all over, the quidnunc kid had finally taken matters into his own hands with a well-timed coup d'état. Just when the most prolific users are basking in holiday good cheer, munching and clicking on cookies, in he swoops. Arab Spring becomes Mefi Winter! Dogs and cats living together! Mass hysteria!

As a loyal Matthowier, I for one am thrilled you were able to stop that barbarian storming the gates! I imagine all the torches and pitchforks lying around on the grey came in handy.

And kudos on the cover story! "SSID failure"! Not bad, not bad at all. Given the time pressure you were all under to get things up and running without unduly panicking the masses, you are to be commended. The nonsensical technical jargon really sells it.

You may want to work on the logistics a bit more, though. In the event another attempted incursion comes along, the mythical wunderkind 'pb' bot will garner greater credibility when you remember that real humans need to sleep occasionally, ha ha. Still, job well done and all that.

Keep Calm and Filter On.
posted by misha at 8:45 PM on December 17, 2013 [4 favorites]

I see zarq's epic Get Smart post missed it by that much :-)

Ah, the ol' post-a-great-Get-Smart-FPP-and-then-take-down-the-server-to-win-the-contest trick.
posted by ceribus peribus at 8:54 PM on December 17, 2013 [1 favorite]

I literally was in bed whispering *oh noes, save us pb* after I put my laptop away until I fell asleep.

yay it worked
posted by Unicorn on the cob at 9:37 PM on December 17, 2013 [1 favorite]

These details on the failure rates of SSDs reported in that Coding Horror post are troubling, and even more troubling is reading a proponent of the things pooh-pooh the fail rate with an argument like "they're so blazing fast you won't care." Not everyone can afford to buy a new hard drive every couple of years, especially ones as expensive as SSDs! It seems Linus Torvalds was bitten by this problem this past September.

But now I'm wondering why fail rates are so high. The only reason I've seen in the thread up to this point is the one I was previously aware of -- individual flash memory cells have a limited number of write cycles. But the figure I had seen before is that each should be good for tens of thousands of writes, and with load balancing that should, under normal use, be good for a long time. For a server application like Metafilter, yeah, lots of changes, but why should it become unreadable once too many cells fail from writes? Shouldn't the failure mode for SSDs be something less catastropic?

Of course I'm not a hard drive technician, but these kinds of failure rates seem like they should be much more widely known, and would be a deal breaker to many. And mobile devices use this kind of storage don't they?
posted by JHarris at 9:46 PM on December 17, 2013

And mobile devices use this kind of storage don't they?

I'd guess the expectation there is that devices will be replaced by newer models every year or two anyway (OMG my $LastYear'sModel phone won't be updated to $LatestAndroidVersion, I must upgrade to stay relevant!!!1!!), so the potential failure rates don't end up mattering enough to approach major Deal-Breaker status. The mobile version of Planned Obsolescence, in other words.
posted by Greg_Ace at 10:27 PM on December 17, 2013 [1 favorite]

The only reason I've seen in the thread up to this point is the one I was previously aware of -- individual flash memory cells have a limited number of write cycles. But the figure I had seen before is that each should be good for tens of thousands of writes, and with load balancing that should, under normal use, be good for a long time.

What. Look, will you guys all stop with your nonsense? Cells and write cycles and load balancing indeed. We all know that the interwebs is actual magik, with information being smuggled into our computer things by teeny tiny faeries. Nothing else makes sense. I mean, sometimes the computers aren't even connected to anything!!??!!? (I'm no technical expert, but I believe they live under the keyboard. How else would they know what we're all saying?) They clearly went on strike due to pay and conditions, and pb had to do some serious negotiating. Glad you got them back on board!
posted by billiebee at 1:06 AM on December 18, 2013 [3 favorites]

Thanks, pb! I'm at my parents' house for the holidays and yesterday morning I had to actually interact with people -- at least today I will be safe!
posted by daisyk at 1:31 AM on December 18, 2013

We all know that the interwebs is actual magik

I rather suspect that the interwebs are looking at me, thinking "is this life form real? Am I just making this up? Hey whoops, there it moves again. Hmmm. What does it do? Must be magic." And so on.
posted by Namlit at 2:15 AM on December 18, 2013

Thanks pb and all the others behind the scenes!

I was witnessing the whole thing from the future in euroland. It was sad.
The status blog and Twitter helped a tiny bit, but when the site came up and disappeared again I knew I had to actually find something else to do - so I worked. Already replaced the one comment and one favorite that was lost. :-)
posted by travelwithcats at 2:29 AM on December 18, 2013

What circumstances will cause a SSD to die?

Electricity, air, water. This applies to most things, actually.
posted by bongo_x at 2:47 AM on December 18, 2013

Thanks pb and all the others behind the scenes!

At this point I should probably specify that my involvement with this particular epic was almost entirely limited to sitting at a desk with a Gmail browser window open, trying to think of new variations on "hang in there pb" and "ooh that sounds bad pb" that would still be meaningful. I did at one point briefly consider emailing a photograph of a kitten hanging on to a clothesline. Then I thought of just typing "hangintherekitty.bmp" in a monospace font. I must admit I was *this* close to "I reiterate my statement of support and yield to the gentleman from Oregon". In the end I think I just went with "take it easy there pb".
posted by goodnewsfortheinsane (staff) at 3:09 AM on December 18, 2013 [13 favorites]

Thanks for those, gnfti. There's nothing like catastrophic failure to get your adrenaline going which isn't always helpful. So a reminder of that once in a while is very helpful.

Does it make sense to do more frequent backups?

We do, but the backups made during the drive failure were not restoring properly. So we had to go back in time a bit to get the most recent reliable backup. We have transaction log backups every 10 minutes, so in theory we should be able to restore to any 10 minute increment. The theory didn't work out with the way this failure worked.
posted by pb (staff) at 8:12 AM on December 18, 2013 [1 favorite]

mobile devices use this kind of storage don't they

I believe that mobile operating systems do most of their volatile I/O in RAM, and are generally pretty brutal about killing processes to accommodate others. No swap: they'll write some state data, but that's it.

Desktop/server OSes are swappier (and log more) by nature, even if you chuck RAM at the problem: that, I suspect, is one motivation behind Apple's 'compressed memory' feature as it moves all its storage to solid state. We're also in a transitional period for those OSes where filesystems are still built with spinnydisks in mind, but with add-ons like TRIM on the driver level. Eventually, we'll see reimplementations that strip out all the logic about disk geometry.

The SSD failures I've seen around me have tended to be filesystem corruption crapouts where no amount of recovery superblocks and whatnot save the day.
posted by holgate at 8:43 AM on December 18, 2013 [1 favorite]

> As a loyal Matthowier, I for one am thrilled you were able to stop that barbarian storming the gates!

As one of the quidnuncians, I am still grumbling about the failure of the coup attempt. We were so close! The "NUMBER ONE" flag was being raised above the server and everything. We would have gotten away with it, too, if it hadn't been for that meddling pb!
posted by languagehat at 8:50 AM on December 18, 2013 [10 favorites]

This is what you splitters get for abandoning the democratic process. I for one will continue to vote #1 quidnunc kid as I have always done.
posted by Rustic Etruscan at 8:56 AM on December 18, 2013 [2 favorites]

Hahahaha, I was like "yikes, I guess those comments came off a lot more offensive than I meant them...?" Server outage, not deletion. Whew.
posted by en forme de poire at 12:51 PM on December 18, 2013 [1 favorite]

You misspelled NEVAR.

posted by languagehat at 7:13 AM on December 17 [+] [!]

This proves what I've long suspected: languagehat is secretly a prescriptivist.
posted by medusa at 1:35 PM on December 18, 2013 [3 favorites]

Time to round up the scapegoats for a ceremonial execution. BRAND NEW DAY!
posted by blue_beetle at 2:28 PM on December 18, 2013 [1 favorite]

Electricity, air, water. This applies to most things, actually.

Yeah, but those things don't cause CPUs or RAM to die with the kind of regularity we see for SSDs. Is it heat? Something inherent to non-volatile RAM?

At least with hard drives it's easy to imagine a spinning mechanical thing breaking down.
posted by Pruitt-Igoe at 4:33 PM on December 18, 2013

Well this thread has taught me a lot about SSD fallibity, although ultimately it comes on the bowed back of pb. So thanks, doubly, for your efforts there! Have a Unicode hamburger on me: 🍔
posted by JHarris at 5:14 PM on December 18, 2013

Time to round up the scapegoats for a ceremonial execution. BRAND NEW DAY!

what kind of brand new day is it when you all are the reactionaries

and what am I even doing here I'm one of the Democratic Quidnun-
posted by Rustic Etruscan at 9:25 PM on December 18, 2013 [2 favorites]

Something inherent to non-volatile RAM?

Something inherent to flash memory. There's an oxide layer in the memory cells that's used to trap the electrons in whichever state they're supposed to be in that degrades over time.
posted by empath at 10:12 PM on December 18, 2013

I was gobsmacked last night to discover Metafilter really was down and my cable company wasn't just screwing up again.

Sadly both were true here (and I still haven't managed to fix my internet.)
posted by Jahaza at 7:51 AM on December 19, 2013 [1 favorite]

This is why we need a microfiche edition of MetaFilter. Archival analog film FTW, yo!
posted by sonascope at 12:19 PM on December 19, 2013 [2 favorites]

If you added a post, comment, mefi mail, or favorite after 7:40 PM Pacific last night that data was lost.

B-b-but... it was my best comment ever!
posted by mazola at 11:00 PM on December 19, 2013

No, it was just a tribute.
posted by Etrigan at 5:06 AM on December 20, 2013 [1 favorite]

Looks like there was another brief outage just now, I confirmed with isupme. I will attribute this outage to MeFi's bad karma due to a very poor moderation decision just moments before the outage.
posted by charlie don't surf at 2:24 PM on December 20, 2013

I'm sure you do.
posted by jessamyn (staff) at 2:26 PM on December 20, 2013 [2 favorites]

I will a tribute of this to the Grand Crusader.
posted by It's Raining Florence Henderson at 2:31 PM on December 20, 2013

There is a big difference between being a moderator and a martinet.
posted by charlie don't surf at 2:36 PM on December 20, 2013

Seriously dude, you know what all the ways are you can contact us or open a discussion about moderation in specific or in general.
posted by jessamyn (staff) at 2:40 PM on December 20, 2013 [2 favorites]