Server Snafu January 9, 2010 1:47 AM   Subscribe

We were performing some scheduled maintenance on the server tonight, and lost some data over at Ask MetaFilter.

We were able to restore from our backup last night at 2am, but we are missing all Ask MetaFilter comments posted between Jan 8th at 2am to Jan 8th at 11pm (Pacific Time). We're working on restoring them from Google's cache. If anyone out there has a full copy of Ask comment RSS from this time period, we could use that to restore too.

We apologize for this huge headache, and working to restore what we can.
posted by pb to Uptime at 1:47 AM (104 comments total)

AHHH I CAN BREATHE AGAIN!
posted by koeselitz at 2:35 AM on January 9, 2010


Yeah for a bit there I could get nothing but error messages, and not having my favorite website handy freaked me out a bit.
posted by vrakatar at 2:45 AM on January 9, 2010


Is it just me or are all the threads on AskMe showing 0 answers, even ones that have the best answer checkmark?
posted by IndigoRain at 2:50 AM on January 9, 2010


Wow, 0 answers goes 2 1/2 pages back. That's a lot of lost comments. :(
posted by IndigoRain at 2:51 AM on January 9, 2010


Wow, how... weird. It's like what Ask would look like if it were unpopular or something.

Anyway, I'm sure that was an unpleasant night for you pb. Thanks for doing whatever you could do and bringing us back to the land of the living.
posted by DarlingBri at 2:53 AM on January 9, 2010 [11 favorites]


Oh, for one of the affected questions to have been "How do I get people to stop ignoring me?"
posted by genghis at 2:53 AM on January 9, 2010 [9 favorites]


Whoa, glad I came here first. Seeing a whole page full of '0 Comments' freaked me out; having some of them with Best Answers Marked checkmarks was even weirder. I mean, that's one of the prophecies of Isaiah about the Second Coming, you know?
posted by po at 3:01 AM on January 9, 2010


That was scary - I thought everyone had suddenly lost confidence in their ability to answer.
posted by paduasoy at 3:09 AM on January 9, 2010 [3 favorites]


Oh, that's what happened. When I saw all these questions with 0 answers, but marked with a tick, I thought it was like some Zen thing - 'the best answer is no answer' or some cosmically trippy paradox like that. But it's just a glitch, you say? I'm slightly disappointed.

If anyone does have the full copy of the RSS, they should demand that every best answer be re-entered under their own username. It's only fair.
posted by eatyourcellphone at 3:16 AM on January 9, 2010 [1 favorite]


The other thing that occurred to me when I saw all those zeroes:

(Michael Palin voice): How can we all have died at the same time?
posted by eatyourcellphone at 3:19 AM on January 9, 2010 [2 favorites]


I happened to be discussing speculative scenarios for a US-China conflict when the server went down and thought that I might have finally typed something that tripped the automatic Internet censorship here - as I'm sure my penetrating online insights are the major source of anxiety for the custodians of China's national security.
posted by Abiezer at 3:38 AM on January 9, 2010 [4 favorites]


Oh yeah great go ahead and lose all my answers that weren't trollish.
posted by turgid dahlia at 3:50 AM on January 9, 2010 [2 favorites]


I just thought, in light of having to deal with all the recent MeTa agita, that the mods had gone in with a wide brush. A really, really wide brush.
posted by availablelight at 4:36 AM on January 9, 2010 [2 favorites]


I have eaten
the answers
that were on
the website

and which
you were probably
expecting
to read in the morning

Forgive me
they were delicious
so helpful
and so informative
posted by sambosambo at 4:55 AM on January 9, 2010 [75 favorites]


Oh no! After staying up late describing my cat-medicating system in super fine detail! I sense kitty sabotage.
posted by Metroid Baby at 5:16 AM on January 9, 2010


Keep the answers where they are; do not re-awake what should not be woken. YOU'VE BEEN WARNED
posted by Think_Long at 5:31 AM on January 9, 2010 [3 favorites]


This is what happens when I skip a day of looking at Ask Metafilter -- all of the answers disappear into the ether. Even the best ones.
posted by armage at 5:36 AM on January 9, 2010


It was hard to wake up and not get my morning cup of MeFi.

I may have to face the fact that I have a problem.
posted by Kattullus at 5:41 AM on January 9, 2010


Good luck on the restore, team!
posted by BeerFilter at 5:48 AM on January 9, 2010


You guys lost my formula that makes your nipples produce lager beer? But I burned that after I posted it.
posted by middleclasstool at 5:54 AM on January 9, 2010 [1 favorite]


I wish I'd seen this before I woke up and (still half asleep) was like "oh, man, these questions need answers!"
posted by paulg at 6:23 AM on January 9, 2010


Aargh. Of course that would be when all but one of the very useful answers to my question about classical music were posted. Gotdammit.

Oh well. I'll live. Good luck on the restore, pb.
posted by mediareport at 6:26 AM on January 9, 2010


We were able to recover 501 comments from Google's cache and they're back on the site. The earliest comment was from 3am Pacific, and the latest we were able to recover was from 9:46pm Pacific. Unfortunately, that's just a fraction of the comments that were posted on Jan 8th.

I sincerely apologize for the loss, and we'll be reviewing/revamping our maintenance process to make sure this doesn't happen again.
posted by pb (staff) at 6:43 AM on January 9, 2010


Looks like comments are coming back. Kudos.
posted by Stewriffic at 6:47 AM on January 9, 2010


pb, I think many of us have had something like this happen in our professional capacity. So we know that horrible sinking feeling when you realize the data is really gone, and there's probably no way to get it all back again, and users are going to have to be told it's gone.

I suspect there are a bunch of us who can really sympathize with you right now, and who are thinking "eh, it happens, life goes on, no big deal," because it's happened to us too.
posted by FishBike at 6:53 AM on January 9, 2010 [39 favorites]


.
posted by DevilsAdvocate at 6:55 AM on January 9, 2010 [3 favorites]


root@metafilter:/$ rm -rf *

oops.
posted by JeffK at 7:03 AM on January 9, 2010 [5 favorites]


The funny thing about AskMe is that the comments will be replaced in no time at all.
posted by smackfu at 7:11 AM on January 9, 2010


Interesting, when the site went down for me, I was just in the process of posting this little ditty, to the oh so popular Whoah thread.

The gods are mefites.
posted by philip-random at 7:12 AM on January 9, 2010


pb, have you tried posting a question on AskMe? Those guys know everything.
posted by shothotbot at 7:31 AM on January 9, 2010 [20 favorites]


This probably needs to go on a bar across the top of AskMe as well, people will be asking questions about it.
posted by jessamyn (staff) at 7:33 AM on January 9, 2010


Your attempt to SILENCE ME has been stymied by the fact that I only answered one question yesterday! Hah!



Seriously, though: that is sad, but these things sometimes happen. Now you know not to do it again, young man!
posted by subbes at 7:35 AM on January 9, 2010


mods, these things happen. There is no absolutely bulletproof DR or backup strategy. We'll cope. Thanks for keeping the site running.
posted by kalessin at 7:36 AM on January 9, 2010


Sigh. I checked here too soon, and missed this before sending off my question about it to the admins. Please feel free to ignore it. Sorry for adding to the noise.
posted by carmen at 7:46 AM on January 9, 2010


Would it be possible to link from the "Shit. Nothing is working." status page to something funny and distracting elsewhere that would keep people busy and amused during down times?
posted by pracowity at 7:47 AM on January 9, 2010


"; delete from ask_table;
posted by DU at 7:53 AM on January 9, 2010


pb, you definitely need to get therapy, but before that, DTFMA. You should also hire a lawyer and go to a doctor as soon as possible.
posted by fuq at 7:54 AM on January 9, 2010 [3 favorites]


Other sources I'd check: Yahoo cache & Bing cache (Warrick just updated to work with those), & Yahoo Pipes may have some feeds saved.
posted by Pronoiac at 8:06 AM on January 9, 2010


I thought to myself, "I should go to bed." Now I'm really glad I stayed up & read all my ask.me answers. (And I finally have the justification I've been looking for for staying up as long as I want. "No, really! It might not be there in the morning!")
posted by eleyna at 8:09 AM on January 9, 2010 [3 favorites]


Seconding Fishbike's comment. Have heart, pb.

Meanwhile, I'm going to see what original answers I can provide. Since I'm otherwise rather useless here.
posted by zennie at 8:11 AM on January 9, 2010 [1 favorite]


Yeah, pb - no need to lie prostrate or anything. These things happen.
posted by mintcake! at 8:12 AM on January 9, 2010


Also: this is the perfect chance to answer every question from this week with "I think you need a massage."
posted by mintcake! at 8:17 AM on January 9, 2010


Without any rancour towards the admins, because these things happen, I thought it might be worth pointing to this really painful anonymous askme that's lost many thoughtful answers. I made a comment pointing anonymous here in case he didn't see the answers yesterday, but I also thought that maybe some of the people who commented in that thread might want to recreate their answers.

Obviously it would be a bit much to point to every thread that lost questions, but I thought in this case, because it's anonymous, and because the answers were very heartfelt, people who posted there might want to know their comments had been lost.
posted by carmen at 8:25 AM on January 9, 2010


Oh man. I had gotten some really good answers to my question, and I had commented in thread about how I was going to implement information from those answers. I can pretty much reconstruct what I was planning to do, but when I headed off to the meetup last night there were comments that I hadn't read yet that I was going to read this morning. And now they're gone.

Gah.

It's frustrating, but I must say, this incident is notable for its rarity. Thanks for everything you do to keep the metafilter reading experience smooth sailing, pb and mathowie.
posted by ocherdraco at 8:29 AM on January 9, 2010


How about a banner suggesting people repose their comments...and maybe something that shows up in our rss feeds as well?

I am very sad about the 19th cen composer thread as well...if no one reposts I'll add what I can remember...at least now I know I'm not crazy for thinking there was a whole set of responses there!
posted by lemonade at 8:34 AM on January 9, 2010


Yeah for a bit there I could get nothing but error messages, and not having my favorite website handy freaked me out a bit.

status.metafilter.com should always resolve when the site's down, and lists scheduled maintenance times (usually a Friday night, 11pm Pacificish, once a month).
posted by carsonb at 8:35 AM on January 9, 2010 [1 favorite]


Ah, never mind... I'm too slow on my phone
posted by lemonade at 8:36 AM on January 9, 2010


Wait, we're not getting out the torches and pitchforks?

*looks sheepish, hides something behind back*
posted by Malor at 8:36 AM on January 9, 2010 [1 favorite]


Oh, man. I saw the askme I posted to yesterday before this and thought "REALLY???? Our answers were deleted?!?! They were so good!"

It is too bad - Grumblebee, Carmen, and Gerard Sorme had some good responses in this thread that were lost.
posted by Tchad at 8:36 AM on January 9, 2010


I need a pony that will let me favorite FishBike's comment a thousand times.

pb, know that many, many, many of us have been in spots just like this before, and we know how you feel.

{{{{{{pb}}}}}}
posted by marsha56 at 8:45 AM on January 9, 2010


For Firefox users, one thing you can do personally is install Text Area Cache.

It keeps locally everything you type into text fields (for a few days to a week I think). Great for situations like this, as well as browser crashes and other annoyances.

(This is an example of a phenomenon I think has been documented before: I always want to impress people here by mentioning cool stuff, but every time I get all excited about doing so, like now, I realize I only know about the cool thing in question because I saw it on Metafilter earlier. Oh well.)

posted by drjimmy11 at 8:47 AM on January 9, 2010 [4 favorites]


I modified a pipe, which has only 16 comments from yesterday, but a couple in the thread carmen pointed out.
posted by Pronoiac at 8:50 AM on January 9, 2010


What fishbike said, lots of us know what that awful, "Oh SHIT, that really happened" moment is like. Good luck with the restore.
posted by usonian at 8:51 AM on January 9, 2010


I'm pretty sure the answers in question were DTMFA, IANAD and IANYL/IANAL.
posted by fixedgear at 8:53 AM on January 9, 2010


Um, hang on, that pipe's not picking up anything other than a different thread.
posted by Pronoiac at 8:54 AM on January 9, 2010


I finally have the justification I've been looking for for staying up as long as I want.

I love that my last post was favorited by Night_owl.
posted by eleyna at 9:06 AM on January 9, 2010


Yikes, what a pain, and what a lot of work. Thanks for keeping my favorite addiction suppled.
posted by theora55 at 9:07 AM on January 9, 2010


But but but I NEED TO KNOW WHAT SALADS TO EAT!!?!
posted by Admiral Haddock at 9:07 AM on January 9, 2010


supplied.
posted by theora55 at 9:07 AM on January 9, 2010


Aww. I nailed this one out of the gate (admittedly an easy question, though) and now my answer is lost in vaporspace. I'll get my check mark back from you nicely or roughly--your choice!
posted by Burhanistan at 9:10 AM on January 9, 2010


I had about ten AskMe threads that I could view in Firefox via the "Work Offline" option so I've sent pb the updates. This is a little different than just the data being missing because, as you've seen Pronoiac, the comment IDs were reused.

pb has a system in place for not only straightening this out moving forward but also integrating the older comments we've been able to salvage.

this incident is notable for its rarity.

This is, in fact, the only time this sort of thing has happened.

I thought it might be worth pointing to this really painful anonymous askme that's lost many thoughtful answers.

I emailed the OP of that question directly to indicate that he might want to see if he had a cached version of the thread. My cached version is from about 5 pm last night. If anyone else does, please feel free to send it to pb.

I reupdated another anonyme -- the one about the sick cat, with the very bad update I made that the cat had died -- this morning so that people aren't confused. We're trying to reassemble what we can, and really appreciate people's patience and understanding.
posted by jessamyn (staff) at 9:11 AM on January 9, 2010


I got the Pipe working better - 502 items, although those might not be cached. (I'd added an extra, useless, filter, & my glance at the subject wasn't careful enough.)

I used Warrick to hit Bing for the first three pages of Ask; grepping shows 34 comments posted yesterday.

I'm mailing pb to see if these might be helpful.
posted by Pronoiac at 9:38 AM on January 9, 2010


this incident is notable for its rarity. (jessamynAdmin)

Yeah, by rarity I guess I meant "it's notable because it's never happened before"; didn't mean to imply that it had.
posted by ocherdraco at 9:39 AM on January 9, 2010


And I meant to quote "This is, in fact, the only time this sort of thing has happened," not my own words. Oy.
posted by ocherdraco at 9:42 AM on January 9, 2010


Also, the "Work Offline" thing is genius! I just got back all the comments from my thread that had been there before I went to the meetup yesterday evening—it was still in my browser's cache. I'll be emailing them to pb. Thanks, jessamyn!
posted by ocherdraco at 9:48 AM on January 9, 2010


I guess I should be used to this, having been SILENCED ALL MY JANUARY 8TH
posted by Greg Nog at 9:49 AM on January 9, 2010 [1 favorite]


On some of the restored threads e.g. it looks like the css formatting for Best Answer and OP Comments have been swapped around.
posted by Lanark at 10:24 AM on January 9, 2010


Thanks Lanark, should be fixed up.
posted by pb (staff) at 10:31 AM on January 9, 2010


For what it's worth, if anyone has any deleted answers from my question about hard drives and backup, I'd really appreciate recovering them, as it's still quite a bundle of stress for me.

Cheers!
posted by Magnakai at 10:43 AM on January 9, 2010


Should have checked here first, left a WTF? in the askme thread about party appetizers. Please repost, people, those were excellent suggestions.
posted by StickyCarpet at 10:46 AM on January 9, 2010


Obviously this is due to the mods' lack of banjos. Where's that Musician's Friend catalog....
posted by Rhomboid at 10:47 AM on January 9, 2010


n.b. i already own a five-string resonator banjo so any idiosyncratic expressions of love would best take the form of an open-back model hth hand
posted by cortex (staff) at 11:51 AM on January 9, 2010 [3 favorites]


January 8th. Never Forget.
posted by IndigoRain at 11:53 AM on January 9, 2010 [2 favorites]


Just FYI, my favorites count did not decrement according to lost comments of mine, but my favorites of others aren't there anymore.
posted by StickyCarpet at 11:55 AM on January 9, 2010


Don't worry, pb. It sucks when this happens, but you're still an awesome webmaster.

Of course, if Matt was like Vlad Dracul, you would be slowly impaled while Matt enjoyed a meal, a glass of wine and your screams.

Of course, if Matt was like Vlad Dracul, we would all be slowly impaled while Matt enjoyed a meal, a glass of wine and our chorus of screams.
posted by double block and bleed at 11:57 AM on January 9, 2010


This is what happens when I decide to contribute several answers in a day, sorry everyone!
pb, we have all had something like this happen to us at some time, you have my sympathy. Not to worry, the MeTa abideth.
posted by arcticseal at 12:31 PM on January 9, 2010


I'm not too upset about the lost comments, but my favorites! My precious, precious favorites! Woe is me.
posted by grouse at 12:40 PM on January 9, 2010


amusingly, I also have a banjo. true story.
posted by jessamyn (staff) at 1:13 PM on January 9, 2010


IndigoRain writes "January 8th. Never Forget."

Elvis' birthday; who could?
posted by Mitheral at 1:14 PM on January 9, 2010


I've sent pb my cached copies of three threads, including the selective-reduction anonyme and the salad bar question, both as of 5 pm ish Jan 8.
posted by LobsterMitten at 2:17 PM on January 9, 2010


Anybody know how to retrieve cached copies of pages in Safari?
posted by LobsterMitten at 2:40 PM on January 9, 2010


Anyone freaking out could listen to this song next time!
posted by schyler523 at 3:03 PM on January 9, 2010


Thanks to everyone who sent in their cached versions of pages. Our recovered comment count is currently at 626.
posted by pb (staff) at 3:35 PM on January 9, 2010


my favorites! My precious, precious favorites!

Easy for you to say, I had a best answer in there.
posted by StickyCarpet at 4:01 PM on January 9, 2010


ALL THESE ANSWERS ARE YOURS EXCEPT ASKME. ATTEMPT NO QUESTIONS THERE.
posted by slogger at 5:35 PM on January 9, 2010 [1 favorite]


Three cheers for pb!
posted by SPrintF at 7:36 PM on January 9, 2010


I tried to summarize the answers I could remember in the question I asked; are other folks doing the same?

At the very least, for archival purposes, there should be a short note added to each thread that lost comments so folks coming to the site in the future won't go, "Damn, why the hell does such a great/easily answerable/interesting question only have one answer?"

Seriously, can you do that, pb?
posted by mediareport at 7:40 PM on January 9, 2010


Just to clarify, that last question was meant as, "Is it possible to do that, pb?" Adding something like "Some answers to this question were lost in a server snafu on January 8, 2010" to all the questions that lost answers?
posted by mediareport at 7:46 PM on January 9, 2010


Thanks to everyone who sent in their cached versions of pages. Our recovered comment count is currently at 626.

Eh, the other 352 were just delmoi anyway.
posted by cjorgensen at 8:08 PM on January 9, 2010 [2 favorites]


About a year ago, I got a MetaFilter t-shirt. Either as a gift or purchased myself, I forget.

Today was the first day that I actually wore it.

So apparently this is all my fault. Sorry for ripping the space-time continuum. I'll go back to my well-worn collection of Atlanta underground culture wear.
posted by intermod at 9:51 PM on January 9, 2010


So, it's just Matt and pb that need banjers? or should I post this on AskMe?
posted by mwhybark at 11:10 PM on January 9, 2010


For a minute, it was like the old days when Metafilter would go down randomly AND I DIDN'T KNOW WHAT TO DO.

Although I should say that it is impressive that this sort of thing has become the exception and not the rule.
posted by Afroblanco at 12:54 AM on January 10, 2010


Is it possible to do that, pb?

It's not my call to make, but it's something we can discuss. One of the problems is that we don't know which questions had answers and which didn't. And threads are open for a year, so it's possible for a question to gather answers over all of those days. Of course, that first day is critical for a thread and I've noticed that in at least a couple threads from January 8th, the OP or another answerer has already stepped in to link to this thread. I think that's a fine way to notify people too.
posted by pb (staff) at 8:02 AM on January 10, 2010


It's not my call to make, but it's something we can discuss.

My feeling is that we're now talking about the area of a few hundred comments I think? In that case, I think the small holes that might have been created by missing comments are a little less disruptive than a note that says "something may have been here" since we really don't know if that's true or not. I've contacted the OPs of complicated threads that I thought were negatively impacted and have been keeping an eye on some of the other ones, but I feel that a link to this thread is sufficient and if people want to ask their questions over when their week rolls around again, that's an option for them.
posted by jessamyn (staff) at 8:23 AM on January 10, 2010


Of course, that first day is critical for a thread

Yeah, I don't think we need to go back to the threads from 4 months ago.

I feel that a link to this thread is sufficient

So do I. I just think it might be a good idea to add one to all the threads that were on the front page that day.
posted by mediareport at 8:39 AM on January 10, 2010


As much as it sucks to have lost that data, I'm not sure there's much need for a long-term notice, which is what I think you're suggesting mediareport. After the immediate "weird, what happened to my question" shock has passed, they'll just be questions with lower-than-expected comment counts and probably a few apparent non-sequitur replies. Odd but probably not worth drawing attention to.

I'm going to go ahead and add a note to the timeline on the wiki about the data loss (if someone else hasn't already), but that's about as far as I was personally thinking we'd need to go with it.
posted by cortex (staff) at 9:04 AM on January 10, 2010


Chrome doesn't have "work offline" functionality, as far as I know. But it still keeps a cache, I just don't know how to access it. (I was reading several ask questions when the scheduled maintenance started, so I might have something useful to send, if only I knew how...)
posted by nat at 9:45 AM on January 10, 2010


but that's about as far as I was personally thinking we'd need to go with it.

Until my lawyers call the MeFi lawyers concerning lost data, emotional distress, and appropriate monetary compensation.
posted by telstar at 9:47 AM on January 10, 2010


Not sure if it's related, but there seems to be some glitching in this question.
posted by TheNewWazoo at 1:29 PM on January 10, 2010


Indirectly related, pb's doing some wrench work right now which that's making things a little bumpy on the short term in order to get things slicker in the long run. We'll probably see some double comments show up as a result of server hiccups and people hitting Post an extra time or three.
posted by cortex (staff) at 1:58 PM on January 10, 2010


Oh, cool. I was just about to send an email with all the server errors I've been getting, but if they're to be expected, I won't.
posted by ocherdraco at 2:16 PM on January 10, 2010


Yeah, for this afternoon don't worry about it unless it seems like unusually notable downtime. Go ahead and flag double comments, we'll catch 'em quick enough that way.
posted by cortex (staff) at 2:18 PM on January 10, 2010


Thanks, I've seen a couple of server errors today.
posted by fixedgear at 2:21 PM on January 10, 2010


Is this what the government is telling you to tell us?
posted by anniecat at 12:55 PM on January 12, 2010


« Older Postcards: short, sweet, and goes on the fridge.   |   More META than META! Newer »

You are not logged in, either login or create an account to post comments