Comments on: MetaFilter Offline

MetaFilter Offline

pb — Tue, 17 Dec 2013 04:05:20 -0800

MetaFilter was offline from around 9:45 PM Pacific Time to 4:00 AM Pacific Time. A hard drive failed in our database server, taking some of the database with it. We were able to restore from a backup that happened at 7:40 PM Pacific. So we lost data during those two hours. We apologize for that, the long downtime, and the bumpiness as we get things running again.

If you added a post, comment, mefi mail, or favorite after 7:40 PM Pacific last night that data was lost. If you added a MetaFilter post during that time we can recover that text. (The full text of posts are sent to admins vie email.) So drop us a line via the Contact Form and we'll send that to you so you can repost.

By: turbid dahlia

turbid dahlia — Tue, 17 Dec 2013 04:06:13 -0800

Thanks for all your hard work.

By: empath

empath — Tue, 17 Dec 2013 04:06:56 -0800

i was working a double shift, and i literally almost died without it.

By: Mrs. Pterodactyl

Mrs. Pterodactyl — Tue, 17 Dec 2013 04:07:14 -0800

I'm just glad we're back. God, that was horrible news to which to wake up. How would I have survived my day?

By: 23

23 — Tue, 17 Dec 2013 04:09:23 -0800

Thank you all so much for putting out the fires. You have my thanks and the thanks of every byte you saved from the bitbucket.

By: flabdablet

flabdablet — Tue, 17 Dec 2013 04:11:00 -0800

I see zarq's epic Get Smart post missed it by that much :-(

By: octothorpe

octothorpe — Tue, 17 Dec 2013 04:12:07 -0800

Yay for backups!

By: flabdablet

flabdablet — Tue, 17 Dec 2013 04:13:03 -0800

(puts on binocular glasses)

I see zarq's epic Get Smart post missed it by that much :-)

By: goodnewsfortheinsane

goodnewsfortheinsane — Tue, 17 Dec 2013 04:13:04 -0800

Thanks so much pb.

By: Doktor Zed

Doktor Zed — Tue, 17 Dec 2013 04:14:53 -0800

Many thanks for the crisis management, pb. (It's also interesting to learn what kind of redundancy systems the mods have in place.)

By: mathowie

mathowie — Tue, 17 Dec 2013 04:15:57 -0800

If you did make a post (there were 2-3 between 7-10PM when the site went down) last night, feel free to repost it now.

By: NoraReed

NoraReed — Tue, 17 Dec 2013 04:18:00 -0800

I was so lonely

By: Anitanola

Anitanola — Tue, 17 Dec 2013 04:18:50 -0800

I am amazed at how efficient you are and thank you for updating the status and getting everything working again. I had a comment zooming off just as the darkness descended so I quickly made a note of it to post later. I loved finding out just how important this place is to me. Night time is my time to play and tonight was a little sad and scary. Thank you MeFi for being here and not letting us down.

By: Devonian

Devonian — Tue, 17 Dec 2013 04:27:27 -0800

Thanks! That looked like a lot of work late at night - hope that the bugs weren't just squashed but will never breed again...

By: Rustic Etruscan

Rustic Etruscan — Tue, 17 Dec 2013 04:27:32 -0800

Thanks, pb! I'm glad to see the site's back up. I'm also glad the site was restored to where it was, because it got rid of a fighty, deraily comment I made in a thread on the Blue. Score one for accidental, blameless moderation.

By: Brandon Blatcher

Brandon Blatcher — Tue, 17 Dec 2013 04:29:39 -0800

Thanks for all your hard work pb, you're a champ.

By: Blasdelb

Blasdelb — Tue, 17 Dec 2013 04:29:58 -0800

Thanks for working through the night on this, watching this all unfold this morning from the future in Europe all I could think about was the sleep pb must not be getting.

By: arcticseal

arcticseal — Tue, 17 Dec 2013 04:37:14 -0800

You all rock and I missed you.

My wife spilled a glass of water on my Macbook around the time MeFi went dark so you can blame her. Now begins 3 days of waiting to see if it dries out ok.

By: billiebee

billiebee — Tue, 17 Dec 2013 04:43:31 -0800

You do realise I had to spend the morning at work actually, you know, working?! It's quite simply not good enough. I shall expect a partial refund of my £3.31 signup fee at your convenience.

By which I mean: thankyouthankyouthankyouohpleaseneverleavemeagain

By: zarq

zarq — Tue, 17 Dec 2013 04:49:10 -0800

Oh man, that must have been a total nightmare. Thank you, pb and everyone else who worked on getting things back up and running so quickly.

By: adamvasco

adamvasco — Tue, 17 Dec 2013 04:51:06 -0800

Happy Christmas pb.

By: pb

pb — Tue, 17 Dec 2013 04:51:25 -0800

Thanks everyone. Just a heads up—chat is still offline. We'll get that running again tomorrow.

By: JHarris

JHarris — Tue, 17 Dec 2013 04:53:00 -0800

Noooo the Get Smart post didn't make it??

Ah, it did survive. Whew!

By: Blasdelb

Blasdelb — Tue, 17 Dec 2013 04:53:08 -0800

We'll live somehow

By: octothorpe

octothorpe — Tue, 17 Dec 2013 04:58:04 -0800

Frustrating that we still have to rely on spinning disks for storage in 2013. Hard drives just suck.

By: Wordshore

Wordshore — Tue, 17 Dec 2013 05:00:00 -0800

MetaFilter down, and we lost the Ashes as well. End of Days.

Kudos to PB for diligently pulling MetaFilter back. Cook, KP, Swann and Anderson (28 off one over, ffs!) should take lessons from his commitment.

By: pointystick

pointystick — Tue, 17 Dec 2013 05:01:44 -0800

pb, you are awesome as are all the metafilter staff. Thanks for all the hard work!

By: you must supply a verb

you must supply a verb — Tue, 17 Dec 2013 05:04:04 -0800

Thank you. And sorry for refreshing the status blog a bazillion times.

I appear to have a little metafilter problem.

By: beryllium

beryllium — Tue, 17 Dec 2013 05:04:56 -0800

pb: "Thanks everyone. Just a heads up—chat is still offline. We'll get that running again tomorrow."

Entirely understood! I hope you get some sleep, pb, and thank you very much for all your hard work!

By: spitbull

spitbull — Tue, 17 Dec 2013 05:09:27 -0800

I figured it was NSA payback.

By: DingoMutt

DingoMutt — Tue, 17 Dec 2013 05:10:56 -0800

Wow, I missed the blackout period completely. I'm impressed! So good of you all to schedule this so as to spare us fragile EST'ers.

By: lampshade

lampshade — Tue, 17 Dec 2013 05:11:29 -0800

i demand a $5 refund.........

......ummm....wait a sec.....nebba mind

carry on

By: mathowie

mathowie — Tue, 17 Dec 2013 05:12:27 -0800

Frustrating that we still have to rely on spinning disks for storage in 2013. Hard drives just suck.

Hilariously, this is all due to a SSD drive we used to build faster search indexes dying. But 1s and 0s still suck.

By: middleclasstool

middleclasstool — Tue, 17 Dec 2013 05:23:14 -0800

Any developer or sysadmin knows what miserable goddamn things computers are. Thanks for the hard work. On the bright side, I was super relieved to discover it wasn't my DSL connection again.

By: Phlegmco(tm)

Phlegmco(tm) — Tue, 17 Dec 2013 05:25:25 -0800

Thanks for all the great work. I rarely comment, but reading this fine, fine site has become such a daily ritual that apparently I get all skeevy and jittery like when I don't have that option.

By: mecran01

mecran01 — Tue, 17 Dec 2013 05:28:02 -0800

I'm living in the Middle East right now and I was afraid it had been blocked. We're all so codependent:

"Metafilter is down--was it something I did or said? How can I change myself to get Metafilter back?"

By: ThePinkSuperhero

ThePinkSuperhero — Tue, 17 Dec 2013 05:28:24 -0800

Thanks for your hard work! I just noticed an answer I gave in an AskMeta Q disappeared, and I was all, whut? At least I know I wasn't deleted for being a jerk :D

By: kimberussell

kimberussell — Tue, 17 Dec 2013 05:28:39 -0800

Thanks, pb!

By: JHarris

JHarris — Tue, 17 Dec 2013 05:29:01 -0800

Hilariously, this is all due to a SSD drive we used to build faster search indexes dying.

Really? I thought they were less prone to doing that, what with having no moving parts. What circumstances will cause a SSD to die?

By: Blasdelb

Blasdelb — Tue, 17 Dec 2013 05:29:53 -0800

Just a heads up for future reference there is the metafilter status blog, which is like somehow hosted separately, that we can always check to see if a problem is from us or the site.

By: Seymour Zamboni

Seymour Zamboni — Tue, 17 Dec 2013 05:30:26 -0800

We were able to restore from a backup that happened at 7:40 PM Pacific.

There goes my plan to actually have a productive day at work.

By: Wordshore

Wordshore — Tue, 17 Dec 2013 05:32:41 -0800

What circumstances will cause a SSD to die?

Long and deeply annoying story, but half a jar of Lingonberry Jam from Ikea and an ex-housemate with "issues" will do this.

Am hoping the MetaFilter situation was rather different.

By: rtha

rtha — Tue, 17 Dec 2013 05:35:40 -0800

Whew. Glad you're back.

By: DigDoug

DigDoug — Tue, 17 Dec 2013 05:38:00 -0800

Goddammit, I'd piss on a spark plug if I thought it'd do any good!

(In all seriousness, I've contributed $5 to Metafilter, for nearly a decade of use of the best place on the web. Can I do anything more to help keep it running smoothly?)

By: Ghidorah

Ghidorah — Tue, 17 Dec 2013 05:38:44 -0800

The downtime coincided with the portion of my return commute where sitting and watching video on the train is no longer possible. That's the time I usually check Ask, and I was so, so alone. Thanks for putting everything back so quickly. I was sure it was just my iPhone being dumb until I remembered the status blog.

By: Devonian

Devonian — Tue, 17 Dec 2013 05:39:08 -0800

SSD - Silicon Soon Dies. Live fast, die young...

By: Chrysostom

Chrysostom — Tue, 17 Dec 2013 05:45:21 -0800

So, first off - Thank you, Matt and pb, for keeping this place running so well in general, and for getting it back online so quickly last night.

Question, though: Does it make sense to do more frequent backups? What would be the impact of backups, say, every 10-15 minutes?

By: octothorpe

octothorpe — Tue, 17 Dec 2013 05:49:14 -0800

What circumstances will cause a SSD to die?

There's a limit to the number of writes to each block but even that takes a long time and if it was used for an index, I'd assume a lot more reads than writes.

By: Mitheral

Mitheral — Tue, 17 Dec 2013 05:51:56 -0800

I missed the black out completely so yay me. But I had to chuckle because, as is my usual pattern, I checked Recent Activity where a single comment had been posted and then clicked over to Ask Me where there were only a couple new questions and I was like "What the heck was the site down". Clicks over to Meta. "Yes, Yes it was."

By: Flunkie

Flunkie — Tue, 17 Dec 2013 05:54:55 -0800

Partial restoration from screenshots:

7:40 PM: DTMFA
7:40 PM: DTMFA
7:40 PM: We've all dated that guy. DTMFA.
7:40 PM: eat it, it's fine
7:41 PM: I don't usually say DTMFA, but DTMFA.
7:41 PM: As long as it smells ok
7:41 PM: DTMFA
7:42 PM: Four weeks?! I've had some of that in the back of my fridge for four years. I plan on serving it to guests on Christmas Eve.
7:42 PM: People here are always so quick to say DTMFA, but in this case I think they're right.
7:43 PM: DTMFA

By: Now there are two. There are two _______.

Now there are two. There are two _______. — Tue, 17 Dec 2013 06:05:18 -0800

Is chat still down for others, or is this a problem on my end?

By: Jacqueline

Jacqueline — Tue, 17 Dec 2013 06:06:10 -0800

12/16/13 NEVER FORGET

By: sammyo

sammyo — Tue, 17 Dec 2013 06:06:26 -0800

Headline from my tech "news" tab this morning: Don't Believe Backup is Broken? Think Again

Get used to data evaporating, the current best dirty little secret of the amazing 4TB drives and huge fast SSD, very hard and expensive to back up, and then restoring TB's of data, sloooow.

By: cortex

cortex — Tue, 17 Dec 2013 06:08:18 -0800

Thank you, pb, for being the dude who is on this shit so I can just go to bed and hope things are better when I wake up.

By: Namlit

Namlit — Tue, 17 Dec 2013 06:09:30 -0800

wow

such offline

many sad happy now

By: arcticseal

arcticseal — Tue, 17 Dec 2013 06:10:25 -0800

Chat will be fixed tomorrow, pb has gone to get some sleep.

For those who want to help maintain the site, I've donated in the past when I've felt flush for Matt to buy doughnuts, beer and whatnot for the mods.

By: Now there are two. There are two _______.

Now there are two. There are two _______. — Tue, 17 Dec 2013 06:12:01 -0800

Thanks pb, thanks arcticseal, and sorry for my lousy reading comprehension.

By: wenestvedt

wenestvedt — Tue, 17 Dec 2013 06:14:30 -0800

IT'S ALIIIIIIIVEE again!!

Thanks, from another sysadmin.

(Hey, do we get a double helping of Favorites today? Because I, uh, used a full day's worth just before the crash. Honestly.)

By: Deathalicious

Deathalicious — Tue, 17 Dec 2013 06:25:40 -0800

How frustrating that must have been. Server hard drive failures suck.

Glad you were able to get everything up and running again.

By: sammyo

sammyo — Tue, 17 Dec 2013 06:27:33 -0800

Backups: I know the process would be slow but can we submit a Freedom of Information request to the NSA for the missing posts?

By: GenjiandProust

GenjiandProust — Tue, 17 Dec 2013 06:29:57 -0800

Dear AskMe, I have a relationship with this special site. It educates, enlightens, and infuriates me. We sometimes run hot and cold, but I always come back, and the site is waiting for me. Then, suddenly, at 9:45 PM Pacific Time, the site went dark. Was it something I did? How can I fix this? Please do not tell me to see a therapist; I am seeing two.

By: Faintdreams

Faintdreams — Tue, 17 Dec 2013 06:31:00 -0800

Thank you mods, pb, and all other behind the scenes peeps, for fixing everything so quickly.

By: Faintdreams

Faintdreams — Tue, 17 Dec 2013 06:33:36 -0800

Me: Metafilter is down, but status blog says they are fixing it.

Boyf: Oh well that's basically the entire internet broken for you then.

Me: Nope it's okay ::lip quivers::

By: desjardins

desjardins — Tue, 17 Dec 2013 06:33:37 -0800

I was reading on my phone in bed when the outage hit (11:45 my time). I figured it was a sign from God to go to sleep, so that's the reason I'm coherent this morning. Blessing in disguise.

By: Etrigan

Etrigan — Tue, 17 Dec 2013 06:36:58 -0800

We couldn't even make it one lousy year between outages. Honestly, what are we paying you people for?

Everyone needs a hug, my ass.

By: chavenet

chavenet — Tue, 17 Dec 2013 06:43:01 -0800

This Is Just to Say

I have damaged
the SSD
that was in
the server rack

and which
you were probably
saving
for Metafilter

Forgive me
It was del.icio.us
so 1337
and so old

By: sciencegeek

sciencegeek — Tue, 17 Dec 2013 06:47:34 -0800

My morning routine includes looking at the Metafilters before I get ready for work in the morning. I was early to work today and I blame you guys.

Glad to see it is up and running again.

By: jason_steakums

jason_steakums — Tue, 17 Dec 2013 06:48:53 -0800

So, opening a new tab and checking Mefi is like my ingrained default action when on the web late at night, so much that "Oh no, Mefi's down! What should I do?" *new tab, check Mefi, like I've got goldfish memory* happened multiple times last night.

By: running order squabble fest

running order squabble fest — Tue, 17 Dec 2013 06:50:49 -0800

The terrifying thing about SSDs is that when they fail, they tend to fail suddenly and without warning. You don't get the clicking and searching that is the HDD's faithful warning to back up your data and replace the drive. Just... nothing.

By: zarq

zarq — Tue, 17 Dec 2013 06:54:17 -0800

desjardins: "I was reading on my phone in bed when the outage hit (11:45 my time). I figured it was a sign from God to go to sleep, so that's the reason I'm coherent this morning. Blessing in disguise."

Heh. My daughter woke me up at 2:30am to tell me that she had taken off her socks because her feet were too hot and now her feet are cold and she can't find her socks anywhere in her bed and maybe they crawled out of her bed while she was asleep because sometimes her socks like to hide and could i please, please get her another pair from her sock drawer because it's too dark in her room and even though she opened the drawer she can't see in the drawer because it's dark and she wants mismatched socks not ones that match because that would be boring. *gasp for air* But they need to be pink, both of them pink but two different kinds.

So I got up, using my phone as a flashlight. And I gave her the #^&%$$! socks. And of course on the way back to bed I checked mefi and it was down. I also took this as a sign that ~~my daughter is an evil, sockless insomniac~~ it was time to go back to bed.

By: Sequence

Sequence — Tue, 17 Dec 2013 06:55:56 -0800

"Less prone to failure" does not mean "never fails", yeah.

Boyf: Oh well that's basically the entire internet broken for you then.

Me: Nope it's okay ::lip quivers::

Even knowing it was down and that I should wait awhile, I found myself clicking on either Metafilter or AskMe about every 2-3 minutes until I went to bed last night, and even though I knew logically there was other stuff out there I could look at, I kept coming back to it.

By: cjorgensen

cjorgensen — Tue, 17 Dec 2013 06:56:00 -0800

Thanks to the team for the hard work.

Sometimes I wonder when these things happen what this place would be like if there had to be a complete reboot. Like if there was no database, no code, no graphics, and it had to reconstructed from scratch. New very first post, new signup process, only people still interested and active would be signing up. The archive of posts and questions gone, so no doubles, no one bragging about their low user numbers, etc.

How long before something was functioning again? What assumptions would be changed? What choices would be made. Which sections would get priority. What would never be revived?

Some things would be possible to reconstruct from people's caches (like the graphics), but pretend nothing was. What would be done differently?

Consider this a case study. I'll take my answer off the air.

By: Sophie1

Sophie1 — Tue, 17 Dec 2013 07:00:44 -0800

pb, thanks for restoring my morning!

By: Nanukthedog

Nanukthedog — Tue, 17 Dec 2013 07:09:41 -0800

I'm not putting my 4 pun tennis movie comment back as a protest. I'm not sure what exactly I'm protesting, but I'm protesting.

Also, thanks for getting things back up and running. I hope somebody is getting a well deserved rest now.

By: lalex

lalex — Tue, 17 Dec 2013 07:10:49 -0800

Thanks pb!

Also, here's a little oddity resulting from the failure.

By: languagehat

languagehat — Tue, 17 Dec 2013 07:13:45 -0800

> 12/16/13 NEVER FORGET

You misspelled NEVAR.

Also, thanks to pb and all the hell-demons he contracts the work out to! (Come on, how else do you think he could possibly accomplish all that?)

By: chavenet

chavenet — Tue, 17 Dec 2013 07:14:52 -0800

Hah, I just checked the status.metafilter.com site and ... Metafilter is still "currently offline".

By: filthy light thief

filthy light thief — Tue, 17 Dec 2013 07:15:49 -0800

mathowie: 1s and 0s still suck.

I'll get you guys a big bucket of 2s and 3s for Northern Hemisphere Winter Holiday/Southern Hemisphere Summer Celebration!

And thanks to the mods for helping out with those of us who didn't read this MetaTalk and were baffled at the disappearance of their activities last night.

By: Etrigan

Etrigan — Tue, 17 Dec 2013 07:36:14 -0800

Hah, I just checked the status.metafilter.com site and ... Metafilter is still "currently offline".

Wait, am I a figment of your imagination, or are you a figment of mine?

By: onlyconnect

onlyconnect — Tue, 17 Dec 2013 07:39:06 -0800

Thanks for fixing everything, pb! Boy, seeing this happen reminds me that MetaFilter being offline rarely happens anymore, it's not like the old days of random Jrun errors. So thanks, Matt, for hiring pb and organizing everything so that the site is such a dependable home to so many.

By: Artw

Artw — Tue, 17 Dec 2013 07:39:30 -0800

Good work getting it back, guys!

By: Fig

Fig — Tue, 17 Dec 2013 07:48:54 -0800

Only after I re-started my phone for the second time this morning did it occur to me to check a website OTHER than metafilter, AskMeFi, or MetaTalk to see why nothing would load.

Thanks for your hard work, pb!

By: penguin pie

penguin pie — Tue, 17 Dec 2013 08:04:41 -0800

*Flings self on MeFi to give it an embarrassingly snotty, tearful, hysterical hug*

Oh thank God you're baaaaack. I was so worried. And on the way to work I had to stare out of the bus window at the real world. Sniff snifffff...

*wipes nose on back of shaky hand, tries to pull self together*

And thanks pb and mathowie and everyone else. You guys...

*moves towards the guys for the next hysterical hug*

By: nat

nat — Tue, 17 Dec 2013 08:11:39 -0800

Insomnia sucks. Insomnia with no job yet is worse. Insomnia with no job and no metafilter... what is the world coming to?

Thanks for fixing the last of those, pb. Any hope on the first two?

By: nathancaswell

nathancaswell — Tue, 17 Dec 2013 08:13:13 -0800

Try adding more decorative gears and copper plating.

By: LobsterMitten

LobsterMitten — Tue, 17 Dec 2013 08:30:19 -0800

Three cheers for pb as usual! What a night - hope you are getting some sleep today.

By: jessamyn

jessamyn — Tue, 17 Dec 2013 08:41:06 -0800

I slept through the whole thing. Glad the loss was minimal. Thanks for doing the heavy overnight lifting getting it working.

By: ambrosen

ambrosen — Tue, 17 Dec 2013 08:44:31 -0800

I'm not putting my 4 pun tennis

Your 4 ton p..., what!? That's gigantic!

By: rabbitrabbit

rabbitrabbit — Tue, 17 Dec 2013 08:44:38 -0800

If only the site went down every night I would consistently get my full 8 hours of sleep. It was like, huh, MetaFilter's down... guess I'll... uh... go to bed or something.

By: slogger

slogger — Tue, 17 Dec 2013 08:53:47 -0800

My comments! My precious comments are lost! You heartless bastards! Gone like so many tears in the rain, I can never get them back! You have betrayed me for the last time MetaFilter!!!

*converts PST to EST*

Oh, I was sleeping the entire time. Sooo...hunh.

Thanks, guys!

By: Devils Rancher

Devils Rancher — Tue, 17 Dec 2013 09:04:33 -0800

Yeah, I had the usual bout of insomnia 4:30-ish, grabbed the phone off the nightstand, hit Safari & panicked - was I going to have to risk waking up the entire household to go re-boot the router? Then I hit status.metafiler.com (which I probably typed wrong) and IT didn't load, & was all oh, shit! but Twitter worked, then status loaded, then the tweets nearly lulled me back to sleep, in time for the coffee maker to do its thing at 5:45.

But really! If you guys could give a little more advance notice so I can adjust my meds on nights when there's gonna be a server outage, I would really appreciate it. PROTIP: Have DiskWarrior email you when it detects a S.M.A.R.T. status irregularity! (hamburger, etc. sorry, I'm punchy due to lack of sleep)

By: Devils Rancher

Devils Rancher — Tue, 17 Dec 2013 09:10:50 -0800

My comments! My precious comments are lost! You heartless bastards! Gone like so many tears in the rain, I can never get them back! You have betrayed me for the last time MetaFilter!!!

I can totally hear this in a Peter Lorre voice. You have to hold your hand out stiffly, like you're holding an invisible globe, though to make it really work.

By: Lynsey

Lynsey — Tue, 17 Dec 2013 09:11:06 -0800

Thank you, pb for this and all the other services you perform. Just like Santa (in the Northern Hemisphere, anyway), you're doing amazing things while we sleep. ♥

By: It's Raining Florence Henderson

It's Raining Florence Henderson — Tue, 17 Dec 2013 09:12:58 -0800

1s and 0s still suck

I'll get you guys a big bucket of 2s and 3s

I felt a great disturbance in the 4s... as if millions of favorites suddenly cried out in terror and were silenced.

By: Elsa

Elsa — Tue, 17 Dec 2013 09:13:22 -0800

Thank you for all your diligent work and care!

By: librarylis

librarylis — Tue, 17 Dec 2013 09:13:32 -0800

Awwww man, I wrote a detailed and well-researched comment for restless_nomad's AskMeta question and I'm not sure I'm up to recreating it. Ok deep breath, and yes, thank you, pb for all your hard work to restore the site. So grateful you're willing to work through the night on this kind of thing!

By: Pruitt-Igoe

Pruitt-Igoe — Tue, 17 Dec 2013 09:14:08 -0800

I blame Shia LaBoeuf.

By: salvia

salvia — Tue, 17 Dec 2013 09:21:07 -0800

Thank you, pb! And thanks for letting us know. I had been wondering why my comment was "deleted" until I saw this.

By: nanojath

nanojath — Tue, 17 Dec 2013 09:21:33 -0800

I feel bad for wasting Jessamyn's time asking about why my comment in the Shia LaBeouf thread got deleted (because I apparently don't have anything more important on my plate this morning...?) but I'm sure glad I used the contact form instead of posting a question to MetaTalk.

By: pb

pb — Tue, 17 Dec 2013 09:33:39 -0800

chat should be back online.

By: lalex

lalex — Tue, 17 Dec 2013 09:41:07 -0800

pb, get some sleep!

By: davejay

davejay — Tue, 17 Dec 2013 09:42:49 -0800

Is this the place where we complain about the downtime because we paid for this service goddammit and there are expectations?

No?

Good.

By: twist my arm

twist my arm — Tue, 17 Dec 2013 09:43:22 -0800

chat should be back online.

thankyouthankyouthankyou

based on the time period you had to be awake, i was expecting this maybe in the afternoon some time after you got a decent 6 hrs or something. above and beyond! as per usual.

By: arcticseal

arcticseal — Tue, 17 Dec 2013 09:56:57 -0800

I slept through the whole thing.

It was horrible.

By: Etrigan

Etrigan — Tue, 17 Dec 2013 09:57:01 -0800

chat should be back online.

Maybe if the guy in charge of the magic internet boxes knew what "tomorrow" meant, we wouldn't be having these problems.

By: carolr

carolr — Tue, 17 Dec 2013 10:13:27 -0800

So is this related to why the search functionality was broken just before the site went down?

By: fearfulsymmetry

fearfulsymmetry — Tue, 17 Dec 2013 10:28:29 -0800

I was busy doing rl stuff and missed it!

By: Decani

Decani — Tue, 17 Dec 2013 10:44:16 -0800

What a relief. I thought you'd all up and moved somewhere else and not told me. Which is the STORY OF MY LIFE.

By: Celsius1414

Celsius1414 — Tue, 17 Dec 2013 10:47:55 -0800

On the plus side, I finally got an "It's not just you!" message from downforeveryoneorjustme.com.

By: Rhaomi

Rhaomi — Tue, 17 Dec 2013 10:50:40 -0800

What a relief. I thought you'd all up and moved somewhere else and not told me. Which is the STORY OF MY LIFE.

What a coincidence! My one lost (recently restored) comment was that Ted Chiang's novella STORY OF YOUR LIFE is getting turned into a movie soon. I suspect the incredibleness of that news is what crashed MeFi. Apologies.

By: Weeping_angel

Weeping_angel — Tue, 17 Dec 2013 10:53:23 -0800

Hey, so when I hit "back" on my iPhone(iOS7), MetaFilter isn't remembering where on the page I was anymore. It takes me back up to the top instead. Is this related to your problems last night?

By: Weeping_angel

Weeping_angel — Tue, 17 Dec 2013 11:00:26 -0800

Never mind, it's working normally now. Weird.

By: maxwelton

maxwelton — Tue, 17 Dec 2013 11:11:12 -0800

It was like, huh, MetaFilter's down... guess I'll... uh... go to bed or something.

As our boys Bret and Jemaine suggest:

"Conditions are perfect: ~~there's nothing good on TV~~ Metafilter is down...conditions are perfect."

By: Nanukthedog

Nanukthedog — Tue, 17 Dec 2013 11:14:16 -0800

Glad to know its done.

By: mannequito

mannequito — Tue, 17 Dec 2013 11:15:10 -0800

I didn't even notice, I'm so happy I bought that hand cranked metafilter.

By: homunculus

homunculus — Tue, 17 Dec 2013 11:52:00 -0800

pb's a jolly good fellow, pb's a jolly good fellow, pb's a jolly good fellow, and so say all of us!

By: NiceKitty

NiceKitty — Tue, 17 Dec 2013 12:02:21 -0800

I had just posted a comment last night criticizing General Alexander to the NSA thread when suddenly I couldn't access metafilter. And upon checking it today, my comment (and at least one other) was gone. Feeling real paranoid for about 5 minutes!!

I'm going to repost the comment and see if your supposed "hard drive" fails on your supposed "database server" again.

By: NiceKitty

NiceKitty — Tue, 17 Dec 2013 12:03:39 -0800

What an NSA thread might look like.

By: ignignokt

ignignokt — Tue, 17 Dec 2013 12:20:44 -0800

Much respect, pb.

Also, yeah, yeah HD failures happen and then you have to restore from backup, but the fact that a huge, solidly performant site like MetaFilter does not have database clustering affirms my crazy idea that not every web app needs to have a backend with DB clustering. Sure, it's cool, but they have their costs, and just maybe they don't have to be gospel for everything.

By: ignignokt

ignignokt — Tue, 17 Dec 2013 12:21:13 -0800

Whoa, I just subtweeted outside of Twitter.

By: filthy light thief

filthy light thief — Tue, 17 Dec 2013 12:28:26 -0800

mannequito: I didn't even notice, I'm so happy I bought that hand cranked metafilter.

Ooh, the rare hand cranked MeFi, the precursor to the Wiki Reader.

By: paleyellowwithorange

paleyellowwithorange — Tue, 17 Dec 2013 12:45:35 -0800

When I first noticed MeFi was down, I had a moment of terror that the IT guys at work had finally cottoned on to how I spend the bulk of my workdays. I was so relieved when I checked the status blog.

By: gauche

gauche — Tue, 17 Dec 2013 12:51:17 -0800

Many thanks for your work, pb.

By: leslies

leslies — Tue, 17 Dec 2013 13:05:39 -0800

Thanks pb! Hope you are vaguely caught up on sleep!

By: Dr Dracator

Dr Dracator — Tue, 17 Dec 2013 13:16:39 -0800

On the upside, the site is now statistically safe from a hardware outage in the middle of the holidays.

By: zarq

zarq — Tue, 17 Dec 2013 13:38:18 -0800

Dr Dracator: "On the upside, the site is now statistically safe from a hardware outage in the middle of the holidays."

Are you out of your mind?!? Go outside right now, turn around three times and spit, curse, do all the things.

*grumble* Kids these days. Tempting the wrath of the whatever from high atop the thing. *grumble*

By: Etrigan

Etrigan — Tue, 17 Dec 2013 13:40:42 -0800

Now he has to wear his pajamas forward.

By: From Bklyn

From Bklyn — Tue, 17 Dec 2013 13:41:00 -0800

Thanks!

That was like quitting smoking all over again.

By: paleyellowwithorange

paleyellowwithorange — Tue, 17 Dec 2013 13:51:11 -0800

Relevant Confession Bear.

By: Tanizaki

Tanizaki — Tue, 17 Dec 2013 14:07:00 -0800

I left a web browser window open on the AskMeFi index page when I went to sleep last night, so I have preserved the AskMeFi questions that were lost. They are as follows:

I Don't Know How to Dress Myself
My "Partner" is an Asshole
Look At This Thing I Do - I'm So Quirky and/or Adorkable
My "Partner" is a Bitch
Should I Yell at my Boss and Then Quit My Job?
My Pet is a Shithead
Where to Find Artisanal Toilet Paper in Williamsburg?
Can Anyone Translate This Novel For Me?

By: cortex

cortex — Tue, 17 Dec 2013 14:10:32 -0800

I feel like you didn't quite manage to keep the wheel on the right side of the joke/jerkiness divide, there.

By: Namlit

Namlit — Tue, 17 Dec 2013 14:16:59 -0800

Where to Find Artisanal Toilet Paper in Williamsburg?

Let's answer that right here. Right beside the guy who makes boxwood rat traps on his turning lathe While You Wait, there's a corner shop where they sell handmade toilet paper arcs and a special spool mechanism for you to assemble your own rolls.

You have the choice between various qualities:
Beethoven ("Rough" and "Harmonious". "Deafening" is out of stock)
Cicero
Sigmund F.
Isadora Duncan
Bart Simpson,

There's a possibility to order custom-made, too; delivery three weeks (which is problematic).

By: billiebee

billiebee — Tue, 17 Dec 2013 14:47:10 -0800

I Don't Know How to Dress Myself

To be fair to Tanizaki, I have a couple of AskMes cued up and this pretty much describes one of them. (Thanks for the question title!)

By: Jacqueline

Jacqueline — Tue, 17 Dec 2013 15:02:37 -0800

On the upside, the site is now statistically safe from a hardware outage in the middle of the holidays.

That's not how statistical probabilities work...

By: Tanizaki

Tanizaki — Tue, 17 Dec 2013 15:58:51 -0800

You have the choice between various qualities:
Beethoven ("Rough" and "Harmonious". "Deafening" is out of stock)
Cicero
Sigmund F.
Isadora Duncan
Bart Simpson,

No Tilda Swinton?

By: Greg_Ace

Greg_Ace — Tue, 17 Dec 2013 16:14:20 -0800

I gave up and read a book until bedtime...but felt like I was somehow betraying an old friend.

By: Namlit

Namlit — Tue, 17 Dec 2013 16:25:24 -0800

No Tilda Swinton?

It's a very small shop; they have to be selective. Nothing too obvious - it's not for the crowds.
Inside rumors have it, though, that a Sherlock version is on its way. High-performance and super fast, but abrasive.

(They tried Legolas, but it wasn't a hit. Neither was Samwise Gamgee. "More Lembas bread," I'm asking you)

By: lordaych

lordaych — Tue, 17 Dec 2013 17:08:56 -0800

SSDs actually kind of suck in a lot of server applications especially with RAID, they don't take a good beating for very long (2 years maybe) like a 15K SAS drive might (4-5 years). The flash doesn't hold up to being written all of the time due to the nature of the silicon / circuitry design and wear leveling algorithms are necessary to keep them from dying prematurely.

We're looking at a storage system that uses them for caching for fast reads and writes but ultimately stores all data on spinning SATA drives. You gotta replace those SSD suckers every 1-2 years, at least the grade we're seeing.

For me the outage happened immediately as I was clicking on the latest NSA story. I was also stoned out of my vulcan mind. You do the math

By: lordaych

lordaych — Tue, 17 Dec 2013 17:10:28 -0800

(on my next thoughts that is)

Then the Google Cache worked and I was like "oh yeah Google Cache baby" *cough*

By: nubs

nubs — Tue, 17 Dec 2013 18:10:10 -0800

Thanks for all the hard work!

And a . for the lost data.

By: George_Spiggott

George_Spiggott — Tue, 17 Dec 2013 18:12:49 -0800

Where to Find Artisanal Toilet Paper in Williamsburg?

In Colonial Williamsburg there isn't any other kind.

By: Namlit

Namlit — Tue, 17 Dec 2013 18:24:56 -0800

ooompf

By: bquarters

bquarters — Tue, 17 Dec 2013 18:26:11 -0800

Yeah! What the heck! In my 3:30am panic mode awake in the dark time I need to check my Metafilter!! Actually maybe I was checking it in my "bored at work" time. Anyway, at no time is it acceptable for MF to not be functioning! Ok?? Thank you! PS I must learn how to convert time zones...

By: double block and bleed

double block and bleed — Tue, 17 Dec 2013 18:27:08 -0800

Namlit: "No Tilda Swinton?

It's a very small shop; they have to be selective. Nothing too obvious - it's not for the crowds.
Inside rumors have it, though, that a Sherlock version is on its way. High-performance and super fast, but abrasive.

(They tried Legolas, but it wasn't a hit. Neither was Samwise Gamgee. "More Lembas bread," I'm asking you)"

Trust me from experience: stay away from the Captain Hook.

By: ersatz

ersatz — Tue, 17 Dec 2013 18:43:51 -0800

At least the 2372 Mefites who favourited my comment on the nature of life, christmas pastries, and the platonic ideal of cat fluffiness got their $5's worth before my contribution was lost in the ether.

By: gingerest

gingerest — Tue, 17 Dec 2013 19:06:16 -0800

I am just going to pretend that the grumpy probably-off-topic answer I left in AskMe fell into the data hole instead of being deleted. The timing is almost right. No great loss there.

Thanks, pb! (For restoring MeFi at great effort, not for providing me a cover story.)

By: hippybear

hippybear — Tue, 17 Dec 2013 19:38:22 -0800

You know those things you hear about where reddit decides to gang up and send a zillion pizzas to someone for some reason?

We should all find a way to have beers delivered to pb to thank him for his hard work on our behalf.

By: holgate

holgate — Tue, 17 Dec 2013 20:08:05 -0800

Jeff Atwood's hot/crazy analogy for SSDs is still pretty accurate.

By: The corpse in the library

The corpse in the library — Tue, 17 Dec 2013 20:17:29 -0800

Is this why I'm getting weird results when I click on things in Inoreader? For example I see in Inoreader that I'm going to get a post about "Betty Dodson's Feminist Sex Wars," but clicking on it takes me to a post about Christmas cookies for singles (which looks great, I'm not complaining, but it's odd).

By: LobsterMitten

LobsterMitten — Tue, 17 Dec 2013 20:24:23 -0800

That Dodson post was one that was deleted last night, and then its existence was erased in the failure... so I imagine its post number got reassigned?

By: stet

stet — Tue, 17 Dec 2013 20:25:52 -0800

cjorgensen: "Sometimes I wonder when these things happen what this place would be like if there had to be a complete reboot. Like if there was no database, no code, no graphics, and it had to reconstructed from scratch. New very first post, new signup process, only people still interested and active would be signing up. The archive of posts and questions gone, so no doubles, no one bragging about their low user numbers, etc."

I'm pretty sure that I'm not the only one that would have a less stupid user name. Sadly, I'm too attached to my pre-$5 user number and favorites count to change the name that guarantees me multiple typos in every. single. fucking. thing, I post.

By: pb

pb — Tue, 17 Dec 2013 20:28:05 -0800

Yeah, the post ID would have been reused. So your newsreader must have picked up and stored the post before the site went offline and doesn't check for updates. There are probably four or five IDs that could happen with and then you'll be set.

By: flabdablet

flabdablet — Tue, 17 Dec 2013 20:36:59 -0800

Jeff Atwood's hot/crazy analogy for SSDs is still pretty accurate.

Neatest way I've seen for enjoying the hot without being hurt by the crazy is to one as an external journal for a disk filesystem.

By: misha

misha — Tue, 17 Dec 2013 20:45:22 -0800

I was gobsmacked last night to discover Metafilter really was down and my cable company wasn't just screwing up again. Bright House, my ass. More like dim bulbs...grumble grumble.

Anyway, then I realized the only logical explanation was that after months of attempting to win us all over, the quidnunc kid had finally taken matters into his own hands with a well-timed coup d'état. Just when the most prolific users are basking in holiday good cheer, munching and clicking on cookies, in he swoops. Arab Spring becomes Mefi Winter! Dogs and cats living together! Mass hysteria!

As a loyal Matthowier, I for one am thrilled you were able to stop that barbarian storming the gates! I imagine all the torches and pitchforks lying around on the grey came in handy.

And kudos on the cover story! "SSID failure"! Not bad, not bad at all. Given the time pressure you were all under to get things up and running without unduly panicking the masses, you are to be commended. The nonsensical technical jargon really sells it.

You may want to work on the logistics a bit more, though. In the event another attempted incursion comes along, the mythical wunderkind 'pb' bot will garner greater credibility when you remember that real humans need to sleep occasionally, ha ha. Still, job well done and all that.

Keep Calm and Filter On.

By: ceribus peribus

ceribus peribus — Tue, 17 Dec 2013 20:54:09 -0800

I see zarq's epic Get Smart post missed it by that much :-)

Ah, the ol' post-a-great-Get-Smart-FPP-and-then-take-down-the-server-to-win-the-contest trick.

By: Unicorn on the cob

Unicorn on the cob — Tue, 17 Dec 2013 21:37:13 -0800

I literally was in bed whispering *oh noes, save us pb* after I put my laptop away until I fell asleep.

yay it worked

By: JHarris

JHarris — Tue, 17 Dec 2013 21:46:18 -0800

These details on the failure rates of SSDs reported in that Coding Horror post are troubling, and even more troubling is reading a proponent of the things pooh-pooh the fail rate with an argument like "they're so blazing fast you won't care." Not everyone can afford to buy a new hard drive every couple of years, especially ones as expensive as SSDs! It seems Linus Torvalds was bitten by this problem this past September.

But now I'm wondering why fail rates are so high. The only reason I've seen in the thread up to this point is the one I was previously aware of -- individual flash memory cells have a limited number of write cycles. But the figure I had seen before is that each should be good for tens of thousands of writes, and with load balancing that should, under normal use, be good for a long time. For a server application like Metafilter, yeah, lots of changes, but why should it become unreadable once too many cells fail from writes? Shouldn't the failure mode for SSDs be something less catastropic?

Of course I'm not a hard drive technician, but these kinds of failure rates seem like they should be much more widely known, and would be a deal breaker to many. And mobile devices use this kind of storage don't they?

By: Greg_Ace

Greg_Ace — Tue, 17 Dec 2013 22:27:06 -0800

And mobile devices use this kind of storage don't they?

I'd guess the expectation there is that devices will be replaced by newer models every year or two anyway (OMG my $LastYear'sModel phone won't be updated to $LatestAndroidVersion, I must upgrade to stay relevant!!!1!!), so the potential failure rates don't end up mattering enough to approach major Deal-Breaker status. The mobile version of Planned Obsolescence, in other words.

By: billiebee

billiebee — Wed, 18 Dec 2013 01:06:39 -0800

The only reason I've seen in the thread up to this point is the one I was previously aware of -- individual flash memory cells have a limited number of write cycles. But the figure I had seen before is that each should be good for tens of thousands of writes, and with load balancing that should, under normal use, be good for a long time.

What. Look, will you guys all stop with your nonsense? Cells and write cycles and load balancing indeed. We all know that the interwebs is actual magik, with information being smuggled into our computer things by teeny tiny faeries. Nothing else makes sense. I mean, sometimes the computers aren't even connected to anything!!??!!? (I'm no technical expert, but I believe they live under the keyboard. How else would they know what we're all saying?) They clearly went on strike due to pay and conditions, and pb had to do some serious negotiating. Glad you got them back on board!

By: daisyk

daisyk — Wed, 18 Dec 2013 01:31:42 -0800

Thanks, pb! I'm at my parents' house for the holidays and yesterday morning I had to actually interact with people -- at least today I will be safe!

By: Namlit

Namlit — Wed, 18 Dec 2013 02:15:39 -0800

We all know that the interwebs is actual magik

I rather suspect that the interwebs are looking at me, thinking "is this life form real? Am I just making this up? Hey whoops, there it moves again. Hmmm. What does it do? Must be magic." And so on.

By: travelwithcats

travelwithcats — Wed, 18 Dec 2013 02:29:06 -0800

Thanks pb and all the others behind the scenes!

I was witnessing the whole thing from the future in euroland. It was sad.
The status blog and Twitter helped a tiny bit, but when the site came up and disappeared again I knew I had to actually find something else to do - so I worked. Already replaced the one comment and one favorite that was lost. :-)

By: bongo_x

bongo_x — Wed, 18 Dec 2013 02:47:08 -0800

What circumstances will cause a SSD to die?

Electricity, air, water. This applies to most things, actually.

By: goodnewsfortheinsane

goodnewsfortheinsane — Wed, 18 Dec 2013 03:09:35 -0800

Thanks pb and all the others behind the scenes!

At this point I should probably specify that my involvement with this particular epic was almost entirely limited to sitting at a desk with a Gmail browser window open, trying to think of new variations on "hang in there pb" and "ooh that sounds bad pb" that would still be meaningful. I did at one point briefly consider emailing a photograph of a kitten hanging on to a clothesline. Then I thought of just typing "hangintherekitty.bmp" in a monospace font. I must admit I was *this* close to "I reiterate my statement of support and yield to the gentleman from Oregon". In the end I think I just went with "take it easy there pb".

By: pb

pb — Wed, 18 Dec 2013 08:12:03 -0800

Thanks for those, gnfti. There's nothing like catastrophic failure to get your adrenaline going which isn't always helpful. So a reminder of that once in a while is very helpful.

Does it make sense to do more frequent backups?

We do, but the backups made during the drive failure were not restoring properly. So we had to go back in time a bit to get the most recent reliable backup. We have transaction log backups every 10 minutes, so in theory we should be able to restore to any 10 minute increment. The theory didn't work out with the way this failure worked.

By: holgate

holgate — Wed, 18 Dec 2013 08:43:09 -0800

mobile devices use this kind of storage don't they

I believe that mobile operating systems do most of their volatile I/O in RAM, and are generally pretty brutal about killing processes to accommodate others. No swap: they'll write some state data, but that's it.

Desktop/server OSes are swappier (and log more) by nature, even if you chuck RAM at the problem: that, I suspect, is one motivation behind Apple's 'compressed memory' feature as it moves all its storage to solid state. We're also in a transitional period for those OSes where filesystems are still built with spinnydisks in mind, but with add-ons like TRIM on the driver level. Eventually, we'll see reimplementations that strip out all the logic about disk geometry.

The SSD failures I've seen around me have tended to be filesystem corruption crapouts where no amount of recovery superblocks and whatnot save the day.

By: languagehat

languagehat — Wed, 18 Dec 2013 08:50:57 -0800

> As a loyal Matthowier, I for one am thrilled you were able to stop that barbarian storming the gates!

As one of the quidnuncians, I am still grumbling about the failure of the coup attempt. We were so close! The "NUMBER ONE" flag was being raised above the server and everything. We would have gotten away with it, too, if it hadn't been for that meddling pb!

By: Rustic Etruscan

Rustic Etruscan — Wed, 18 Dec 2013 08:56:22 -0800

This is what you splitters get for abandoning the democratic process. I for one will continue to vote #1 quidnunc kid as I have always done.

By: en forme de poire

en forme de poire — Wed, 18 Dec 2013 12:51:24 -0800

Hahahaha, I was like "yikes, I guess those comments came off a lot more offensive than I meant them...?" Server outage, not deletion. Whew.

By: medusa

medusa — Wed, 18 Dec 2013 13:35:22 -0800

You misspelled NEVAR.

posted by languagehat at 7:13 AM on December 17 [+] [!]

This proves what I've long suspected: languagehat is secretly a prescriptivist.

By: blue_beetle

blue_beetle — Wed, 18 Dec 2013 14:28:39 -0800

Time to round up the scapegoats for a ceremonial execution. BRAND NEW DAY!

By: Pruitt-Igoe

Pruitt-Igoe — Wed, 18 Dec 2013 16:33:02 -0800

Electricity, air, water. This applies to most things, actually.

Yeah, but those things don't cause CPUs or RAM to die with the kind of regularity we see for SSDs. Is it heat? Something inherent to non-volatile RAM?

At least with hard drives it's easy to imagine a spinning mechanical thing breaking down.

By: JHarris

JHarris — Wed, 18 Dec 2013 17:14:09 -0800

Well this thread has taught me a lot about SSD fallibity, although ultimately it comes on the bowed back of pb. So thanks, doubly, for your efforts there! Have a Unicode hamburger on me:

By: Rustic Etruscan

Rustic Etruscan — Wed, 18 Dec 2013 21:25:50 -0800

Time to round up the scapegoats for a ceremonial execution. BRAND NEW DAY!

what kind of brand new day is it when you all are the reactionaries

and what am I even doing here I'm one of the Democratic Quidnun-

By: empath

empath — Wed, 18 Dec 2013 22:12:54 -0800

Something inherent to non-volatile RAM?

Something inherent to flash memory. There's an oxide layer in the memory cells that's used to trap the electrons in whichever state they're supposed to be in that degrades over time.

By: Jahaza

Jahaza — Thu, 19 Dec 2013 07:51:46 -0800

I was gobsmacked last night to discover Metafilter really was down and my cable company wasn't just screwing up again.

Sadly both were true here (and I still haven't managed to fix my internet.)

By: sonascope

sonascope — Thu, 19 Dec 2013 12:19:04 -0800

This is why we need a microfiche edition of MetaFilter. Archival analog film FTW, yo!

By: mazola

mazola — Thu, 19 Dec 2013 23:00:35 -0800

If you added a post, comment, mefi mail, or favorite after 7:40 PM Pacific last night that data was lost.

B-b-but... it was my best comment ever!

By: Etrigan

Etrigan — Fri, 20 Dec 2013 05:06:31 -0800

No, it was just a tribute.

By: charlie don't surf

charlie don't surf — Fri, 20 Dec 2013 14:24:58 -0800

Looks like there was another brief outage just now, I confirmed with isupme. I will attribute this outage to MeFi's bad karma due to a very poor moderation decision just moments before the outage.

By: jessamyn

jessamyn — Fri, 20 Dec 2013 14:26:40 -0800

I'm sure you do.

By: It's Raining Florence Henderson

It's Raining Florence Henderson — Fri, 20 Dec 2013 14:31:21 -0800

I will a tribute of this to the Grand Crusader.

By: charlie don't surf

charlie don't surf — Fri, 20 Dec 2013 14:36:00 -0800

There is a big difference between being a moderator and a martinet.

By: jessamyn

jessamyn — Fri, 20 Dec 2013 14:40:42 -0800

Seriously dude, you know what all the ways are you can contact us or open a discussion about moderation in specific or in general.