Why can't I reach Metafilter? March 14, 2006 5:28 PM   Subscribe

What is being done to curb the continuing amount of times that MetaFilter dies in the arse each day (e.g. when it's inaccessible)? Just wondering.
posted by sjvilla79 to Uptime at 5:28 PM (64 comments total)

Do we all have to pitch in and get Matt some decent hardware and hosting or something? Matt, would this even help?
posted by sjvilla79 at 5:28 PM on March 14, 2006


Visit less often. The fewer visits, the less overloaded.
posted by five fresh fish at 5:30 PM on March 14, 2006


"Dies in the arse"? Is that an Aussie thing?
posted by languagehat at 5:40 PM on March 14, 2006


You can't be serious. Perhaps it's just me, but isn't there something strange about a website that has worldwide popularity but can't even maintain uptime? I just find that weird. Excuse my frankness too. I mean I'm not trying to have a dig or cause shit, but this all seems a bit too cyclic.
posted by sjvilla79 at 5:40 PM on March 14, 2006


"Dies in the arse"? Is that an Aussie thing?

Bloody oath, mate. Just call me Mick Taylor.
posted by sjvilla79 at 5:42 PM on March 14, 2006


Sorry, it's my fault. In a drunken haze I keep mistaking the server for that well-known purveyor of tasty treats: The Easy Bake Oven.

It does a fine job of toasting up those gummy, nummy little snacks but I'm pretty sure that the CPU doesn't like running without it's heat sink.
posted by loquacious at 5:44 PM on March 14, 2006


No no, it's me. I made a voodoo doll of MetaFilter last night and poked it in the arse. Does anyone know how to reverse this? Apologies.
posted by _sirmissalot_ at 5:47 PM on March 14, 2006


Drew Curtis has been crashing at Matt's house lately. Beer will wreak havoc on a server.
posted by mr_crash_davis at 5:50 PM on March 14, 2006


Matt only has one outlet where his server is, so when he goes in there to vacuum, he has to unplug the server. And he has OCD, which explains the frequency.
posted by my sock puppet account at 5:52 PM on March 14, 2006


apparently you're unaware that the blue is driven by MetaGnomes...unionized MetaGnomes. "JRun Error" is a gnomish holiday. Or smoke break.
posted by rollbiz at 5:58 PM on March 14, 2006


Drew Curtis has been crashing at Matt's house lately

THE FUCK YOU SAY!
posted by loquacious at 6:02 PM on March 14, 2006


I am kinda curious... is it that the server is overloaded, or that the code is unstable, or that coldfusion in general is unstable, or that the ISP is having trouble?

As an owner of a colocated server myself, I'm seriously just curious what kinds of things i should expect to run into as load increases over time...
posted by twiggy at 6:02 PM on March 14, 2006


...unionized MetaGnomes

Indeed, I'm sure they're unionized. How would you ionize a gnome?
posted by kindall at 6:03 PM on March 14, 2006


I'm disappointed the urinating elephant hasn't made an appearance yet. Gee, one would think that all the concern for Matt's bandwidth and strain on the server would be the least of all your worries. The comments thus far have at least proven that. Anyway, although slightly funny (but also kind of lame), does anyone actually have anything serious to add?
posted by sjvilla79 at 6:03 PM on March 14, 2006


does anyone actually have anything serious to add?

Sure, both of us can now kick the server so that there are probably five or six more hours in the day when one of us will notice if the site is down and fix it. I don't know mathowie's angle. It seems like there's a correlation between new feature rollouts and unexpected server strain/death.
posted by jessamyn (staff) at 6:06 PM on March 14, 2006


+++ Welcome to the MetaFilter datacenter +++

posted by furtive at 6:08 PM on March 14, 2006


There has been a direct corellation between bitchy callouts and improvements in uptime, historically.

If I were Matt I'd instigate a little, uh, "downtime" right about now and maybe, uh, "lose some data" (namely, this post).
posted by scarabic at 6:18 PM on March 14, 2006


Scott, buddy, I think the reason for all the snark is that this question comes up frequently. From memory, the answers to your questions are: it's being worked on, the hardware is fine, jrun and coldfusion are unreliable but fortunately the Portland Coldfusion User Group are on the case, it would take a lot of work to move the site over to a more stable platform and it's a lot better than it used to be.
posted by blag at 6:19 PM on March 14, 2006


How would you ionize a gnome?

Rub him against a cat, then stick him to the wall?
posted by five fresh fish at 6:21 PM on March 14, 2006


Rub him against a cat, then stick him to the wall?

Ok, now that's just mean. And kinky.


sjvilla79: While I'm going to stop short of apologizing for deploying my incredibly sexy wit in this thread, it's not meant to harm or attack you. The question itself, yeah, maybe we're attacking that, but not you.

The uptime has been a lot better lately. An frankly, a bit of downtown and uncertainty adds a bit of charm to MeFi.
posted by loquacious at 6:44 PM on March 14, 2006


Downtown? Uh, downtown is a whole different kind of charm of a sort MeFi has plenty of. Sorry. Downtime.

Also, this is a sarcasm detector calibration target. The words "incredibly sexy wit" in my previous were used in a wholly sarcastic and self-deprecating manner.
posted by loquacious at 6:47 PM on March 14, 2006


PMSL at furtive. I'll pay that. I hear you too, loquacious. And thanks jessamyn and blag. scarabic, get some fresh air perhaps. Will a hug do? I'm good at them.
posted by sjvilla79 at 6:56 PM on March 14, 2006


Downtown Detroit, maybe.
posted by five fresh fish at 7:37 PM on March 14, 2006


Beware of those who say they are good at hugs.

They usually are lousy at them.
posted by _sirmissalot_ at 7:49 PM on March 14, 2006


Seems to me Metafilter has been up more than Gmail lately, and that's nothing to sneeze at.
(Bless you)
posted by Roger Dodger at 7:50 PM on March 14, 2006


you rang?
posted by quonsar at 7:52 PM on March 14, 2006


Seems to me Metafilter has been up more than Gmail lately

OH SNAP! No shit, huh? And that's gotta sting. I'd love to see that kind of headline on the GOOG stock info page.

"Blog Served From Closet Beats Google's Uptime, Investors Shit Pants Thrice and Sell Till Their Anuses Bleed"

Google is having a bad week. It looked like they dropped my site out of the index, but the buzz on the SEO forums is that they accidentally REVERSED one of their spam filtering controls, sending all kinds of bogus linkfarms skyrocketing and penalizing a great many legit sites.

I guess they can do wrong after all.
[ducks]
posted by scarabic at 8:05 PM on March 14, 2006


quonsar, I believe that's the one. How lovely. And I'm having my lunch right now too. Perfect. Of course sirmissalot is just jealous though. But how many times have I told you that I'm not exclusive? Gee! Geek it up.
posted by sjvilla79 at 8:20 PM on March 14, 2006


The worst part is that right now, Coldfusion is really a black box type server. I can only write simple CF code on my end, which then gets thrown into a java server and it's treated as a complete mystery to CF coders. Macromedia's documentation on Java server tuning and reliability is almost nil -- they just focus on making sure people know how to write the simple code.

So it's a mix of my crappy code, the high server load, and the mystery of CFMX servers. Weekdays where the server stays up result in over 300k pageviews (about 9 million pages a month) so the loads are ever increasing. I've heard that myspace adds a dozen CF servers a day to keep up with their loads, but I hope I'm not at the limit of what I can do with one web server and one db server. It seems like the web server is ok and not under too heavy of a load when it is up. Every so often the internal java server just dies though.
posted by mathowie (staff) at 8:21 PM on March 14, 2006


Coldfusion is really a black box type server

This is entirely true.

Thankfully I've opened that sealed black box and peered at it's mystical innards. There's a bunch of monkeys in there. Bonobo chimps, specifically. One half of them run on beer and they're all drunk and making out all the time. (Hence, ColdFusion.) The other half run on coffee and they're all hopped up and hyper all the time. (Hence, Java).

The Java half dies from sheer boredom and frustration when the ColdFusion half gets too drunk to keep the Java half fed with commands and information. And then eventually they get sick of each other, tired of the same old stinky monkeys from the same old stinky tribe, and never anyone new to make out with.

If you replace a few of the monkeys every so often, or feed both sides a steady diet of Esctasy, they won't get bored with making out with each other and they won't get falling down drunk so often.
posted by loquacious at 8:34 PM on March 14, 2006


...but I hope I'm not at the limit of what I can do with one web server and one db server.

This is an interesting statement. I mean I appreciate your forwardness given so far, but at what point do you come to the realisation that the above is a major factor (along with the code issues)? And where to from there? That's basically why I posted today. Again, didn't mean to sound snarky at all, so thanks for seeing it that way. Um, probably sounds fucking lame too, but I guess I just want to know what MetaFilter (e.g. its members) can do to make shit better. Yeah, most of the above comments aren't helpful (including some of my own). What do you think though, Matt?
posted by sjvilla79 at 8:34 PM on March 14, 2006


I for one am amazed one guy can run something like this out of his house without specialized server admin, redundancy, or someone to work on the ColdFusion code and make java and CF play nicely together while he posts and moderates and changes diapers, and it still works most of the time. And that it's all coming off of one web server and one db server. This is not meant to be a kiss-up. I am seriously amazed. Anyone who's been in any end of the IT business knows what I'm talking about. Look at DailyKos -- he's got a darn crew of coders working for him and seems to add a server once a month. (It's cool that he keeps users posted on tech developments, which Matt does here too.) I think the opensource crowd here has also done cool work with browser extensions and stuff, but as far as I can tell there is no crew of MeFi members debugging scripts for Matt on a volunteer basis. Or am I wrong about that?

The price of some downtime for the value MeFi delivers is minimal. No doubt Matt's DIY ethic is one reason we don't see ads all over registered user pages. And it makes for better community too if users feel the DIYness of the site enhances its alternative qualities, which I certainly do. But it's damn impressive DIY. Plus, the times its down are the times I can exercise some discipline and actually get real work done!!!

Sorry to go all serious. Here's some compensation:


posted by fourcheesemac at 8:39 PM on March 14, 2006


Ideally, I would love to port the server to some other more reliable language/platform. I would also love to have a coder to help me add new features and keep things up and running so I could focus more on ideas and moderation stuff.

The reality is that to do that I need to find someone I trust as much as my wife -- it can't just be an employee that converts some code and leaves, I sort of need to get someone to devote the better part of their life to the work like I have for the past 7 years. To do that I need to be able to pay someone well and at the moment I'm covering myself and I'm able to pay jessamyn a bit here and there, but my hope is that someday soon when ad revenue increases a bit I'll be able to support a full time coder employee to help rewrite the site in something more manageable.

I can't see any way to improve the site greatly without changing the backend and copying the functionality in php or rails or something like that. Hopefully by the end of the year I can find someone to do that and be able to support them.
posted by mathowie (staff) at 8:46 PM on March 14, 2006


myspace adds a dozen CF servers a day to keep up with their loads

:O

Good lord. I can understand how MeFi grew itself to where it is now, slowly, over time, with no good opportunity to rip out core architecture and start over, but SHIT! MySpace is running CF? I am not an engineer, architect, or much of a technologist, but I tend to think of CF like I think of Lasso: featureful and utterly unscalable. I'm shocked to hear that MySpace is using it. If they have money to spend on 12 boxes a day, they have money to spend on rearchitecting. Who owns their ROI? Gaddam!
posted by scarabic at 8:53 PM on March 14, 2006


If they have money to spend on 12 boxes a day, they have money to spend on rearchitecting. Who owns their ROI? Gaddam!

Who is thinking about the gnomes?!?
posted by rollbiz at 9:08 PM on March 14, 2006


Hopefully by the end of the year I can find someone to do that and be able to support them.

That's awesome, Matt. Again, thanks for being so open with this feedback.
posted by sjvilla79 at 9:13 PM on March 14, 2006


and be able to support them

Yeah, it seems like a great idea to just open up the whole effort to the membership, but if you've ever tried anything like that, you know it's a briar patch.

For one thing, when you're not paying someone, they might get sucked off into a higher-priority PAYING project at any time, leaving you high and dry. That project might turn into a permanent job, and you'll be really screwed, trying to figure out half-finished code you were counting on someone else to finish.

Barring that, you can just get seriously sub-standard work. People's enthusasim is often skin-deep when it comes down to it, and it's not much help to get substandard code you have to figure out and improve before launching. In fact, it's often more trouble for less ultimate quality.

The final bad scenario is the passionate helper who does good work but wants to interfere with every design idea you have. You can't blame someone for having ideas, but when they're working for you for free, they feel much more liberty to force their ideas on you, whether those ideas are good or not. Good web products usually come from a collaboration between different types of people (users, engineers, designers, product managers, etc) and it can be trouble to give one person "hero working for free" status. They lose the collaborative spirit and things become much more "my baby" and "I'm not volunteering my time so you can boss me around" at worst.

Oddly, paying someone is the easiest way to get the most and best work done. This is not an ideal axiom, but it probably is a good rule as long as you're working within this capitalist economy and culture of ours.
posted by scarabic at 9:26 PM on March 14, 2006


Is anyone monitoring backend stuff with Nagios or something similar? Something that can SSH in and restart dead processes? Or at least send someone an SMS when things go tits up?

Just wondering.. :-)
posted by drstein at 9:48 PM on March 14, 2006


Yeah, it seems like a great idea to just open up the whole effort to the membership

Um, no it doesn't.

Oddly, paying someone is the easiest way to get the most and best work done.

No way. Paying someone is the best way to get work done? You're talking crazy now!
posted by gtr at 9:54 PM on March 14, 2006


Hopefully by the end of the year I can find someone to do that and be able to support them.

Matt, if you want a helping hand, one of my friends is a Javascript and CF guru with lots of experience developing apps in CF. He wrote the CMS that used to run evolt.org (in CF) and withstood a Slashdotting. I'm sure he can help you fix a few key things.
posted by madman at 10:16 PM on March 14, 2006


No way, I remember Jeff from the evolt list! I should get in touch with him.
posted by mathowie (staff) at 10:21 PM on March 14, 2006


Scarbic: "if you've ever tried anything like that, you know it's a briar patch."

He did try years ago to open it up to the membership and so begat PHP-Mefi... a project that never resulted in a line of code. The group expected leadership from Matt but he wasn't interested, no one was nominated to lead, and no one stood up to take over. It was everyone's fault.

Over the years I've offered to fix this thing. My ideas have changed as I become a better coder and I've now built pipeline architectures that scale across boxes very well. I've outlined metafilter upgrade approaches that mitigate the need for a marital-level of trust -- such as setting up nextversion.metafilter.com with
a LAMP/WAMP+Python and read-only access the current database. Give it to half a dozen potentials on their own subdomain and it's only $500 worth of hardware to see if any of them can dig their way out of coldfusion and replace all the functionality..
"someone to devote the better part of their life to the work like I have"
Because of that my offer still stands but it's now ammended it to exclude metafilter worker blood pacts.



ps. Anyone need to convert MSWord to HTML or any XML?

posted by holloway at 11:19 PM on March 14, 2006


No way, I remember Jeff from the evolt list

Yeah, we are both admins there, though both of us haven't found much time for it lately.

But yeah, I've already pointed him to this thread, so do give him a holler. He's ranked in the top 0.5% by Brainbench on CF.
posted by madman at 11:28 PM on March 14, 2006


I was going to mention holloway's offer in this thread, but I needed to track down the username to make sure I had the right person, and then I got distracted by a shiny thing and forgot.

loquacious didn't describe CF upthread, he described my brain.
posted by stavrosthewonderchicken at 11:45 PM on March 14, 2006


Oooh, that is shiny. Finally a link aggregator that doesn't actually add a whole new page for me to monitor.

Yeah, I think I was using my own brain as inspiration, so, small wonder, stavros. Perhaps we should copy our brains and send them to each other so we can have mad monkey brain sex. Or not. Whichever. In any case it's worth it just for the turn of phrase "mad monkey brain sex".
posted by loquacious at 12:09 AM on March 15, 2006


Something that can SSH in and restart dead processes?

IANAP by any stretch, but I do recall someone in one of these threads talking about how easy it would be to set something up that would automatically restart the server after it goes down. Matt, is that similar to what drstein is asking about? Can it be done?
posted by mediareport at 1:12 AM on March 15, 2006


It's a Java application server on Windows. It's unreasonable to expect a site to remain running in the face of either one of these handicaps, let alone both.
posted by majick at 1:29 AM on March 15, 2006


mediareport: yeah it can be done and it's easy enough if there's a command line interface to boot the server. Does this site run in a servlet container like tomcat or what?


ps. just got an email from the [php-mefi] mailing list, ah... my dear friend belle-gobble973@gramsofcoke.com decided to stay in touch.
posted by holloway at 2:11 AM on March 15, 2006


"I've now built pipeline architectures that scale across boxes very well"

Now you're just bragging.
posted by mr_crash_davis at 6:39 AM on March 15, 2006


How would you ionize a gnome?

This made me very happy.
posted by languagehat at 6:44 AM on March 15, 2006


penisfilter
posted by Dreamghost at 7:00 AM on March 15, 2006


mathowie writes "Every so often the internal java server just dies though."

Is this why we get pages that keep loading without throwing errors?
posted by Mitheral at 7:08 AM on March 15, 2006


This made me very happy.

I figured it would take a linguist, a chemist, or an Asimov fan to get that. (Asimov once offered it as a litmus test for chemists.)
posted by kindall at 10:58 AM on March 15, 2006

Now you're just bragging.
Fucking A  I am.

Like Asimov.
posted by holloway at 3:08 PM on March 15, 2006


Q: What is the point of circumcision?
A: So the brain can breath.
posted by mischief at 3:30 PM on March 15, 2006


... which helps cognitive functions like spelling.
posted by mischief at 3:43 PM on March 15, 2006


goodness me....
posted by strawberryviagra at 4:07 PM on March 15, 2006


Did I offend thee?
posted by strawberryviagra at 4:10 PM on March 15, 2006


It was a stupid photo of some guy pissing. What was the point of it, strawberryviagra?
posted by mathowie (staff) at 4:37 PM on March 15, 2006


Well, that would be an abstract concept to get into, but I guess the answer was yes.
I'm sorry if I caused any offense, and I won't post anymore pissing photos in the future.
posted by strawberryviagra at 4:55 PM on March 15, 2006


ahem. that should be "... any more pissing photos ..."
posted by strawberryviagra at 4:57 PM on March 15, 2006


....but posting big pictures of drunken idiots with their dicks out is a real conversation starter everywhere else I hang out!
posted by stavrosthewonderchicken at 5:20 PM on March 15, 2006


It's clear to me that you should only hang out near urinals. It's certainly a better place to take the piss.
posted by strawberryviagra at 6:14 PM on March 15, 2006


Touché! (but stop standing so close to me...)
posted by stavrosthewonderchicken at 6:18 PM on March 15, 2006


« Older NYC Meetup?   |   Recommand some books set in Israel. Newer »

You are not logged in, either login or create an account to post comments