Comments on: Why all the crashes?

Why all the crashes?

empath — Wed, 04 Oct 2006 11:03:20 -0800

Why is metafilter so slow and crash-prone? I know it gets a huge amount of traffic, but what causes the regular outages? Do you guys need a new server or what? I can't think of another major site that experiences as much downtime as metafilter does. (i'm not criticizing, I know how difficult it is to run a site of this scale... just curiouis)

By: wheelieman

wheelieman — Wed, 04 Oct 2006 11:16:20 -0800

I was wondering the same thing. Whats the deal?

By: mathowie

mathowie — Wed, 04 Oct 2006 11:28:51 -0800

I'll ignore the past (mostly coldfusion related -- it's a shitty language and technology that doesn't scale well) and say that lately, things have been slow due to high traffic and a heck of a lot of search bots. Something like 30% of the site's total traffic (millions of pages a month) are search bots sucking down text. My error logs are filled with data about when they hit a bad link or load up a malformed URL and they'll hit the site hundreds of times in a short time, locking up the db and filling the logs. I've squashed a fair number of them, but there are still some I haven't tracked down.

I'm working with some sysadmin members about the possibility of moving the web server to something a bit more robust and failsafe.

By: blue_beetle

blue_beetle — Wed, 04 Oct 2006 11:54:32 -0800

I blame communism.

By: Dunwitty

Dunwitty — Wed, 04 Oct 2006 12:37:29 -0800

Heh. I just tried to post an agreement, and got an Internal Server Error.

I was saying, I also manage t see a lot of outages. I tend to visit from 3 to 5 am PST. Are those generally planned outages? If so, you could consider posting an "We're Away" page...

By: caddis

caddis — Wed, 04 Oct 2006 12:43:36 -0800

As far as I know it is still running on a PC server in Matt's abode. For five bucks and no advertising it's hard to complain about what Matt has managed to do.

(It's a bit slow as I type this)

By: mathowie

mathowie — Wed, 04 Oct 2006 12:45:52 -0800

3 to 5 am is generally a "everyone's asleep" outage.

By: SirStan

SirStan — Wed, 04 Oct 2006 12:50:23 -0800

Matt,

Perhaps you could make a simple script that detects the server having crashed, and restarts the service? It should take about 15 lines of Perl.

(ActiveState Perl for Windows).

Though there is that pesky -- whats the real issue that should be resolved as well.

By: muddgirl

muddgirl — Wed, 04 Oct 2006 12:50:54 -0800

I like to think of the 3 to 5 am outage as an incentive to move back to the West Coast someday.

By: SirStan

SirStan — Wed, 04 Oct 2006 12:52:17 -0800

Also enable coldfusions slow page load log, though I am sure you have :~)

By: mathowie

mathowie — Wed, 04 Oct 2006 13:02:02 -0800

As far as I know it is still running on a PC server in Matt's abode.

Stuff of legends. Here's the entire server history:

December 1998: I take my old 300Mhz celeron home computer to my office at UCLA. Working in a computer group at UCLA means that you're free to run a server on the massive OC12 bandwidth for free as long as it is for personal projects. It runs under my desk as I start playing with ColdFusion. It runs here until 2000.

March 2000: I accept employment at Pyra and they have an ISP in the building and a free T1 connection. I put the box into my car and drive up to SF and put it online. It stays here until May of 2001 (after a month or so, I upgrade the server with a monster tower my dad actually pieces together with a friend)

May 2001: In the middle of the Kaycee drama, the ashes of Pyra loses the free T1 and delfuego steps in with a free T1 in his NYC apartment. An article about Kaycee comes out in the NYT just as the server is being overnighted to NYC. The page saying the server is moving runs off my personal PC on my home DSL for all of 24 hours. The server sits in NYC for 2 years or so.

Sometime in 2003: Server moves to Boston along with delfuego, eventually I upgrade it to a dual athlon box

Oct 2004: I finally start treating mefi like a real business and I get two dual xenon servers at servermatrix.com, hosted in racks somewhere in Texas. It has been there ever since, with one upgrade to the db server and web server since. To pay my hosting bills (which are now over a grand a month), I re-enable signups for a $5 one-time cover.

Oct 2006: I'm getting a beefy linux box setup at servermatrix and a member here is helping admin the box and build up some failover. All the Coldfusion developer lists I'm on do say that CF runs faster and more stable on redhat, so I'm setting that up now and hoping to move a few of the subsites to the new box as a test, before moving the whole thing over and wiping out the windows web server.

By: caddis

caddis — Wed, 04 Oct 2006 13:30:51 -0800

I have been trying to post a comment for 15 minutes or so that nevertheless your accomplishment is pretty damn impressive for so little cost to the members, and it is.

By: blue_beetle

blue_beetle — Wed, 04 Oct 2006 13:37:58 -0800

October 29, 2010 - MeFiNet comes online. Nuclear war follows.

By: reklaw

reklaw — Wed, 04 Oct 2006 13:47:30 -0800

ColdFusion is, like, the RealPlayer of scripting languages. That's why.

By: timeistight

timeistight — Wed, 04 Oct 2006 13:49:47 -0800

Your going to run SQL Server on Red Hat? Or is the database box staying on Windows?

By: SteveInMaine

SteveInMaine — Wed, 04 Oct 2006 14:03:24 -0800

I blame today's slowdown on the Woz sighting in Ask.

By: seanyboy

seanyboy — Wed, 04 Oct 2006 15:11:11 -0800

And when metafilter goes down, people flock over to metachat and then that goes down.

which gives the phrase "pay it forward" seem ironic and a little bit sad.

By: timeistight

timeistight — Wed, 04 Oct 2006 15:14:17 -0800

~~Your~~You're

By: mathowie

mathowie — Wed, 04 Oct 2006 15:22:00 -0800

I tweaked a couple things and now it seems downright zippy, but maybe everyone on the east coast finally went home from work and lightened the load.

By: dersins

dersins — Wed, 04 Oct 2006 15:27:50 -0800

And when metafilter goes down, people flock over to metachat and then that goes down.

In light of that, I'm proud to announce MetaMetaChat, a very formal place for strictly controlled discussion of MetaChat outages.

By: timeistight

timeistight — Wed, 04 Oct 2006 15:35:30 -0800

In light of that, I'm proud to announce MetaMetaChat, a very formal place for strictly controlled discussion of MetaChat outages.

...seems to be down...

By: It's Raining Florence Henderson

It's Raining Florence Henderson — Wed, 04 Oct 2006 15:40:53 -0800

MetaMetaMetaChat

By: snsranch

snsranch — Wed, 04 Oct 2006 15:46:08 -0800

Matt, thanks for the history lesson. Very interesting stuff, especially for relative noobz like me.

By: fandango_matt

fandango_matt — Wed, 04 Oct 2006 15:53:50 -0800

posted by blue_beetle October 29, 2010 - MeFiNet comes online. Nuclear war follows.

Uh oh, does this mean MetaFilter has become self-aware?

By: It's Raining Florence Henderson

It's Raining Florence Henderson — Wed, 04 Oct 2006 15:55:29 -0800

No fate.

By: matteo

matteo — Wed, 04 Oct 2006 16:10:22 -0800

hosted in racks somewhere in Texas

By: Civil_Disobedient

Civil_Disobedient — Wed, 04 Oct 2006 16:36:42 -0800

I'm working with some sysadmin members about the possibility of moving the web server to something a bit more robust and failsafe.

Dual Xeons and a fat pipe could more than handle the hits if you could just use a decent web engine.

All the Coldfusion developer lists I'm on do say that CF runs faster and more stable on redhat, so I'm setting that up now...

Nooooooo...

By: smackfu

smackfu — Wed, 04 Oct 2006 16:46:26 -0800

Is there a cache for the static content, like for non-logged in users?

That would be easier to do on Linux, I think.

By: Brak

Brak — Wed, 04 Oct 2006 17:16:53 -0800

All the Coldfusion developer lists I'm on do say that CF runs faster and more stable on redhat, so I'm setting that up now...

Followed by a wholesale conversion to a PHP-based solution? Actually, I know very little about PHP, but I figure I'd be run out of town if I dropped my credentials as a (*gasp!*) .NET developer and suggested that conversion....

Then again, it worked for MySpace.

Matt, thanks for all you do. If you ever do convert MeFi to .NET and need any development/optimization help, I'd gladly repay my years of enjoyment of the MeFi community you've fostered. :)

By: twiggy

twiggy — Wed, 04 Oct 2006 18:44:47 -0800

Matt: You do know that you can limit [the observant/respectful] web bots to a certain number of hits within a given timeframe, right? If 30% of your traffic is web crawling bots, I'd say now's a good time to consider that.

Google for robots.txt to find out more...

By: delmoi

delmoi — Wed, 04 Oct 2006 19:02:07 -0800

Any thoughts of rewriting the system with the same DB schema in a language that doesn't suck?

By: mathowie

mathowie — Wed, 04 Oct 2006 19:05:08 -0800

Yeah, delmoi, moving to linux would mean moving to php/mysql would be the next steps. This is a stopgap measure before I move the db server to linux/mysql and the webserver from cf to php. It's something that would take quite a while, probably in stages as each subsite is converted.

By: smackfu

smackfu — Wed, 04 Oct 2006 19:39:03 -0800

Any thoughts of rewriting the system with the same DB schema in a language that doesn't suck?

Woo! Total rewrite! That always works wonders.

By: mock

mock — Wed, 04 Oct 2006 20:29:16 -0800

No, he said a language that doesn't suck - not php.

By: Mr. Gunn

Mr. Gunn — Wed, 04 Oct 2006 22:02:31 -0800

The Woz sighting, as well as the 200+ post mystery thread probably have a lot to do with the suckage today.

I appreciate your efforts, Matt.

By: strawberryviagra

strawberryviagra — Thu, 05 Oct 2006 01:01:52 -0800

Biggest of props, Matt.

By: lemonfridge

lemonfridge — Thu, 05 Oct 2006 01:19:48 -0800

No, he said a language that doesn't suck - not php.

So what language would you use? I was always under the impression that PHP+MySQL contained the least amount of suck.

By: Civil_Disobedient

Civil_Disobedient — Thu, 05 Oct 2006 03:47:21 -0800

No, it contains the possibility of huge amounts of suck, but for a simple message board like this, it's definately the right way to go.

By: cmonkey

cmonkey — Thu, 05 Oct 2006 03:57:19 -0800

PHP is so horrifically insecure that anyone planning on using PHP for anything should be forced to use Windows 3.1 for a year as penance.

By: greycap

greycap — Thu, 05 Oct 2006 04:37:32 -0800

That was a fascinating potted history of the life and servers of MeFi. I hadn't realised how expensive the hosting bills were - thank you, as ever, for all you do for the community.

By: yerfatma

yerfatma — Thu, 05 Oct 2006 05:34:48 -0800

Oh my. Double-plus developer love: a post about how the current application sucks and a flame war about tools instead of approach.

By: empath

empath — Thu, 05 Oct 2006 06:34:35 -0800

I didn't mean to say it sucks at all, FWIW. Sometimes not being able to access mefi is a feature, not a bug.

By: srboisvert

srboisvert — Thu, 05 Oct 2006 10:41:06 -0800

3 to 5 am is generally a "everyone's asleep" outage.

I think of it more as an early morning chance for UK knowledge workers to try and correct the productivity imbalance but then I load up Dicewars and wait...

By: mock

mock — Thu, 05 Oct 2006 18:30:13 -0800

PHP is the visual basic of our time. It allows semi-trained people to create useful applications which solve their business needs. None of this code is of the sort that you'd want to maintain even if you were being paid. It's great if your job is finding security vulnerabilities though.

It's not that it's impossible to create good code in PHP, clearly a great programmer can create great software despite the shortcomings of any tool. However the odds are against you being that programmer (for whatever value of 'you').

Personally I quite like the Catalyst Framework. Apparently the SomethingAwful forums are being rewritten to use it. You could always drink the DHH koolaid, and there are some nice frameworks (Django and TurboGears) in python if you find some of the idiosyncracies of perl objectionable.

Also while I'm on the tool hate, MySQL sucks, either use SQLite or PostgreSQL depending on the feature set you need. There is no good reason to continue using MySQL.

By: oxford blue

oxford blue — Thu, 05 Oct 2006 18:38:27 -0800

The hamsters are only human, afterall.