What would it take to cure it?

January 23, 2003 1:26 PM

A new server in June, a new version of Cold Fusion in July, a new hard drive in January and it's still an ailing mefi server. What would it take to cure it?
posted by timeistight to MetaFilter-related at 1:26 PM (105 comments total)

Wild guess: Money?
posted by Stan Chin at 1:36 PM on January 23, 2003

Half of us to die.
posted by crunchland at 1:43 PM on January 23, 2003

but i use both halfs of me...
posted by quonsar at 1:44 PM on January 23, 2003

Wild guess: Money?

Money for what? DNS-round robining? Redundant servers? What do the big boys do?
posted by timeistight at 1:48 PM on January 23, 2003

I'm sure this has been suggested before, but why not money? I mean, instead of voluntary donation, which apparently isn't making mathowie rich, why not take the top 200 posters (via the contribution index) and ask for $10 or $20? Or are we talking about much larger sums of money than that and I should just shut up?
posted by jonson at 1:49 PM on January 23, 2003

What would it take to cure it
A big frickin' hammer. That'll cure it. Permanently.
posted by PenDevil at 1:50 PM on January 23, 2003

What would it take to cure it?

Linux, PostgreSQL, Python.
posted by four panels at 2:02 PM on January 23, 2003

why not take the top 200 posters (via the contribution index) and ask for $10 or $20?

I'm ready to pay my share. but a flat tax? shouldn't it be progressive? (just don't tell the WSJ editorial board)
posted by matteo at 2:05 PM on January 23, 2003

Linux, PostgreSQL, Python.

Is that it? ColdFusion, SQL Server and IIS can't scale up? I thought some pretty large sites used these technologies.
posted by timeistight at 2:09 PM on January 23, 2003

why not take the top 200 posters (via the contribution index) and ask for $10 or $20?

While I see what you're getting at with this, I don't think it's a good idea. No.

I'd like to hear what Matt has to say about all the hardware problems and then speculate as to the solution.
posted by ashbury at 2:09 PM on January 23, 2003

"I thought some pretty large sites used these technologies."

I'm aware of some that do, but they don't try to run off a single server. MeFi may be getting to the point where it needs multiple boxes and replication.

Of course, this is all hypothetical until Matt drops by this thread.
posted by mr_crash_davis at 2:16 PM on January 23, 2003

Just use a LAMP platform. That will increase your speed.

On Preview:
Is that it? ColdFusion, SQL Server and IIS can't scale up? I thought some pretty large sites used these technologies.

IIS is a pig compared to Apache, at least if you're trying to conserve limited resources. I'm not the biggest fan of PostgreSQL (I like MySQL a lot, and once 4.0 gets transaction support, I probably wont even bother with other databases) but it's a lot better than SQLServer 2000, in my opinion.

And as far as Cold Fusion is concerned, I remember reading some benchmark results where perl, python and php blew CF out of the water.
posted by SweetJesus at 2:18 PM on January 23, 2003

But is this realistic, to expect the site to be recoded from scratch in a new scripting language, on a new web server using a new database on a new operating system?
posted by jonson at 2:31 PM on January 23, 2003

What would it take to cure it?

Fewer MetaTalk threads sucking up bandwidth.
This comment excepted
posted by yerfatma at 2:38 PM on January 23, 2003

my brother says he can get bits and bobs at cost,
would have to post it though from uk.
i know nowt about computers- he's a developer or something.
anyway worth a try.
posted by sgt.serenity at 2:46 PM on January 23, 2003

A couple of suggestions in lieu of a hardware or OS upgrade:

- For users not logged in, auto-archive threads and the front page every 5 or 10 minutes and display the cache to these users instead of pulling from the database. (I think this may already be done.)
- Display the cache to users who are logged in, but switch to database query mode when the user becomes "active" in a particular thread. A user can become active by previewing a comment or clicking a button somewhere on the page. The active status would be stored in a cookie that expires in x hours from the last activation.
- Eliminate auto-refresh when a thread page is reloaded but not re-requested (i.e. clicking "Back" from an outside link back into Metafilter.)

Basically, the amount of server time dedicated to a single end-user should be a function of that user's importance to the specific page. An end user who is not logged in is least critical to the operation of the particular page at time x, while someone in the act of clicking "post" to submit a comment is the most important user at the same time x. The server prioritizes accordingly.

Those who know better about how these things work (i.e. most of you) should feel free to flame away on anything technical I just said.
posted by PrinceValium at 2:54 PM on January 23, 2003

But is this realistic, to expect the site to be recoded from scratch in a new scripting language, on a new web server using a new database on a new operating system?

The only pain in the ass might be translating the code in to a new language, and even that isn't that hard. Changing databases isn't really that hard, as long as the databases are in SQL format, like Mifi's is. Changing servers is also not a problem, unless you use fontpage extensions or some crazy IIS shit that Apache doesn't support.

I used to have a peice of Mifi's code back from when someone posted a link to it on Metatalk (for some reason). With a few weeks of CFML practice, I'm pretty sure I could port it over to PHP.
posted by SweetJesus at 2:54 PM on January 23, 2003

If the free perl, python and php blow the expensive ColdFusion out of the water, how are Macromedia able to make money selling ColdFusion? (Same question for IIS vs Apache and MS-SQL vs MySQL/PostgreSQL.)
posted by timeistight at 3:14 PM on January 23, 2003

But is this realistic, to expect the site to be recoded from scratch in a new scripting language, on a new web server using a new database on a new operating system?

Sure. There are many programmers here who I expect would donate their time. Programming is fun. (I've been working on Phpilfer for a while now).

Apparently high-profile sites run Coldfusion, but then you don't know how many boxes they've got running behind the scenes to make it usable. I've used Coldfusion and it was acceptable, but from what I've read Coldfusion's performance compared to PHP or Perl is sluggish.

I think I've suggested it before, but starting from scratch doesn't mean you can't look at the old code and the algorithms it uses. A way to approach a software change like this is to make a beta.metafilter.com and give it access to a copy of the database. Allow programmers to make the site piece by piece and switch over when it's ready.

Matt's done a lot of work though, and it would take several months to replicate it. Replicating is always easier, you don't have to think.
posted by holloway at 3:14 PM on January 23, 2003

Racing stickers. And lots of them. At least it can look fast.
posted by hama7 at 3:16 PM on January 23, 2003

how are Macromedia able to make money selling ColdFusion?

If Titantic was a crummy movie how did it make money? If N*Sync are a generic band how do they make money? If BeOS was superior why did the company fold?

Coldfusion is mediocre. Mediocre + legacy users + marketing + commercial support and backing = profit.
posted by holloway at 3:20 PM on January 23, 2003

If the free perl, python and php blow the expensive ColdFusion out of the water, how are Macromedia able to make money selling ColdFusion? (Same question for IIS vs Apache and MS-SQL vs MySQL/PostgreSQL.)

2 Words: Development and Support

ColdFusion comes with a nice IDE, full debugging support, a paper manual, and a number to call if something goes wrong.

Apache, PHP, and MySQL don't (although the people who created PHP created Zend Inc, and now offer a nice, cross-platform IDE with debugging support, and contract-based tech support)
posted by SweetJesus at 3:28 PM on January 23, 2003

crunchland: Half of us to die.
quonsar: but i use both halfs of me...

maybe we could just die inside?
posted by blue_beetle at 3:59 PM on January 23, 2003

[quonsar and crunchland stare at blue beetle with fear and suspicion, having never before encountered a being who wasn't already dead inside]
posted by quonsar at 4:26 PM on January 23, 2003

What would it take to cure it?

If Sgt. Serenity behaves, we should be OK.
posted by planetkyoto at 4:48 PM on January 23, 2003

Everyone has raised major points (I knew saying the word "ailing" was a bit much, but again, I'm tired of pointless chat on the site) I would agree with.

I have converted much of the site over to pulling static files from a cache, but there is a lot more I could be doing to minimize db access (the ticketstubs site requires very little db utilization because I could build it from the ground up).

So yeah, in the short term the server would do well to have the memory increased and some code rewritten. Coldfusion actually does run faster on linux with mysql versus windows, and I've thought about moving over, though I'm far from knowing what I'm doing running a linux box (I think I do a decent job of security and admin duties on a windows box, and I've also got the help of friends there).

In the long term, the site would benefit greatly from moving to php/apache/linux/mysql and going GPL, but my php knowledge is pretty basic, and I'm not really interested nor do I have the time to run a software project I'm not the lead contributor on.
posted by mathowie at 5:37 PM on January 23, 2003

This thread is the equivalent of asking your dad when he's going to tune the engine on the classic car he restored for you, 'cause it's just not purring right.
posted by yerfatma at 6:02 PM on January 23, 2003

This thread is the equivalent of asking your dad when he's going to tune the engine on the classic car he restored for you, 'cause it's just not purring right.

A classic car exists by itself. A community weblog requires a community.
posted by the fire you left me at 6:07 PM on January 23, 2003

This thread is the equivalent of asking your dad when he's going to tune the engine on the classic car he restored for you, 'cause it's just not purring right.

A classic car exists by itself. A community weblog requires a community.

But sometimes you have a cranky dad that doesn't let his kids touch his car, much less help him tune it up. Then the kids talk about the fiberglass replica cars that can be built from the ground up cheaply, and run more reliably and faster than the old heap of junk Dad pushes around.

Dad should get rid of the old car and just get a PT Cruiser, y'know.
posted by mathowie at 6:11 PM on January 23, 2003

Matt, I'm sorry but that string of analogies totally lost me.

Are you saying that you're trading the new server in on a Chrysler?
posted by mr_crash_davis at 6:17 PM on January 23, 2003

my new leaf is all turned over.
posted by sgt.serenity at 6:28 PM on January 23, 2003

Are you saying that you're trading the new server in on a Chrysler?

I think I'm supposed to move to detroit. I lost myself in the analogy
posted by mathowie at 6:56 PM on January 23, 2003

Why is Metafilter down for me for the last few hours, but not Metatalk up?
posted by ParisParamus at 7:13 PM on January 23, 2003

(only Metatalk is on line?)

------------------------------------------------------------------------
posted by ParisParamus at 7:15 PM on January 23, 2003

Never, never discuss buying a PT Cruiser, even in jest.
posted by Hildago at 7:18 PM on January 23, 2003

<web developer mode>
On a site this size, the primary bottlenecks are going to be expensive database queries and maybe some code topheaviness, although from what I can tell, the application layer of this site is virtually nil, and the front end is as graceful as it could be.

RAM will certainly help with IIS performance, but I'd be willing to bet that there are db optimizations yet to be made--cluster indexing FPP's and comments in date order, double-checking all your query execution plans. With SQL Server, you can improve response time drastically if all of your queries run in stored procs and there's no dynamic SQL going from the app.

Also, the idea of running the front page as static HTML that gets updated by the server every n seconds is a good one. IIS can serve up quite a few (as in, like, hundreds, according to performance tests that we performed at my last job) static HTML pages in the time it takes to serve up one interpreted (ASP or CF) page.

Sorry to ramble; this is a core competency of mine.
</web developer mode>

So who here likes pancakes?
posted by vraxoin at 7:28 PM on January 23, 2003

Me! Me! I like pancakes!
posted by ac at 7:34 PM on January 23, 2003

The front page is static for all non-members, updated every five minutes. All archived threads are now flat files, and I'd like to make all thread pages flat files that get appended whenever someone comments.

I bet the db can use some optimization. Most common queries are stored procedures and not queries on the fly. I run queries through the analyzer once every few months, and most things are already optimized to the hilt, but once in a while I find a non-indexed table that has its query improved 10 to 100 fold by a quick new index. If anyone is a MS SQL guru and would want to take a crack at it, I'd love the help.
posted by mathowie at 7:55 PM on January 23, 2003

Well, OK, the first answer is always "more RAM", and RAM's cheap — so, can the current Athlon box hold more than 1 GB?
posted by nicwolff at 7:59 PM on January 23, 2003

Metafilter is still down....
posted by ParisParamus at 8:08 PM on January 23, 2003

Not for us, I think your computer gained sentience and refuses to load madamjujujive's latest post.
posted by Stan Chin at 8:09 PM on January 23, 2003

Actually, I just discovered that IE for Mac isn't working, but Mozilla is...
posted by ParisParamus at 8:13 PM on January 23, 2003

All archived threads are now flat files, and I'd like to make all thread pages flat files that get appended whenever someone comments.

This would probably make an enormous difference, as I would venture to guess that thread pages get more hits in aggregate than the front page does. On the /comments.mefi/xxxxx pages, the "preview" and "post" functions are really the only things that the database needs to get involved with on demand for an end user. On a thread with 50 comments this means that the page would make maybe 200 to 300 total requests to the database.
posted by PrinceValium at 8:35 PM on January 23, 2003

I keep expecting to see a MeTa post saying, "There's a big damn schlong bouncing all over the front page."
posted by Shane at 8:42 PM on January 23, 2003

I think mathowie should add one of those oversized racing spoilers to the server.

And a big, fucking R sticker too.

Those things make cars go like a muthafucka, boyee!
posted by grum@work at 8:51 PM on January 23, 2003

posted by the fire you left me at 9:28 PM on January 23, 2003

Do you feel it? It did get a little faster.


posted by mathowie at 10:03 PM on January 23, 2003

Metafilter: Fast. Furious.
posted by Stan Chin at 10:12 PM on January 23, 2003

/me bows before mathowie
The server is blazing fast now!
posted by riffola at 10:13 PM on January 23, 2003

Hey! love that flame job. Now isn't that more satisfying than buying a PT Cruiser?
posted by dg at 10:16 PM on January 23, 2003

jonson can add this to his list now:

mathowie: lives his life a quarter-mile at a time
posted by mr_crash_davis at 10:17 PM on January 23, 2003

There are many programmers here who I expect would donate their time.

No, they won't.
posted by anildash at 10:17 PM on January 23, 2003

I don't believe it--that's the box? Down on the right? Maybe I was expecting something more like this (note color). It's like finally seeing what the Car Talk guys look like...

No, I knew it was a regular PC all along, but still you form images. I used to be a sysadmin here. Fellow students were visibly underwhelmed, crestfallen even, when they finally laid eyes on the beige, undistinguished-looking merlin.sccs.swarthmore.edu.

Can't you paint it Cerulean Blue or something?
posted by tss at 10:42 PM on January 23, 2003

Oh yeaaah.
posted by stavrosthewonderchicken at 10:53 PM on January 23, 2003

Can't you paint it Cerulean Blue or something?

well, the old one was black, which was kind of cool. I considered getting a blue anodized aluminum chassis like the gamerz with mad skillz but it seemed like a frivolous waste of money for something that lives in a friggin closet.
posted by mathowie at 11:06 PM on January 23, 2003

Well, you could have just shown a photo of a blue anodized aluminum chassis like the gamerz with mad skillz and we would never have known the difference :-) The blue would show up the flames better, as well.
posted by dg at 11:21 PM on January 23, 2003

crunchland: Half of us to die.

Still the most compelling answer.

SweetJesus: Just use a LAMP platform. That will increase your speed.

Why change everything? The app is already well coded in CFML. IIS can definitely handle the load. The bottleneck is the DB, I hear. Keep what's working. Change what's not.

Now, let's start the open source DB religious war!

Would a dedicated DB server help in any way? I wouldn't know, I have never dealt with excessive DB load issues.
posted by qbert72 at 11:51 PM on January 23, 2003

So, static for non-members, eh? Just curiosity, but what percentage of visitors are members? Has this been asked already?
posted by Wood at 1:29 AM on January 24, 2003

Nah, phpmefi just didn't even start. Lots of people were interested but they didn't know where to fit in.
posted by holloway at 1:36 AM on January 24, 2003

Man, that thing looks fast even sitting at the stoplight!

I can barely reach my mouse!
posted by hama7 at 1:58 AM on January 24, 2003

Did disabling the "search comments" option speed things up much? I always had the feeling I was bringing the entire server to its knees whenever I tried a "since day one" comment search, and it usually timed out anyway...

I have no idea how many searches are being done—a couple a minute? dozens? But if searching is still putting an inordinate load on the server, one thing to consider might be caching search result pages for frequently searched-for terms. (When a new post is made, you could check its text against all the terms for which cached pages currently exist, and clear them if necessary, or even insert the post into the relevant cached results pages.)
posted by staggernation at 6:56 AM on January 24, 2003

i don't know how search is implemented now, but if all pages will be static soon then it might be worth looking at a simple indexing/search engine that processes pages once (ie when each static page is created) rather than chugging through the whole database each time.
posted by andrew cooke at 7:10 AM on January 24, 2003

I believe you can use Indexing Services to search a SQL Server dateabase. Would that take some load of the database itself?
posted by timeistight at 8:29 AM on January 24, 2003

Why change everything? The app is already well coded in CFML. IIS can definitely handle the load. The bottleneck is the DB, I hear. Keep what's working. Change what's not.

If you're looking for the best way to make the site run as fast as it can, then rebuilding the app from scratch (or close to scratch) is a good idea. I'm sure Matt has made a number of upgrades on the site's code since the initial version was written, and this makes the code not as efficient as it was when it was first written.

PHP is really fast. With a Zend Optimizer and a Zend Cache add-on, its like fucking lightning. Way, way faster than CF on Windows.

PHP 4.3 is a fully featured programming language designed just to build web applications (although you can use it to write shell scripts, and with a GTK add-on, GUI applications in X). It's object oriented, which CF isn't (I think), and that makes a whole world of difference when it comes to making an efficient application.

It may not be worth Matt's time to invest his energy on this, but it would squeeze every last bit of power out of the web server. Then again, if it is, I'm willing to help.
posted by SweetJesus at 8:43 AM on January 24, 2003

ColdFusion's engine is written in Java, which means that it is object-oriented by rule. php's strength is in the number of different databases which it supports, which by far blows most others out of the water.

so what is the most database intensive query that gets used often? (i feel deja-vu. i think i participate in too many let's-soup-up-metafilter threads.) searching metafilter is a back-breaker, but by now most people know to use google and not mefi's built-in search procedure. otherwise, the bottleneck is probably the number of pageviews it gets from members on the front page and in comment pages. there's no real cure for that, short of a way to distribute queries across several servers.
posted by moz at 9:09 AM on January 24, 2003

And when I say PHP with the Zend Optimizer and the Zend Cache is like lightning, I'm not kidding. PHP is already faster that CF, and with the Zend Optimizer it can get up to 100% faster, so already you're talking about doubling the speed. Matt mentioned something about wanting to serve the all the pages up as static until changes or comments are made. With the Zend Cache, that's already done for you, so you don't even need to bog down your application code with rouitines to cache files.

And if you feel like spending 500 bucks, you can blow your speed through the roof. It is the fastest web scripting language out there, by far. That's probably one reason why it's the number 1 Apache mod by far (almost 20% more popular than any other Apache module add-on). PHP also runs on over 10,000,000 web sites, and with CPAN-like support from PEAR, there are tons of pre-built objects to use.

It's a really neat language, and it makes web development infinitly easier.
posted by SweetJesus at 9:10 AM on January 24, 2003

ColdFusion's engine is written in Java, which means that it is object-oriented by rule. php's strength is in the number of different databases which it supports, which by far blows most others out of the water.

That's not what I'm talking about. I assumed that CF was written in an object oriented language, although I didn't know it was Java. That must slow it down a whole bunch (having to run the byte code though a Java VM is much slower than native C or C++).

I was talking about the language supporting objects. CFML is a tag based language, similar to HTML in syntax. PHP is very similar to C, and pretty similar to perl. It has object support, which would make redesigning MetaFilter easier, and increase reusability, which increases speed. CFML doesn't have object support (as far as I know).
posted by SweetJesus at 9:18 AM on January 24, 2003

Whoops, my bad Moz. I misread the beginning of that article, and though that just CF's engine was object oriented, and not the language. But I think this is the first time it's had this support, so Metafilter probably doesn't take advantage of it.
posted by SweetJesus at 9:24 AM on January 24, 2003

SweetJesus:

But I think this is the first time it's had this support, so Metafilter probably doesn't take advantage of it.

matt had a bitch of a time, a while ago, getting metafilter to work with the new version of ColdFusion. i don't know if that qualifies as matt "taking advantage" of OO support.

It has object support, which would make redesigning MetaFilter easier, and increase reusability, which increases speed.

1. how would object-oriented programming support make redesigning metafilter easier?
2. reusability and speed do not have anything to do with each other, unless by speed you imply speed-of-development. (and i am not of the opinion that OOP provides a dramatic increase in that kind of speed, either.)

if you are talking about speed-of-redevelopment, keep in mind matt is not a programmer by profession. in that case, i think OOP begins to become a hinderance. luckily, you aren't forced to use it in PHP the way you are with java.

PHP also runs on over 10,000,000 web sites, and with CPAN-like support from PEAR, there are tons of pre-built objects to use.

PEAR is very nice. i much prefer to use it over PHP's (in my opinion, rather poorly designed) builtin functions.
posted by moz at 9:39 AM on January 24, 2003

I know PHP is better. Trust me, I know. It's free, it's fast, it's easy, and it's powerful. While my knowledge of it is pretty basic, I am convinced.

My problem with using it (or having others do all the code for me) is that if at 3am I want to add another scholarship contest, rearrange the way trackbacks work, and/or add new member features, I can't code them because I don't have the experience with the language. Granted, most of moving from one language to another is just syntax, the logic should stay the same ("on this page, we grab some data from the database and display it"), but I don't really have the time to move over to php entirely, and would hate to throw 4 years of experience away.

I'm going to be a stick in the mud about this because I'm not quite motivated enough yet to sit down and recode the whole thing.
posted by mathowie at 9:41 AM on January 24, 2003

SweetJesus: back in June 2001, mathowie invited us to develop the MetaFilter functionality in open source tools, something he's has made clear that he's not going to do himself. As anildash points out, the response to that has been underwhelming.

It may be a question of having lots of cats, but no one to herd them.
posted by timeistight at 9:49 AM on January 24, 2003

I kind of flubbed that project from the start, and take the blame for not leading it. If there's any wonder why it failed, it'd be my non-leadership on it.

I think I really wanted to say this on that original post about a php version of mefi. I just wanted to see others build it themselves, for themselves (you could build the basics in a single day, really). It's not magic, it's not that hard, anyone can do it.
posted by mathowie at 10:13 AM on January 24, 2003

1. how would object-oriented programming support make redesigning metafilter easier?

Well, object oriented programming makes adding new features to existing pieces of object-specific code much easier (just add a new method to your object, and access it where needed in your inline code. Much cleaner and easier to understand than adding it to a library file). It's (arguably) easier for other programmers to understand your code, and quickly get a grasp of how everything fits together. It also makes for generally cleaner code, which is always good.

It would also probably be easy to separate MetaFilter's basic features, such as comments, user's profiles, user's display settings, which would speed up design (Again, this is arguably, but in my experience I find it much easier to build an app from an OO perspective than from a structured one.)

2. reusability and speed do not have anything to do with each other, unless by speed you imply speed-of-development. (and i am not of the opinion that OOP provides a dramatic increase in that kind of speed, either.)

I do imply speed of development, as well as ease and speed of maintenance.

(and i am not of the opinion that OOP provides a dramatic increase in that kind of speed, either.)

Take a look at Object Oriented and Classical Software Engineering (I read it as part of a software engineering course, last spring.), specifically this[PDF file, scroll down to page 17]. Matt's site might not be 500,000 lines of code, but the speed benefits are still there.

My problem with using it (or having others do all the code for me) is that if at 3am I want to add another scholarship contest, rearrange the way trackbacks work, and/or add new member features, I can't code them because I don't have the experience with the language.

I don't blame you at all for this Matt. I wouldn't want to run a web site in a language I don't know, and be at the mercy of others. That would really be a pain in the ass. But if you ever get any free time, and need a project, take a look at PHP. I think it's probably one of the easiest languages to learn, in my opinion. It's probably the fastest I've ever learned a programming language.

SweetJesus: back in June 2001, mathowie invited us to develop the MetaFilter functionality in open source tools, something he's has made clear that he's not going to do himself. As anildash points out, the response to that has been underwhelming.

Back in Summer 2001 I was an un-employed software engineer, moonlighting as a medieval english/tropical paradise-themed hotel bartender. I was kind of lying low, and staying away from the internet and computers for a while. If I had know about it, I probably would have helped out.
posted by SweetJesus at 10:24 AM on January 24, 2003

SweetJesus:

Well, object oriented programming makes adding new features to existing pieces of object-specific code much easier (just add a new method to your object, and access it where needed in your inline code. Much cleaner and easier to understand than adding it to a library file).

adding a new method to an object is no different than defining a function that accepts an object as its first parameter, nor is it easier. moreover, most OOP languages encourage you to use "library" files for the storage of object methods.

It would also probably be easy to separate MetaFilter's basic features, such as comments, user's profiles, user's display settings, which would speed up design

what makes you feel that these are not already seperated?

Take a look at Object Oriented and Classical Software Engineering (I read it as part of a software engineering course, last spring.), specifically this[PDF file, scroll down to page 17].

there's a lot of overhead to OOP design. you have to lay out your objects; design their classes. for small-to-mid level projects, i think a fully-realized OOP design may not be necessary.

i do like object oriented design, but it has to be the right situation. is metafilter that? comments and posts aren't generally represented as objects, i wouldn't think: rather as records in a database. comments have their own table; posts their own table. does it make much sense to define them formally as objects in a language?
posted by moz at 11:15 AM on January 24, 2003

we're here to welcome you to the Inn with Personality..
posted by sgt.serenity at 11:38 AM on January 24, 2003

Regardless of the programatical solution, I still think we need to kill half of you.
posted by crunchland at 11:44 AM on January 24, 2003

adding a new method to an object is no different than defining a function that accepts an object as its first parameter, nor is it easier. moreover, most OOP languages encourage you to use "library" files for the storage of object methods.

This just comes down to a matter of preference. I've spent more time with object oriented languages, and I have a better design background when it comes to them.

what makes you feel that these are not already separated?

Nothing. I imagine they are. But if you're going to rebuild something, you start from close to scratch. Might as well sperate everything into objects.

there's a lot of overhead to OOP design. you have to lay out your objects; design their classes. for small-to-mid level projects, i think a fully-realized OOP design may not be necessary.

Not necessarily. You can keep the overhead down if you design for maximum reusability. That's one of the benefits of OOP design.

i do like object oriented design, but it has to be the right situation. is metafilter that? comments and posts aren't generally represented as objects, i wouldn't think: rather as records in a database. comments have their own table; posts their own table. does it make much sense to define them formally as objects in a language?

Yes, I think so. If you're going to generate a page such as this, you're going to be grabbing a large amount of information from the database. You'll probably run a loop on a database query, and assign the results into an array. At that point you've got 5 or 6 variables that you're dealing with, and printing out in specific areas. Instead you could create an array with a set of objects, and add the database info to these objects. Now you don't have to deal with multiple for-loops and several double arrays. You can create a method in the object called display_comment(), and can contain all the HTML print out routines for the comment. Now all you have to do is use 2 loops to generate a comment page - 1 for the database, and one for display_comment() function. You've just cut down your inline code a bunch, and streamlined your code.
posted by SweetJesus at 11:50 AM on January 24, 2003

Instead you could create an array with a set of objects, and add the database info to these objects. Now you don't have to deal with multiple for-loops and several double arrays.

you actually needn't bother with multiple for-loops when you query from a database to start with. when you perform a query -- for example, using mysql_query(...) -- the results of the query are assigned to an array that is passed by parameter to the function. all that is left is to iterate over the result set, printing the values of each result. this can be accomplished, quite simply, without any need for a class definition.

now, mind you, i'm not trying to say that class definition is horrible and/or needless. i'm an advocate of lazy programming: i hate doing things that i don't have to. in this case, you just haven't demonstrated to me that classes are necessary.
posted by moz at 12:14 PM on January 24, 2003

I'm a terminal lurker, but I feel obliged to speak up for ColdFusion. I also love reading here, so this is my knowledge dump donation. My creds: I'm a programmer on the CF server. About six months ago I was stolen from the JRun team, where I worked on the servlet engine. I also helped out on "Neo" while CF5 was still being created.

CF speed: I have a real hard time believing php, Python or Perl is faster than CF. I've never noticed anything lightning fast about php or Python, or any webapp technology for that matter. Sockets, threads, and databases are where the bottlenecks are, which doesn't change much from technology to technology.

CF pages, under the hood, are servlets. Most of the value of the CF core is syntactic sugar and add-ons. If you think servlets are slow, you'll think CF is slow. Benchmarking JRun 4, however, we came within 20% of static pages for a simple servlet. Unless PHP can run faster than static pages, I don't think switching is going to make much of a difference.

PHP: CF is object-orientated in CFCs and in Java add-ons. I'd guess CF supports more databases as PHP, since this has been it's bread-and-butter for the past decade. Benchmarks, as everyone probably already knows, are generally worthless. Google for "petstore performance" if you want to see the latest server bogusness. Benchmark your own app's most used areas, and then decide for yourself.

Java: JVMs are wicked fast these days. Saying C/C++ is always faster just isn't true with hotspot VMs.

CF can scale up: many people in the past used ClusterCats (must have enterprise edition), many people today use their J2EE server's clustering (must have J2EE edition, if that's not obvious). There's a technote out there on using JRun's session replication for clustering/failover with CF. I've used this myself, and it works quite well.

CF stability: we do MTBF tests and always beat our numbers. Hundreds of fixes have been in the two updaters, so apply these ASAP if you haven't already. As for CF on Linux, please don't mention the eweek article. Oh shoot, I just did. Some users have reported problems with CFMX on Linux. All of these issues are on a tech note on our site. Almost all of these issues are addressed in Updater 3, which will be out shortly.

Also, CF is also a thousand times easier to use than other webapp technologies. Of course that's just my completely unbiased opinion. :)
posted by luyon at 12:21 PM on January 24, 2003

for people who are curious about oo i (self link!) once spent a lot of time on an essay (pdf) (trying to) explain it to non-computer-literate audience. in retrospect, it wasn't as good as i'd hoped (i suspect i aimed a tad high), but anyway...

and on a more technical note, adding objects extends the type system which gives you more ways to describe relationships between entities than a similar language without it. the overheads are normally negligible in most apps because (1) programmer time is usually more important than run time (2) run time problems are usually in small isolated areas (hot-spots) which can be tweaked or, if desperate, implemented without objects and (3) they (the overheads) are usually small. and if you're writing about something then using a more descriptive language often helps - but then there's usually a shed-load of things more important in choosing a language.
posted by andrew cooke at 12:21 PM on January 24, 2003

Well, ok. Here are some other reasons. For one, objects can be serialized. If you're trying to cut down on wasteful database queries, only grab a comment from the database when one is added, serialize the comment as an object, and generate the pages from the serialized objects. The comments on these pages are static, so there is no reason to keep hitting the database for comments. You could have a script on a crontab to run, say, 3am every night to clean up the objects and re-generate the first 100 font page posts in the database. The rest of the threads aren't accessed as much, so you can just query the database as needed.

This would cut down on the database stress a bunch, and is not possible (with out pain) in most programming languages, and it's not possible to do without objects in PHP. This might take up more hard drive space, but that's the expense.

This process could be applied to just about every aspect of the site, making all the pages database-static, but php-dynamic.

Another advantage of a pure OOP design, in php atleast, is that all PEAR packages are OOP. It's a coding standard. If you're going to be using PEAR packages, they're very useful for things such as logging, and sessions, you might as well go all out OOP.

now, mind you, i'm not trying to say that class definition is horrible and/or needless. i'm an advocate of lazy programming: i hate doing things that i don't have to. in this case, you just haven't demonstrated to me that classes are necessary

Classes aren't necessary, but I think they're the best way to go. You could created Metafilter in an non-OOP way if you wanted. But at this point in PHP's evolution, not going OO for an web site like this is not the smart thing to do.

Maybe I'll take a look at implementing this.
posted by SweetJesus at 12:36 PM on January 24, 2003

So far the only thing that's come out of this thread is that we all know we can do a better job. However none of us have built a site like MetaFilter, so why not put our money(time) where our mouth is?

(and yes, I include myself in that, I've built a whole knowledge base application that's based on publishing articles, commenting and editing them, am I gonna build a new Mefi? eeeuhhmmm...)
posted by sebas at 12:36 PM on January 24, 2003

Maybe I'll take a look at implementing this.

Three seconds... Three seconds between preview and posting. Now my whole comment is useless ;)
posted by sebas at 12:39 PM on January 24, 2003

Regardless of the programatical solution, I still think we need to kill half of you.

Anyone without a 1 as their first digit. Strip out the dead wood...;p
posted by inpHilltr8r at 12:50 PM on January 24, 2003

Another $0.02 to the mix:

I develop in CF. I'm curious about PHP, but I haven't found a huge justification to switch my company's intranet codebase over. I like, as others have pointed out, having a real company with real support behind it.

Regarding performance: I can't speak to details, but I'm inclined to agree with luyon. Performance differences these days, until you reach massive load, vary wildly by condition. MeFi's a fairly successful site, but I don't think it gets the volume of hits that would highlight any substantial difference at the app server level.

My thoughts-

- Ditch Windows. I know you're not a Linux guy, but CF on Solaris, at least, blows Windows away.

- RAM, RAM, RAM. Especially for the db. You can't have enough.

- Are you running CF and MSSQL on the same box? If so, that's a problem. Windows really wants to be set up as a single-purpose machine (e.g. one for your app server, one for your db). The multitasking just can't compare to UNIX, which is much better suited to a "one box does it all" setup.

- How's your disk? It's an often-overlooked piece, and it's always the slowest part of any machine. Of course, having oodles of RAM gets around this issue.

- BBS sites are always going to be poor performers. Large chunks of text are meant to be stored in files, not db fields. The sheer volume of data you're pushing between your db and the app server is going to hurt performance.

- Going OO will do *zilch* for this site. It's well put together, but it's not rocket science, and doesn't have ongoing code expansion.

Hey luyon- is CFLDAP going to be fixed in the next patch?
posted by mkultra at 12:52 PM on January 24, 2003

I haven't OO'd (pronounced "oo oopted", by stupid people) Phpilfer because the OO in PHP is arse.

PHP 4.3, which fixes some OO problems, was released on... *counts-on-fingers*... a few days ago, but everyone is waiting for PHP 5 as the OO release.

Mkultra is probably right. OO won't help much more than a sensible function naming scheme. I would imagine that this site is around 5000 lines. Use OO only when it'll help.

OO, like functions, are just another way to help you organise and reuse your code. Some people will say that organisation is neccessary, and others will say that organisation gets in the way for something of this size.

5000 lines is a desk. I don't organise my desk unless I'm going away for christmas.
posted by holloway at 1:59 PM on January 24, 2003

because the OO in PHP is arse.

Word.
posted by yerfatma at 3:01 PM on January 24, 2003

Anyone without a 1 as their first digit. Strip out the dead wood...;p

I was actually thinking of suggesting we put all the even numbered users up against the wall.
posted by crunchland at 3:10 PM on January 24, 2003

Let's do both: that'd twice as good.
posted by timeistight at 3:17 PM on January 24, 2003

be twice as good.
posted by timeistight at 3:59 PM on January 24, 2003

I was actually thinking of suggesting we put all the even numbered users up against the wall.

Uh oh.

Language features and speed arguments are not helping Matt. Matt is not going to redo all his work in another language. On top of that, the "application server" is not what's slowest. You'll just get faster scripts waiting a little longer for the DB. Net gain: nil.

I've never noticed anything lightning fast about php or Python, or any webapp technology for that matter. Sockets, threads, and databases are where the bottlenecks are, which doesn't change much from technology to technology.
Amen to that.

So we need to cut down even more on DB usage:
serialize the comment as an object, and generate the pages from the serialized objects
Interesting suggestion. How complicated to implement?

Or, allow the DB more resources:
Are you running CF and MSSQL on the same box? If so, that's a problem. Windows really wants to be set up as a single-purpose machine (e.g. one for your app server, one for your db).
My thoughts, exactly. This is where money could help, and Matt's time investment would be "minimal". Where's that donation page again?

[/project manager]
posted by qbert72 at 3:59 PM on January 24, 2003

Ahem, as a professional developer, I can say this and be reasonable comfortable with it: It really doesn't matter which of the major platforms (OS, DB, Web) you use, it's what you do with it. You can make anything work (just about). I work mainly with .NET so of course that's what I'd use. The site wasn't built with that, so it'd be silly to actually use it here. Same for Java, cobol, punch cards or whatever technology someone wants to recommend. It's not the tech that makes it scale, it's the person writing it, and their abilities. It's the big M that got us here so far, and his skills will get us the rest of the way.

That being said. I vote we change Metafilter to a snail-mail newsletter, printed on a hand-cranked printing press. Limit the distribution only to people without a computer (that way spelling mistakes in the URLs wont be noticed). Over time, as readership increases, we leave it up to the post
office to "scale the delivery system".

Either that or outsource the whole thing to slashdot.org.
posted by blue_beetle at 6:13 PM on January 24, 2003

blue_beetle, I'm pretty sure that's the first time anyone's ever put the phrase "I work mainly with .NET" in the same post as "outsource the whole thing to slashdot."
posted by vraxoin at 7:28 PM on January 24, 2003

If the solution is more hardware and processing power (and it seems to be the case), can anyone quantify the $$ involved to make it happen? I think a fundraising among the community would be quite successful, but some kind of quantifiable goal would empower more people to participate.
posted by PrinceValium at 1:28 PM on January 25, 2003

Does anyone have any money bet on the Superbowl? If you win, how about donating some or all of your winnings to MeFi? If not to upgrade the servers, just because there is almost no advertising and it has been a while since there has been a fundraising campaign. Think of it kind of like PBS or NPR, only better - its interactive, more interesting and actually has less advertising.
posted by caddis at 1:44 PM on January 25, 2003

If anyone is interested in a Perl/MySQL clone of MetaFilter, check the site in my user page (no self links here!). Source code isn't available yet, until it gets a few weeks of bug tracking and testing.. but then it'll be going completely open source.

Alternatively, you PHP people could code your own. It's not particularly hard. It's the scalability and optimization that ends up becoming the problem, as it appears Matt has experienced.
posted by wackybrit at 10:16 PM on January 25, 2003

Oh, and maybe the nice people over at BuffyFilter who appear to have made another MeFi clone in PHP will release their code at some time.
posted by wackybrit at 10:22 PM on January 25, 2003

I'm generally in favor of Linux/Postgres/PHP solution but I doubt it's worth the time and it would take time, not just for Matt to learn but to optimize it all so that it would be as good as it is now, and mroe time to make it better. It might be worth it in the long run (assuming a long run!) but those first steps are doozies.

A couple of friends of mine talked about this thread at breakfast this morning and my friend Scott wanted to know how many network cards were in the server and suggested that one possible solution would be to put more in to serve traffic through.
posted by wobh at 9:49 AM on January 26, 2003

PHP Zend Cache is one of many Opcode caches
posted by holloway at 10:52 AM on January 26, 2003

It might be worth it in the long run (assuming a long run!) but those first steps are doozies.

It would be, if Matt was planning to sell and setup these sorta systems for large companies. But there have often been words about trimming the site down, shutting it down, etc, and this is just a small community.

I can't speak for Matt, but I don't know it'd be worth his time learning all of that stuff, unless he planned to actually get into Linux/Postgres/PHP/whatever for his real work (in which case, MeFi would be a great warmup project!)
posted by wackybrit at 7:29 PM on January 26, 2003

Matt doesn't seem interested in this. The technical argument is a small speed increase and that more eyes which probably means fewer bugs.

I haven't donated any money to Mefi* but the most appealing idea to me is that Metafilter's users could distribute the effort instead of lumping it on Matt, and we'd get a site where the routine "balanceTags" had been declared only once in a different template.

*because I don't have a credit card so it's not easy, though I did buy "Usability: The site speaks for itself" which was dang nifty.
posted by holloway at 8:45 PM on January 26, 2003

What are the licensing considerations of using the Metafilter concept even without adapting the original code? Does Matt's adoption of the Creative Commons license have any implications for people wanting to set up a FooFilter of their choice?
posted by PrinceValium at 7:18 AM on January 27, 2003

The best ideas are always the simplest and I think that crunchland has hit upon the perfect solution with his "even numbers against the wall" idea. Except, of course, that registered users are only part of the load on the server, as there are a lot of lurkers. Still, the idea of getting rid of the even-numbered users has attractions (no names mentioned)....
posted by dg at 2:51 PM on January 27, 2003

A little slow on the response, but if anyone is still reading this: a separate db machine sounds like a great, easy-to-do idea that just needs the money. Also I'll mention the whole Updater thing again just in case. As for the LDAP fix, mkultra, don't know if its what you're looking for but all I see is "cfldap tag with a modify attribute now works correctly when passing multi-value attributes."
posted by luyon at 7:12 PM on January 27, 2003

luyon- YES! That's exactly what I've been waiting for! Thanks for the heads-up :)
posted by mkultra at 7:35 AM on January 28, 2003