server donation request May 7, 2004 2:49 PM   Subscribe

Here's a direct approach to the constant metafilter downtime : what would it cost (roughly) to pay someone to port Metafilter to a more reliable platform, and are there enough site members who care sufficiently to foot the bill ? Couldn't Matt just put a little permanent add/plea for $ donations - along with the amount collected and the amount still needed - on the front page ? Why not ?
posted by troutfishing to Bugs at 2:49 PM (62 comments total)

I'm not sure I understand what's actually broken. The downtime never used to be as bad as this, so what's changed?
posted by bonaldi at 2:51 PM on May 7, 2004


It's users 17359 and up. We warned you all about reopening registrations, and did you listen? NOOoOoOOOoo.
posted by mr_crash_davis at 2:58 PM on May 7, 2004


they killed mefi? you bastards! ; >
posted by amberglow at 3:05 PM on May 7, 2004


There's really not much need for porting metafilter, there are plenty of free clones available. Adding stuff like Ask MeFi might need to be coded in, but there's plenty of competent coders on MeFi to not make it a money issue. Database migration might be a bit of an issue, but as someone pointed out earlier, if you're willing to convert the old pages to HTML it's just user data, which isn't much. Even putting everything in the database shouldn't be too much effort.

On the other hand, even though I'm fond of neither, there is a certain charm in being just about the only major community blog out there running IIS/ASP.
posted by fvw at 3:08 PM on May 7, 2004


As Matt has said, the problem is the frontend and backend software (IIS/SQL Server/ColdFusion). The hardware is secondary - the code needs to be rewritten in PHP/MySQL and moved to Apache.
posted by PrinceValium at 3:08 PM on May 7, 2004


I'm not sure I understand what's actually broken. The downtime never used to be as bad as this, so what's changed?

since when does something need to change for mickeysoft to decide to get flakey?
posted by quonsar at 3:26 PM on May 7, 2004


they killed mefi? you bastards! ; >

sorry
posted by mr.marx at 3:40 PM on May 7, 2004


So sorry. I'm sure it's mostly me. No, really. You know someone would say it, so I thought I'd take the initiative.
posted by Ethereal Bligh at 3:42 PM on May 7, 2004


/me waves to quonsar, wishing he could IRC from work....
posted by Steve_at_Linnwood at 3:44 PM on May 7, 2004


My understanding is that the problem lies with IIS and the move is mostly a time issue for Matt, since Apache is free.
posted by gwint at 3:45 PM on May 7, 2004


just don't do it again mr.marx (or EB), mmkay?
posted by amberglow at 4:00 PM on May 7, 2004


I don't need to port it to a new platform, I just need to work on basic server stability. I'm moving to apache and/or win2003 server soon, but have to boot some friends off the box that were relying on the old IIS install (and ASP). I swear things will get better in June, though if I can figure out why IIS 5 is killing itself every few hours, it'll get better sooner.

Who here has ever run a site serving about 3 million pages out per month running on IIS/MSSQL/CF?
posted by mathowie (staff) at 4:06 PM on May 7, 2004


only you, master! :-)
posted by quonsar at 4:10 PM on May 7, 2004


/me waves back at S@L!
posted by quonsar at 4:11 PM on May 7, 2004


Who here has ever run a site serving about 3 million pages out per month running on IIS/MSSQL/CF?

Maybe this guy?
posted by Civil_Disobedient at 5:30 PM on May 7, 2004


Dammit, the user search said that was bill_gates's account. 1 am s0 l4me.
posted by Civil_Disobedient at 5:33 PM on May 7, 2004


LOL civil, I was wondering about that. Seeing as how that guys homepage is about women hangliding. I didn't think that was such a popular sport nowadays, but I was about to reconsider my whole perspective on the issue.
posted by lazaruslong at 5:53 PM on May 7, 2004


Hey, perhaps it's some malicious european! The crashes don't happen during the US day, (do they matt?) only at night. Which is our morning. Perhaps some pissed-off fucker is doing *something* to kill the box. Grr!

Anything suspicious in logs, matt?
posted by bonaldi at 6:34 PM on May 7, 2004


well, I'm european
posted by mr.marx at 7:16 PM on May 7, 2004


Who here has ever run a site serving about 3 million pages out per month running on IIS/MSSQL/CF?

I could have sworn we had a place for questions like this.
posted by anathema at 7:53 PM on May 7, 2004


It seemed such a straightforward question....

Oh well.
posted by troutfishing at 11:03 PM on May 7, 2004


I worked for a fairly high traffic cf/iis/mssql site for awhile... nothing in the 3 million range, probably close to about a third of that. 'Course, this was on CF5, so MX probably behaves a lot differently...

I'd imagine the standard rules apply to any site like this, though. If possible, move your db server to a different box and index your tables properly. Take advantage of CF's built-in ability to cache queries where you can... little efficiencies in code can make a big difference in the long run.
posted by ph00dz at 11:24 PM on May 7, 2004


But the problem is the IIS service crashing and not re-starting automatically though isn't it? If it were me, I'd just uninstall IIS and re-install it.
posted by chill at 1:59 AM on May 8, 2004


There's really not much need for porting metafilter, there are plenty of free clones available.

I can assure you, based on personal experience with some FAR less popular sites, none of the free clones are up to the job.
posted by mmoncur at 2:23 AM on May 8, 2004


Or pay someone to do it. God bless you Matt, but you're clearly busy with life, your website's uptime is abysmal, and you have dozens, if not hundreds, of people willing to either solve the problem themselves or contribute money toward it. Why not just unleash the hounds on this problem? Is it that you don't love us?
posted by stupidsexyFlanders at 2:27 AM on May 8, 2004


stupidsexyFlanders - I'm glad you said that. Here's my train of logic on this :

1) The constant downtime discourages Metafilter participation and readership. This seems a shame to me and either it is a desired effect for unknown ends - perhaps Metafilter is a secret sociological experiment in intermittent reinforcement ? - or not. If not then :

2) There are solutions to this. I'm not a programmer, but I refuse to believe that the basic software challenge is a profound one so :

3) I have to assume that Matt doesn't derive enough benefit from the site for it to be worthwhile (in his own personal utilitarian calculus) to either put in the time to wrestle the existing code into submission, change to a different platform, or pay someone else to do the job.

4) Hence, my question - are there enough committed Metafilter members, willing to contribute five, ten or twenty dollars each, to raise enough money to buy a solution ?
posted by troutfishing at 6:01 AM on May 8, 2004


Who here has ever run a site serving about 3 million pages out per month running on IIS/MSSQL/CF?

could someone stick this question on ask.metafilter?
posted by mecran01 at 7:08 AM on May 8, 2004


Perl code for auto-restarting IIs

Not sure if that will work in Matt's context...
posted by mecran01 at 7:11 AM on May 8, 2004


Sacrilege in saying this, but I think mathowie doesn't want to solicit outside help because it must might foster an implicit obligation towards users. That would limit his options.
posted by Gyan at 7:29 AM on May 8, 2004


why IIS 5 is killing itself every few hours

of course you can set IIS to restart itself as well.
posted by specialk420 at 8:25 AM on May 8, 2004


Matt has said that it is set to restart itself, but it is behaving badly.
posted by anathema at 8:45 AM on May 8, 2004


Gyan, I think you hit on something here.

Despite the frequent threads concerning the downtime and server status, user #1 has yet to solicit outside help (at least publicly). I think there might be a message here. Matt has never struck me as being shy and I'm sure if there was anything he wanted from us he would let us know, until that happens, get off the guys back.
posted by cedar at 10:17 AM on May 8, 2004


I purchase hosting services from Michal Wallace (Cornerhost) whose full-time job is hosting small websites on eight LAMP servers. I doubt if the aggregate traffic of all his clients approaches that of MetaFilter. He purchases co-location and monitoring services for these servers at a large professional facility whose name escapes me. Michal keeps these machines updated, patched and monitored for problems. In spite of all this, sites or whole servers occasionally go down until Michal can restart them.

MetaFilter has a part-time admin. It's housed in a busy medical student's closet. Why on earth would we expect it to stop having problems just because of a change in server software?

If there really is some magic combination of software that would flawlessly run a high traffic site like this if minimal human intervention I like to hear about it. I'll start a part-time hosting company and get rich.
posted by timeistight at 10:23 AM on May 8, 2004


sites or whole servers occasionally go down

the operative word, timeistight, is "occasionally".
posted by quonsar at 12:17 PM on May 8, 2004


Without spending some time looking at the specifics of the Metafilter setup, it's silly to dismiss any problems as simply the fault of IIS. Either Apache or IIS are perfectly capable of run high-traffic sites. Here's a survey suggesting that a significant portion of top companies are running it. That survey is probably biased, but not enough to completely dismiss IIS as an unwokable solution to high-traffic sites.
posted by normy at 12:31 PM on May 8, 2004


unwokable ...unworkable, even.
posted by normy at 12:37 PM on May 8, 2004


a significant portion of top companies are running it

...with a paid babysitting staff, yes.
posted by quonsar at 12:59 PM on May 8, 2004


...with a paid babysitting staff, yes.

Yes, of course. I doubt those running Apache stick the server in a closet and pretend it's not there, either.
posted by normy at 1:16 PM on May 8, 2004


nobody pretends a server isn't there. however, put side by side, in the same closet, running equivalent db-driven sites, an apache/php/postgresql platform is far more likely to be forgotten about by the admin because it will, far more often than not, simply keep working. the IIS/ASP/CF/WHATEVER box will be requiring regular restarts. it's an indisputable fact.
posted by quonsar at 2:00 PM on May 8, 2004


normy: "I doubt those running Apache stick the server in a closet and pretend it's not there, either."

I don't really think that is the case.

It's clear that a great deal of attention has been paid to this closet over the last few years. It's gotta be among the most popular closets in Boston. Anyway, how could it possibly be forgotten when it is brought up in Meta at least five times a month and reset on a daily basis?

All these, "what can I do", and "what would it cost," threads are just silly. You can't do anything and it won't cost much -- except the only thing that matters, time. Since it's not *your* time, you don't have much to say now, do you? Matt has promised improvements as early as next month so just deal with it for a few more weeks. There is always the great outdoors, Monkeyfilter and Fark.

You want to do something, beautiful. Buy a text ad, click a Google link or donate via Paypal. But don't sit back and throw stones at the playground equipment someone else was kind enough to provide. Even better, you want to help Metafilter? Cool. Take the time your spending worrying about *how* it works, and apply it to *making* it work. That you can do. Show me the best of the web rather than a bunch of half-baked notions about how all that finery should be presented.
posted by cedar at 2:02 PM on May 8, 2004


I'm not here to defend microsoft, quonsar, but that's FUD. There are many more factors that will cause web site faliure than which particular web server software it runs on. Site and database engineering and design is way more important. Web servers are not difficult to engineer - any competent computer science undergraduate can make one.

On preview: cedar, I never suggested that was the case. I was replying to quonsar. I've previously donated cash to Metafilter. I have no comment to make about how well Metafilter may or may not be coded, designed, administered or maintained. How could I?
posted by normy at 2:38 PM on May 8, 2004


normy, FUD is the only language microsoft apologists speak!

:-) <---- smiley
posted by quonsar at 2:43 PM on May 8, 2004


no, wait.

i forgot FUD++ and Visual FUD.NET.
posted by quonsar at 2:44 PM on May 8, 2004


My company runs almost exclusively on IIS, and it rarely gives us any grief.

The problem, as I see it, is in the connection between the front-end and the database. IIS is inherently unstable with a non-MS backend (by design, maybe? Naahhhh.) I'm almost positive (99.9784%, to be exact) that if Matt was running MSSQL on the backend there would be fewer IIS crashes.

But that's the nature of MS, isn't it? They make their products just that much more stable with their own stuff, so they can get you into a lucrative licensing pact.
posted by mr_crash_davis at 3:14 PM on May 8, 2004


I'm 99.9784% that MetaFilter *is* running MSSQL!
posted by timeistight at 3:21 PM on May 8, 2004


crash, the server is running mssql, tables are indexed up the wazoo and I'm caching where I can. The server's got a ton of ram (4Gb of it now) and MS SQL gets about 2-3Gb of it to pool queries.

I have IIS set to restart itself, and in checking the event viewer after a crash, I'll see messages like "IIS crashed unexpectedly after 23 restarts" with no other flags or errors.

I'm going to start testing out apache on a different port this week, and if possible, start moving some stuff over to it sooner than June 1.
posted by mathowie (staff) at 3:30 PM on May 8, 2004


Site and database engineering and design is way more important.

Exactly. Each client demands an awful lot from the Mefi server, with it's unpaginated lengthy pages that reload every single time. How about just show the last 15 or so replies to a post, with a button that says "show all" if you want to read all. It would cut out a massive overhead. Ditto the front page (show last 10 FPPs with option to view all). Also the "new" posts that get mentioned for those that login - click on the "X new" and it shows X number of posts only. Just food for thought.
posted by SpaceCadet at 3:35 PM on May 8, 2004


I can say with some authority that IIS is less reliable a web server than Apache. I say this as someone with professional technical experience with enterprise class web software that runs on several server platforms, with several databases, and with several web servers. We're talking about major corporate internet and intranet web sites, companies that are very familar and web sites that you visit every day. Win/MSSQL/IIS required a lot more hand-holding than other platforms and fails more often. Period. It's irrefutable. Yes, many high volume sites use this platform, and some of those are ones with which I have direct experience.

I prefer Solaris/Oracle/Apache, but that's just me.

Cold Fusion is not up to the task of a high-volume website, and certainly not mission-critical.
posted by Ethereal Bligh at 3:39 PM on May 8, 2004


Oh, and I will echo the points about site design being absolutely critical for high-volume sites. You want to limit the workload for each page view, making use of cached data as much as possible, and optimizing your db access. Stored procedures and other things help. Hell, just badly written db queries can kill a high-volume site.

With all due respect to Matt, and I have a lot of respect for Matt, he simply didn't build MeFi with all the considerations in mind. And I'll mention again that hardware is very important, and we're talking commodity hardware.

As I said earlier, you could probably build a much, much more stable site around MeFi's hardware config and platform, but you'd need to do so with a lot of expertise from the ground up.
posted by Ethereal Bligh at 3:43 PM on May 8, 2004


Matt, I've been thinking about this this afternoon, and I'm sure that if you're able to restart IIS etc. manually, there's a way to automate this. What may be failing is a dependency, or just not waiting long enough, or something else.

But Win2K, which is what I am assuming you're on, has a variety of options for restarting stopped services. But, more to the point, there are service monitoring utilities that are very sophisticated (one of which I have), not to mention something home-grown, if necessary. I would be glad to help of advise, or whatever, if you'd like.

This kind of thing is a good idea regardless. But the long-term solution is making your platform and installation more robust.
posted by Ethereal Bligh at 7:27 PM on May 8, 2004


"IIS crashed unexpectedly after 23 restarts"

*ROFL's helplessly*

and this fine product costs how much?
posted by quonsar at 7:52 PM on May 8, 2004


"crash, the server is running mssql..."

Ah, my mistake. For some reason I thought you were running on MYSQL.

Carry on.
posted by mr_crash_davis at 7:53 PM on May 8, 2004


It's a simple fact that there are many of us who are members of this community who have enterprise class network administration experience. I know. I'm one of those people.
But, community or no, it's Matt's project. I say this as someone with a project of their own that happens to have a community attached.
When I want help I ask for it. When I don't want help, I don't ask for it. Notice that Matt isn't asking for help right now.
<Bill Hicks>That's called logic. It'll help you.</Bill Hicks>
posted by snarfodox at 11:59 PM on May 8, 2004


the IIS/ASP/CF/WHATEVER box will be requiring regular restarts. it's an indisputable fact.

Sure thing, chief. I run Apache at home and I love it, but we run everything at work off of IIS/ MSSQL and I've never known this to be true. And we do host sites that see millions of hits per month (those sites each have their own box and typically have a separate db server too-- not sure if that's how MeFi rolls). Simply blaming the problem on MS and screaming "Switch" with you fingers in your ears is why Linux doesn't get the attention it deserves from non-zealots.

Cold Fusion is not up to the task of a high-volume website, and certainly not mission-critical.

I hate writing in ColdFusion; even so, I can think of two clients of ours whose sites run on CF that see high traffic volumes and provide "mission-critical" applications (software licensing and project registration respectively). Of course, your specific experience must hold true for the world at large. Perhaps one could be a little less assertive?

Matt, a stupid question: are all the big queries in stored procedures?
posted by yerfatma at 10:40 AM on May 9, 2004


I had a problem with this with one of my servers at work. Some script kiddies were causing the IIS service to crash by causing memory overflow errors. The service would start immediately after, making the problem fairly transparent, that is until IIS reaches it's maximum restart limit when you all you can do is reboot.

I only noticed when I looked through the web log file (not the event log) matching the times with the time of the crash and seeing a huge stream of crap (usually in bunches of ten).

However this information is of no use to anyone as the solution for me was to apply all the latest security fixes at Windows Update and I'm pretty sure Matt would have done that anyway. Maybe if he checks his logs he'll see suspicious activity though and that may help.
posted by dodgygeezer at 11:30 AM on May 9, 2004


With all due respect to those with some experience with high-volume sites; there's a difference between that sort of limited experience and experience planning, maintaining, and troubleshooting dozens of corporatebehemoth.com websites, doing many of the same things, with some of the same propietary software, but on parallel platforms where a reasonably direct comparison of efficiency and robustness is possible. The Windows platform is simply not as robust as a UNIX server. It's not. I'm not an MS basher, I couldn't possibly care less (other than annoyance) at the freakin' OS wars and whatever people's product loyalties are.

Q's and other people's reflexive MS bashing was irritating; but this "who are you to offer advice?" dick-swinging is like a bad Slashdot flashback. Grow the fuck up. MeFi is experiencing problems that a lot of people have complained about lately, Matt has discussed some of his plans, and he even asked a question in this thread. A few people—me, for example—commented and offered help simply because, you know, we wanted to be helpful, not compare dick length.
posted by Ethereal Bligh at 2:42 PM on May 9, 2004


this "who are you to offer advice?" dick-swinging is like a bad Slashdot flashback. Grow the fuck up.

Sure thing. Could you please be a little more grating and condescending? My only axe here is that people constantly come in and offer advice and it amounts to, "Start over and do it how I would have done it." I have no idea if Matt even wants any help, but if he does, small changes that would make a difference on every transaction (e.g., stored procedures) could make a big difference right away are more likely to get implemented than porting it to someone else's language of choice.
posted by yerfatma at 5:41 PM on May 9, 2004


You know, I'm starting to think it could be a buffer overflow that is causing the crashes. I haven't applied any IIS hotfixes in months but I haven't seen any get announced on the technet bulletins MS sends me.

What's really surprising is that MS has re-architected their website and it took me almost 20 minutes to end find a security bulletin page for IIS, where nothing has been announced for months. I ran the baseline security analzyer thing and it only found one component out of date.

When I switch to apache, we'll know for sure if it was just random script kiddies killing the IIS process.
posted by mathowie (staff) at 6:13 PM on May 9, 2004


Everything is a stored proc, by the way.
posted by mathowie (staff) at 6:15 PM on May 9, 2004

"Sure thing. Could you please be a little more grating and condescending?"—yerfatma
Probably not. I'm in a bad mood and thin-skinned this evening. Apologies to all.
posted by Ethereal Bligh at 8:33 PM on May 9, 2004


Cold Fusion is not up to the task of a high-volume website, and certainly not mission-critical.

*sigh*, that's just not the case. Like any system, you just need hardware appropriate for the load. If MeFi is still running on that one box in a closet, my first suggestion would be to move the db to another box- no matter what MS says, Windows works best with a "one task per box" design.

I've been with ColdFusion since v1.5 (dbml.exe, anyone?). Since I started using it under Solaris, I'm never going back to Windows- the integration is SO MUCH SMOOTHER under Apache.

Potential performance gains under PHP are an arguable point (and impossible for anyone to say without knowing the code), but I think the more significant issue is trying to run everything on one box.
posted by mkultra at 9:39 AM on May 10, 2004


yerfatma - "Simply blaming the problem on MS and screaming "Switch" with you fingers in your ears is why Linux doesn't get the attention it deserves from non-zealots.

apache runs just fine on a windows box too, you know. it's just (in my opinion) more secure to run a server that isn't potentially tied into major operating system components. microsoft has a long history of "bundling" things, so now for example when their freebie email program has a security hole, it is suddenly affecting mission-critical OS components. (IE i can see, sort of, but outlook express? how is this a vital component of the OS? what will they tie into the OS next - solitaire?)

used to run our lab website on IIS, win2k box. nothing huge but it got hits, including scripts and worms that exploit IIS vulnerabilities. we were exploited out the wazoo, until i switched over to apache. still running on the same windows box, still getting hit by scripts but now they actually have no effect, as apache is simply a service and has no ability to access critical OS components through the MS standard "good system integration / bad security protocols" combination.

i don't think anyone's blaming the OS here, just the robustness of the web server software. i did that apache install about three years ago and haven't touched it since. to give MS credit they must have a damn hard time patching holes in the system with so many different projects all bundled together... amazing that they can get it working as well as they do.
posted by caution live frogs at 9:06 AM on May 11, 2004


« Older "Pre" Tags   |   No comment area for logged in user on newly... Newer »

You are not logged in, either login or create an account to post comments